Handling Software Icebergs

Humanity is obsessed with GROWTH

For GROWTH are needed resources

The natural resources are running out and the most important resource has become DATA

This data is processed by programs.
Worldwide spending on IT technology is growing > 5% per year

Software (programs) have become services
Different programs share many components, which are stored in world-wide cloud storage

The global cloud computing size grows yearly > 19%. The growth of cloud computing in education is expected 25.6%

We do not have any more monolithic programs hosted on your own server

Programs are collections of independent components/packages/libraries
Those components/libraries are responsible for 60..99% of program's functionality

Nobody does not know exactly what these inserted components/libraries do

"Modern software development is a study in entropy" (E. Freeman, AWS)

“It has never been more difficult to be a software developer than it is today” (N. Simpson)

Example 1

A Python 3 program to create graph diplaying growth of the SQAMIA conference:

For experienced 'classical' programmers the above code should look rather strange:
no loops ???

The 'classical' program, i.e. using only commands from a programming language without any libraries would have to use several loops – using the x_sqamia iterator, projecting values to axes, solving the system of equations for creating square approximation in order to calculate correlation parameters a2,a1,a0. Totally is hidden the AI (Artificial Intelligence) – calculating the size and visually pleasing placement of axes and their scales.

The whole 'magic' is in the first two lines:

In these two libraries are loaded other libraries, which again load others etc

Thus the script imports alltogether 162 modules (and 28 items appear as MissingModule):

_bootlocale • _collections_abc • _weakrefset • abc • codecs • collections • collections.abc • copyreg • encodings • encodings.aliases • encodings.ascii • encodings.base64_codec • encodings.big5 • encodings.big5hkscs • encodings.bz2_codec • encodings.charmap • encodings.cp037 • encodings.cp1006 • encodings.cp1026 • encodings.cp1125 • encodings.cp1140 • encodings.cp1250 • encodings.cp1251 • encodings.cp1252 • encodings.cp1253 • encodings.cp1254 • encodings.cp1255 • encodings.cp1256 • encodings.cp1257 • encodings.cp1258 • encodings.cp273 • encodings.cp424 • encodings.cp437 • encodings.cp500 • encodings.cp65001 • encodings.cp720 • encodings.cp737 • encodings.cp775 • encodings.cp850 • encodings.cp852 • encodings.cp855 • encodings.cp856 • encodings.cp857 • encodings.cp858 • encodings.cp860 • encodings.cp861 • encodings.cp862 • encodings.cp863 • encodings.cp864 • encodings.cp865 • encodings.cp866 • encodings.cp869 • encodings.cp874 • encodings.cp875 • encodings.cp932 • encodings.cp949 • encodings.cp950 • encodings.euc_jis_2004 • encodings.euc_jisx0213 • encodings.euc_jp • encodings.euc_kr • encodings.gb18030 • encodings.gb2312 • encodings.gbk • encodings.hex_codec • encodings.hp_roman8 • encodings.hz • encodings.idna • encodings.iso2022_jp • encodings.iso2022_jp_1 • encodings.iso2022_jp_2 • encodings.iso2022_jp_2004 • encodings.iso2022_jp_3 • encodings.iso2022_jp_ext • encodings.iso2022_kr • encodings.iso8859_1 • encodings.iso8859_10 • encodings.iso8859_11 • encodings.iso8859_13 • encodings.iso8859_14 • encodings.iso8859_15 • encodings.iso8859_16 • encodings.iso8859_2 • encodings.iso8859_3 • encodings.iso8859_4 • encodings.iso8859_5 • encodings.iso8859_6 • encodings.iso8859_7 • encodings.iso8859_8 • encodings.iso8859_9 • encodings.johab • encodings.koi8_r • encodings.koi8_t • encodings.koi8_u • encodings.kz1048 • encodings.latin_1 • encodings.mac_arabic • encodings.mac_centeuro • encodings.mac_croatian • encodings.mac_cyrillic • encodings.mac_farsi • encodings.mac_greek • encodings.mac_iceland • encodings.mac_latin2 • encodings.mac_roman • encodings.mac_romanian • encodings.mac_turkish • encodings.mbcs • encodings.oem • encodings.palmos • encodings.ptcp154 • encodings.punycode • encodings.quopri_codec • encodings.raw_unicode_escape • encodings.rot_13 • encodings.shift_jis • encodings.shift_jis_2004 • encodings.shift_jisx0213 • encodings.tis_620 • encodings.undefined • encodings.unicode_escape • encodings.unicode_internal • encodings.utf_16 • encodings.utf_16_be • encodings.utf_16_le • encodings.utf_32 • encodings.utf_32_be • encodings.utf_32_le • encodings.utf_7 • encodings.utf_8 • encodings.utf_8_sig • encodings.uu_codec • encodings.zlib_codec • enum • functools • heapq • io • keyword • linecache • locale • matplotlib.pyplot • numpy • operator • pyi_rth__tkinter.py • pyi_rth_inspect.py • pyi_rth_mplconfig.py • pyi_rth_multiprocessing.py • pyi_rth_pkgres.py • pyi_rth_pkgutil.py • pyi_rth_subprocess.py • pyi_rth_traitlets.py • re • reprlib • scipy.stats • sre_compile • sre_constants • sre_parse • tokenize • traceback • types • warnings • weakref

But the 'magic' has cost

A simple Python program reveals, that the imported 162 modules contain 18378 LOC (Lines Of Code)

The difference between 18 LOC in the presented previously program listing with the all LOC in imported modules can be represented by index of visibility:
$visibility=\frac{LOC(visible\text{\_}code)}{LOC(invisible\text{\_}code)}=\frac{18}{18378}=0.0000979431$

Thus students can inspect only 0.0000979431-th size part of the above ('loop-less') program

And from the code presented above it is not clear, what should be done with the program for the above graph in order to include two missing SQAMIA years 2020, 2021 - if x_data, y_data have different size, then how to mark the corresponding values?

Example 2

The introduction to library React ("React Getting Started" in popular Web W3Schools) seems rather short :

But loaded above three Javascript files contain 73571+3358+26275 = 103204 lines of JavaScript code

The 103204 lines of code imply index of visibility:
$$ visibility=\frac{LOC(visible\text{\_}code)}{LOC(invisible\text{\_}code)}=\frac{17}{103204}=0.000164723$$

This large amount of invisible/inspectable code is not needed, since the following program will produce exactly the same output (without any external scripts):

Example 3

Three lines of Tensorflow example from Google load three libraries:

These libraries import 158 other modules:
_bootlocale • _collections_abc • _weakrefset • abc • codecs • collections • collections.abc • copyreg • encodings • encodings.aliases • encodings.ascii • encodings.base64_codec • encodings.big5 • encodings.big5hkscs • encodings.bz2_codec • encodings.charmap • encodings.cp037 • encodings.cp1006 • encodings.cp1026 • encodings.cp1125 …•

$visibility=\frac{LOC(visible\text{\_}code)}{LOC(invisible\text{\_}code)}=\frac{52}{24465}=0.002125485$

But the real complexity of this program comes from code: Tensorflow creates a model with nearly 4 million parameters:

Pandemic has forsed many IT students to self-learning from WWW

Self-learning students can not understand the 'hidden-layers' of modern software
The examples/tutorials in WWW are recepies, where the 'inner working' is not explained

Thus the pandemic has created 'top-down' programmers:
First: import all libraries (what seem to be usefull)
Then: start googling

They do not undersand the problem or how to solve it