Tuesday, December 3, 2013

Essential & Accidental complexity of Software Engineering: Eliminating “Cognitive Dissonance” is a silver bullet


This is continuation to this previous blog dated 26th Nov, 2013. It is useful to read up to the point where the previous blog refers “Cognitive Dissonance”. Today “cognitive dissonance” for each SCC (i.e. component) builds up as more and more spaghetti code is added for other SCCs in the application. In case of real-CBD, componentization and hierarchy of components eliminates such cognitive dissonance. That is, componentization (i.e. encapsulating each SCC in a RCC) allows focusing on a given problem at hand (i.e. each of the RSCCs in the component-hierarchy), without any overbearing interference and huge distractive noise from surrounding issues (i.e. code sections for other SCCs).

The CBD-structure(or hierarchy of replaceable modules) is single most essential aspect universally shared by each and every large physical CBD-product (exists today or in the future). If I were designer of Hard Drive or Power Supply of an experimental computer, how long would it take for me to find the all the parts (i.e. devices) of such large container component for a making minor redesign to the component or to replace the component? It would not take more than few minutes to find the component for either replacing the component or to redesign and test the component individually outside of the product.

Today the computer is designed as shown in FIG-2, as a hierarchy of replaceable modules, where each component can be disassembled (or plugged-out) and re-assembled (or plugged-in) in mater of minutes. But imagine if the computer were designed as shown in FIG-3, where each device of a large component (comprising hundreds of devices) is hard wired into the system-board along with the devices for other such large components (comprising hundreds of devices). The FIG-3 illustrates, how complex software having multiple SCCs is being designed today. It is essential to keep in mind that even large components (i.e. SCCs) by nature can be loosely coupled, while other kinds of parts (even small parts) used for building large components by nature tightly coupled.

Compare the cognitive dissonance for the components in a CBD-structure with the cognitive dissonance of a large software component such as City_ATC having 5,000 lines of code or container component such as City_Landmarks having 15,000 lines of code in a medium size application having 500,000 lines of code. How long it would take for me to find all the code-section for City_ATC that are spread across multiple non-exclusive files after a gap of few months to redesign the component? Is it practical to effectively remove all the code-section to effective remove the component? What is the cognitive dissonance for a newly hired software engineer?

Compare the cognitive dissonance for above components (i.e. City_ATC and City_Landmarks), if each of the components is designed as a RCC and assembled into the application by implementing 2 to 3 lines as illustrated in Listings-1 and Listings-2 (where the coupling interfaces and communication code are respectively managed and created by an intelligent CASE-tool).

The cognitive dissonance of each component represents non-essential or accidental complexity wasted for making a small change to the component. In his famous seminal paper "No Silver Bullet — Essence and Accidents of Software Engineering" Dr. Brooks persuasively argued that there is no ‘cognitive dissonance’ (i.e. accidental complexity) and/or not possible to eliminate ‘cognitive dissonance’ (i.e. accidental complexity).

I respectively disagree based on the easily verifiable 3-CBD-facts. His famous seminal paper and famous book “Mythical Man Month” has huge influence on shaping software engineering research and our belief that software is unique/different and all the complexity is essential (so not possible to eliminate). This is true in the existing paradigm, which is build by relying on a huge error that prevented us from discovering innate nature of real-components.

However discovering unique essential characteristics universally shared by the physical functional-components not only expose the error but also help invent real-software-components equivalent to the physical functional-components for achieving real-CBD (e.g. CBD-structure) for eliminating cognitive dissonance for each of the components in the component-hierarchy. I always admire and still a big fan of Dr. Brooks, but I have to respectfully disagree with him now (based on my recent discoveries).

Few years back in his email response to me, he said that reasonable people can disagree. Unfortunately I have to even disagree with that statement, since in the context of basic science there can be only one accurate answer for nature of physical beings (e.g. diverse spices of components or animals) or physical phenomenon (e.g. CBD or controlled powered flight). The laws of nature are immutable and they can’t be changed or defined by a committee (as we have been doing by defining many kinds of so called software components & CBSDs).

Can respected scientists disagree, if the Earth or the Sun at the center, without any scientific basic and factual support. Can the scientist debate or disagree on Newton’s laws of motion, without any scientific basic and factual support? Therefore, I believe that the scientific process is broken in the computer science and that must be fixed: http://real-software-components.blogspot.in/2013/11/the-process-of-scientific-research-is.html.

Of course, respected scientists still could say that all this is just a theory, so there still could be flaw. But I don’t know what kind of excuse possible, when I can show practical results by demonstrating countless applications created as hierarchy of replaceable components on my laptop? I created hundreds of applications as hierarchy of replaceable components, when I accidentally discovered replaceable components more than a decade ago, which induced irresistible curiosity that driven me to focus on this long and arduous research.


Tuesday, November 26, 2013

The scientific process is broken and to fix it I must find real software scientists & researchers who really care about science & technology

The CBD (Component-Based Design) is not a peripheral issue or afterthought, but the very heart of designing and building complex physical products (either in step-1 or step-3 of CBD-process). Although physical components and CBD are taken for granted (as free oxygen in the air for breathing), its value would be realized if designers of complex physical products are prevented from using components and hierarchy of replaceable components for designing complex one-of-a-kind physical products (e.g. experimental automobiles, jet-fighters or spaceships) either in step-1 or step-3 of CBD-process (e.g. designers of physical products face crisis even larger than the crisis being faced by the designers of software).

The software industry is inventing hundreds of new kinds of software products each year. Likewise, mankind is inventing many new kinds of products each year. For example, please click here to review the design of a physical-product artificial-kidney that is in the process of being invited. This CBD-process must include (i) partitioning the product into hierarchy of replaceable components (i.e. CBD-structure), (ii) designing and building each component individually and (iii) periodically assembling the components to build the product to make sure that the components are functioning as intended and the product design is progressing as expected.

The process of designing any (software or physical) product and each of its large functional components is an iterative process, where each component is improved individually little-by-little to make it as best as it can be. No large functional component is neither magically works perfectly first time nor magically appears in a fine day. For example, in case of City_GIS application on the first day each component is empty whitespace or a pretty-GIF image as a place holder. In the next first iteration, if we assemble all the RCCs, only 10% to 20% of the features might be working in each of the components.

Today no other GUI-API is capable of encapsulating such large SCC in a RCC. If each SCC is not encapsulated in a RCC, the custom application code implemented for the application would begins as a spaghetti code even in the first iteration (having 15% features), since many application files contain source code for more than one SCC. This spaghetti code grows each day as each developer adds more and more code each day (in the non-exclusive files). No component can be tested individually, for example to test newly added features. So it makes nearly impossible to predict what sort of effect a small change in the custom application code of a SCC would have on other parts of the ever growing spaghetti code (comprising application code of all the SCCs of the application). In this case, each file comprises code-sections for multiple SCCs and code-sections for each SCC spread across multiple such files.

See: Cognitive dissonance: Essential & Accidental complexity of Software Engineering


Any changes made for unpredictable future needs, wisdom gained from use-cases, changing technologies or sparks-of-innovation would cost many times more, while adding more and more spaghetti code to one or more of these files. For me, most of the innovation-sparks come after completing 80% or most of the implementation (because I gain more and more insights and expertise as I work every day by intensely focus on each feature and problem domain); or by looking at how the features are being used by users, and feedback of users help me realize what matters most to them in day-to-day use. I am sure, the basic need for agile program is to provide a mechanisms to make incremental improvements.

Today's spaghetti-code is worst possible mechanism for agile-programming. The componentization of a complex problem into hierarchy of replaceable self-contained modules is most effective and efficient mechanisms known to mankind for improving each module little-by-little to make it as best as it can be individually. I am repeatedly using phrase "refining each component 'little-by-little'", which is from famous book on Evolution of Useful Things; and also using phrase  "making each component 'as best as it can be' individually", which is from video of designing new artificial-kidney. This is the only way for useful components to get closer and closer to perfection. A product can get closer and closer to perfection by making each component gets closer and closer to perfection.

Please consider this spaghetti code in contrast with the CBD of any physical product or City_GIS that is partitioned as hierarchy of replaceable components. The source code for each RCC/SCC is implemented in a set of exclusive files, where the exclusive files do not contain code for any other SCC. I can readily find all the code for each SCC even after few years, if I need to add more features (I don't need to see even a single line of code for any other SCC in the exclusive files of it's RCC). Each component can be tested individually outside of the product. If it works outside as expected and interfaces are not changed, it works predictably when assembled.  It is much simple to predict the impact of even extensive redesign of a RCC.

In case of software, most of the need for such need for prediction or guessing by using intelligent CASE-tools to manage and automatically detect any changes resulting in inadvertent incompatibilities in the coupling interfaces. So real-CBD for software can be far superior than CBD of physical products, once real-software-components are discovered.

True componentization requires sandboxing and versioning of each component for refining little-by-little individually to satisfy evolving needs. An ideal mechanism for agile-programming must let the programmer focus and work only on essential complexity, by eliminate all the non-essential overhead (or accidental complexity). For example, in case of physical components, the designers can disassemble each component and can focus only on the component. If each SCC is not encapsulated in a RCC, the spaghetti code (e.g. code-sections of other SCCs) surrounding the code-sections of the SCC adds  huge amount of accidental complexity.

The replaceable-components offer many other kinds of benefits, such as experimentation (which is extremely useful for innovations). For example, if I had a spark-of-innovation and asked an engineer to redesign a component and he ruined the code structure due to inexperience or a big customer prefers older version, I can still restore older version of the RCC in minutes from version-control-system. It is not practical in the existing spaghetti code, since each of the files that contained code-sections of the SCC also contain other code-sections for other SCCs and these other code-sections likely have been changed for redesigning the other SCCs.

It is not possible to list all this kinds of benefits of real-components and real-CBD, but it is sufficient to say componentization and hierarchy of replaceable components offer perfect mechanism for agile-programming. Many software experts give baseless excuses that software is different, because we can't see or touch each component as we can do with physical components. If I am designer of City_ATC of City_GIS, why can't I see or touch? I can touch and see any software component better than, I can see or inspect the circuits in semiconductor-chips. One may think and argue, they are different things. But please understand that I know what I am talking about (it is a long story, but let me briefly explain why I am qualified to say this):

I worked for Advantest (the largest test equipment company) for more than 3 years designing software for parametric testing of IC-chips. Parametric testing requires inputting test values at input and testing output (0 or 1 by using a given range of volts) after waiting some time (to allow propagation delay). The wave-forms at both input and output are highly analog (can take a range of values). I also worked for KLA-tencor (the largest maker of optical inspection equipment) and I was responsible for a software product called KLA25xx, which was installed in 90% of the semiconductor-fabs in the world during mid-1990s.

The inspection equipment of KLA-tencor optically-inspected the chips at various steps of manufacturing and stored defects and images in software product KLA25xx (e.g. for online monitoring at real-time and offline analysis for improving the process at later time). I was solely responsible for this product for 4 years and intimately knew each and every module of this product. It is amazing that each computer is working perfectly, when billions of things could go wrong in imperfect etching of the circuits and resultant analog waveforms due to even minuscule imperfections in etching. So it feels silly to hear such excuses that, we can't see or touch software. I can see RCC of my City_ATC far better than I can see circuits. I can use debugger tool to step through each line of code. if I need to add more features one year after releasing the City_GIS, only thing makes my life miserable is searching and finding all the sections of the code spread across the spaghetti code in ever changing non-exclusive multiple files for other SCCs, if the SCC is not encapsulated in a RCC (exclusive files).

On the other hand, today even large physical functional-components are designed as software. The soft-design of the physical components are called blueprints. The designers of physical components redesign each of the blueprints to redesign each of the physical-components, such as engine or gearbox of cars. Many of the components for Boeing Dreamliner or Airbus-380 are designed and tested as blueprints. The component interfaces and compatibilities for Airplane models are validated using CAD-tools similar to our proposed intelligent CASE-tools. Once real-software-components are discovered, I feel no one could see much difference between designing and building physical functional components and software components. The real-software-components certainly shall have clear and distinct advantage, because we can design software components more intelligent than mechanical components made of metals (e.g. for collaborating with CASE-tools to automate couplings).

I can write this kind of things endlessly (if any one has intellectual curiosity and cares to know), and for many aspects I don't even know where to start. If one wanders for 10 years in an vast uncharted area no one else don't even know exist, one get to learn very little about countless new species (i.e. thing or aspects) belongs to the ecosystem of this new vast uncharted area (or new paradigm). It is much bigger that one man or even 100 men could explore. I am eager to make the world aware of this vast uncharted area and start exploring countless possibilities.

I found my treasure (i.e. patents for real-software-components), but it is just a beginning and I feel other early explorers would find other treasures (and I am hoping that other treasures won't undermine value of our patented treasures). Although it is not simple and challenging to master such Kuhnien paradigm-shift, I am sure there shall be plenty to go for early explorers, who took the challenge. I feel, if I am successful in exposing this huge error, most software experts would be forced to master this new paradigm sooner or later (either by choice or for survival in the long-term).

The main question is when and how I could succeed in exposing the error (if ignore the possibility that I might fail). The next question is, if the US leverages or left to a foreign competition to exploit these new discoveries. India has only few research organizations and researchers, and I found most of them are devoid of any accountability (i.e. they simple don't care, and most are busy self-promoting before their political bosses, since top posts are political appointments based on cronyism than merits).

I am beginning to feel the US research organizations are becoming as bad as Indian research organizations. There is no way for making the research organizations aware of this kind of huge errors. For example, if we try to call, executive secretary answers phone to say he is very busy. Although the job of SEI of CMU and SDP of NITRD is to search every cave and even far corners for this kind of basic discoveries and must be shocked to know this kind of huge error and scramble for finding reasons for breakdown of basic scientific process. Instead they refuse to pay attention or listen to what I wanted to say.

The scientific process is broken, so we can and must fix it, by making real respected researches aware of broken scientific process and resultant huge errors. How or where can I find real software scientists & researchers who care about science & technology?

Monday, November 11, 2013

Most successful agile process for designing and building complex software

I would like to propose a new innovative mechanism for agile programming. Fortunately we all know this mechanism, which has been widely used and extremely successful in designing and building complex newly invented or one of a kind product such as experimental Jet-fighter or spacecraft. For example, please kindly see design and development of a simple newly invented product at: http://real-software-components.com/CBD/CBD_of_new_product.html

The physical products are designed as component-hierarchies (or CBD-structure), where each component can be designed and tested individually outside of the product. Once all the components are ready, the total cost of disassembling or reassembling could never be more that 5% of the total cost of the product throughout the life of the complex product (i.e. in step-1 or step-3 of CBD-process). Please let me summarize two essential aspects CBD-Structure and CBD-process of real-CBD of physical products in separate web pages:

It is extremely important to recognize two irrefutable facts about the CBD-design:

  1. It is not necessary that even a single large functional component in the CBD-structure (or component-hierarchy) of a complex product conform to any known so called component models or have any useful properties (e.g. reuse or standardized) erroneously attributed to software components today. 
  2. Either complexity or uniqueness of one-of-a-kind product (e.g. experimental spacecraft) can’t prevent the designers from partitioning the product as component-hierarchy.

Please kindly look at design of a sample application that is designed as hierarchy of real-software-components that are equivalent to the physical functional-components for achieving CBD-structure (or component-hierarchy): http://real-software-components.com/CBD/City_GIS.html

Of course, pre-requisite to this mechanism is discovering innate nature and essential properties that are uniquely and universally shared by the physical functional components. This discovery is essential not only to expose huge error in the existing software engineering paradigm but also to invent real-software-components equivalent to the physical functional components capable of achieving CBD-structure: http://real-software-components.com/RSCC/Real-Components.html 

The software engineering researchers committed a huge mistake not analyzing all the facts and valid observations for discovering the innate nature and essential properties of the physical functional components. It is not a small error: http://real-software-components.com/forum_blogs/year2013/IsItSmallError.html 

I have been doing research on this unknown area of software engineering for nearly 12 years, and made fascinating discoveries. I openly published all the evidence in my website: http://www.real-software-components.com/moredocs.html

The componentization is the most effective and efficient method known to mankind for addressing a complex problem (i.e. by a team of experts) by partitioning the complex problem in smaller (or components) and smaller (or subcomponents) self-contained problems, where each smaller self-contained problem can be addressed individually (by each member of the team in manner consistent with his domain expertise, knowledge and skills): http://real-software-components.com/technologies/componentization_purpose.html

Stating the fact that “the Sun is at the center” offended common sense or insulted deeply entrenched collective conventional wisdom of respected scientists 500 years ago. Likewise, unfortunately few software researchers might feel offended, when I try to point out certain errors. At any time since 1970s tens of thousands of researchers have been working very hard (e.g. applying brute force) with passion for advancing the software engineering by relying on this unsubstantiated flawed root postulation (without ever validating or even aware of the huge error). This brute force resulted in evolution of complex paradoxical paradigm with ecosystem comprising 3-dimensional web of interdependent concepts (many of them are no different from epicycles & retrograde motions resulted form the error in the root postulation).

Please kindly remember, the software engineering paradigm has been evolving since 1960s, by relying on a huge undetected error. Mankind not made this kind of error in basic seed axiomatic premises since exposing the error of then deeply entrenched Geocentric-paradigm 400 years ago. Please kindly remember, it is invalid circular logic to use any thing (e.g. epicycles and retrograde motion) derived from geocentric-paradigm to discredit heliocentric-paradigm.

Any real scientist must agree: the scientific progress is discovering new facts for expanding the boundaries of human knowledge. Pursuit of the absolute truths (or facts) is the basic responsibility and sacred duty of each and every real scientist or researcher. Unfortunately, many software researchers abdicated their sacred responsibility. So I decided to openly publish all the evidence and irrefutable proof on the web and respectfully challenge the brilliant minds around the world to find a flaw: http://www.real-software-components.com/moredocs.html

Please kindly don’t forget basic scientific principles: Any real science ends up in a contradiction or paradox if and only if there is an error in the reasoning or basic facts. It is an error to rely on any subjective concept without sound basis in reality and fact, since any error certainly leads to a paradox.

"By denying scientific principles, one may maintain any paradox" ---Galileo Galilee

The real scientific progress depends on researchers pursuing Truth (objective facts) with passion, since no real scientific progress is possible by relying flawed subjective concepts. For example, this error in CBSD made in 1960s might have a cost of a trillion dollars to the world economy (so far). Each and every concept in computer science can be and must be objective facts, but today computer science ended up with many subjective concepts (many of them contradict reality).


I shall standby and can defend each and every concept in my website. In fact, an irrefutable proof is provided for each and every concept in one or more web-pages in website: http://www.real-software-components.com/moredocs.html. I am more than happy to explain any questions by using only objective facts (I shall not use subjective reasoning and excuses). My commitment is, I will never abdicate my basic duty to perusing the truth. Is it wrong to expect that other researchers to not abdicate their basic duty to only rely on facts?