Home » SyEN Newsjournal » PPI SyEN 51

PPI SyEN 51

Read monthly Project Performance International's Systems Engineering Newsjournal, named "PPI SyEN". PPI SyEN presents for the engineering professional 30-60 pages of valuable technical articles on topical subjects, shorter technical pieces, in-depth coverage of the month's news in systems engineering and directly related fields, pointers to useful resources and relevant industry events, plus limited information on PPI's activities.

Welcome to PPI SyEN 51

In This Edition

Quotations to Open On

Feature Article

Avoiding Off-ramps on the Way to Good Systems Engineering, by David Long

Article

The Root Causes of Poor Software Quality: Bad Quality Metrics, Incomplete Quality Data, Inadequate Quality Education, by Capers Jones

Systems Engineering News

National Academy of Engineering – Engineering Challenges of the 21^st Century
Research at the Stevens School of Systems of Enterprises (SSE)
SSE Faculty Share Insights on Modern Systems
Engineering Smart Factories of the Future
Creating a Less Wasteful Economy
3 Principles to Guide Designing for Social Change
IIBA releases new multi-level, competency-based Certification
Short Comedy Sketch Concerning System Requirements – “The Expert”

INCOSE

Revised INCOSE Vision and Mission Statements
INCOSE SE Vision 2025
INCOSE Working Groups
Empowering Women as Leaders in Systems Engineering (EWLSE)
INSIGHT

Featured Organizations

IEEE Technology and Engineering Management Society (TEMS)
International Association of Engineers (IAENG)

Systems Engineering Tools News

INCOSE Tool Integration and Model Lifecycle Management Working Group
The COBIT 5/CMMI Practices Pathway Tool

Systems Engineering Publications

Systems Engineering Innovation and Design
The Seeds of Innovation
Team of Teams
Guideline for Systems Engineering within the Dutch Civil Engineering Sector
Systems Engineering Jr. Handbook

Education and Academia

Worcester Polytechnic Institute (WPI) Systems Engineering and Leadership Institute (SELI)
Research positions available at the University of Toyko in Formal Methods and Cyber-Physical Systems

Some Systems Engineering-Relevant Websites

Standards and Guides

Ford Chief Engineer Nand Kochhar named Vice Chair of SAE International Technical Standards Board
Object Management Group Chairs take Standards Work to Next Level at Coronado Membership Meeting
Techstreet Facilitates Communication Concerning Standards
National Electrical Manufacturers Association

Some Definitions to Close On

Engineering
Quality

Conferences and Meetings

PPI News

PPI and CTI Events

PPI Upcoming Participation in Professional Conferences

Quotations to Open On

“Don’t assume that the original statement of the problem is necessarily the best, or even the right one.”

One pragmatic principle of Systems Engineering – INCOSE

“Systems Engineering focuses on ensuring the pieces work

together to achieve the objectives of the whole.”

Systems Engineering Body of Knowledge (SEBoK)

“Amazing things are all around us – one needs only to glance through the

Systems Engineering Newsletter (SyEN) and follow up on something of interest!”

Adapted from google.com

Feature Article

Image Source

Avoiding Off-ramps on the Way to Good Systems Engineering

by David Long

15 December 2016

There is no doubt that systems engineering is growing. It is growing in applicability as the degree of complication and complexity rises across many different application domains – aerospace & defense, transportation, automotive, energy, medical, consumer products, and more. It is growing in importance as we seek to deliver value efficiently and effectively, free from unintended consequences. But as systems engineering grows, there is a groundswell of frustration as well – a groundswell that increasingly what people call systems engineering is not really systems engineering at all.

This complaint has increasingly been raised in private discussions, small meetings, and public forums, and I share that sense and frustration. It is often framed in terms of individual practitioners and organizations performing “partial systems engineering” or “checklist systems engineering”, sometimes tinged with a sense of nostalgia for the early days of the practice. Rarely do I hear it framed in an “us versus them” mindset but more often in a constructive question concerning how do we make good systems engineering more accessible to all those who could benefit from good application and good practice.

It starts by recognizing that there are multiple off-ramps on the way to good systems engineering. The first off-ramp is seeing, intellectually grasping, and implementing only a subset of systems engineering – most often requirements engineering. There is absolutely nothing wrong with requirements engineering when it is performed well. It is an essential part of good problem solving and good systems engineering. However, it is a very limited subset and is by no means a substitute for systems engineering.

How do we fall into this trap – creating and then taking this off-ramp? Quite often it comes from returning to our reductionist roots, which have served us so well since the Scientific Revolution. We break problems – and disciplines – down into smaller and smaller pieces in order to increase understanding and advance. We hope to assemble the pieces into a working whole. This approach has served us well for centuries but breaks down when addressing the interconnected, interdependent, highly complicated, and often complex problems, solutions, and environments of today. Hence the birth of systems engineering, but even this practice based on holism is subjected to reductionism as people seek to break systems engineering down into its constituent parts, practice each independently, and then assemble the right outcome. The moment we break systems engineering into parts, we both violate its fundamental principles and run the risk of individuals substituting a part for the whole – knowingly or, more often, unknowingly. This is exactly the case with requirements engineering as a decoy for systems engineering.

The second off-ramp is substituting process for thinking and creating so-called “checklist systems engineering.” Again, process is a critical component of performing anything complex in an appropriate and repeatable manner. However, checklist engineering is neither purpose nor totality, only a partial enabler. Here we need to be very aware and very careful of the way we position our practice because we are part of the problem. INCOSE’s most cited definition of systems engineering is “an engineering discipline whose responsibility is creating and executing an interdisciplinary process to ensure that the customer and stakeholder’s needs are satisfied in a high quality, trustworthy, cost efficient, and schedule compliant manner throughout a system’s entire life cycle” – a process-centric mentality. The most cited systems engineering standard is ISO/IEC/IEEE 15288:2015 – Systems and software engineering — System life cycle processes. We must stop enabling – if not encouraging – this misinformed substitution of process for practice.

For those lacking a deeper understanding, process easily becomes a proxy. The thinking is that if I follow the process, I am doing the practice and will deliver the results. If the process is tailored to the circumstance and performed properly in the context of principles, supporting methods, and experience, the conclusion is absolutely true. However, we all know that the process executed in isolation is useless at best, harmful at worst – and I suspect we have all observed practitioners doing just this in the name of systems engineering.

The most recent off-ramp to emerge between many practitioners and good systems engineering is the rise of model-based systems engineering – or more specifically the misunderstanding of MBSE. Like requirements engineering and process, MBSE done right is an enabler to the modern practice of systems engineering to address the reality of today’s problems and solutions. However, MBSE is easily misunderstood – as a distinct approach, as a constrained practice to be performed in specific ways with specific notations, or most often simply as a series of diagrams. As is the case with the first two off-ramps, sometimes the misunderstanding is simply naivety on the part of the practitioner. In other cases, again we share the blame in the way we have framed (or miss-framed) this tool in our systems engineering toolkit.

These off-ramps emerge – and are taken – for a variety of reasons. The first is by far the most innocent but also the most damaging – simple misunderstanding. Misunderstanding on the part of the practitioner who believes he is doing the right thing but simply lacks the greater knowledge. Misunderstanding on the part of the individual, group, or organization who builds the off-ramp thinking that they are enabling good practice when in reality they are creating a false substitute. The second is more frustrating to those committed to advancing the state of the practice – personal or commercial gain. Approximately fifteen years ago the developer of a then-popular requirements management tool told the story of starting with systems engineering and then “dumbing the tool down” (his language, not mine) until procurement could understand what was being sold. Today, I know many spearheading the charge of MBSE within organizations who are knowingly incorrectly positioning the implementation and approach in furtherance of their interests at the ultimate expense of the organization. The third case is errantly substituting what is expedient or efficient for what is effective. Sometimes this is cloaked in the language of simplification (after all, systems engineering can be quite complex). Sometimes it begins with the thought “if I can only get them started, then I can correct the path later.” Too often, later never comes.

There are many other off-ramps on the way to good systems practice. We could attempt to enumerate them all, but the moment we do so other off-ramps would emerge. The answer is not to identify every possible error in the hopes of avoiding them. The answer is to return to first principles which will keep us on track throughout the journey. The answer is also to reduce the barriers to entry for good systems engineering, simplifying without becoming simplistic in order to help practitioners stay on course.

Whereas the oft-cited INCOSE focuses on interdisciplinary process, I prefer descriptions that highlight an interdisciplinary or transdisciplinary approach and practice that can be appropriately tailored to different levels of complication and complexity. Jon Wade of Stevens Institute emphasize two critical aspects – the lifecycle perspective and the creative act of design noting that “there is art in engineering, not just science and analysis.” Richard Beasley of Rolls Royce says “systems engineering collects and organizes all the information to understand the whole problem, explores it from all angles, and then finds the most appropriate solution.” Stanley McChrystal in Team of Teams describes systems engineering and addresses examples from across many different application domains without resorting to “tech speak”. We can learn from all of these individuals as we work not to reduce or simplify systems engineering but instead to make the critical concepts more accessible to those with different levels of experience and varied backgrounds.

Resolutions are generally made in the New Year, but let us begin (early) with the end in mind. Let us resolve to reach out and welcome individuals and organizations to this systems engineering practice, recognizing that more today than ever before the world needs good systems engineering. Let us resolve to make systems engineering more intellectually accessible, not by creating decoys and becoming simplistic but by emphasizing first principles and good communication. Let us resolve to combat off-ramps at every opportunity – refusing to take them ourselves, teaching and coaching those that we see moving off-track, and rejecting the creation of new off-ramps whatever the purpose.

If we are willing to make and follow through on these resolutions, we can move systems engineering from an oft misunderstood and misapplied process to an important and impactful practice, changing the world for the better. And isn’t that exactly what resolutions – and systems engineering – are for?

Article

The Root Causes of Poor Software Quality: Bad Quality Metrics, Incomplete Quality Data, Inadequate Quality Education

by Capers Jones

Version 1.0 August 10, 2016 update 2^nd January 2017

Abstract

The software industry has a bad and justified reputation for poor quality and low reliability. A key reason for poor quality is that quality measurements are embarrassingly bad. Metrics such as cost per defect and lines of code distort reality and conceal software quality. Software measures omit important data, use hazardous metrics, and are not sufficient to show the effectiveness of various quality control methods such as static analysis, inspections, and testing.

Software quality depends upon two important variables. The first variable is that of “defect potentials” or the sum total of bugs likely to occur in requirements, architecture, design, code, documents, and “bad fixes” or new bugs in bug repairs. The second important measure is “defect removal efficiency (DRE)” or the percentage of bugs found and eliminated before release of software to clients.

The metrics of Defect Potentials and Defect Removal Efficiency (DRE) were developed by IBM circa 1970 and are widely used by technology companies and also by insurance companies, banks, and other companies with large software organizations.

Web: www.Namcook.com

Email: Capers.Jones3@gmail.com

Introduction

Poor software quality is an endemic problem of the software industry. Many large software system projects are canceled due to poor quality. Almost all large system projects run late and are over budget due to poor quality. There are many lawsuits for poor quality. There are billions of dollars wasted by software customers due to customer damages caused by poor quality. There are also rapidly increasing numbers of cyber-attacks most due to poor software quality. Why do these endemic problems occur?

An important root cause is that most software quality companies are “one-trick-ponies” that only care about one subject. Some quality companies sell automated testing, some sell static analysis, some sell automated proofs, or whatever; no company sells a full suite of software quality tools and methods that encompass all sources of software defects and all forms of software defect removal.

Effective quality control needs a synergistic combination of defect prevention, pre-test defect removal, and formal testing with certified test personnel. Worse, most quality companies have zero empirical data as to the efficacy of their tools and methods. They make vast claims of better quality but provide no case studies or validated results.

Software quality data should be based on function point metrics because function points can show defects from all sources such as requirements, design, etc. Table 1 shows approximate U.S. averages for defect potentials for poor quality and high quality software projects.

If drugs and pharmaceutical products were released to the public with as little validation as software quality tools, the U.S. death rate would probably be twice what it actually is today.

Table 1 provides a list of 56 software topics that include defect origins, defect prevention methods, and defect removal stages that run from early requirements to post-delivery for a large system of a nominal 10,000 function points and 1,500 SNAP points in size in Java.

Note: all data in this article come from studies by the author and colleagues, which are based on about 26,000 projects from over 750 companies and 45 government organizations between 1984 and 2016. Additional facts and data were published in the author’s 19 books and over 100 journal articles.

Table 1: Results of Poor Quality and High Quality Software

(Nominal 10,000 function points; 1,500 SNAP points)

			Poor Quality	High Quality
	U.S. Software Defect Potentials per Function Point 2016
1	Requirements defects (functional and non-functional)		0.90	0.30
2	Architecture defects		0.20	0.05
3	Design defects		1.10	0.35
4	Code defects		1.55	0.60
5	Security flaw defects		0.35	0.10
6	Document defects		0.45	0.20
7	Bad fix defects (new bugs in bug repairs)		0.55	0.05
	Total		5.10	1.65

	Application Defect Removal Efficiency Results
8	Defect removal efficiency (DRE)		91.00%	98.50%
9	Defects removed per function point		4.64	1.63
10	Defects removed – actual total		46,410	16,253
11	Defects delivered per function point		0.46	0.02
12	Defects delivered – actual total		4,590	247
13	High severity defects delivered per function point		0.46	0.02
14	High severity defects delivered – actual total		689	27
15	Security flaws delivered – actual total		55	1

			Poor Quality	High Quality
	Application Defect Prevention Stages
16	Joint application design (JAD)		No	Yes
17	Prototype		Yes	Yes
18	Requirements models (primarily functional)		No	Yes
19	Quality function deployment (QFD)		No	Yes
20	Automated proofs		No	Yes
21	SEMAT (Software Engineering Methods and Theory)		No	Yes
22	Six-Sigma for software		No	Yes
23	Capability Maturity Model Integration (CMMI) – defense only		No	No
	Total		1	7

			Poor Quality	High Quality
	Application Pre-Test Defect Removal Stages
24	Formal inspections of requirements		No	Yes
25	Formal inspection of architecture (large systems)		No	Yes
26	Formal inspections of design		No	Yes
27	Formal inspections of new/changed code		No	Yes
28	Formal quality assurance reviews		No	Yes
29	Pair programming (not recommended)		No	No
30	Independent verification & validation (defense only)		No	No
31	FOG readability index of requirements, design		No	Yes
32	Static analysis of application code and changed code		No	Yes
33	Ethical hackers on high-security software		No	Yes
34	Cyclomatic complexity analysis and reduction		No	Yes
35	SANS Institute defect category analysis and removal		No	Yes
	Total		1	10

			Poor Quality	High Quality
	Application Test Defect Removal Stages
36	Unit test – automated		Yes	Yes
37	New function test		Yes	Yes
38	Regression test – automated		Yes	Yes
39	Stress/performance test		Yes	Yes
40	Usability test		No	Yes
41	Component test		Yes	Yes
42	Independent test (defense only)		No	No
43	Security test		No	Yes
44	System test – automated		Yes	Yes
45	Multi-platform test		Yes	Yes
46	Global nationalization test		Yes	Yes
47	Beta/acceptance test		Yes	Yes
	Total		9	11

			Poor Quality	High Quality
	Application Post-Release Quality Stages
48	Static analysis of all code changes/bug repairs		No	Yes
49	Formal inspection of large changes		No	Yes
50	Cyber-attack defenses (firewalls, antivirus, etc.)		Yes	Yes
51	Penetration teams (high security applications)		No	Yes
52	Maintainability analysis of legacy applications		No	Yes
53	Test library analysis (defective test case removal)		No	Yes
54	Error-prone module (EPM) analysis and removal		No	Yes
55	Race-condition analysis and correction		No	Yes
56	Cyclomatic complexity analysis and correction		No	Yes
	Total		1	9
56	TOTAL QUALITY CONTROL FACTORS	19		53

All of these quality control factors are important for large systems in the 10,000 function-point size range. The problem today is that no known software quality company sells more than one or two of these 56 quality methods or even knows about the others!

Few quality companies and even fewer of their clients know about the other factors! A narrow focus on testing and basic ignorance of the suite of effective software defect prevention and defect removal methods is an endemic and chronic problem in the software industry.

If a reader wanted to learn about all 56 quality factors, he or she would probably need a dozen courses from at least half a dozen software quality training companies because none of the courses cover the full spectrum of effective quality tools and methods.

None of the major international standards such as ISO and IEEE standards on software quality are fully adequate because none of them ask for quantitative quality data and all ignore basic software quality factors such as defect potentials and defect removal efficiency (DRE).

Software Quality Education Curricula

The curricula of the major software quality training companies are embarrassing because of the gaps, omissions, and topics that are not covered. Even worse, not a single quality education company has actual quantified data on software defect origins, defect densities, defect prevention, or defect removal efficiency (DRE) levels. You would have to go back almost 200 years in medical education to find such skimpy knowledge of the basic topics needed to train physicians as we have for training software quality and test personnel in 2016.

You might take quality courses from companies such as Construx, from CAST, from IBM, from ITMPI, from SQE, from QAI, from the SANS Institute, from Parasoft, from Smart Bear, and probably from other local educators, but these would likely be single-topic courses such as static analysis or automated testing. The courses, while useful by themselves, would not be part of a full software quality curriculum because none of the quality companies know enough about software quality to have effective overall curricula!

Worse, these courses even from major quality companies would lack quantitative data on defect potentials, defect removal efficiency (DRE), bad-fix injections, error-prone modules or any other of the critical topics that quality professionals should know about. The software industry is running blind due to a widespread lack of quantitative quality data.

Software quality data is available from some benchmark organizations such as Davids Consulting, Gartner Group, Namcook Analytics LLC, TIMetricas, Q/P Management Group/ QSM, and several others. But the combined set of clients for all current quality benchmark organizations is less than 50,000 customers in an industry employing close to 20,000,000 people on a global basis.

Software quality data can be predicted by some parametric software estimation tools such as Software Risk Master (SRM), KnowledgePlan, SEER, SLIM, and COCOMO, but the combined market for all of these parametric tools is less than 25,000 customers in an industry employing almost 20,000,000 people on a global basis.

In other words, even companies that offer accurate quality data have comparatively few clients who are interested in that data, even though it could save companies and governments billions of dollars in reduced defect repairs and reduced cyber-attack recovery costs!

It is professionally embarrassing about how unsophisticated software quality education is compared to medical school curricula for training physicians.

You probably could not take courses on this set of 56 topics from any university because their curricula tend to deal only with a few of the more common methods and concentrate primarily on testing. I have yet to see a university with quantitative data on software defect volumes, severity levels, origins, or effective defect removal methods with quantitative results.

You might take some courses from non-profit associations such as the American Society for Quality (ASQ), the Society for Information Management (SIM) or the Project Management Institute (PMI). But no single organization in 2016 covers more than a small fraction of the total intellectual content of effective software quality control. None of the software non-profit organizations have quantitative data on defect volumes, severity levels, or defect removal efficiency. To illustrate the kind of quality education that is needed, Table 2 shows a sample curriculum for software quality assurance testing:

Table 2: Software Quality Assurance Curricula Circa 2016

	Software Quality Assurance Courses	Days	Value
1	Hazardous quality metrics: cost per defect	0.50	10.00
2	Hazardous quality metrics: lines of code	0.50	10.00
3	Hazardous quality metrics: technical debt	0.50	10.00
4	Hazardous quality metrics: story points	0.50	10.00
5	Effective quality metrics: function points	0.50	10.00
6	Effective quality metrics: defect removal %	0.50	10.00
7	Effective quality metrics: defect severity levels	0.50	10.00
8	Effective quality metrics: defect origin analysis	0.50	10.00
9	Emerging quality metrics: SNAP points	0.50	10.00
10	Overview of Major Software Failures	1.00	10.00
11	Overview of Major Software Cyber-Attacks	1.00	10.00
12	Error Prone Module (EPM) Analysis	1.00	10.00
13	Software Defect Detection Efficiency (DDE)	1.00	10.00
14	Software Defect Removal Efficiency (DRE)	1.00	10.00
15	Software Defect Tracking	1.00	10.00
16	Software Defect Prevention (JAD, QFD, etc.)	1.00	10.00
17	Software Pre-Test Defect Removal	1.00	10.00
18	Software Test Defect Removal	1.00	10.00
19	Software Requirements Modeling	1.00	10.00
20	Functional and non-Functional Requirements	2.00	10.00
21	Software Static Analysis: text	1.00	10.00
22	Software Static Analysis: code	1.00	10.00
23	Software correctness proofs: manual	1.00	10.00
24	Software correctness proofs: automated	1.00	10.00
25	Software Security and Quality in 2016	2.00	10.00
26	Quality Benchmarks: Namcook, Q/P, etc.	1.00	10.00
27	Software Security Inspections	3.00	10.00
28	Security flaw removal (hacking, test, etc.)	3.00	10.00
29	Error Prone Module (EPM) Analysis	2.00	9.95
30	Software Test Case Design	2.00	9.75
31	Software Test Library Management	1.00	9.75
32	Reducing Bad-Fix Injections	1.00	9.75
33	Test Case Conflicts and Errors	1.00	9.75
34	Software Requirement Inspections	1.00	9.75
35	Software Design Inspections	2.00	9.50
36	Software Code Inspections	2.00	9.50
37	Software Test Inspections	2.00	9.50
38	Defect removal using Pair Programming	1.00	9.50
39	Defect removal using Container Development	1.00	9.50
40	Defect removal using DevOps	2.00	9.50
41	Defect removal using TSP/PSP	2.00	9.00
42	Defect removal using Agile	2.00	9.00
43	Defect removal using RUP	2.00	9.00
44	Automated software testing	2.00	9.00
45	Quality Assurance of Software Reuse	1.00	9.00
46	Quality Assurance of COTS and ERP	1.00	9.00
47	Quality Assurance of Open Source	1.00	9.00
48	Tools: Quality Assurance	1.00	9.00
49	Tools: Defect Prediction	1.00	9.00
50	Defect removal using Waterfall Development	1.00	8.00
51	Cost of Quality (COQ)	1.00	8.00
52	Overview of the CMMI	1.00	8.00
53	ISO and IEEE Quality Standards	1.00	7.00
54	Six Sigma: Green Belt	3.00	7.00
55	Six Sigma: Black Belt	3.00	7.00
	Total	70.50	9.49

In today’s world, software quality assurance has an expanding role in cyber defenses and cyber-attack recovery. Software quality assurance personnel need much more knowledge on security topics in 2016 than they did 30 years ago in 1986.

Table 3: Software Testing Topics Circa 2016

	Software Topics Courses	Days	Value
1	Test Case Design Optimization	2.00	10.00
2	Test Cases – Design of Experiments	2.00	10.00
3	Test Cases – Cause/Effect Graphing	2.00	10.00
4	Test cases and requirements	2.00	10.00
5	Risk-based test case design	2.00	10.00
6	Analysis of gaps and errors in test case designs	2.00	10.00
7	Cyclomatic complexity and test coverage	2.00	10.00
8	Test Library Control	2.00	10.00
9	Security Testing Overview	2.00	10.00
10	Advanced Security Testing	3.00	10.00
11	Test Schedule Estimating	1.00	10.00
12	Software Defect Potential Estimating	1.00	10.00
13	Defect Removal Efficiency (DRE) Measurement	1.00	10.00
14	Software Build Planning and Control	1.00	10.00
15	Big Data Test Design	2.00	10.00
16	Cloud Test Design	2.00	10.00
17	Removal of Incorrect Test Cases	1.00	10.00
18	Test Coverage Analysis	1.00	9.50
19	Identifying Error-Prone Modules (EPM)	2.00	9.50
20	Data Base Test Design	1.00	9.50
21	Test Case Conflicts and Errors	1.00	9.25
22	Static analysis and testing	1.00	9.00
23	Reducing Bad-Fix Injections	1.00	9.00
24	Basic black box testing	1.00	9.00
25	Basic white box testing	1.00	9.00
26	Basic gray box testing	1.00	9.00
27	Fundamentals or Risk-Based Testing	1.00	9.00
28	Fundamentals of Unit Testing	1.00	9.00
29	Fundamentals of Regression Testing	1.00	9.00
30	Fundamentals of Component Testing	1.00	9.00
31	Fundamentals of Stress Testing	1.00	9.00
32	Fundamentals of Virus Testing	2.00	9.00
33	Fundamentals of Lab Testing	1.00	9.00
34	Fundamentals of System Testing	2.00	9.00
35	Fundamentals of External Beta Testing	1.00	9.00
36	Fundamentals of Acceptance Testing	1.00	9.00
37	Testing Web Applications	1.00	9.00
38	Tools: Automated Testing	2.00	9.00
39	Tools: Test Case Design	1.00	9.00
40	Tools: Test Library Control	1.00	9.00
41	Tools: Defect Tracking	1.00	9.00
42	Tools: Complexity Analysis	0.50	9.00
43	Tools: Test Coverage Analysis	0.50	9.00
44	Fundamentals of reusable test materials	1.00	9.00
45	Testing Cloud, SOA, and SaaS	2.00	8.80
46	Testing COTS Application Packages	1.00	8.75
47	Testing ERP Applications	1.00	8.75
48	Testing Reusable Functions	1.00	8.75
49	Supply Chain Testing	1.00	8.50
50	Function Points for Test Measures	1.00	7.00
	TOTAL	67.00	9.31

Software testing has become a barrier to cyber-attacks, so special attention is needed for testing software security flaws. Table 3 lists topics that should be addressed in software testing courses.

Between software quality assurance training and software test personnel training there is a need to expand on both university curricula and the limited curricula from software quality companies, neither of which are fully adequate as of 2016.

If you wanted to acquire actual supporting tools for these 56 quality topics you would probably need to go to at least 15 commercial quality companies, static analysis companies, and test tool companies and another half dozen open source quality groups.

Nobody in 2016 sells all the tools that are needed to control software quality! Most quality tool vendors don’t even know about effective quality tools other than the ones they sell. There are no software quality companies in 2016 that have the depth and breadth of medical companies such as McKesson or Johnson & Johnson.

The static analysis companies only sell static analysis; the testing companies only sell test tools; to get quality metrics and measurement tools you need additional vendors; to get ordinary defect tracking tools you need still other vendors; to get quality benchmark data you need another set of vendors; to get software quality predictions via commercial estimating tools you need yet another set of vendors.

The software industry has nothing like a Mayo Clinic where advanced medical treatments are available for a wide spectrum of medical conditions. In fact software quality control in 2016 is closer to the days 200 years ago when doctors were also barbers and sterile surgical procedures had not yet been discovered. Software has nothing even close to CAT scans and MRI exams for finding quality and security problems.

No known software company in 2016 covers the full spectrum of software quality tools, technologies, topics, and effective quality methods, although a few large companies such as IBM, Microsoft, and Hewlett Packard may sell perhaps 12 to 15 out of the set of 56.

No university in 2016 has a truly effective software quality curriculum. In fact many universities still teach courses using “cost per defect” and “lines of code” metrics and hence have no accurate quality data available at all, since these metrics distort reality.

Of course no pharmaceutical company sells medicines for all diseases and no physicians can treat all medical conditions; however, physicians at least learn about almost all common medical conditions as a basic part of their education. There are also specialists available who can deal with uncommon medical conditions.

Medicine has the Index Medicus that provides an overall description of the use of thousands of prescription drugs, their side effects, and dosage. There is no exact equivalent to the Index Medicus for software bugs and their treatment, but the closest is probably Capers Jones’ and Olivier Bonsignour’s book on The Economics of Software Quality, published in 2012.

Medicine also has many wide-ranging books such as Control of Communicable Diseases in Man published by the U.S. Surgeon General’s office which show the scope of common infectious diseases such as polio and smallpox as well as their known treatments. Software has nothing like the breadth and depth of the medical literature.

A book recommended by the author to all clients and colleagues is The Social Transformation of American Medicine by Paul Starr. This book won a Pulitzer Prize in 1982. It also won the Bancroft Prize. This book provides an excellent guide concerning how medicine was transformed from a poorly educated craft into one of the top learned professions in world history.

Surprisingly, at one time about 150 years ago, medicine was even more chaotic than software is today. Medical schools did not require college degrees or even high-school graduation to enter. Medical students never entered hospitals during training because the hospitals used private medical staff. There was no monitoring of medical malpractice and quacks could become physicians. There were no medical licenses or board certifications.

There was no formal evaluation of prescription drugs before release and harmful substances such as opium could be freely prescribed. (A Sears-Roebuck catalog in the 1890’s offered liquid opium as a balm for quieting noisy children. This product was available without prescription.)

Paul Starr’s excellent book shows how the American Medical Association (AMA) transformed itself and also medical practice to improve medical education and introduce medical licenses and board certifications.

This book by Paul Starr provides a full guide for the set of steps needed by the software industry in order to become a true profession.

One of the interesting methods used by the AMA was reciprocal membership with all state medical societies. This had the effect of raising AMA membership from below 800 to more than 80,000 which finally gave physicians enough political clout to lobby for medical licenses. It would be interesting to see the impact if the Institute of Electrical and Electronic Engineers (IEEE), Society for Information Management (SIM), Association for Computing Machinery (ACM), and other software professional organizations also had reciprocal memberships instead of more or less competing with one another.

Poor software quality is a sociological problem as well as a technology problem. Starr’s book showed how medicine gradually improved both the sociology of medical practice and the underlying technology of medical offices and hospitals over about a 50-year period.

With Starr’s book as a guide, software engineering might be able to accomplish the same results in less than 25 years instead of the 50 years required to professionalize medicine.

The three major problems facing the software industry in 2016 are these:

Software has poor quality control due to lack of knowledge of effective software quality techniques.
Software has embarrassingly poor education on software quality due to lack of empirical data.
Software has embarrassingly bad and incomplete quality data due to use of ineffective and hazardous metrics such as “cost per defect,” combined with the failure to use effective metrics such as function points and defect removal efficiency (DRE).

The sad thing about poor software quality is that all three of these problems are treatable conditions that could be eliminated in less than 10 years if one or more major software companies became proactive in 1) Effective quality metrics and measures, 2) Fact-based education with quantitative data, and 3) Expanded quality control that encompassed effective quality measures, effective defect prevention, effective pre-test defect removal, and effective formal testing.

The author’s hope is that vaccinations and effective treatments for poor software quality will be developed soon such as vaccinations for smallpox and polio were developed, and anti-biotics were developed for many bacterial infections. There are still untreatable medical conditions, but overall medicine has made huge advances in prevention and control of hundreds of formerly serious and common diseases. Software has not yet made any major advances in quality control although some modern methods such as static analysis hold promise.

Author’s end note: The author’s first job after college was editing a medical journal and medical papers for the Office of the Surgeon General of the U.S. Public Health Service. The author’s first programming job was also for the USPHS working on medical applications for the National Institutes of Health (NIH).

As an editor of many medical papers and then an author and editor for many software papers, the differences between medical writing and software writing are striking. In medical papers about a third of the text is devoted to descriptions of the measurement methods used to collect the data that form the main opinions of the articles and lead to the key conclusions.

In software papers, including refereed papers for major journals, there is barely a mention of measurement methods. Worse, many software articles use flawed metrics such as cost per defect and lines of code (LOC) without any apparent awareness that these metrics distort reality. Some software articles that use LOC metrics do not even state whether physical lines or logical statements were used by the author in spite of the 500% variances between the two!

The reason that the author has developed such a large collection of quantitative software data is due to my early experience with medical reports and medical writing, where quantitative results are the norm and not a rare exception as they are with the software literature.

The author’s first book on software quality in 1993, Software Quality: Analysis and Guidelines for Success, followed the same format as the medical book, Control of Communicable Diseases in Man, and the format worked quite well.

The author’s most recent book in 2014 on The Technical and Social History of Software Engineering, was inspired by Paul Starr’s book on The Social Transformation of American Medicine, although it did not use quite the same format. For one thing, software has not yet been transformed from a craft into a true profession while Starr’s book was written after the transformation of medicine from a craft into a top profession.

There is much the software industry could learn from the structure and nature of medical education and the medical literature which is far more professional than the software literature and software education. Software could also learn from sociological medical factors such as reciprocal memberships among professional associations, and the need for licenses and board certifications for software specialists.

Above all, software should learn that progress depends upon accurate quantified information and not upon unproven claims and questionable assertions. It has been over 100 years since a new medicine could be released to physicians without testing and validation. New software methods and new programming languages come out almost every month without any validation at all.

List of Acronyms Used in this Paper

Acronym Explanation

ACM Association for Computing Machinery

ACR Applied Computer Research

AMA American Medical Association

ASQ American Society for Quality

CAST CAST is a multinational technology corporation, with headquarters in France and New York City.

COCOMO Constructive Cost Model

Construx Construx is a company in Bellevue WA that provides training and consulting services concerning software development.

CMMI Capability Maturity Model Integration

COQ Cost of Quality

COTS Commercial Off-The-Shelf

DRE Defect Removal Efficiency

EPM Error Prone Module

ERP Enterprise Resource Planning

IBM International Business Machines

IEEE Institute of Electrical and Electronic Engineers

ISBSG International Software Benchmarking Standards Group

ISO International Standards Organization

IT Information Technology

ITMPI Information Technology Metrics and Productivity Institute

JAD Joint Application Design

JAVA Java is a programming language and computing platform first released by Sun Microsystems in 1995.

LOC Lines of Code

NIH National Institutes of Health

Parasoft Parasoft is a company headquartered in Monrovia CA that researches and develops software solutions.

PMI Project Management Institute

PSP Personal Software Process

QAI Quality Assurance Institute

QFD Quality Function Deployment

Q/P Q/P Management Group, Inc., located in Stoneham MA, is a provider of software measurement, benchmarking, quality and productivity consulting services.

QSM Quantitative Software Management is a company located in McLean VA that provides software measurement and estimation services.

RUP Rational Unified Process

SaaS Software as a Service

SANS System Administration, Networking, and Security Institute

SEER A tool utilized for software estimation

SEMAT Software Engineering Methods and Theory

SIM Society for Information Management

SLIM A systems and software project estimation tool

Smart Bear A company headquartered in Somerville MA that provides software testing tools.

SNAP Points Software sizing is an activity in software engineering that is used to estimate the size of a software. The non-functional size is measured by SNAP Points.

SOA Service Oriented Architecture

SPR Software Productivity Research

SRM Software Risk Manager

SQE Software Quality Engineering

TSP Team Software Process

USPHS United States Public Health Service

References and Readings in Software Quality Control

Abrain, Alain. Software Estimating Models. Wiley-IEEE Computer Society, 2015.

Abrain, Alain. Software Metrics and Metrology. Wiley-IEEE Computer Society, 2010.

Abrain, Alain. Software Maintenance Management: Evolution and Continuous Improvement. Wiley-IEEE Computer Society, 2008.

Abrain, A. and P. N. Robillard. “Function Point Analysis: An Empirical Study of its Measurement Processes.” IEEE Transactions on Software Engineering, Vol. 22, No. 12, December 1996, pp. 895-909.

Albrecht, Allan. AD/M Productivity Measurement and Estimate Validation. Purchase, NY: IBM Corporation, May 1984.

Austin, Robert D. Measuring and Managing Performance in Organizations. New York, NY: Dorset House Press, 1996, 216 pages. ISBN 0-932633-36-6.

Beck, Kent. Test-Driven Development. Boston, MA: Addison Wesley, 2002, 240 pages. ISBN 10: 0321146530.

Black, Rex. Managing the Testing Process: Practical Tools and Techniques for Managing Hardware and Software Testing. Wiley, 2009, 672 pages. ISBN-10 0470404159.

Boehm, Barry. Software Engineering Economics. Englewood Cliffs, NJ: Prentice Hall, 1981, 900 pages. ISBN-10 0138221227.

Brooks, Fred. The Mythical Man-Month. Reading, MA: Addison-Wesley, 1974, revised in 1995.

Chelf, Ben and Raoul Jetley. “Diagnosing Medical Device Software Defects Using Static Analysis.” San Francisco, CA: Coverity Technical Report, 2008.

Chess, Brian and Jacob West. Secure Programming with Static Analysis. Boston, MA: Addison Wesley, 2007, 624 pages. ISBN 13: 978-0321424778.

Cohen, Lou. Quality Function Deployment: How to Make QFD Work for You. Upper Saddle River, NJ: Prentice Hall, 1995, 368 pages. ISBN 10: 0201633302.

Crosby, Philip B. Quality is Free. New York, NY: New American Library, Mentor Books, 1979, 270 pages. ISBN-10 0070145121.

Charette, Bob. Software Engineering Risk Analysis and Management. New York, NY: McGraw Hill, 1989.

Charette, Bob. Application Strategies for Risk Management. New York, NY: McGraw Hill, 1990.

Constantine, Larry L. Beyond Chaos: The Expert Edge in Managing Software Development. ACM Press, 2001, ISBN-10 0201719606.

DeMarco, Tom. Peopleware: Productive Projects and Teams. New York, NY: Dorset House, 1999, 245 pages. ISBN 10: 0932633439.

DeMarco, Tom. Controlling Software Projects. New York, NY: Yourdon Press, 1982, 284 pages. ISBN 0-917072-32-4.

Everett, Gerald D. and Raymond McLeod. Software Testing. Hoboken, NJ: John Wiley & Sons, 2007. ISBN 978-0-471-79371-7, 261 pages.

Ewusi-Mensah, Kweku. Software Development Failures. Cambridge, MA: Massachusetts Institute of Technology Press, 2003, ISBN 0-26205072-2, 276 pages.

Flowers, Stephen. Software Failures: Management Failures, Amazing Stories, and Cautionary Tales; John Wiley & Sons; 1996, ISBN-10 0471951137.

Gack, Gary. Managing the Black Hole: The Executives Guide to Software Project Risk. Thomson, GA: Business Expert Publishing, 2010, ISBN10: 1-935602-01-9.

Gack, Gary. Applying Six Sigma to Software Implementation Projects http://software.isixsigma.com/library/content/c040915b.asp.

Galorath, Dan and Michael Evans. Software Sizing, Estimation, and Risk Management: When Performance is Measured, Performance Improves. Philadelphia, PA: Auerbach, 2006.

Garmus, David and David Herron. Function Point Analysis – Measurement Practices for Successful Software Projects. Boston, MA: Addison Wesley Longman, 2001, ISBN 0-201-69944-3, 363 pages.

Gibbs, T. Wayt. “Trends in Computing: Software’s Chronic Crisis”. Scientific American Magazine, 271(3), International edition, pp 72-81, September 1994.

Gilb, Tom and Dorothy Graham. Software Inspections. Reading, MA: Addison Wesley, 1993, ISBN 10: 0201631814.

Harris, Michael D.S., David Herron, and Stasia Iwanacki. The Business Value of IT: Managing Risks, Optimizing Performance and Measuring Results. Boca Raton, FL: CRC Press, Auerbach Publications; ISBN-10 1420064748, 2009.

Hill, Peter R. Practical Software Project Estimation: A Toolkit for Estimating Software Development Effort & Duration. New York: McGraw Hill, 2010, ISBN -10 0071717010.

Hill, Peter, Capers Jones, and Don Reifer. The Impact of Software Size on Productivity. Melbourne, Australia: International Software Benchmarking Standards Group (ISBSG), September 2013.

Howard, Alan (Ed.). Software Metrics and Project Management Tools. Phoenix, AZ: Applied Computer Research (ACR), 1997, 30 pages.

Humphrey, Watts. Managing the Software Process. Reading, MA: Addison Wesley, 1989.

International Function Point Users Group (IFPUG). IT Measurement – Practical Advice from the Experts. Boston, MA: Addison Wesley Longman, 2002, ISBN 0-201-74158-X; 759 pages.

Jacobsen, Ivar, Martin Griss, and Patrick Jonsson. Software Reuse – Architecture, Process, and Organization for Business Success. Reading, MA: Addison Wesley Longman, ISBN 0-201-92476-5, 1997, 500 pages.

Jacobsen, Ivar et al. The Essence of Software Engineering; Applying the SEMAT Kernel. Boston, MA: Addison Wesley Professional, 2013.

Jones, Capers. Software Risk Master (SRM) Tutorial. Narragansett RI: Namcook Analytics LLC, 2015.

Jones, Capers. Software Defect Origins and Removal Methods. Narragansett RI: Namcook Analytics LLC, 2015.

Jones, Capers. The Mess of Software Metrics. Narragansett RI: Namcook Analytics LLC, 2015.

Jones, Capers. The Technical and Social History of Software Engineering. Boston, MA: Addison Wesley, 2014.

Jones, Capers and Olivier Bonsignour. The Economics of Software Quality; Boston, MA: Addison Wesley, 2011, ISBN 978-0-13-258220-9, 587 pages.

Jones, Capers. Software Engineering Best Practices. New York, NY: McGraw Hill, 2010, ISBN 978-0-07-162161-8, 660 pages.

Jones, Capers. Applied Software Measurement. New York, NY: McGraw Hill, 3rd edition 2008, ISBN 978-0-07-150244-3, 662 pages.

Jones, Capers. Critical Problems in Software Measurement. Information Systems Management Group, 1993, ISBN 1-56909-000-9, 195 pages.

Jones, Capers. Software Productivity and Quality Today — The Worldwide Perspective. Information Systems Management Group, 1993, ISBN -156909-001-7, 200 pages.

Jones, Capers. Assessment and Control of Software Risks. Englewood Cliffs, NJ: Prentice Hall, 1994, ISBN 0-13-741406-4, 711 pages.

Jones, Capers. New Directions in Software Management. Information Systems Management Group, ISBN 1-56909-009-2, 150 pages.

Jones, Capers. Patterns of Software System Failure and Success. Boston, MA: International Thomson Computer Press, December 1995, ISBN 1-850-32804-8, 292 pages.

Jones, Capers. “Sizing Up Software.” Scientific American Magazine, December 1998:279(6)104-111.

Jones, Capers. Conflict and Litigation between Software Clients and Developers. Narragansett, RI: Software Productivity Research Technical Report, 2007, 65 pages.

Jones, Capers. Software Quality – Analysis and Guidelines for Success. Boston, MA: International Thomson Computer Press, ISBN 1-85032-876-6, 1997, 492 pages.

Jones, Capers. Estimating Software Costs, 2^nd edition. New York, NY: McGraw Hill, 2007, ISBN-10 1850328676700, 700 pages.

Jones, Capers. “The Economics of Object-Oriented Software.” Burlington, MA, Software Productivity Research: SPR Technical Report, April 1997, 22 pages.

Jones, Capers. “Software Project Management Practices: Failure versus Success.” Crosstalk, October 2004.

Jones, Capers. “Software Estimating Methods for Large Projects.” Crosstalk, April 2005.

Kan, Stephen H. Metrics and Models in Software Quality Engineering, 2^nd edition. Boston, MA: Addison Wesley Longman, ISBN 0-201-72915-6, 2003, 528 pages.

Land, Susan K; Douglas B Smith, and John W Walz. Practical Support for Lean Six Sigma Software Process Definition: Using IEEE Software Engineering Standards. Wiley-IEEE Computer Society Press, 2008, ISBN 10: 0470170808, 312 pages.

McConnell, Steve. Software Project Survival Guide. Redmond, WA: Microsoft Press, 1997, ISBN-10 1572316217, 288 pages.

Mosley, Daniel J. The Handbook of MIS Application Software Testing. Englewood Cliffs, NJ: Yourdon Press, Prentice Hall, 1993, ISBN 0-13-907007-9, 354 pages.

Nandyal; Raghav. Making Sense of Software Quality Assurance. New Delhi, India: Tata McGraw Hill Publishing, 2007, ISBN 0-07-063378-9, 350 pages.

Pressman, Roger. Software Engineering – A Practitioner’s Approach, 6^th edition. New York, NY: McGraw Hill, 2005, ISBN 0-07-285318-2.

Radice, Ronald A. High Quality Low Cost Software Inspections. Andover, MA: Paradoxicon Publishing, ISBN 0-9645913-1-6, 2002, 479 pages.

Royce, Walker E. Software Project Management: A Unified Framework. Reading, MA: Addison Wesley Longman, 1998, ISBN 0-201-30958-0.

Starr, Paul. The Social Transformation of American Medicine. New York, NY: Basic Books; Perseus Group, 1982, ISBN 0-465-07834-2. AUTHOR’S NOTE: This book won a Pulitzer Prize in 1982 and is highly recommended as a guide for improving both professional education and professional status. There is much of value for the software community in this book.

Strassmann, Paul. Information Payoff. Stamford, CT: Information Economics Press, 1985.

Strassmann, Paul. Governance of Information Management: The Concept of an Information Constitution; 2^nd edition (eBook). Stamford, CT: Information Economics Press, 2004.

Strassmann, Paul. Information Productivity. Stamford, CT: Information Economics Press, 1999.

Weinberg, Gerald M. The Psychology of Computer Programming. New York, NY: Van Nostrand Reinhold, 1971; ISBN 0-442-29264-3, 288 pages.

Weinberg, Gerald M. Becoming a Technical Leader. New York, NY: Dorset House, 1986, ISBN 0-932633-02-1, 284 pages.

Weinberg, Gerald M. Quality Software Management – Volume 2 First-Order Measurement. New York, NY: Dorset House Press, ISBN 0-932633-24-2, 1993, 360 pages.

Wiegers, Karl A. Creating a Software Engineering Culture. New York, NY: Dorset House Press, 1996, ISBN 0-932633-33-1, 358 pages.

Wiegers, Karl E. Peer Reviews in Software – A Practical Guide. Boston, MA: Addison Wesley Longman, ISBN 0-201-73485-0, 2002, 232 pages.

Yourdon, Ed. Outsource: Competing in the Global Productivity Race. Upper Saddle River, NJ: Prentice Hall PTR, ISBN 0-13-147571-1, 2005, 251 pages.

Yourdon, Ed. Death March – The Complete Software Developer’s Guide to Surviving “Mission Impossible” Projects. Upper Saddle River, NJ: Prentice Hall PTR, ISBN 0-13-748310-4, 1997, 218 pages.

Systems Engineering News

National Academy of Engineering: The Engineering Challenges of the 21^st Century

A diverse committee of experts from around the world, some of the most accomplished engineers and scientists of their generation, proposed 14 challenges. Rather than attempt to include every important goal for engineering, the panel chose opportunities that are both achievable and sustainable to help people and the planet thrive. The panel’s conclusions were reviewed by more than 50 subject-matter experts. The effort received worldwide input from prominent engineers and scientists and the general public. Here are the challenges:

Make solar energy economical
Provide energy from fusion
Develop carbon sequestration methods
Manage the nitrogen cycle
Provide access to clean water
Restore and improve urban infrastructure
Advance health informatics
Engineer better medicines
Reverse-engineer the brain
Prevent nuclear terror
Secure cyberspace
Enhance virtual reality
Advance personalized learning
Engineer the tools of scientific discovery

PPI SyEN 51

Subscribe to PPI SyEN

Image Source

PPI SyEN 134

PPI SyEN 133

PPI SyEN 132

PPI SyEN 131

FREE Original SE Articles and Industry News?

Subscribe to PPI

MORE About Us