Home » SyEN Newsjournal » PPI SyEN 52

PPI SyEN 52

Read monthly Project Performance International's Systems Engineering Newsjournal, named "PPI SyEN". PPI SyEN presents for the engineering professional 30-60 pages of valuable technical articles on topical subjects, shorter technical pieces, in-depth coverage of the month's news in systems engineering and directly related fields, pointers to useful resources and relevant industry events, plus limited information on PPI's activities.

Welcome to PPI SyEN 52

In This Edition

Quotations to Open On

Feature Article

The Influence of Systems Engineering on the Strategy of a Research University by Stephen E. Cross

Article

Agile Software Quality – A Quantitative Assessment by Donald J. Reifer

Systems Engineering News

Integrating Program Management and Systems Engineering
How Systems Engineering Can Help Fix Health Care
“The Father of Process Systems Engineering” – Roger Sargent to Receive Sir Frank Whittle Medal
Project in Gambia Helps Combat Coastal Erosion and Flooding
Legacy Systems are Problems for Boardrooms Not Computer Geeks
CIMdata Announces the Expansion of Manufacturing Systems Engineering Consulting Practice
U. S. Air Force Charts Wideband Global Satellite Future
DARPA Licenses Emerging Chip Technology
Nordic Systems Engineering Voyage 2017
Newcomer’s View of INCOSE’s International Workshop 2017

Featured Organizations

National Academy of Engineering (NAE)
Institute of Industrial and Systems Engineers (IISE)
Georgia Tech Research Institute (GTRI)
United States Air Force Scientific Advisory Board (SAB)

Systems Engineering Tools News

PragmaDev Studio

Systems Engineering Publications

Systems Engineering Journal
The Art and Science of Systems Engineering
Made to Stick: Why Some Ideas Survive and Others Die
The Routledge Companion to Lean Management
Balanced Scorecard Step-by-Step: Maximizing Performance and Maintaining Results (2nd Edition)
A Primer for Model-Based Systems Engineering

Education and Academia

MIT’s Architecture and Systems Engineering Professional Certificate Program
Industrial Engineering and Systems Engineering at Florida Tech
Southeast Missouri State University Launching New Engineering Program in Fall 2017
Institute for Mathematics and its Applications, University of Minnesota College of Science & Engineering
Welcome to the EIT Digital Professional School
solidThinking Aviation Case Study Teaches Embedded System Design
University of Maryland Baltimore Campus Systems Engineering Graduate Programs

Some Systems Engineering-Relevant Websites

Standards and Guides

NIST Releases Internet of Things (IoT) Security Guidance

Definitions to Close On

A Simple Explanation of ‘The Internet of Things’

Conferences and Meetings

PPI News

PPI and CTI Events

PPI Upcoming Participation in Professional Conferences

Quotations to Open On

“People who have “systems engineer” in their title, regardless of the modifiers – program, project, flight system, and so on – are responsible for everything.”

Gentry Lee, Jet Propulsion Laboratory

“A great systems engineer completely understands and applies the art of leadership and has the experience and scar tissue from trying to earn the badge of leader from his or her team.”

Harold Bell, NASA Headquarters

“Great networking includes genuinely helping people, and staying in touch when you don’t need something.”

Ronnette Meyers

Feature Article

The Influence of Systems Engineering on the Strategy of a Research University

Stephen E. Cross

Georgia Institute of Technology

Version 2 February 8, 2017

Abstract

Publicly supported colleges and universities in the United States are expected to educate students, conduct research, and support economic growth within their states and regions. One such university is the Georgia Institute of Technology which over the past 40 years has gained a global reputation for its leading-edge research programs. Its strategy for research and economic development is based on system engineering principles which are described in this paper. Also described is an ongoing effort to create a balanced scorecard to guide and assess performance under this strategy.

Website: www.gatech.edu/

Email: evpr@gatech.edu

Introduction

The Georgia Institute of Technology (Georgia Tech), is a research university in Atlanta, Georgia (USA). It has a legacy culture of addressing real world problems and of embedding innovation throughout its education and research programs. It drives economic development in the southeast United States. A new strategic vision, introduced in 2010 to guide Georgia Tech’s role nationally and internationally, included a systems approach to faculty-led research to concurrently pursue transformational research and economic/societal impact. The systems approach guides research infrastructure investments in core research areas and institute-wide support for discovery, application, and deployment functions. Significant results have been achieved over the past five years. An effort to define and implement a balanced scorecard to guide future work under the strategy is currently underway. This paper explains the systems-focused approach to the university’s research strategy, the progress that has been achieved, and some initial thoughts on the balanced scorecard. Feedback concerning the scorecard from readers of the Systems Engineering Newsletter (SyEN) is most welcome and will further strengthen and improve the value of the scorecard (please provide feedback to evpr@gatech.edu).

Systems Engineering

Systems engineering is defined as “… an interdisciplinary approach and means to enable the realization of successful systems.” Systems engineering and management concepts are described in (Sage & Armstrong, 2000); (Sage & Rouse, 2009); and the INCOSE Body of Knowledge. There are many variations. For example, an evolutionary approach (e.g., intensive interaction with end-users to understand the requirements and prototype possible solutions) is described in Cross & Estrada, 1994. Four key elements of systems engineering – the systems description, requirements, risk management, and leadership of teams – are used in the research strategy described in this paper.

A key skill of the systems engineer is the ability to translate and decompose a description of a system vision, based on customer need, into realizable modules and requirements, each with a known technical solution and with predictable implementation risk, cost, and schedule. This has been characterized as both art and science (Bay et al., 2009). The science involves the use of well-established tools and methods to aid in the decomposition, modeling, integration, and testing of the system (Weiss & Cross, 2009). The art is often compared to the skill of an orchestra conductor who is sufficiently knowledgeable and proficient with the various instruments though perhaps not as talented with any one instrument as any given musician. But the successful conductor is able to coax the best from each musician in order to achieve an overall desired performance. This style of leadership is important, and also very challenging within a university. Universities are organizations with shared governance between administrators, faculty, students, and staff. Faculty have the freedom to pursue scholarship activities of their choosing (referred to as academic freedom). It is within such an environment that Georgia Tech created a new research and economic development strategy focused on innovation, interdisciplinary research, and greater collaboration between faculty and professional staff spanning discovery-focused basic research, applied and translational research, and economic development activities.

Georgia Institute of Technology

Georgia Tech was created in 1888 with the mission to educate a cadre of technical leaders to build a manufacturing and economic base in the State of Georgia. Georgia Tech’s strong engineering culture resulted from the blending of two standard approaches for engineering education of that era. Commonly called the school and the shop approaches, they differed in the order in which theoretical and practical work were introduced (i.e., theoretical work was mastered first in the school approach; practical work was mastered first in the shop approach). Georgia Tech sought to blend these two approaches by having students study science and engineering simultaneously with practical work in its shops and foundries. Thus, from its beginning, Georgia Tech pursued a concurrent approach to theory and practice. This has produced a culture where collaboration across traditional areas of scholarship (e.g., interdisciplinarity) facilitated bi-directional benefits between research and economic development activities.

Today, Georgia Tech consists of six colleges (engineering, architecture, computing, business, science, and liberal arts). The Georgia Tech Research Institute (GTRI), an outgrowth from the original shops and foundries, was created in 1934 to conduct applied, industry focused research. In addition, the Enterprise Innovation Institute (EII) supports more than 800 companies across the State of Georgia and manages the largest and oldest public incubator in the United States (U.S.). Through 2009, the colleges, GTRI, and EII largely pursued their activities independently. Research results were deployed through a slow and linear process not atypical of most American universities. A new systems approach was instituted as part of the strategic planning process in 2009 to create more synergy between competencies across the university.

A Systems Approach to Research and Economic Development Strategy

As shown in Figure 1, the strategy focuses on synergy between competencies across the university in discovery focused research, applied research, and economic development. These efforts are focused in core thematic areas such as biomedicine, manufacturing, and electronics through interdisciplinary research institutes that report centrally. Each is led by a well-respected research active faculty member and each provides both an intellectual crossroads for faculty across the university and a portal into the university for industry and government sponsors. The strategy operates under three guiding principles: research is led by faculty, powered by ideas, and supported by professionals. It has three objectives: create transformative opportunities, strengthen collaborative partnerships, and maximize economic and societal impact. More information is available on the Georgia Tech research web site. Within this construct, the key systems engineering principles are now described.

Systems Description. Grand challenge statements are used as system descriptions. A well-known grand challenge about the moon landing was expressed by U.S. President John Kennedy in a speech at Rice University on 12 September 1962. Such statements are descriptions of hard problems and provide an effective way to communicate the potential for leading edge research. Wallace Coulter, the inventor of the blood counter and a famous American entrepreneur once said it is easy to solve hard problems. “You break them into a set of smaller problems and solve them one at a time.” In this spirit, the grand challenges descriptions are decomposed into a set of smaller problems. Those for which there is either no known solution or some potentially significantly new or improved approach for solving that problem become high priority candidates for research. There are many public sources for grand challenge statements. Roland and Shiman (Roland & Shiman, 2002) describe the quest of the Defense Advanced Research Projects Agency to create intelligent machines in the 1980s. The US National Academy of Engineering maintains a description of engineering grand challenges in engineering. At Georgia Tech, such descriptions are maintained as part of the internal planning documents in each core area of research.

Figure 1: Georgia Tech Research and Economic Development Strategy

Requirements management. A research endeavor within a university does not proceed in isolation as its faculty are part of a larger research community. Often those organizations that sponsor research seek to create a community consensus for articulating requirements related to grand challenges and key unsolved problems. A roadmap is a tool for communicating this consensus and agreement on possible solution approaches. For example, a US robotics research roadmap was developed by leading research universities and industry in 2009 (and recently updated in 2016). Another recent example is a roadmap for cell-based manufacturing. How the research community pursues research varies. At Georgia Tech, the strategy illustrated in Figure 1 guides how research is pursued and evaluated. That is, it motivates an approach for managing risk.

Risk management. The key process for managing risk relates to how research is performed and how results are translated into use. For Georgia Tech, this means being innovative and creative in support of experimentation and maturation. At the core are curiosity, experimentation, and maturation.

Curiosity is a key attribute linking interesting and challenging societal problems to basic research. Curiosity is also an attribute of a more individualized research process called use-inspired research as described in (Stokes, 1997). The goal of use-inspired research is to maximize the generation of new knowledge and solutions to important societal problems. This is illustrated in Figure 1. Grappling with the societal problems (curiosity) often leads to insight into new fundamental research questions. At Georgia Tech, faculty councils are used to facilitate discussions across academic disciplines in ways that promote discussion with outside partners who seek to deploy research results. For example, in the area of health care technologies, a faculty council works closely with health care providers from Children’s Healthcare of Atlanta and biomedical device companies in order to understand clinical problems and to use that understanding to guide research in nanomedicine, regenerative medicine, and health systems. Ensuring an environment in which curiosity flourishes is essential for mitigating the risk of “not missing the next big thing.”
Experimentation is crucial for both providing a laboratory or test bed in which to support discovery-focused research and in supporting the applied research needed to test research discoveries and to integrate them into a systems solution (Fouse & Cross, 2006). At Georgia Tech, facilities for interdisciplinary work with shared equipment are used in the core research areas. Such facilities provide “intellectual crossroads” where faculty can explore new ideas and their potential application. Disruptive innovations often arise from this work. Such experiments incorporate a challenge-competitive environment to maximize depth of innovation and exploration. As an example, Georgia Tech has a smart grid laboratory as part of its energy systems programs. This lab is used both to educate students and to conduct research. It is also used to stage a competition between student teams supervised by a faculty member and an industry representative. Industry challenge problems are presented for which there are no known solutions. The student team that produces the most useful results receives a cash prize (and a good grade!). Over the past three years, companies have hired many of the students who have competed in the competitions. One company produced 26 patent applications with a return on investment (ROI) six times their own internal ROI. Senior leaders often find themselves in the position of “giving permission to take risk.” Risk taking in discovery-focused research and related experimentation is crucial, yet more managed approaches are prudent in maturation pursuits. Common practices used in discovery and applied research include use of “seed grant” funds to provide initial support for exploring a new idea.
Maturation activities involve the use of commercial experts engaging earlier in the research process to look for promising ideas that have market potential. At Georgia Tech, technology may be deployed through creation of new companies, licensing technology to existing companies, or conducting value-added services for a company interested in accelerating maturation. Risk management differs in a large research university from that in a typical systems engineering process. While in a systems engineering process, one wants to mitigate technical, cost, and schedule risk; in research one wants to promote risk taking to support the discovery and application of game changing ideas. Systems engineering techniques have proven to be directly applicable in the maturation of promising research ideas available for widespread use. One method involves the use of readiness levels as typified in Technology Readiness Levels or TRLs and Manufacturing Transition Levels (MRLs). Levels 1-3 typically deal with basic and applied research and Levels 7-9 deal with more application ready technologies. Universities typically deal in the Levels 1-3 area and industry is more committed to Levels 7-9. Often called “the valley of death,” Levels 4-6 deal with the risk reduction issues necessary to migrate a technology from promise in a laboratory or test bed application to hardened industrial use. Risk reduction for technical adoption is thus mitigated.

Team leadership. Lastly, the author has found one’s leadership approach, while often a matter of individual style, differs in terms of successful application in leading a systems engineering project versus leading a university’s research program. In a commercial systems engineering project, especially in a large undertaking, a top down management style is often appropriate where system management tools ranging from Gantt charts, modeling, and formal design reviews are appropriate means for ensuring progress towards realizing the desired systems capability. In a university research environment, where grand challenge statements are used to motivate and guide research investment and progress, a much more personal and supportive leadership style often proves to be more appropriate. Sometimes called servant leadership, it is important that leaders lead from behind, letting the faculty and students take center stage. A model used at Georgia Tech which blends attributes of servant and adaptive leadership, the Research Leadership Model, is described further in (Cross, 2013).

In summary, Table 1 compares and contrasts the system approaches used in the Georgia Tech research strategy with what is typically done in industry.

Table 1: Compare and Contrast Systems Engineering (SE) Approaches: Industry and Research

Process	Industry SE Approach	Research SE Approach
System Description	End product description	Grand challenge description
Requirements Management	Decompose into statements of what is needed; define known approaches for addressing them	Decompose into statements of what is needed; define open research issues to address in order to eventually achieve needs
Risk Management	Select technically viable approaches with predictable cost and schedule	Promote risk taking in exploration, experimentation, and evaluation of novel ideas; pursue accelerated maturation using a risk management approach
Team Leadership	Disciplined management approach, often directive	Anticipatory, influencing, and support of others

Towards a Balanced Scorecard^{^[1]}

As previously stated, the Georgia Tech strategy for research and economic development has three objectives: (1) faculty perform transformative research (2) done through collaborative partnerships (3) to achieve economic development and societal benefit. As a result of this strategy, Georgia Tech’s sponsored research awards have increased 10-12% per year over the past 6 years and commercialization activity has also advanced. It created 57 start-up ventures in 2016 as compared to 17 in 2010. It now ranks second in the State of Georgia in patent production, out distanced only by AT&T. A recent article in the online Harvard Business Review describes how the co-location of industry and a research university in an urban environment enhances such impact. Georgia Tech is used as the example in that article.

Of late, there has been an active discussion within the University concerning how best to assess the overall success of its research and economic development strategy. Colleges and universities are often assessed based on rating services such as the annual rankings of the US News and World Report. Often such rankings deal with “size” versus “quality.” For example, annual research awards and expenditures that increase from year to year would generally be viewed as a good thing. But how does one assess the quality of the awards and the actual work done under them? An analogy that is well understood by individual faculty is the number of publications in a curriculum vitae (i.e., an academic resume) versus the impact of those publications. Concepts such as h-factor or Google scholar metrics provide insight into the citations of the faculty member’s presentations. Presumably, those publications that are cited most often are a good indicator of significance and impact. These measures are absolutely fundamental and critical to conveying the scholarship and thought leadership impact of faculty at Georgia Tech. The equivalent kind of measures are desired for assessing performance under the strategy for financial and economic/society impact.

Using the balanced scorecard concepts initially advanced by Norton and Kaplan, Georgia Tech is defining a balanced set of measures with which to assess the impact of its strategy in four areas – finance, economic development, reputation, and process. A notional view is presented in Figure 2 and a brief description of each quadrant is presented. Note that qualitative and quantitative dashboards can be provided within each quadrant.

Financial

Clearly revenue growth is important, but so is diversification of the revenue base so industry awards and revenues are tracked. Another key measure is the cost of research reflected in the university’s audited indirect costs.

Economic & Societal Impact

The university has tracked Intellectual Property (IP) decelerations, patent application and issuances, licenses, and start-up formation for years. Now the IEEE Patent Power ranking and a new measure termed Patent Velocity are key measures. With respect to start-ups, while the number created is important, so too are the number that are still in business three years after formation and the total jobs each created within the region.

Reputational

A key measure that indicates the quality of research awards is the kind of large multi-faculty award resulting from extensive peer review (e.g. NSF Engineering Research Center). Another key indicator is the degree to which influential sponsors, both from industry and government, seek an embedded presence with the university and rely upon its faculty for advice.

Processes

Reducing faculty administrative burden and providing more effective research support are an important part of the overall strategy. Each year three or four processes are selected for intense improvement focus, for example, legal approvals, shared use facilities, and financial management.

Figure 2: Notional Balanced Scorecard for a University’s Research Strategy

Finance. As noted above, financial measures have typically dealt with “size.” As long as research awards and expenditures increased each year, everyone was happy. An important part of the strategy is to be more relevant to industry so measuring the increase in industry awards and diversification of awards and expenditures across federal, state, industry, and philanthropic sources is important. So too is assessing the cost of research by tracking audited indirect costs.

Economic Development and Reputation. Measures related to economic and societal impact have in the past typically dealt with media citations about that impact. For example, Forbes recently described the start-up culture emerging in Atlanta and the role of Georgia Tech. Periodic economic studies commissioned by the State of Georgia routinely cite the economic impact of the university. International rankings have listed the university as a top 10 global technological university and #24 in terms of innovation impact. Again, these measures largely deal with “size.” The University has begun to explore use of metrics that better communicate the impact and the “first derivative of size” (the speed by which research results are adopted). With respect to impact, the university now closely tracks the IEEE Patent Power index which assesses not just the number of patents but also their significance and impact (analogous to the faculty publication quality rankings). The university has also defined a new measure, patent velocity, to measure the number of patents issued in a given year that lead to an industry license in that same year.

Process. As part of the engagement of the entire Georgia Tech research and economic community, an annual offsite meeting and periodic workshops are held to continually update an operations plan and a SWOT analysis (strengths, weakness, opportunities, and threats). These are used to select 3 to 4 key internal processes each year for improvement. For example, the university is currently supporting two task forces to improve its commercialization process and its management of shared equipment facilities.

Summary

The Georgia Institute of Technology is a major research university with a strategic vision that spans discovery-focused research, applied research, and deployment. While research is largely an activity that resides with individual faculty investigators, the overall strategy is managed based on systems engineering principles where the entire enterprise is viewed as a system. Intentional support for curiosity, experimentation, and maturation create synergy between previously disconnected units. Key principles include the articulation of futuristic systems based on grand challenge problem statements, the strategic investment into research capabilities based on unresolved open issues (requirements) related to these descriptions, an intentional approach to pursue big ideas (taking risk), reduction to practice and deployment (managed risk), and a leadership approach in support of all activities and the people who conduct them. The work is aligned with strategic markets being developed in the southeast United States. The results to date have been impressive. A primary finding is that a systems engineering approach is a useful means by which to manage such a diverse program, but that the key functions of system description, requirements management, risk management, and team leadership differ from the conventional systems engineering approach in a typical industrial setting. Current refinement of the strategy is focused on definition and use of impact measures that can be incorporated into a balanced scorecard.

Acknowledgements

This paper was based on a keynote talk given at the annual Systems Engineering University Affiliated Research Center meeting in Washington D.C. in November 2016. The author is indebted to the mentoring and advice over the years of Barry Boehm, Jean Lou Chameau, Hal Falk, Scott Fouse, Paul Kaminski, Raj Reddy, Bill Rouse, and many others who directly influenced the thoughts described in this paper. The author is also indebted to numerous senior faculty and staff at the Georgia Institute of Technology who have provided incredible leadership to yield the results seen to date.

References

Bay, M., Gerstenmaier, W., Griffin, M., Knight, J., Larson, W., Ledbetter, K., Gentry, L., Menzel, M., Muirhead, B., Muratore, J., Ryan, R., Ryschkewitsch, M., Schaible, D., Scolese, C., & Williams, C. (2009). “The Art and Science of Systems Engineering: “Personal Characteristics of a Good Systems Engineer”. NASA Working Paper.

Cross, S. & Estrada, R. (1994). “DART: An Example of Accelerated Evolutionary Development”. Proceedings of the Fifth IEEE International Workshop on Rapid System Prototyping, pages 177-183.

Cross, S. (2013). “A Leadership Model for the Research University”. Proceedings of the 3^rd International Conference on Leadership, Technology, and Innovation Management, Istanbul TR, xix-xxvii..

Fouse, S. & Cross, S. (2006). “System Level Experimentation”. A report for the Unites States Air Force Scientific Advisory Board..

Roland, A. & Shiman, P. (2002). DARPA and the Quest for Machine Intelligence, 1983-1993. Cambridge, MA: MIT Press.

Sage, A. and Armstrong, A. (2000). Introduction to Systems Engineering. New York: John Wiley & Sons.

Sage, A. and Rouse, W. (Eds.) (2009). Handbook of Systems Engineering and Management, 2^nd Ed. New York: John Wiley & Sons.

Stokes, D. (1997). Pasteur’s Quadrant – Basic Science and Technological Innovation. Washington, DC: Brookings Institution Press.

Roland, A. & Shiman, P. (2002). DARPA and the Quest for Machine Intelligence, 1983-1993. Cambridge, MA: MIT Press.

Weiss, L., Roberts, R. and Cross, S. (2009). “The Systems Engineering and Test (TSET) Approach for Unprecedented Systems”. Journal of the International Test and Evaluation Association, 30(2), 386-394.

List of Acronyms Used in this Paper

EII – Enterprise Innovation Institute

GTRI – Georgia Tech Research Institute

IEEE – Institute of Electrical and Electronic Engineers

INCOSE – International Council on Systems Engineering

IP – Intellectual Property

MRLs – Manufacturing Transition Levels

ROI – Return on Investment

SyEN – Systems Engineering Newsletter

TRLs – Technology Readiness Levels

U.S. – United States

About the author

In addition to serving as Georgia Tech’s Executive Vice President for Research, Dr. Stephen E. Cross is a professor in the H. Milton Stewart School of Industrial and Systems Engineering and an adjunct professor in the College of Computing and the Ernest J. Scheller College of Business. From 2003-2010 he served as a Vice President and Director of the Georgia Tech Research Institute. Previously, he was at Carnegie Mellon University as the Director and CEO of the Software Engineering Institute. He serves on the executive committee of the Government-University-Industry Research Roundtable, an organization sponsored by the National Academies, and on the Administrative Committee of the Institute of Electrical and Electronic Engineers (IEEE) Technology and Engineering Management Society. Dr. Cross is a Life IEEE Fellow and a former Editor-in-Chief of IEEE Intelligent Systems. He received his B.S. in Electrical Engineering from the University of Cincinnati (1974), his M. S. in Electrical Engineering from the Air Force Institute of Technology (1976), and his PhD from the University of Illinois at Urbana-Champaign (1983).

Article

Agile Software Quality – A Quantitative Assessment

Donald J. Reifer

February 2017

Introduction

This note summarizes the findings of an analysis by Reifer Consultants LLC of “hard” data from over 2,000 completed software projects. This data was collected by about 200 firms from 26 nations that used agile methods to quantitatively assess the quality of software products developed by applications domain. Software quality was measured during both the development phase and the first year of operations. These two periods were chosen because they provide our readers with insights into the quality of the software product as seen by different users as it transitions from development through the first year of operations.

Definition of Software Quality

While there were some differences in quality observed across applications domains, the software generated using agile methods was in general much better than that produced using traditional software development methods. To develop this conclusion, we assessed the relative reliability of the software using defect densities during software development. After the software was delivered, we switched and used the following three measures to get a more rounded view of software quality during the first year of operations:

Reliability – measured by the number of defects in the defect backlog for each release or delivery as a function of defect density and find and fix rate.
Maintainability – measured by the time required to fix a defect and the quality of the fix per release as measured by the number of fixes to repairs.
Fitness for use – measured by the number of stories/features/functions delivered versus the number promised to the user.

Reliability during Software Development

When looking at software reliability during development, we found a sharp contrast between agile and traditional methods fostered by the manner in which software products were delivered. When traditional software development approaches were used, software was designed and developed in stages and then integrated and tested towards the end of the development cycle. Defects were captured during each stage and the goal was to minimize defects that escaped from one stage to another; i.e., design to coding and coding to testing, because they were costly to repair. Acceptance testing was conducted at the end of the cycle to qualify the software for delivery. Defects were captured using software trouble reports and logs were kept to track closures.

In contrast, agile developments continuously integrated and tested the working software products that were delivered sprint-by-sprint. Developers used test-first principles to identify the test criteria along with the requirements at the start of each sprint. Automated test cases were generated and executed whenever possible as the software was integrated and tested. Automated regression tests were developed, baselined, and used to requalify the releases sprint-by-sprint. Acceptance testing was performed to qualify the final version of the software as products were delivered.

To illustrate the differences between approaches, we summarized the reliability experience of 1,500 agile and 1,500 traditional projects as averages in Table 1 and Table 2 respectively. As expected, the defect densities and escape rates got better and stabilized for projects using both agile and traditional methods by the time the software product was readied for delivery.

Agile	Auto	C&C	Defense	Fin	IT	Medical	Mobile	Tools	Telecom	Web
Defect Density¹	0.182	0.118	0.076	0.112	0.328	0.121	0.335	0.336	0.135	0.375

Table 1: Defect Density (Defects/UFP) during Software Development by Applications Domain (Agile Methodology)

Legend

¹ Computed as the average number of defects per Unadjusted Function Points (UFP) when the software was released to operations.

Domains: Auto = Automation; C&C = Command & Control; Fin = Finance; IT = Information Technology

Traditional	Auto	C&C	Defense	Fin	IT	Medical	Mobile	Tools	Telecom	Web
Defect Density¹	0.216	0.135	0.085	0.126	0.401	0.140	0.376	0.396	0.168	0.468

Table 2: Defect Density (Defects/UFP) during Software Development by Applications Domain (Traditional Methodology)

Legend

¹Computed as the average number of defects per UFP when the software was released to operations.

Domains: Auto = Automation; C&C = Command & Control; Fin = Finance; IT = Information Technology

The improvements in defect density made using agile methods as illustrated by the two Tables are dramatic. They stem from the fact that the composite defect find & fix rates experienced during software development by those organizations that provided us the data were between 10 to 23 percent better when agile methods were used. In addition, for whatever reason, many projects that used traditional methods did not pass defect lists forward to operations when software acceptance testing was completed. This led to confusion and additional work during the first year of operations because software maintenance teams had to find and fix defects rather than make repairs using defect backlogs created for this purpose.

Reliability during First Year of Operations

As expected, the results of our assessment for defect densities during the first year of operations, as shown in Figure 1, were also favorable when agile methods were used in the area of reliability as measured by number of defects per 100 UFP across our ten applications domains. Based on data from 1,726 completed projects, defect densities as measured by the number of defects per hundred unadjusted function points (UFP) were lower when agile methods were used with the exceptions noted in this report. This should come as no surprise because the agile practices used by these projects focused (test-first concepts, continuous integration and test, etc.) on early defect identification and removal. As a result, escapes were minimized and there were fewer defects passed from development to operations as a result. In addition, the software maintenance effort was minimized as defect backlogs were passed forward when agile methods were used along with prioritized lists of repairs that needed to be completed as functionality was added to the release.

Figure 1 – Reliability by Domain Measured by

Defect Density (Defects/100 UFP)

Automation C&C Defense Financial IT

Medical Mobile

Tools

Density – Agile

Density – Traditional

Telecom

Web

Based on data from 1,119 projects, as shown in Figure 2, defect rates during the same time period were also lower for agile developments than for similar projects that used traditional development methods as measured by number of defects found and fixed per month across all ten applications domains. As with defect densities, there were exceptions in domains where quality was not the primary concern of stakeholders. These exceptions were centered in applications like games where bugs were perceived as adding value by some stakeholders.

Figure 2 – Reliability by Domain Measured by

Number of Defects Found and Fixed Per Month

Automation C&C Defense Financial IT

Medical

Mobile Tools

Rate – Agile

Rate – Traditional

Telecom

Web

Table 3 shows the number of escapes or number of defects that were found but not fixed during development and were passed on into operations. For agile projects, escapes could be easily identified because defects passed from development to operations were reported in the defect backlog. More importantly, these defects were consciously deferred often to make deadlines or for budgetary or schedule reasons. In contrast, no such backlog existed for traditional projects.

Traditional developments most often tracked defects using some bug tracking system like Bugzilla. Not all defects were identified. This was especially true when teams were rushed as they neared delivery. As a consequence, traditional teams often reported a larger number of defects than their agile counterparts during the first year of operation. As evidenced by Table 1, the number of escapes was larger for traditional developments than for their agile counterparts.

	Auto	C&C	Defense	Fin	IT	Medical	Mobile	Tools	Telecom	Web
Agile	0.25	0.15	0.1	0.2	0.5	0.1	0.4	0.25	0.15	1
Traditional	0.33	0.22	0.18	*	*	0.14	*	0.33	0.25	*

Table 3: Number of Escapes per Unadjusted Function Points (UFP) by Applications Domain

Legend

* No data available

Domains: Auto = Automation; C&C = Command & Control; Fin = Finance; IT = Information Technology

Maintainability during First Year of Operations

In general, as pictured in Figure 3, the fix response times for defects during the first year of operations, as measured in days, ranged widely when projects that used agile methods were likened to similar ones that used traditional approaches. Based on data from 1,009 projects, projects using agile methods had from 6 to 18 percent better fix response time. The two major agile principles cited that facilitated this improvement were simplicity and refactoring.

More important, as shown in Figure 4, fewer defects were generated as fixes were made. This is an important finding because it shows the positive effect that agile practices have on software maintenance activities. Improvements achieved, according to those polled, stem the refactoring practices used during development and the repair and testing practices used during maintenance.

Figure 3 – Maintainability by Domain Measured

by Fix Time in Days

Time – Agile

Automation

C&C Defense Financial IT

Medical Mobile Tools Telecom

Web

Time – Traditional

Figure 4 – Maintainability by Domain Measured by

Average Number of Defects Introduced during Fixes

Automation C&C Defense Financial IT

Medical Mobile

Tools

Fixes – Agile

Fixes – Traditional

Telecom

Web

The telling result, as shown in Figure 4, is the percentage of critical defects that made their way through the maintenance process. For projects using traditional software development methods, about ten percent of the defects introduced as applications were repaired were critical; i.e. caused the system to crash. These had to be fixed immediately because no work-arounds were possible. In contrast, the corresponding percentage of critical defects introduced during fixes on agile projects was just seven percent. While this may not seem to be a large difference, its impact on a 24/7 software operations is significant, especially when high availability is the goal.

“Fitness for Use” During First Year of Operations

As shown in Figure 5, agile developments delivered less functionality than traditional methods as measured by our “fitness for use” metric. The result is based on data mined from 912 completed projects. This finding partially explains why many firms have reported that they could develop software much more quickly using agile methods. Getting product to market is much easier for organizations that backlog the release of non-critical functionality until the maintenance phase. Backlogging functionality to preserve schedule is a common practice for agile developments. The good news is that the functionality that was delayed was implemented during maintenance almost 100 percent of the time. One can argue that getting working software with as much as 90 percent of the important functionality released at delivery is a positive result. One can also conclude the opposite when the goal is to realize all of the user’s expectations relative to functionality.

We will discuss this and other observations made relative to “fitness for use” and “delivered functionality” in the main body of our full report. As already mentioned, the percent of the functionality promised that was actually delivered often determines how the customer views the software’s delivered value. Because of this controversy, our findings and conclusions in this area, some of which run counter to the claims being made in the agile literature, are important. Based on the high reported failure rate of software projects reported in the literature, we believe that value increases when it is delivered on time and within budget even when some functionality is delayed.

Figure 5 – Fitness for Use by Domain Measured

by Percentage of Stories/Features/Functions Delivered as Promised

100%

80%

60%

40%

20%

Deliveries – Agile

Deliveries – Traditional

Automation

C&C Defense Financial IT

Medical Mobile Tools Telecom

Web

In Summary

The data that we have gathered provides convincing evidence that the quality of software developed using agile methods is better than that developed using more traditional software development approaches. The data gathered makes it apparent that software generated using agile methods has better software reliability and maintainability. Defect densities and rates are lower as are the response time for fixes and the number of defects introduced by them in fixes. However, this conclusion does lead one to wonder how much of this result is based on the fact that the best software performers today work in an agile environment because that is what they prefer.

In contrast, the results for “fitness for use” have to be rated in terms of whether readers view deferring software functionality to meet deadlines and budgetary constraints as a positive or negative. As stated earlier in the document, getting working software delivered on-time and budget with as much as 90 percent of the critical functionality in place can be viewed as a success. However, critics may argue that failure to meet one’s commitments for functionality and performance results in user dissatisfaction. In contrast, supporters of such deferrals might question how much of this added functionality was needed in the first place. Many times this extra functionality is never completed because it was not deemed necessary.

Of course, our final conclusion is that more analysis needs to be performed to determine whether or not these findings are valid for a range of unexplored regions. For example, the question can be raised: “Will these findings hold true when agile methods are scaled to address very large software systems or systems-of-systems?” While we have generated some clues relative to answering this question, the data collected thus far on this topic is still inconclusive. That is the reason why we need to continue our analysis.

Another question could be raised as to whether our agile findings are valid for subcategories of applications domains like Information Technology (e.g., banking, communications, finance, insurance, manufacturing, professional services, real estate, retail trade, travel, etc.) and defense (airborne, ground, missile, shipboard, space, etc.). For example, do the findings hold for ERP (Enterprise Resource Planning) and other enterprise-wide systems? Do they hold for embedded systems where software is developed to run across a wide range of secure platforms in real-time? Are there exceptions to the findings like safety-critical systems? To develop answers to these and other questions, we need to collect more data and continue to analyze and report the results.

To conclude, the analysis conducted to date provides empirical evidence that quality will be improved when agile methods are used to develop and maintain software products. This case is based on improved reliability and maintainability and acceptable “goodness of fit.”

Acknowledgements

We wish to thank those organizations who supplied the data upon which this study is predicated and those expert reviewers who supplied us comments for this publication.

Get the Full Report

For those interested, the full report upon which this summary is based is available on our website for a nominal fee at: http://www.reifer.com/products. Further information is available from:

Reifer Consultants LLC

1042 Willow Creek Road, Suite A101-485 Prescott, AZ 86301

Phone: (310) 922-7043

Website: http://www.reifer.com

Systems Engineering News

Integrating Program Management and Systems Engineering

One major effort of the INCOSE-PMI-MIT alliance has been to develop a book incorporating research the organizations have supported over the past three years focused on the integration of systems engineering and project/program management practices. This important new work is now becoming available to the world. The book will be available very soon through the INCOSE and PMI bookstores.

In the meantime, as an INCOSE member you can attend the INCOSE webinar where the lead representatives from each alliance organization introduce the background behind this work, overview the contents, and share their vision for opportunities that the book creates.

How Systems Engineering Can Help Fix Health Care

Hospitals and clinics are typically built in a piecemeal, patchwork approach. Institutions purchase hundreds of individual, siloed technologies — each with its own work processes, training, and user interfaces — based on what the market offers. They are then plopped into an ICU or operating room and hope that they somehow work together.

The result is a constellation of technologies that rarely connect, to the detriment of patient safety, quality, and value. For example:

Different monitors emit alarms that compete with one another for the attention of clinicians, who must sort out which signify serious conditions and which don’t. Sometimes they miss critical alarms amid the noise.
Devices, electronic medical records, and even patient beds have electronic information that can help diagnose conditions and assess risks. However, clinicians must consult each one individually, rather than seeing a unified display of information from them.
Time that could be spent with patients and their loved ones is instead squandered in front of computer monitors, as clinicians click through dozens of screens in search of relevant information.

All of this leads to needless patient harm, low productivity, excessive costs, and clinician burnout. Doctors and nurses feel as though they’re serving technology, not the other way around. Preventing complications, errors, and other harm too often depends on the heroism of clinicians rather than the design of safe systems.

An approach is needed that puts the needs of patients and clinicians first. One needs to integrate technology, people, and processes so that they are seamlessly joined in pursuit of a shared goal. At Johns Hopkins, they have experienced how powerful systems engineering can be when they set out to improve patient safety and quality of care in intensive care units. Patient safety researchers and clinicians from Johns Hopkins Medicine partnered with the systems engineers and systems integrators of the Johns Hopkins University Applied Physics Laboratory (APL). For 75 years APL has supported the Department of Defense and other government agencies as a “trusted agent” to solve critical challenges, such as building satellites and weapons systems on ships.

PPI SyEN 52

Subscribe to PPI SyEN

PPI SyEN 134

PPI SyEN 133

PPI SyEN 132

PPI SyEN 131

FREE Original SE Articles and Industry News?

Subscribe to PPI

MORE About Us