Read monthly Project Performance International's Systems Engineering Newsjournal, named "PPI SyEN". PPI SyEN presents for the engineering professional 30-60 pages of valuable technical articles on topical subjects, shorter technical pieces, in-depth coverage of the month's news in systems engineering and directly related fields, pointers to useful resources and relevant industry events, plus limited information on PPI's activities.

Welcome to PPI SyEN 52

In This Edition

Quotations to Open On

Read More…

Feature Article

The Influence of Systems Engineering on the Strategy of a Research University by Stephen E. Cross

Read More…


Agile Software Quality – A Quantitative Assessment by Donald J. Reifer

Read More

Systems Engineering News

  • Integrating Program Management and Systems Engineering
  • How Systems Engineering Can Help Fix Health Care
  • “The Father of Process Systems Engineering” – Roger Sargent to Receive Sir Frank Whittle Medal
  • Project in Gambia Helps Combat Coastal Erosion and Flooding
  • Legacy Systems are Problems for Boardrooms Not Computer Geeks
  • CIMdata Announces the Expansion of Manufacturing Systems Engineering Consulting Practice
  • U. S. Air Force Charts Wideband Global Satellite Future
  • DARPA Licenses Emerging Chip Technology
  • Nordic Systems Engineering Voyage 2017
  • Newcomer’s View of INCOSE’s International Workshop 2017

Read More…

Featured Organizations

  • National Academy of Engineering (NAE)
  • Institute of Industrial and Systems Engineers (IISE)
  • Georgia Tech Research Institute (GTRI)
  • United States Air Force Scientific Advisory Board (SAB)

Read More…

Systems Engineering Tools News

  • PragmaDev Studio

Read More…

Systems Engineering Publications

  • Systems Engineering Journal
  • The Art and Science of Systems Engineering
  • Made to Stick: Why Some Ideas Survive and Others Die
  • The Routledge Companion to Lean Management
  • Balanced Scorecard Step-by-Step: Maximizing Performance and Maintaining Results (2nd Edition)
  • A Primer for Model-Based Systems Engineering

Read More…

Education and Academia

  • MIT’s Architecture and Systems Engineering Professional Certificate Program
  • Industrial Engineering and Systems Engineering at Florida Tech
  • Southeast Missouri State University Launching New Engineering Program in Fall 2017
  • Institute for Mathematics and its Applications, University of Minnesota College of Science & Engineering
  • Welcome to the EIT Digital Professional School
  • solidThinking Aviation Case Study Teaches Embedded System Design
  • University of Maryland Baltimore Campus Systems Engineering Graduate Programs

Read More…

Some Systems Engineering-Relevant Websites

Read More…

Standards and Guides

NIST Releases Internet of Things (IoT) Security Guidance

Read More…

Definitions to Close On

  • A Simple Explanation of ‘The Internet of Things’

Read More…

Conferences and Meetings

Read More…

PPI News

Read More…

PPI and CTI Events

Read More…

PPI Upcoming Participation in Professional Conferences

Read More…

Quotations to Open On

“People who have “systems engineer” in their title, regardless of the modifiers – program, project, flight system, and so on – are responsible for everything.”

Gentry Lee, Jet Propulsion Laboratory

“A great systems engineer completely understands and applies the art of leadership and has the experience and scar tissue from trying to earn the badge of leader from his or her team.”

Harold Bell, NASA Headquarters

“Great networking includes genuinely helping people, and staying in touch when you don’t need something.”

Ronnette Meyers

Feature Article

The Influence of Systems Engineering on the Strategy of a Research University


Stephen E. Cross

Georgia Institute of Technology

Version 2 February 8, 2017


Publicly supported colleges and universities in the United States are expected to educate students, conduct research, and support economic growth within their states and regions.  One such university is the Georgia Institute of Technology which over the past 40 years has gained a global reputation for its leading-edge research programs.  Its strategy for research and economic development is based on system engineering principles which are described in this paper.  Also described is an ongoing effort to create a balanced scorecard to guide and assess performance under this strategy.

Website: www.gatech.edu/

Email: evpr@gatech.edu


The Georgia Institute of Technology (Georgia Tech), is a research university in Atlanta, Georgia (USA). It has a legacy culture of addressing real world problems and of embedding innovation throughout its education and research programs. It drives economic development in the southeast United States. A new strategic vision, introduced in 2010 to guide Georgia Tech’s role nationally and internationally, included a systems approach to faculty-led research to concurrently pursue transformational research and economic/societal impact. The systems approach guides research infrastructure investments in core research areas and institute-wide support for discovery, application, and deployment functions.  Significant results have been achieved over the past five years. An effort to define and implement a balanced scorecard to guide future work under the strategy is currently underway. This paper explains the systems-focused approach to the university’s research strategy, the progress that has been achieved, and some initial thoughts on the balanced scorecard. Feedback concerning the scorecard from readers of the Systems Engineering Newsletter (SyEN) is most welcome and will further strengthen and improve the value of the scorecard (please provide feedback to evpr@gatech.edu).

Systems Engineering

Systems engineering is defined as “… an interdisciplinary approach and means to enable the realization of successful systems.” Systems engineering and management concepts are described in (Sage & Armstrong, 2000); (Sage & Rouse, 2009); and the INCOSE Body of Knowledge. There are many variations. For example, an evolutionary approach (e.g., intensive interaction with end-users to understand the requirements and prototype possible solutions) is described in Cross & Estrada, 1994. Four key elements of systems engineering – the systems description, requirements, risk management, and leadership of teams – are used in the research strategy described in this paper.

A key skill of the systems engineer is the ability to translate and decompose a description of a system vision, based on customer need, into realizable modules and requirements, each with a known technical solution and with predictable implementation risk, cost, and schedule. This has been characterized as both art and science (Bay et al., 2009). The science involves the use of well-established tools and methods to aid in the decomposition, modeling, integration, and testing of the system (Weiss & Cross, 2009). The art is often compared to the skill of an orchestra conductor who is sufficiently knowledgeable and proficient with the various instruments though perhaps not as talented with any one instrument as any given musician. But the successful conductor is able to coax the best from each musician in order to achieve an overall desired performance. This style of leadership is important, and also very challenging within a university. Universities are organizations with shared governance between administrators, faculty, students, and staff. Faculty have the freedom to pursue scholarship activities of their choosing (referred to as academic freedom). It is within such an environment that Georgia Tech created a new research and economic development strategy focused on innovation, interdisciplinary research, and greater collaboration between faculty and professional staff spanning discovery-focused basic research, applied and translational research, and economic development activities.

Georgia Institute of Technology

Georgia Tech was created in 1888 with the mission to educate a cadre of technical leaders to build a manufacturing and economic base in the State of Georgia. Georgia Tech’s strong engineering culture resulted from the blending of two standard approaches for engineering education of that era. Commonly called the school and the shop approaches, they differed in the order in which theoretical and practical work were introduced (i.e., theoretical work was mastered first in the school approach; practical work was mastered first in the shop approach). Georgia Tech sought to blend these two approaches by having students study science and engineering simultaneously with practical work in its shops and foundries. Thus, from its beginning, Georgia Tech pursued a concurrent approach to theory and practice. This has produced a culture where collaboration across traditional areas of scholarship (e.g., interdisciplinarity) facilitated bi-directional benefits between research and economic development activities.

Today, Georgia Tech consists of six colleges (engineering, architecture, computing, business, science, and liberal arts). The Georgia Tech Research Institute (GTRI), an outgrowth from the original shops and foundries, was created in 1934 to conduct applied, industry focused research. In addition, the Enterprise Innovation Institute (EII) supports more than 800 companies across the State of Georgia and manages the largest and oldest public incubator in the United States (U.S.). Through 2009, the colleges, GTRI, and EII largely pursued their activities independently. Research results were deployed through a slow and linear process not atypical of most American universities. A new systems approach was instituted as part of the strategic planning process in 2009 to create more synergy between competencies across the university.

A Systems Approach to Research and Economic Development Strategy

As shown in Figure 1, the strategy focuses on synergy between competencies across the university in discovery focused research, applied research, and economic development. These efforts are focused in core thematic areas such as biomedicine, manufacturing, and electronics through interdisciplinary research institutes that report centrally. Each is led by a well-respected research active faculty member and each provides both an intellectual crossroads for faculty across the university and a portal into the university for industry and government sponsors. The strategy operates under three guiding principles: research is led by faculty, powered by ideas, and supported by professionals. It has three objectives: create transformative opportunities, strengthen collaborative partnerships, and maximize economic and societal impact.  More information is available on the Georgia Tech research web site. Within this construct, the key systems engineering principles are now described.

Systems Description. Grand challenge statements are used as system descriptions. A well-known grand challenge about the moon landing was expressed by U.S. President John Kennedy in a speech at Rice University on 12 September 1962. Such statements are descriptions of hard problems and provide an effective way to communicate the potential for leading edge research. Wallace Coulter, the inventor of the blood counter and a famous American entrepreneur once said it is easy to solve hard problems. “You break them into a set of smaller problems and solve them one at a time.” In this spirit, the grand challenges descriptions are decomposed into a set of smaller problems. Those for which there is either no known solution or some potentially significantly new or improved approach for solving that problem become high priority candidates for research. There are many public sources for grand challenge statements. Roland and Shiman (Roland & Shiman, 2002) describe the quest of the Defense Advanced Research Projects Agency to create intelligent machines in the 1980s. The US National Academy of Engineering maintains a description of engineering grand challenges in engineering. At Georgia Tech, such descriptions are maintained as part of the internal planning documents in each core area of research.

Figure 1: Georgia Tech Research and Economic Development Strategy

Requirements management. A research endeavor within a university does not proceed in isolation as its faculty are part of a larger research community. Often those organizations that sponsor research seek to create a community consensus for articulating requirements related to grand challenges and key unsolved problems. A roadmap is a tool for communicating this consensus and agreement on possible solution approaches. For example, a US robotics research roadmap was developed by leading research universities and industry in 2009 (and recently updated in 2016). Another recent example is a roadmap for cell-based manufacturing. How the research community pursues research varies. At Georgia Tech, the strategy illustrated in Figure 1 guides how research is pursued and evaluated. That is, it motivates an approach for managing risk.

Risk management. The key process for managing risk relates to how research is performed and how results are translated into use. For Georgia Tech, this means being innovative and creative in support of experimentation and maturation. At the core are curiosity, experimentation, and maturation.

  • Curiosity is a key attribute linking interesting and challenging societal problems to basic research. Curiosity is also an attribute of a more individualized research process called use-inspired research as described in (Stokes, 1997). The goal of use-inspired research is to maximize the generation of new knowledge and solutions to important societal problems. This is illustrated in Figure 1. Grappling with the societal problems (curiosity) often leads to insight into new fundamental research questions. At Georgia Tech, faculty councils are used to facilitate discussions across academic disciplines in ways that promote discussion with outside partners who seek to deploy research results. For example, in the area of health care technologies, a faculty council works closely with health care providers from Children’s Healthcare of Atlanta and biomedical device companies in order to understand clinical problems and to use that understanding to guide research in nanomedicine, regenerative medicine, and health systems. Ensuring an environment in which curiosity flourishes is essential for mitigating the risk of “not missing the next big thing.”
  • Experimentation is crucial for both providing a laboratory or test bed in which to support discovery-focused research and in supporting the applied research needed to test research discoveries and to integrate them into a systems solution (Fouse & Cross, 2006). At Georgia Tech, facilities for interdisciplinary work with shared equipment are used in the core research areas. Such facilities provide “intellectual crossroads” where faculty can explore new ideas and their potential application. Disruptive innovations often arise from this work. Such experiments incorporate a challenge-competitive environment to maximize depth of innovation and exploration. As an example, Georgia Tech has a smart grid laboratory as part of its energy systems programs. This lab is used both to educate students and to conduct research. It is also used to stage a competition between student teams supervised by a faculty member and an industry representative. Industry challenge problems are presented for which there are no known solutions. The student team that produces the most useful results receives a cash prize (and a good grade!). Over the past three years, companies have hired many of the students who have competed in the competitions. One company produced 26 patent applications with a return on investment (ROI) six times their own internal ROI. Senior leaders often find themselves in the position of “giving permission to take risk.” Risk taking in discovery-focused research and related experimentation is crucial, yet more managed approaches are prudent in maturation pursuits. Common practices used in discovery and applied research include use of “seed grant” funds to provide initial support for exploring a new idea.
  • Maturation activities involve the use of commercial experts engaging earlier in the research process to look for promising ideas that have market potential. At Georgia Tech, technology may be deployed through creation of new companies, licensing technology to existing companies, or conducting value-added services for a company interested in accelerating maturation. Risk management differs in a large research university from that in a typical systems engineering process. While in a systems engineering process, one wants to mitigate technical, cost, and schedule risk; in research one wants to promote risk taking to support the discovery and application of game changing ideas. Systems engineering techniques have proven to be directly applicable in the maturation of promising research ideas available for widespread use. One method involves the use of readiness levels as typified in Technology Readiness Levels or TRLs and Manufacturing Transition Levels (MRLs). Levels 1-3 typically deal with basic and applied research and Levels 7-9 deal with more application ready technologies. Universities typically deal in the Levels 1-3 area and industry is more committed to Levels 7-9. Often called “the valley of death,” Levels 4-6 deal with the risk reduction issues necessary to migrate a technology from promise in a laboratory or test bed application to hardened industrial use. Risk reduction for technical adoption is thus mitigated.

Team leadership. Lastly, the author has found one’s leadership approach, while often a matter of individual style, differs in terms of successful application in leading a systems engineering project versus leading a university’s research program. In a commercial systems engineering project, especially in a large undertaking, a top down management style is often appropriate where system management tools ranging from Gantt charts, modeling, and formal design reviews are appropriate means for ensuring progress towards realizing the desired systems capability. In a university research environment, where grand challenge statements are used to motivate and guide research investment and progress, a much more personal and supportive leadership style often proves to be more appropriate. Sometimes called servant leadership, it is important that leaders lead from behind, letting the faculty and students take center stage. A model used at Georgia Tech which blends attributes of servant and adaptive leadership, the Research Leadership Model, is described further in (Cross, 2013).

In summary, Table 1 compares and contrasts the system approaches used in the Georgia Tech research strategy with what is typically done in industry.

Table 1: Compare and Contrast Systems Engineering (SE) Approaches: Industry and Research

Process Industry SE Approach Research SE Approach
System Description End product description Grand challenge description
Requirements Management Decompose into statements of what is needed; define known approaches for addressing them Decompose into statements of what is needed; define open research issues to address in order to eventually achieve needs
Risk Management Select technically viable approaches with predictable cost and schedule Promote risk taking in exploration, experimentation, and evaluation of novel ideas; pursue accelerated maturation using a risk management approach
Team Leadership Disciplined management approach, often directive Anticipatory, influencing, and support of others

Towards a Balanced Scorecard[1]

As previously stated, the Georgia Tech strategy for research and economic development has three objectives: (1) faculty perform transformative research (2) done through collaborative partnerships (3) to achieve economic development and societal benefit. As a result of this strategy, Georgia Tech’s sponsored research awards have increased 10-12% per year over the past 6 years and commercialization activity has also advanced. It created 57 start-up ventures in 2016 as compared to 17 in 2010. It now ranks second in the State of Georgia in patent production, out distanced only by AT&T. A recent article in the online Harvard Business Review describes how the co-location of industry and a research university in an urban environment enhances such impact. Georgia Tech is used as the example in that article.

Of late, there has been an active discussion within the University concerning how best to assess the overall success of its research and economic development strategy. Colleges and universities are often assessed based on rating services such as the annual rankings of the US News and World Report. Often such rankings deal with “size” versus “quality.” For example, annual research awards and expenditures that increase from year to year would generally be viewed as a good thing. But how does one assess the quality of the awards and the actual work done under them? An analogy that is well understood by individual faculty is the number of publications in a curriculum vitae (i.e., an academic resume) versus the impact of those publications. Concepts such as h-factor or Google scholar metrics provide insight into the citations of the faculty member’s presentations. Presumably, those publications that are cited most often are a good indicator of significance and impact. These measures are absolutely fundamental and critical to conveying the scholarship and thought leadership impact of faculty at Georgia Tech. The equivalent kind of measures are desired for assessing performance under the strategy for financial and economic/society impact.

Using the balanced scorecard concepts initially advanced by Norton and Kaplan, Georgia Tech is defining a balanced set of measures with which to assess the impact of its strategy in four areas – finance, economic development, reputation, and process. A notional view is presented in Figure 2 and a brief description of each quadrant is presented. Note that qualitative and quantitative dashboards can be provided within each quadrant.


Clearly revenue growth is important, but so is diversification of the revenue base so industry awards and revenues are tracked. Another key measure is the cost of research reflected in the university’s audited indirect costs.

Economic & Societal Impact

The university has tracked Intellectual Property (IP) decelerations, patent application and issuances, licenses, and start-up formation for years. Now the IEEE Patent Power ranking and a new measure termed Patent Velocity are key measures. With respect to start-ups, while the number created is important, so too are the number that are still in business three years after formation and the total jobs each created within the region.


A key measure that indicates the quality of research awards is the kind of large multi-faculty award resulting from extensive peer review (e.g. NSF Engineering Research Center). Another key indicator is the degree to which influential sponsors, both from industry and government, seek an embedded presence with the university and rely upon its faculty for advice.


Reducing faculty administrative burden and providing more effective research support are an important part of the overall strategy. Each year three or four processes are selected for intense improvement focus, for example, legal approvals, shared use facilities, and financial management.

Figure 2: Notional Balanced Scorecard for a University’s Research Strategy

Finance. As noted above, financial measures have typically dealt with “size.” As long as research awards and expenditures increased each year, everyone was happy. An important part of the strategy is to be more relevant to industry so measuring the increase in industry awards and diversification of awards and expenditures across federal, state, industry, and philanthropic sources is important. So too is assessing the cost of research by tracking audited indirect costs.

Economic Development and Reputation. Measures related to economic and societal impact have in the past typically dealt with media citations about that impact. For example, Forbes recently described the start-up culture emerging in Atlanta and the role of Georgia Tech. Periodic economic studies commissioned by the State of Georgia routinely cite the economic impact of the university. International rankings have listed the university as a top 10 global technological university and #24 in terms of innovation impact. Again, these measures largely deal with “size.” The University has begun to explore use of metrics that better communicate the impact and the “first derivative of size” (the speed by which research results are adopted). With respect to impact, the university now closely tracks the IEEE Patent Power index which assesses not just the number of patents but also their significance and impact (analogous to the faculty publication quality rankings). The university has also defined a new measure, patent velocity, to measure the number of patents issued in a given year that lead to an industry license in that same year.

Process. As part of the engagement of the entire Georgia Tech research and economic community, an annual offsite meeting and periodic workshops are held to continually update an operations plan and a SWOT analysis (strengths, weakness, opportunities, and threats). These are used to select 3 to 4 key internal processes each year for improvement. For example, the university is currently supporting two task forces to improve its commercialization process and its management of shared equipment facilities.


The Georgia Institute of Technology is a major research university with a strategic vision that spans discovery-focused research, applied research, and deployment. While research is largely an activity that resides with individual faculty investigators, the overall strategy is managed based on systems engineering principles where the entire enterprise is viewed as a system. Intentional support for curiosity, experimentation, and maturation create synergy between previously disconnected units. Key principles include the articulation of futuristic systems based on grand challenge problem statements, the strategic investment into research capabilities based on unresolved open issues (requirements) related to these descriptions, an intentional approach to pursue big ideas (taking risk), reduction to practice and deployment (managed risk), and a leadership approach in support of all activities and the people who conduct them. The work is aligned with strategic markets being developed in the southeast United States. The results to date have been impressive. A primary finding is that a systems engineering approach is a useful means by which to manage such a diverse program, but that the key functions of system description, requirements management, risk management, and team leadership differ from the conventional systems engineering approach in a typical industrial setting. Current refinement of the strategy is focused on definition and use of impact measures that can be incorporated into a balanced scorecard.


This paper was based on a keynote talk given at the annual Systems Engineering University Affiliated Research Center meeting in Washington D.C. in November 2016. The author is indebted to the mentoring and advice over the years of Barry Boehm, Jean Lou Chameau, Hal Falk, Scott Fouse, Paul Kaminski, Raj Reddy, Bill Rouse, and many others who directly influenced the thoughts described in this paper. The author is also indebted to numerous senior faculty and staff at the Georgia Institute of Technology who have provided incredible leadership to yield the results seen to date.


Bay, M., Gerstenmaier, W., Griffin, M., Knight, J., Larson, W., Ledbetter, K., Gentry, L., Menzel, M., Muirhead, B., Muratore, J., Ryan, R., Ryschkewitsch, M., Schaible, D., Scolese, C., & Williams, C. (2009). “The Art and Science of Systems Engineering: “Personal Characteristics of a Good Systems Engineer”. NASA Working Paper.

Cross, S. & Estrada, R. (1994). “DART: An Example of Accelerated Evolutionary Development”. Proceedings of the Fifth IEEE International Workshop on Rapid System Prototyping, pages 177-183.

Cross, S. (2013). “A Leadership Model for the Research University”. Proceedings of the 3rd International Conference on Leadership, Technology, and Innovation Management, Istanbul TR, xix-xxvii..

Fouse, S. & Cross, S. (2006). “System Level Experimentation”. A report for the Unites States Air Force Scientific Advisory Board..

Roland, A. & Shiman, P. (2002). DARPA and the Quest for Machine Intelligence, 1983-1993. Cambridge, MA: MIT Press.

Sage, A. and Armstrong, A. (2000). Introduction to Systems Engineering. New York: John Wiley & Sons.

Sage, A. and Rouse, W. (Eds.) (2009). Handbook of Systems Engineering and Management, 2nd Ed. New York: John Wiley & Sons.

Stokes, D. (1997). Pasteur’s Quadrant – Basic Science and Technological Innovation. Washington, DC: Brookings Institution Press.

Roland, A. & Shiman, P. (2002). DARPA and the Quest for Machine Intelligence, 1983-1993. Cambridge, MA: MIT Press.

Weiss, L., Roberts, R. and Cross, S. (2009). “The Systems Engineering and Test (TSET) Approach for Unprecedented Systems. Journal of the International Test and Evaluation Association, 30(2), 386-394.

List of Acronyms Used in this Paper

EII – Enterprise Innovation Institute

GTRI – Georgia Tech Research Institute

IEEE – Institute of Electrical and Electronic Engineers

INCOSE – International Council on Systems Engineering

IP – Intellectual Property

MRLs – Manufacturing Transition Levels

ROI – Return on Investment

SyEN – Systems Engineering Newsletter

TRLs – Technology Readiness Levels

U.S. – United States

About the author

In addition to serving as Georgia Tech’s Executive Vice President for Research, Dr. Stephen E. Cross is a professor in the H. Milton Stewart School of Industrial and Systems Engineering and an adjunct professor in the College of Computing and the Ernest J. Scheller College of Business. From 2003-2010 he served as a Vice President and Director of the Georgia Tech Research Institute. Previously, he was at Carnegie Mellon University as the Director and CEO of the Software Engineering Institute. He serves on the executive committee of the Government-University-Industry Research Roundtable, an organization sponsored by the National Academies, and on the Administrative Committee of the Institute of Electrical and Electronic Engineers (IEEE) Technology and Engineering Management Society. Dr. Cross is a Life IEEE Fellow and a former Editor-in-Chief of IEEE Intelligent Systems. He received his B.S. in Electrical Engineering from the University of Cincinnati (1974), his M. S. in Electrical Engineering from the Air Force Institute of Technology (1976), and his PhD from the University of Illinois at Urbana-Champaign (1983).


Agile Software Quality – A Quantitative Assessment


Donald J. Reifer

February 2017


This note summarizes the findings of an analysis by Reifer Consultants LLC of “hard” data from over 2,000 completed software projects. This data was collected by about 200 firms from 26 nations that used agile methods to quantitatively assess the quality of software products developed by applications domain. Software quality was measured during both the development phase and the first year of operations. These two periods were chosen because they provide our readers with insights into the quality of the software product as seen by different users as it transitions from development through the first year of operations.

Definition of Software Quality

While there were some differences in quality observed across applications domains, the software generated using agile methods was in general much better than that produced using traditional software development methods. To develop this conclusion, we assessed the relative reliability of the software using defect densities during software development. After the software was delivered, we switched and used the following three measures to get a more rounded view of software quality during the first year of operations:

  • Reliability – measured by the number of defects in the defect backlog for each release or delivery as a function of defect density and find and fix rate.
  • Maintainability – measured by the time required to fix a defect and the quality of the fix per release as measured by the number of fixes to repairs.
  • Fitness for use – measured by the number of stories/features/functions delivered versus the number promised to the user.

Reliability during Software Development

When looking at software reliability during development, we found a sharp contrast between agile and traditional methods fostered by the manner in which software products were delivered. When traditional software development approaches were used, software was designed and developed in stages and then integrated and tested towards the end of the development cycle. Defects were captured during each stage and the goal was to minimize defects that escaped from one stage to another; i.e., design to coding and coding to testing, because they were costly to repair. Acceptance testing was conducted at the end of the cycle to qualify the software for delivery. Defects were captured using software trouble reports and logs were kept to track closures.

In contrast, agile developments continuously integrated and tested the working software products that were delivered sprint-by-sprint. Developers used test-first principles to identify the test criteria along with the requirements at the start of each sprint. Automated test cases were generated and executed whenever possible as the software was integrated and tested. Automated regression tests were developed, baselined, and used to requalify the releases sprint-by-sprint. Acceptance testing was performed to qualify the final version of the software as products were delivered.

To illustrate the differences between approaches, we summarized the reliability experience of 1,500 agile and 1,500 traditional projects as averages in Table 1 and Table 2 respectively. As expected, the defect densities and escape rates got better and stabilized for projects using both agile and traditional methods by the time the software product was readied for delivery.

Agile Auto C&C Defense Fin IT Medical Mobile Tools Telecom Web
Defect Density1 0.182 0.118 0.076 0.112 0.328 0.121 0.335 0.336 0.135 0.375

Table 1: Defect Density (Defects/UFP) during Software Development by Applications Domain (Agile Methodology)


1 Computed as the average number of defects per Unadjusted Function Points (UFP) when the software was released to operations.

Domains: Auto = Automation; C&C = Command & Control; Fin = Finance; IT = Information Technology

Traditional Auto C&C Defense Fin IT Medical Mobile Tools Telecom Web
Defect Density1 0.216 0.135 0.085 0.126 0.401 0.140 0.376 0.396 0.168 0.468

Table 2: Defect Density (Defects/UFP) during Software Development by Applications Domain (Traditional Methodology)


1Computed as the average number of defects per UFP when the software was released to operations.

Domains: Auto = Automation; C&C = Command & Control; Fin = Finance; IT = Information Technology

The improvements in defect density made using agile methods as illustrated by the two Tables are dramatic. They stem from the fact that the composite defect find & fix rates experienced during software development by those organizations that provided us the data were between 10 to 23 percent better when agile methods were used. In addition, for whatever reason, many projects that used traditional methods did not pass defect lists forward to operations when software acceptance testing was completed. This led to confusion and additional work during the first year of operations because software maintenance teams had to find and fix defects rather than make repairs using defect backlogs created for this purpose.

Reliability during First Year of Operations

As expected, the results of our assessment for defect densities during the first year of operations, as shown in Figure 1, were also favorable when agile methods were used in the area of reliability as measured by number of defects per 100 UFP across our ten applications domains. Based on data from 1,726 completed projects, defect densities as measured by the number of defects per hundred unadjusted function points (UFP) were lower when agile methods were used with the exceptions noted in this report. This should come as no surprise because the agile practices used by these projects focused (test-first concepts, continuous integration and test, etc.) on early defect identification and removal. As a result, escapes were minimized and there were fewer defects passed from development to operations as a result. In addition, the software maintenance effort was minimized as defect backlogs were passed forward when agile methods were used along with prioritized lists of repairs that needed to be completed as functionality was added to the release.






Figure 1 – Reliability by Domain Measured by

Defect Density (Defects/100 UFP)

Automation C&C Defense Financial IT

Medical Mobile


Density – Agile

Density – Traditional



Based on data from 1,119 projects, as shown in Figure 2, defect rates during the same time period were also lower for agile developments than for similar projects that used traditional development methods as measured by number of defects found and fixed per month across all ten applications domains. As with defect densities, there were exceptions in domains where quality was not the primary concern of stakeholders. These exceptions were centered in applications like games where bugs were perceived as adding value by some stakeholders.







Figure 2 – Reliability by Domain Measured by

Number of Defects Found and Fixed Per Month

Automation C&C Defense Financial IT


Mobile Tools

Rate – Agile

Rate – Traditional



Table 3 shows the number of escapes or number of defects that were found but not fixed during development and were passed on into operations. For agile projects, escapes could be easily identified because defects passed from development to operations were reported in the defect backlog. More importantly, these defects were consciously deferred often to make deadlines or for budgetary or schedule reasons. In contrast, no such backlog existed for traditional projects.

Traditional developments most often tracked defects using some bug tracking system like Bugzilla. Not all defects were identified. This was especially true when teams were rushed as they neared delivery. As a consequence, traditional teams often reported a larger number of defects than their agile counterparts during the first year of operation. As evidenced by Table 1, the number of escapes was larger for traditional developments than for their agile counterparts.

Auto C&C Defense Fin IT Medical Mobile Tools Telecom Web
Agile 0.25 0.15 0.1 0.2 0.5 0.1 0.4 0.25 0.15 1
Traditional 0.33 0.22 0.18 * * 0.14 * 0.33 0.25 *

Table 3: Number of Escapes per Unadjusted Function Points (UFP) by Applications Domain


* No data available

Domains: Auto = Automation; C&C = Command & Control; Fin = Finance; IT = Information Technology

Maintainability during First Year of Operations

In general, as pictured in Figure 3, the fix response times for defects during the first year of operations, as measured in days, ranged widely when projects that used agile methods were likened to similar ones that used traditional approaches. Based on data from 1,009 projects, projects using agile methods had from 6 to 18 percent better fix response time. The two major agile principles cited that facilitated this improvement were simplicity and refactoring.

More important, as shown in Figure 4, fewer defects were generated as fixes were made. This is an important finding because it shows the positive effect that agile practices have on software maintenance activities. Improvements achieved, according to those polled, stem the refactoring practices used during development and the repair and testing practices used during maintenance.

Figure 3 – Maintainability by Domain Measured

by Fix Time in Days










Time – Agile


C&C Defense Financial IT

Medical Mobile Tools Telecom


Time – Traditional









Figure 4 – Maintainability by Domain Measured by

Average Number of Defects Introduced during Fixes

Automation C&C Defense Financial IT

Medical Mobile


Fixes – Agile

Fixes – Traditional



The telling result, as shown in Figure 4, is the percentage of critical defects that made their way through the maintenance process. For projects using traditional software development methods, about ten percent of the defects introduced as applications were repaired were critical; i.e. caused the system to crash. These had to be fixed immediately because no work-arounds were possible. In contrast, the corresponding percentage of critical defects introduced during fixes on agile projects was just seven percent. While this may not seem to be a large difference, its impact on a 24/7 software operations is significant, especially when high availability is the goal.

“Fitness for Use” During First Year of Operations

As shown in Figure 5, agile developments delivered less functionality than traditional methods as measured by our “fitness for use” metric. The result is based on data mined from 912 completed projects. This finding partially explains why many firms have reported that they could develop software much more quickly using agile methods. Getting product to market is much easier for organizations that backlog the release of non-critical functionality until the maintenance phase. Backlogging functionality to preserve schedule is a common practice for agile developments. The good news is that the functionality that was delayed was implemented during maintenance almost 100 percent of the time. One can argue that getting working software with as much as 90 percent of the important functionality released at delivery is a positive result. One can also conclude the opposite when the goal is to realize all of the user’s expectations relative to functionality.

We will discuss this and other observations made relative to “fitness for use” and “delivered functionality” in the main body of our full report. As already mentioned, the percent of the functionality promised that was actually delivered often determines how the customer views the software’s delivered value. Because of this controversy, our findings and conclusions in this area, some of which run counter to the claims being made in the agile literature, are important. Based on the high reported failure rate of software projects reported in the literature, we believe that value increases when it is delivered on time and within budget even when some functionality is delayed.

Figure 5 – Fitness for Use by Domain Measured

by Percentage of Stories/Features/Functions Delivered as Promised







Deliveries – Agile

Deliveries – Traditional


C&C Defense Financial IT

Medical Mobile Tools Telecom


In Summary

The data that we have gathered provides convincing evidence that the quality of software developed using agile methods is better than that developed using more traditional software development approaches. The data gathered makes it apparent that software generated using agile methods has better software reliability and maintainability. Defect densities and rates are lower as are the response time for fixes and the number of defects introduced by them in fixes. However, this conclusion does lead one to wonder how much of this result is based on the fact that the best software performers today work in an agile environment because that is what they prefer.

In contrast, the results for “fitness for use” have to be rated in terms of whether readers view deferring software functionality to meet deadlines and budgetary constraints as a positive or negative. As stated earlier in the document, getting working software delivered on-time and budget with as much as 90 percent of the critical functionality in place can be viewed as a success. However, critics may argue that failure to meet one’s commitments for functionality and performance results in user dissatisfaction. In contrast, supporters of such deferrals might question how much of this added functionality was needed in the first place. Many times this extra functionality is never completed because it was not deemed necessary.

Of course, our final conclusion is that more analysis needs to be performed to determine whether or not these findings are valid for a range of unexplored regions. For example, the question can be raised: “Will these findings hold true when agile methods are scaled to address very large software systems or systems-of-systems?” While we have generated some clues relative to answering this question, the data collected thus far on this topic is still inconclusive. That is the reason why we need to continue our analysis.

Another question could be raised as to whether our agile findings are valid for subcategories of applications domains like Information Technology (e.g., banking, communications, finance, insurance, manufacturing, professional services, real estate, retail trade, travel, etc.) and defense (airborne, ground, missile, shipboard, space, etc.). For example, do the findings hold for ERP (Enterprise Resource Planning) and other enterprise-wide systems? Do they hold for embedded systems where software is developed to run across a wide range of secure platforms in real-time? Are there exceptions to the findings like safety-critical systems? To develop answers to these and other questions, we need to collect more data and continue to analyze and report the results.

To conclude, the analysis conducted to date provides empirical evidence that quality will be improved when agile methods are used to develop and maintain software products. This case is based on improved reliability and maintainability and acceptable “goodness of fit.”


We wish to thank those organizations who supplied the data upon which this study is predicated and those expert reviewers who supplied us comments for this publication.

Get the Full Report

For those interested, the full report upon which this summary is based is available on our website for a nominal fee at: http://www.reifer.com/products. Further information is available from:

Reifer Consultants LLC

1042 Willow Creek Road, Suite A101-485 Prescott, AZ 86301

Phone: (310) 922-7043

Website: http://www.reifer.com

Systems Engineering News

Integrating Program Management and Systems Engineering

One major effort of the INCOSE-PMI-MIT alliance has been to develop a book incorporating research the organizations have supported over the past three years focused on the integration of systems engineering and project/program management practices. This important new work is now becoming available to the world. The book will be available very soon through the INCOSE and PMI bookstores.

In the meantime, as an INCOSE member you can attend the INCOSE webinar where the lead representatives from each alliance organization introduce the background behind this work, overview the contents, and share their vision for opportunities that the book creates.

How Systems Engineering Can Help Fix Health Care

Hospitals and clinics are typically built in a piecemeal, patchwork approach. Institutions purchase hundreds of individual, siloed technologies — each with its own work processes, training, and user interfaces — based on what the market offers. They are then plopped into an ICU or operating room and hope that they somehow work together.

The result is a constellation of technologies that rarely connect, to the detriment of patient safety, quality, and value. For example:

  • Different monitors emit alarms that compete with one another for the attention of clinicians, who must sort out which signify serious conditions and which don’t. Sometimes they miss critical alarms amid the noise.
  • Devices, electronic medical records, and even patient beds have electronic information that can help diagnose conditions and assess risks. However, clinicians must consult each one individually, rather than seeing a unified display of information from them.
  • Time that could be spent with patients and their loved ones is instead squandered in front of computer monitors, as clinicians click through dozens of screens in search of relevant information.

All of this leads to needless patient harm, low productivity, excessive costs, and clinician burnout. Doctors and nurses feel as though they’re serving technology, not the other way around. Preventing complications, errors, and other harm too often depends on the heroism of clinicians rather than the design of safe systems.

An approach is needed that puts the needs of patients and clinicians first. One needs to integrate technology, people, and processes so that they are seamlessly joined in pursuit of a shared goal. At Johns Hopkins, they have experienced how powerful systems engineering can be when they set out to improve patient safety and quality of care in intensive care units. Patient safety researchers and clinicians from Johns Hopkins Medicine partnered with the systems engineers and systems integrators of the Johns Hopkins University Applied Physics Laboratory (APL). For 75 years APL has supported the Department of Defense and other government agencies as a “trusted agent” to solve critical challenges, such as building satellites and weapons systems on ships.

More information

“The Father of Process Systems Engineering” – Roger Sargent to Receive Sir Frank Whittle Medal

A leading chemical engineer who has been advancing the field of process systems engineering since the 1950s is to be honored with the Royal Academy of Engineering’s Sir Frank Whittle Medal.

Roger Sargent FREng, Emeritus Professor and Senior Research Fellow at Imperial College London and former director of its Centre for Process Systems Engineering, has spent his career of over 60 years championing the application of mathematics and computing to the solution of engineering problems in the process industries. He is to receive the Sir Frank Whittle Medal, for outstanding and sustained achievement, at the Royal Academy of Engineering’s Annual General Meeting on 8 September, 40 years since the inaugural meeting of the Academy at which he was a founding Fellow. Professor Sargent was among the first to recognize the need for a branch of chemical engineering concerned with the issues of how to design and operate processing facilities, leading the way in developing an approach based on mathematical techniques and developing the first commercially-available simulation software for the industry, SPEEDUP. He went on to found the Centre for Process Systems Engineering (CPSE) in 1989 at Imperial College London, which created the spin-out company Process Systems Enterprise (PSE Ltd). In 2007, PSE Ltd won the Academy’s MacRobert Award for its innovative modeling software. As an academic, Professor Sargent has supervised over fifty PhD students, many of whom have gone on to become the world’s leading chemical engineers, in turn inspiring a network of over 2,000 students with his vision and approach. Professor Sargent served as President of the Institution of Chemical Engineers from 1973-74, and in 1976 was one of the 130 engineers to form the Fellowship of Engineering, later to become the Royal Academy of Engineering. In 1993 he was elected as a Foreign Associate of the US National Academy of Engineering.

More information

Project in Gambia Helps Combat Coastal Erosion and Flooding

The West African country of The Gambia is prone to flooding and coastal erosion. The River Gambia, which passes through it, is both a source of livelihoods for communities and at the same time can become an environmental hazard severely affecting the capital, Banjul, and almost half the country. Better ecosystems management can help address this challenge. To this end, the Global Environment facility supported a UN Environment-led project titled Adoption of an Ecosystems Approach for Integrated Implementation of Multilateral Environmental Agreements at National and Divisional Levels.

The “ecosystems approach” to natural resource planning was applied in the pilot villages of Darsilameh and Tumani Tenda. This approach applies systems thinking to gain a better understanding of how ecosystems function. Systems thinking is the use of various techniques to study and understand systems of many kinds. In nature, examples of the objects of systems thinking include living systems in which various levels interact (cell, organ, individual, group, organization, community, earth). The approach contributes to people’s everyday life by identifying potential solutions to environmental issues, such as soil erosion, thus enhancing community livelihoods.

More information

Legacy Systems are Problems for Boardrooms Not Computer Geeks

It is a paradox that many companies, whose boards would prefer their businesses to be causing disruption with digital platforms, face disruption because of unwieldy and ageing platforms.

For instance, a technical glitch last August forced Delta Air Lines to cancel about 2,300 flights, annoying passengers and forcing the company to cut its profit guidance for the third quarter. The problems came about after back-up systems failed to kick in during disruption to the power supply at its Atlanta technology centre. Once power was restored, the back-end reservation system could not connect properly with the check-in and boarding system for several hours. At the time Paul Jacobson, its chief finance officer, declared it was as much a technology company as an airline and it needed improve its systems. As the existential threat, inherent in older “legacy” systems becomes clearer to executives, there is a danger such difficulties will be seen solely as IT problems rather than as wider business concerns. Legacy systems are a reflection of a company’s past and present; they mirror both the complexity of the world they were developed for and that they currently operate in. If you peel away a system’s layers you see code and data flows that reflect rules governing the business — some nuanced, some long forgotten — which determine how a computer should process information. As the business changes, new code is layered over existing code.

More information

CIMdata Announces the Expansion of Manufacturing Systems Engineering Consulting Practice

CIMdata, Inc., a global PLM strategic management consulting and research firm, announces the expansion of its Manufacturing Systems Engineering (MSE) Consulting Practice. This consulting practice focuses on bridging the flow of data and integrating processes between manufacturing and the design, fabrication, and operation of facilities and infrastructure to improve processes and products through the product life cycle. The MSE practice will concentrate on new and emerging technologies, including the Internet of Things (IoT) and Industry 4.0 practices, to achieve the factory of the future goals.

More information

U.S. Air Force Charts Wideband Global Satellite Future

The U.S. Air Force has embarked upon a formal Analysis of Alternatives (AoA) to determine a path for its constellation of Wideband Global Satellites (WGS) – a course which could result in wider use of existing commercial technologies or an effort to engineer and build a new dedicated constellation of satellites.

More information

DARPA Licenses Emerging Chip Technology

The U.S. Defense Advanced Research Projects Agency (DARPA) will license emerging “embedded” programmable chip technology from a Silicon Valley startup that specializes in chip intellectual property cores and accompanying software. Flex Logix Technologies Inc. of Mountain View, California, said this week DARPA would license its embedded FPGA chip technology for use by any contractor or U.S. agency designing chips for government programs. FPGA stands for field programmable gate arrays, an emerging chip technology that allows hardware to be reconfigured. In what company officials called a shift in DARPA procurement policies, the chips will be manufactured using process technology provided by Taiwan Semiconductor Manufacturing Co. (TSMC), the world’s largest chip foundry. Previously, DARPA and other branches of the U.S. military used “trusted” labs to manufacture components for weapons and other gear.

More information

Nordic Systems Engineering Voyage 2017

The German, Swedish, Danish, Finnish, and Polish chapters of INCOSE are sponsoring a voyage including five destinations for May 2017. This is the fifth year that this program has been provided. The objective is to provide a program that contains a balanced mix of talks addressing both general and local systems engineering challenges, including talks addressing current SE trends such as:

  • Systems Engineering as a company’s strategy
  • Engineering of Smart Systems (Autonomous vehicles, IoT, Industry 4.0, …)
  • Lean and Agile Systems Engineering
  • Safety and Security
  • MBSE

More information

Newcomer’s View of INCOSE’s International Workshop 2017

The International Council on Systems Engineering (INCOSE) annual International Workshop (IW) was held in Torrance CA USA January 28-31, 2017. Gonçalo Esteves, Business Process and Change Management Lead at John Wiley & Sons authored an article, “IW2017 – Four Days of Discovery” that provides his insights, observations, and learnings from his experience there. Read the full article here.

Featured Organizations

National Academy of Engineering (NAE)

The National Academy of Engineering (NAE) is an American nonprofit, non-governmental organization. The National Academy of Engineering is part of the National Academies of Sciences, Engineering, and Medicine, along with the National Academy of Sciences (NAS), the National Academy of Medicine, and the National Research Council. New members are elected by current members, based on their distinguished and continuing achievements in original research. The election process for new members is conducted annually. The NAE is autonomous in its administration and in the selection of its members, sharing with the rest of the National Academies the role of advising the federal government. The NAE operates engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. The current president is Dr. C. Daniel Mote, Jr.

More information

Institute of Industrial and Systems Engineers (IISE)

Systems world view. Productivity. Efficiency. These are words that describe the distinctive attributes of industrial engineering, and IISE is the world’s largest professional society dedicated solely to the support of the industrial engineering profession and individuals involved with improving quality and productivity. Founded in 1948, IISE is an international, nonprofit association that provides leadership for the application, education, training, research, and development of industrial engineering. ISEs figure out a better way to do things and work in a wide array of professional areas, including management, manufacturing, logistics, health systems, retail, service and ergonomics. They influence policy and implementation issues regarding topics such as sustainability, innovation, and Six Sigma. And like the profession, ISEs are rooted in the sciences of engineering, the analysis of systems, and the management of people.

More information

Georgia Tech Research Institute (GTRI)

The pioneering Georgia Legislators and Regents who founded the State Engineering Experiment Station – now known as the Georgia Tech Research Institute (GTRI) – would no doubt be impressed and amazed with what they started in 1934. While they might not recognize today’s modern GTRI, the organization’s fundamental purpose still rings familiar – real world research that solves tough problems for government and industry. STEM@GTRI’s purpose is to inspire, engage and impact Georgia educators and students by providing access to experts in the fields of science, technology, engineering, and math. Through this interaction, they hope to improve academic performance in STEM subjects, encourage students to pursue educational and career opportunities in these areas, as well as provide materials for teachers to strengthen their STEM-related curriculum.

More information

United States Air Force Scientific Advisory Board


The United States Air Force Scientific Advisory Board (SAB) is a Federal Advisory Committee that provides independent advice on matters of science and technology relating to the Air Force mission, reporting directly to the Secretary of the Air Force and Chief of Staff of the Air Force.

In the past, it has provided advice on technologies such as supersonic aircraft, weather forecasting, satellite communications, medical research, crewless airplanes, and defenses against aircraft and missiles. Today, the SAB performs in-depth reviews of the Air Force Research Laboratory’s science and technology research, and performs studies on topics tasked by the Secretary and Chief of Staff.

The SAB is tasked each year by the Secretary of the Air Force and the Chief of Staff of the Air Force to conduct studies on topics deemed critical to the Air Force mission and recommend applications of technologies that can improve Air Force capabilities.

The SAB also conducts five annual in-depth reviews of the science and technology programs in the Air Force Research Laboratory (AFRL).  Each of these week-long reviews addresses programs in one of the AFRL Technical Directorates, with essentially all AFRL research programs being reviewed over every four-year cycle.  These reviews have informed Air Force leadership and influenced science and technology pursued and adopted by the Air Force.

More information

Systems Engineering Tool News

PragmaDev Studio

PragmaDev Studio helps managing complexity inherent to developing today’s communicating systems. It integrates four different tools based on international standard technologies. Each tool is dedicated to a user profile: architects/system engineers, developers, and testers.

PragmaDev Specifier helps system engineers to unambiguously specify and verify the functionalities of the system, and define the best architecture for performance or energy efficiency. PragmaDev Developer helps software designers to write maintainable and self-documented code. PragmaDev Tester helps testers to write validation and integration tests with an abstract dedicated language. PragmaDev Tracer is a stand-alone tracing tool that is also integrated in the other modules.

The complete tool set includes bridges from one tool to the other such as automatic test case generation out of a functional model (model-based testing).

PragmaDev has established partnership with key players in the real time domain. Customers include Airbus, Renault, Alcatel-Lucent, ST, ABB Group, the French Army, the European Space Agency, Toshiba, Korean Telecom, and LG Electronics.

More information

Systems Engineering Publications

Systems Engineering Journal

Systems Engineering, INCOSE’s scholarly journal, is a primary source of multidisciplinary information for the systems engineering and management of products and services, and processes of all types. Systems engineering activities involve the technologies, processes, and systems management approaches needed for: definition of systems, including identification of user requirements and technological specifications; development of systems, including conceptual architectures, tradeoff of design concepts, configuration management during system development, integration of new systems with legacy systems, and integrated product and process development; and deployment of systems, including operational test and evaluation, maintenance over an extended life cycle, and reengineering. Modern systems, including both products and services, are often very knowledge intensive, and are found in both the public and private sectors. The journal emphasizes strategic and program management of these, and the information and knowledge base for knowledge principles, knowledge practices, and knowledge perspectives for the engineering of systems. Definitive case studies involving systems engineering practice are especially welcome.

The Art and Science of Systems Engineering

This is a short paper written in 2009 that summarizes the collective wisdom of fifteen of NASA’s best technical minds. The work culminates a collective 390 years of experience in systems engineering and focused discussions among NASA leadership. The objectives of the paper are to provide a clear definition of systems engineering, to describe the highly-effective behavioral characteristics of NASA’s best systems engineers, and to make explicit the expectations of systems engineers at NASA.

Download the paper.

Made to Stick: Why Some Ideas Survive and Others Die

C:\Users\Ralph\Pictures\INCOSE and SyEN\Made to Stick.jpg

Image source

by Chip Heath and Dan Heath

Book Description (from the Amazon website):

Why do some ideas thrive while others die? And how do we improve the chances of worthy ideas? In Made to Stick, accomplished educators and idea collectors Chip and Dan Heath tackle head-on these vexing questions. Inside, the brothers Heath reveal the anatomy of ideas that stick and explain ways to make ideas stickier, such as applying the “human scale principle,” using the “Velcro Theory of Memory,” and creating “curiosity gaps.” In this guide, we discover that sticky messages of all kinds – from the infamous “kidney theft ring” hoax to a coach’s lessons on sportsmanship to a vision for a new product at Sony – draw their power from the same six traits. Made to Stick is a book that will transform the way you communicate ideas. It’s a fast-paced tour of success stories (and failures) – the Nobel Prize-winning scientist who drank a glass of bacteria to prove a point about stomach ulcers; the charities who make use of “the Mother Teresa Effect”; the elementary-school teacher whose simulation actually prevented racial prejudice. Provocative, eye-opening, and often surprisingly funny, Made to Stick shows us the vital principles of winning ideas – and tells us how we can apply these rules to making our own messages stick.

More information

The Routledge Companion to Lean Management

C:\Users\Ralph\Pictures\INCOSE and SyEN\Routledge Companion to Lean Management.jpg

Image source

Edited by Tobjorn H. Netland and Daryl J. Powell

Reviews (from the Amazon website):

“Netland and Powell have done a wonderful job in bringing together the key writers on the Lean agenda. Anyone seeking to dig deeper in the field, either in the underlying philosophy or specific applications, will find much of value here. The book is unique in its breadth, giving an accurate reflection of the state of the art.”

Steve New, University of Oxford, UK

“When Quincy Jones produced the song We Are the World, written by Michael Jackson and Lionel Richie, he gathered what he called ‘the supergroup.’ The most admired singers in the world contributed with their best tones in ONE song. This is a book where Netland and Powell have done the very same thing, but on the ‘lean arena.’ They have gathered the most admired ‘lean singers’ in the world where everyone contributes with their best ‘lean tones.’ This is a megahit! Enjoy the rhythm!”

Niklas Modig, Stockholm School of Economics, Sweden

More information

Balanced Scorecard Step-by-Step: Maximizing Performance and Maintaining Results (2nd Edition)

C:\Users\Ralph\Pictures\INCOSE and SyEN\Balanced Scorecard 2.jpg

Image source

by Paul R. Niven

A customer’s review from Amazon:

This book gives you the power to say “Thank you, but no thank you” to the legions of high-priced consultants who offer to help design, establish and implement your company’s balanced scorecard. While Robert Kaplan did write the introduction, he and Norton must be very disappointed that they didn’t write this book — I’m certainly sorry that I didn’t write it. Author Paul R. Niven provides all the tools, methodologies, and steps necessary to create and execute your own balanced scorecard. He’s held nothing back. It’s all here. Whether you lead a local non-profit or are the CEO of a Fortune Global 500 corporation, Niven has provided detailed step-by-step instructions for both of the key phases of the balanced scorecard project:

I. Planning Stage:

Step One: Develop objectives.

Step Two: Determine appropriate organizational unit(s).

Step Three: Gain executive sponsorship.

Step Four: Build your balanced scorecard team.

Step Five: Formulate your project plan.

Step Six: Develop a communication plan.

II. Development Stage:

Step One: Gather and distribute background material.

Step Two: Develop mission, values, vision, and strategy.

Step Three: Conduct extensive interviews.

Step Four: Develop objectives and measures.

Step Five: Develop cause-and-effect linkages.

Step Six: Establish targets for your measures.

Step Seven: Develop the implementation plan.

This book is fantastic. In fact, I’m thinking about placing a bulk order for the book to give to all my clients. On second thought, why give away all the “secrets”? Seriously, this is one of the best business and management books I’ve read this year. Overall grade: AAA+++/Highly Recommended.

More information

A Primer for Model-Based Systems Engineering

C:\Users\Ralph\Pictures\INCOSE and SyEN\MBSE Primer.png

Image source

by David Long and Zane Scott

Vitech is pleased to make electronic copies of the 2nd Edition MBSE Primer freely available. The primer addresses the basic concepts of model-based systems engineering. It covers the Model, Language, Behavior, Process, Architecture, and Verification and Validation. It is a call to the consideration of the foundational principles behind those concepts. It is not designed to present novel insights into MBSE so much as to provide a guided tour of the touchstones of systems design. It is a guide to the new MBSE acolyte and a reminder to the experienced practitioner. It is suitable for use by a systems engineer new to the practice of “model-based” systems engineering, an experienced systems engineer who has been introduced to “model-based” concepts in an ad hoc fashion, or by any professional knowledgeable of systems thinking and practice. It is not intended as a comprehensive guide or as practice handbook. Comments and suggestions concerning how to improve this primer are welcome. Much of what has been learned about how it should be organized and presented has come from thoughtful contributions from the readers of the 1st edition.

An electronic copy is available and can be can be downloaded free here.

Education and Academia

MIT’s Architecture and Systems Engineering Professional Certificate Program

The friction between Model-Based Systems Engineering theory and practice is one of the themes explored in MIT’s new four-course online certificate program led by Dr. Bruce Cameron that began this month. “Our aim is to put theoretical ideas alongside real-world examples – from companies like Boeing, GM, and GE,” says Cameron, who serves as director of MIT’s System Architecture Lab. “You can see examples of MSBE being used to drive business outcomes across many different fields.”

More information

Industrial Engineering and Systems Engineering at Florida Tech

The mission of the department of engineering systems at the Florida Institute of Technology is to prepare engineers and scientists for leadership roles in business organizations. The educational objectives are to achieve steady enrollment growth and pursue practical funded research; to provide engineers and scientists the skills to expand their areas of responsibility in the workplace; and to update the skills of engineers and scientists in their fields of specialization.

More information

Southeast Missouri State University Launching New Engineering Program in Fall 2017

Southeast Missouri State University, USA, will launch a new engineering degree program beginning with the fall 2017 semester to help meet workforce demands and offer access to students seeking STEM education opportunities in southeast Missouri. “Southeast has a long history of delivering engineering-related programs in areas such as Engineering Physics, Engineering Technology, Industrial Technology and Technology Management, in addition to a minor in Engineering Physics and Southeast’s Pre-Engineering Program,” said Dr. Carlos Vargas, president of Southeast Missouri State University.  “The ability to offer this degree at Southeast will provide access to a high-skill program in a part of Missouri where some students are more place-bound due to financial constraints or familial responsibilities, and where other students are more likely to leave Missouri to pursue their education at schools in neighboring states that are closer than other institutions in Missouri.  Perhaps most importantly, this program will help respond to national, state and local workforce needs.”

More information

Institute for Mathematics and its Applications,

University of Minnesota College of Science & Engineering

The Institute for Mathematics and its Applications connects scientists, engineers, and mathematicians in order to address scientific and technological challenges in a collaborative, engaging environment, developing transformative, new mathematics and exploring its applications, while training the next generation of researchers and educators. This mission requires mathematicians and scientists skilled at thinking in new ways and across disciplinary boundaries. Thus, it is essential that the IMA engage participants from the largest talent base available, and to strengthen this base. For this reason, the IMA strives to increase the involvement of scientists from traditionally underrepresented groups with IMA programs at all levels — as workshop organizers, participants, and speakers; visiting scientists; and postdoctoral fellows. The IMA’s success in fostering new connections and collaborations gives its outreach efforts particular force.

More information

Welcome to the EIT Digital Professional School

The EIT Digital Professional School brings digital innovations to the market and supports the digital transformation of companies and organizations by creating a learning ecosystem. The initiative is powered by continuing education courses, specially designed to provide critical digital knowledge, insights and skills to European professionals and executives, leveraging the EIT Digital Action Lines and partners’ research activities.

The Challenges addressed by EIT Digital: The digital transformation requires constant technology updates in various fields. Also, the convolution of different digital technologies requires integrated knowledge in application domains currently not delivered by generic programs (e.g. systems knowledge for architects in urban development where telecommunications, citizen engagement, mobility and environmental sustainability come together). But today’s professionals have only limited flexibility in their schedules and continuing education often makes it only on the lower levels of the priority list. Intensive programs that require a very significant time investment with limited flexibility do not match their needs. Instead, suitable education programs will need to be wrapped around the existing commitments.

Together with its partners, EIT Digital has developed and operates blended education courses that optimally match the needs of busy professionals and their employers, combining online elements that can be followed asynchronously whenever a suitable amount of time becomes available with focused presence modules that offer opportunities for collecting hands-on experience. Both online as well as presence modules can incorporate peer education elements. EIT Digital provides relevant training courses independently certified, documenting successful course participation.

More information

solidThinking Aviation Case Study Teaches Embedded System Design

Embedded systems are digital computing systems programmed to perform a specific task within a larger mechanical or electrical system. Typically, these embedded systems perform their duties on limited hardware resources. They sense a real-world condition, do some computations, and then produce an output or state change. Embedded systems are now ubiquitous and account for the clear majority of CPUs manufactured globally. Growth in demand is expected to accelerate with the increased adoption of hybrid and electric vehicles, autonomous navigation and the growing complexity of other transportation vehicles from subways to jet planes. A recent example of the efficient design process enabled with model-based system design of this sort and using Altair’s solidThinking Embed software has been provided by AMETEK. AMETEK is a global manufacturer of electronic instruments and devices and often uses virtual prototyping software in developing its products.

More information

University of Maryland Baltimore Campus Systems Engineering Graduate Programs

The Systems Engineering Program at UMBC offers a Master of Science in Systems Engineering, a Master of Science in Systems Engineering with a Certificate in Cybersecurity, and a Certificate in Systems Engineering. Our graduate programs are designed for working engineers. UMBC’s program is designed to accelerate the development of systems engineers by providing practical, real-world experience that can be immediately applied on the job. Students learn from industry experts how to develop operable systems that meet customer requirements, while successfully navigating the complexities of system design. UMBC’s rich curriculum covers all aspects of a system’s life cycle using state-of-the-art principles, practices and technologies. This program, designed in collaboration with some of the leading employers in this field, balances practical application and theoretical understanding.

More information

Some Systems Engineering-Relevant Websites

For more information on Systems Engineering-relevant websites, please proceed to our website.

Standards and Guides

NIST Releases Internet of Things (IoT) Security Guidance

Late 2016, the National Institute of Standards and Technology (“NIST”) released  Special Publication 800-160 (the “Guidance”) on implementing security in Internet-of-Things (“IoT”) devices.  The Guidance was released following several highly-publicized Distributed Denial-of-Service (“DDoS”) attacks in 2016 and is intended to provide a framework for software engineers to better address security issues and to develop more defensible and survivable systems in a sustainable manner throughout the life cycle of these devices. IoT devices have become increasingly prevalent, both in the U.S. and worldwide, with one information technology research and advisory company forecasting that the count of IoT devices in use will reach 20.8 billion by 2020.

Ericsson projects that in 2018, IoT devices will surpass mobile phones as the largest category of connected devices, with a growth projection of 23 percent annually between 2015 and 2021. However, with this increased adoption comes a greater potential for misuse, as evidenced by their use in a number of recent DDoS attacks.  The attackers have been able to exploit the relative security weaknesses in IoT devices, like internet-connected cameras and DVRs, using malware to create networks of these computers, known as botnets, which report to a central control server that can be used as a staging ground for launching powerful DDoS attacks.  This malware is able to gain control over numerous IoT devices by continuously scanning the Internet for IoT systems protected by factory default or hard-coded usernames and passwords.

More information

Definition to Close On

A Simple Explanation of ‘The Internet of Things’

The “Internet of Things” (IoT) is becoming an increasingly growing topic of conversation both in the workplace and outside of it. It’s a concept that not only has the potential to impact how we live but also how we work. But what exactly is the “Internet of Things” and what impact is it going to have on you, if any? There are a lot of complexities around the “Internet of Things”. Lots of technical and policy-related conversations are being had but many people are still just trying to grasp the foundation of what the heck these conversations are about.

Source: http://www.forbes.com/sites/jacobmorgan/2014/05/13/simple-explanation-internet-things-that-anyone-can-understand/#33f8ae786828

Conferences and Meetings

For more information on systems engineering-related conferences and meetings, please proceed to our website.

PPI News

CTI Presenting at Tianjin, China

CTI is excited to be presenting its first course in Tianjin. Tianjin is a major port city in northeastern China and one of the five national central cities of China. Tianjin boasts the largest artificial harbor in northern China with 30 different sea routes and is a famous tourist city in China.

PPI Hosts an Evening Presentation by Simon Fearnley

Simon Fearnley from Agile Controls gave an evening presentation to PPI delegates in Adelaide, Australia on Thursday, 2 March 2017. The presentation was based on Cradle, an integrated requirements management and model-based system engineering tool with features, flexibility and scalability for the full life cycle of today’s complex agile and phase-based projects.

PPI returns to Canberra

PPI returned this month to Canberra, the Australian national capital, with our flagship Systems Engineering course, the first delivery of many in response to changes in the structure and priorities of the public sector. Requirements and military capability development will also be emphasised in this program to contribute to improved outcomes from public sector expenditure.

Software Engineering 5 – Day Training Course Launch

PPI is excited to announce the launch of a major update to our Software Development 5-Day Course, the course having been through several months of intense redevelopment, verification and validation to reflect the best in software development, today and beyond. The many longstanding and new PPI clients who have been waiting for this release may now build public course dates into their training schedules, or seek the training on-site with committed dates. Created by software development experts, the course examines the many elements that contribute towards success in software-intensive projects, not just in the areas of software development methodologies and languages, requirements, architecture, detailed design, coding, integration and test, but across all aspects of the software life cycle.

The course also embraces the organizational and cultural aspects of successful software development. This 5-day course in software engineering provides understanding of the elements for the effective realization of software-intensive systems that are cost-effective, on schedule and meet stakeholder requirements and needs over the full life cycle, whilst managing the inevitable risk. Please contact the PPI team for further information about having this course presented on-site to your organisation.

Being copied is a compliment, except when it is criminal

PPI has become aware that its course descriptions and courseware, created by PPI at a seven figure cost to PPI, have been illegally copied and used by training companies in the United States, the Netherlands and the United Arab Emirates. We are sure that you, the reader, will share our disgust at the immorality displayed by these organisations. Regarding the illegality, of course we are taking action. We will share news with you, including the names of these organisations, at the appropriate times. In the meantime, if you have information on these illegal practices that you can share with us, we would love to hear from you, in confidence of course. If you, as an individual, have taken one of these illegally delivered training courses, we would be pleased to exchange your illegal courseware for a set of legal PPI courseware together with a complimentary attendance at one of PPI’s training courses at any location worldwide. If you as a company have taken one of these illegally delivered training courses on-site, we would like to work with you to establish circumstances under which you can legally hold and use the course materials. In this context, we will be seeking information from you. Please note that, in most jurisdictions, it is illegal to hold counterfeit materials in violation of intellectual property law. If you hold such materials in which PPI owns copyright, but would prefer not to work with us on this matter, you as an individual or as a company are instructed to destroy such materials immediately to avoid yourselves becoming the subject of the action that PPI is taking.

PPI and CTI Events

Systems Engineering Public 5-Day Courses

Upcoming locations include:

  • Zurich, Switzerland
  • London, United Kingdom
  • Ankara, Turkey

Requirements Analysis and Specification Writing Public Courses 

Upcoming locations include:

  • Amsterdam, the Netherlands
  • Ankara, Turkey
  • Munich, Germany

Systems Engineering Management Public 5-Day Courses 

Upcoming locations include:

  • Bristol, United Kingdom
  • Orlando, Florida, United States of America
  • Melbourne, Australia

Requirements, OCD and CONOPS Public 5-Day Courses

Upcoming locations include:

  • Melbourne, Australia
  • Auckland, New Zealand
  • Amsterdam, The Netherlands

Human Systems Integration Public 5-Day Courses

Upcoming locations include:

  • Sydney, Australia

CSEP Preparation 5-Day Courses (Presented by Certification Training International, a PPI company)

Upcoming locations include:

  • Denver, Colorado, United States of America
  • Orlando, Florida, United States of America
  • North Ryde, New South Wales, Australia

PPI Upcoming Participation in Professional Conferences

PPI will be participating in the following upcoming events.

IEEE International Systems Engineering Conference (SysCon)

24 – 27 April 2017

Montreal, Canada

IISE Annual Conference and Expo

20 – 23 May 2017

Pittsburgh, PA, USA

15th Annual Conference on System Engineering Research (CSER)

23 – 25 March 2017

Redondo Beach, CA, USA

27th Annual INCOSE International Symposium (IS2017)

July 15 – 20, 2017

Adelaide, Australia

13th INCOSE SA Conference 2017

11 – 13 October 2017

Pretoria, South Africa

Kind regards from the SyEN team:

Robert Halligan, Editor-in-Chief, email: rhalligan@ppi-int.com

Ralph Young, Editor, email: ryoung@ppi-int.com

Suja Joseph-Malherbe, Managing Editor, email: smalherbe@ppi-int.com

Project Performance International

2 Parkgate Drive, Ringwood North, Vic 3134 Australia Tel: +61 3 9876 7345 Fax: +61 3 9876 2664

Tel Brasil: +55 11 3958 8064

Tel UK: +44 20 3608 6754

Tel USA: +1 888 772 5174

Web: www.ppi-int.com

Email: contact@ppi-int.com

Copyright 2012-2017 Project Performance (Australia) Pty Ltd, trading as Project Performance International.

Tell us what you think of SyEN. Email us at syen@ppi-int.info.

  1. A Balanced Scorecard defines what management means by “performance” and measures whether management is achieving desired results. The Balanced Scorecard translates Mission and Vision Statements into a comprehensive set of objectives and performance measures that can be quantified and appraised.
Scroll to Top