Abstract

Discussion frequently occurs concerning the question “what does a systems engineer do?” This paper looks at the nature of systems engineering and its relationship to the job title or role description “systems engineer”. This concept of a job title or role is contrasted with a view that systems engineering is (or should be) an integral part of the discipline of engineering. The latter view suggests that differentiation between systems engineers and other engineers is artificial. Is the use of the term “systems engineer” just a semantics issue, or does its use have a deeper significance? What are the options and their implications?

Introduction

The term “systems engineer” is widely used within a context of “systems engineering”. This usage contrasts with usage of the terms “systems engineer” and “system engineer” in the IT sector, often referring to somebody who implements or maintains IT systems. This paper will examine definitions of systems engineering, then expose sets of principles of systems engineering from two different sources. The applicability of each of these principles will be examined with respect to the engineering of systems of different complexities and technologies, and to what we will call “non-systems”, unitary objects. From the applicability analysis, the paper draws conclusions regarding the generality of the applicability of systems engineering practice within engineering practice. The paper concludes by discussing the positive and the potentially negative consequences of use of the terms “systems engineer” and “systems engineering” in some contexts, and recommends context of usage.

Systems Engineering

The International Council on Systems Engineering (INCOSE) is believed to be the largest professional society concerned solely with systems engineering. INCOSE defines systems engineering as:

“Systems Engineering is a transdisciplinary and integrative approach to enable the successful realization, use, and retirement of engineered systems, using systems principles and concepts, and scientific, technological, and management methods.” [1]

Significantly, the definition embraces only a statement of purpose (enablement – making possible), and is ambiguous as to whether the principles, concepts and methods used apply to the approach or to the realization, use and retirement. It is unclear whether, under this definition, systems engineering incorporates any practice of engineering at all. Unfortunately, this definition is not going to be useful for the purpose of this paper.

NASA has defined systems engineering as follows: “System engineering is a robust approach to the design, creation, and operation of systems. In simple terms, the approach consists of identification and quantification of system goals, creation of alternative system design concepts, performance of design trades, selection and implementation of the best design, verification that the design is properly built and integrated, and post-implementation assessment of how well the system meets (or met) the goals.” [2]

In the experience of the author, this NASA definition aligns well with the use of the term “systems engineering” in industry worldwide. The definition also strongly aligns with the focus of the 50+ INCOSE Working Groups on aspects of systems engineering, and with the ISO/IEC/IEEE standard 15288:2015 on systems engineering. [3]

The author’s preferred definition of systems engineering aligns closely with the NASA definition: “Systems engineering is an interdisciplinary, collaborative approach to the engineering of systems (of any type) that aims to capture stakeholder needs and objectives and to transform these into a holistic, life-cycle balanced system solution that both satisfies the minimum requirements, and optimizes overall project and system effectiveness according to the values of the stakeholders. Systems engineering incorporates both technical and management processes. Systems engineering excludes construction, except for construction within development.” [4]

Another attractive, more concise definition is “Systems engineering is the problem-independent and solution technology-independent principles and methods for the successful engineering of systems, based on systems thinking”. [5]

These last three definitions of systems engineering provide the basis for the subsequent content of this paper.

Systems Engineering Principles And Their Applicability

The hypothesis that systems engineering is applicable to the conduct of engineering in general, irrespective of the system being engineered, may be tested against each of a set of representative systems engineering principles. In this paper, two sets of principles are considered. The first is a set of principles developed by the author in 2002 and slightly refined over subsequent years. The second set was developed by the INCOSE Principles Action Team and published by INCOSE in 2019. [6]

For each principle in each set, the principle is rated by the author “Yes/No” for applicability to the engineering of:

1. ST: a large socio-technical system, such as the country of Singapore.
2. CO: a complex technology item such as an aircraft.
3. SI: a simple technology item such as a pen.
4. SO: software of any size and complexity.
5. NS: a non-system such as a USA quarter coin, produced by forming a material into a shape and without plating.

NO.	PRINCIPLE	ST	CO	SI	SO	NS
1	Capture and understand the requirements, measures of effectiveness and goals (the problem) before committing to the solution.	Y	Y	Y	Y	Y
2	Try to ensure that the requirements are consistent with what is predicted to be possible in solutions, at the time of required supply, i.e., are feasible.	Y	Y	Y	Y	Y
3	Treat as goals desired characteristics that may not be feasible, but not at the expense of the requirements. Note: “treat as goal” means that effort will be expended to achieve the goal which is related to the importance of the goal, and the probability of success. Where conflicts between goals exist, the goals will be traded off to maximize overall effectiveness.	Y	Y	Y	Y	Y
4	Define system requirements, measures of effectiveness, goals and solutions having regard to the whole of the (remaining) life cycle of the system of interest.	Y	Y	Y	Y	Y
5	Maintain a distinction between the statement of the problem and the description of the solution to that problem, for the system of interest, and for each subsystem/component/system element of that system. “Note: “Maintain a distinction” means “ensure that each is separately identifiable.”	Y	Y	Y	Y	Y
6	Baseline each statement of the problem (requirements, measures of effectiveness and goals set) and description of the solution to that problem (design). Control changes to requirements and design.	Y	Y	Y	Y	Y
7	Identify and develop descriptions of solutions alternatives (designs) that are both feasible (i.e., can meet requirements) and potentially are the most effective. Put aside from further consideration, as potential solutions, all other alternatives (unless the assessment of that potential solution changes). Note: MOEs could include development cost, time to market or other measures unrelated to system capabilities.	Y	Y	Y	Y	Y
8	Develop solution descriptions for enabling systems concurrently and in balance with the solution description for the system of interest. Note: an “enabling system” is a system which enables some phase of the life cycle of the system of interest. The internal design of an enabling system must be related to the internal design of the system of interest.	Y	Y	Y	Y	Y
9	Except for simple solutions, develop logical solution descriptions (description of how the system solution is to meet requirements) as an aid to developing physical solution descriptions (description of how to build the system).	Y	Y	Y	Y	Y
10	Be prepared to iterate in design to drive up overall effectiveness, but not at the expense of the requirements.	Y	Y	Y	Y	Y
11	Decide between feasible solution alternatives based on evaluation of the overall effectiveness of each of these alternatives. Limit alternatives to be evaluated to those that have potential to be the most overall effective. Take risk and opportunity into consideration in the evaluation.	Y	Y	Y	Y	Y
12	Subject to level of risk, independently verify work products (is the job being done right?)	Y	Y	Y	Y	Y
13	Subject to level of risk, validate work products from the perspective of the stakeholders whom the work products serve (is the right job being done?)	Y	Y	Y	Y	Y
14	The act of managing is needed to plan and implement the effective and efficient transformation of requirements and goals into solutions.	Y	Y	Y	Y	Y
15	Decide early on development strategy, between waterfall, incremental, evolutionary and spiral, based on ability to define good, stable requirements up front; risk due to technology; risk due to complexity; and other sources and levels of risk and opportunity.	Y	Y	Y	Y	Y

Table 1: Applicability Mapping from PPI Systems Engineering Principles

NO.	PRINCIPLE	ST	CO	SI	SO	NS
1	Systems engineering in application is specific to stakeholder needs, solution space, resulting system solution(s), and context throughout the system life cycle.	Y	Y	Y	Y	Y
2	Systems engineering has a holistic system view that includes the system elements and the interactions amongst themselves, the enabling systems, and the system environment.	Y	Y	Y	Y	N
3	Systems engineering influences and is influenced by internal and external resources, and political, economic, social, technological, environmental, and legal factors.	Y	Y	Y	Y	Y
4	Both policy and law must be properly understood to not over-constrain or under-constrain the system implementation.	Y	Y	Y	Y	Y
5	The real physical system is the perfect model of the system.	?	?	?	?	?
6	A focus of systems engineering is a progressively deeper understanding of the interactions, sensitivities, and behaviors of the system, stakeholder needs, and its operational environment.	Y	Y	Y	Y	Y
7	Sub-Principle 6(a): Mission context is defined based on the understanding of the stakeholder needs and constraints.	Y	Y	Y	Y	Y
8	Sub-Principle 6(b): Requirements and models reflect the understanding of the system	Y	Y	Y	Y	Y
9	Sub-Principle 6(c): Requirements are specific, agreed to preferences within the developing organization.	Y	Y	Y	Y	Y
10	Sub-Principle 6(d): Requirements and system design are progressively elaborated as the development progresses.	Y	Y	Y	Y	Y
11	Sub-Principle 6(e): Modeling of systems must account for system interactions.	Y	Y	Y	Y	Y
12	Sub-Principle 6(f): Systems engineering achieves an understanding of all the system functions and interactions in the operational environment.	Y	Y	Y	Y	Y
13	Sub-Principle 6(g): Systems engineering achieves an understanding of the system’s value to the system stakeholders.	Y	Y	Y	Y	Y
14	Sub-Principle 6(h): Understanding of the system degrades during operations if system understanding is not maintained.	Y	Y	Y	Y	Y
15	Stakeholder needs can change and must be accounted for over the system life cycle.	Y	Y	Y	Y	Y
16	Systems engineering addresses stakeholder needs, taking into consideration budget, schedule, and technical needs, along with other expectations and constraints.	Y	Y	Y	Y	Y
17	Sub-Principle 8(a): Systems engineering seeks a best balance of functions and interactions within the system budget, schedule, technical and other expectations and constraints.	Y	Y	Y	Y	Y
18	Systems engineering decisions are made under uncertainty.	Y	Y	Y	Y	Y
19	Decision quality depends on knowledge of the system, enabling system(s), and interoperating system(s) present in the decision-making process.	Y	Y	Y	Y	Y
20	Systems engineering spans the entire system life cycle.	Y	Y	Y	Y	Y
21	Sub-Principle 11(a): Systems engineering obtains an understanding of the system.	Y	Y	Y	Y	Y
22	Sub-Principle 11(b): Systems engineering defines the mission context (system application).	Y	Y	Y	Y	Y
23	Sub-Principle 11(c): Systems engineering models the system.	Y	Y	Y	Y	N
24	Sub-Principle 11(d): Systems engineering designs and analyzes the system.	Y	Y	Y	Y	Y
25	Sub-Principle 11(e): Systems engineering tests the system.	Y	Y	Y	Y	Y
26	Sub-Principle 11(f): Systems engineering supports the production of the system.	Y	Y	Y	Y	Y
27	Sub-Principle 11(g): Systems engineering supports operations, maintenance, and retirement.	Y	Y	Y	Y	Y
28	Complex systems are engineered by complex organizations.	Y	Y	N	Y	N
29	Systems engineering integrates engineering disciplines in an effective manner.	Y	Y	Y	Y	Y
30	Systems engineering is responsible for managing the discipline interactions within the organization.	Y	Y	Y	Y	Y
31	Systems engineering is based on a middle range set of theories.	?	?	?	?	?
32	Sub-Principle 15(a): Systems engineering has a systems theory basis.	?	?	?	?	?
33	Sub-Principle 15(b): Systems engineering has a physical/logical basis specific to the system.	?	?	?	?	?
34	Sub-Principle 15(c): Systems engineering has a mathematical basis.	?	?	?	?	?
35	Sub-Principle 15(d): Systems engineering has a sociological basis specific to the organization.	?	?	?	?	?

Table 2: Applicability Mapping from INCOSE Systems Engineering Principles

Deductions From Applicability Of The Principles

If the mapping of applicability of systems engineering principles to various types and sizes of project is valid, one would have to conclude that systems engineering is not something different or special. Systems engineering is just about engineering things, any thing.

Intuitively, one would expect that the more novel, complex or critical a system were, the more valuable a systems engineering approach to its development would be. This intuition is substantiated by the conclusions of a 2012 joint study led by the Carnegie Mellon University (CMU) Software Engineering Institute (SEI) using data from 148 development projects. [7] The projects, ranging from small to very large, were divided into two nominally equal-sized sets, the less-challenging half and the more challenging half.

Correlation coefficients were developed between project performance, incorporating cost, schedule, and quality outcomes, and systems engineering practice. The correlation coefficients of +0.34 for the less challenging and +0.62 for more challenging projects indicate very significant benefit regardless of project challenge, but much greater benefit for the more-challenging projects. A more extensive summary of some results of the CMU study is provided in Figure 1.

For the study, the relationships between the variables were summarized using Goodman and Kruskal’s Gamma, a measure of association that expresses the strength of relationship between two ordinal variables. A description of Goodman and Kruskal’s Gamma appears in the book Elementary Applied Statistics by Linton Freeman. [8]

Source: “The Business Case for Systems Engineering Study: Results of the Systems Engineering Effectiveness Survey”, CMU/SEI-2012-SR-009, November 2012

Figure 1: Correlation between SE Practices and Project Performance

The study also provides a useful benchmark for the group of practices defined, for the purpose of the study, to constitute systems engineering practice. Significantly, these are not enabling practices according to the current INCOSE definition of systems engineering, but are mostly “doing” practices are aligned with both sets of systems engineering principles (my set and the INCOSE set).

Personally, I rarely use the term “systems engineer”, but I have lived and breathed systems engineering since 1985, the year when I realized how incompetent I was as an engineer. I was a good technologist in my field – radio engineering – but I didn’t have a clue as to how to go about applying that expertise in a way that consistently produced great results.

My view has solidified over the years that every engineer is, or should be, a systems engineer within the scope of their assigned engineering role(s). Systems engineering is not something separate from engineering in general; it is a way of practicing engineering in general. The variables are the scope of the task, the degree of systems engineering formality that is appropriate to that scope, and the balance of technical versus social skills needed. The scope of the task has variables of the specific technology or technologies involved, the diversity of technologies involved, the degree of integration of different technologies needed, and the degree of integration needed of specialty disciplines such as safety or cyber-security.

I am not alone in this view of systems engineering as an integral part of the discipline of engineering, that is, the principles and process component that supports the technology, math and physics component of the discipline of engineering. Many companies implement the ethos of every engineer being a systems engineer. That affects their hiring, their engineering procedures, their learning culture, their learning strategy, and their reward systems. And their performance!

My colleague Paul Davies added some further thoughts, observing from 40 years of experience that there are two essentially different approaches that organizations utilize concerning systems engineering:

Type 1 is a “stove-piped” industry, where the executive power is in the hands of single-discipline engineering empires. [9] Systems engineering is recognised as a necessary annoyance to glue the designs together, but the practitioners are subservient to whichever single-discipline expert happens to be in charge of the particular project. Career prospects of the “systems engineer” are limited. I can relate to Type 1 easily, having had many engineers but few managers from “Type 1” organizations participate in my systems engineering training courses, and having consulted to such organizations. Thankfully I have never been an employee of a Type 1 organization!

Type 2 is an organization whereby systems engineering is firmly embedded and the lead engineers on all projects are recognised as experts in both the technologies involved and in the process and integrative discipline of systems engineering. Indeed, single-discipline engineers are allocated to projects, to have their work directed by the cross-discipline “systems engineers”, and systems engineering is seen as a desirable career aspiration.

I share Paul’s views in the sense of Type 1 and Type 2 organizations being common. I add a Type 3 organization which is also common, where engineers work mainly in multidisciplinary teams, making the more important decisions collaboratively, creating a learning environment in which engineers may start as single-discipline and ignorant of systems engineering, and evolve to become multi-disciplinary and competent in systems engineering, usually while maintaining their original discipline strength. Some change discipline in the process, for example from electronics to software. The further along this evolutionary path an engineer travels, the more suited that person becomes to a role of leading a multi-disciplinary team. But engineers at any point on this journey can be, should be, and often are practicing systems engineering. Enterprises in sectors that develop complex systems in competitive markets or complex environments tend towards Type 3.

Some systems are engineered to comprise a diversity of technologies, for example a car. Other systems are engineered to comprise essentially a single technology, examples being a mechanical Swiss watch, a drug, a consulting company, Sydney Harbour Bridge, and MS Word.

The principles of systems engineering described in this paper can easily be checked-off as applying to all of these engineered systems; the benefits accrue accordingly. It is intuitive that the more complex a system is, the greater the risk due to that complexity, and the more valuable systems engineering practices become. Also intuitive is that, everything else being equal, the greater the diversity of technologies involved, the more complex a system solution is likely to be, and therefore, the more valuable systems engineering practices will be.

All of these aspects are thoroughly discussed in the 2012 CMU study. [7] That study also looked at the proportion of projects that implemented systems engineering as a separate group of “systems engineers”, versus systems engineering skills and responsibilities being distributed throughout the organization. The study concluded that the percentages were roughly equal, and the project performances were not different to a statistically significant degree. The study hypothesized that while centralized systems engineering is more effective at performing SE tasks, those efforts are less integrated with the work of the project and therefore have less impact on project performance.

The study would seem to suggest that my advocacy of “systems engineering to be practiced by all engineers” is neither supported nor negated by the data. However, there is more to the story that is not investigated in the study. My experience is that there is a big difference between having a policy of “systems engineering to be practiced by all engineers”, and having an engineering workforce in which that practice is occurring at a significant level of competence. The main challenge is the cost of educating the whole workforce. As a consequence, the education tends to be superficial, at least initially. Training a small group of SE leaders and having them try to influence the work of others requires less investment than training of a whole engineering workforce, notwithstanding the high return on investment observed from doing the latter.

With implementation of a sound systems engineering socialization strategy, involving among other strategies genuinely expert leadership, systems engineering champions, and systems engineering focus groups, the growth of systems engineering competence increases towards a “way of life” level at a rewarding pace.

When systems engineering becomes omnipresent in undergraduate engineering degree programs, the need for industry to train an engineering workforce in systems engineering will disappear. This can be considered a desirable end state. The education does not need to be called systems engineering, and arguably, there may be benefit in avoiding use of the term. I am encouraged to see many initiatives to integrate systems engineering into undergraduate engineering education, at an increasing rate. I am excited to be involved in some of these initiatives.

Systems Engineers, Systems Engineering And Professional Societies

A number of professional societies and industry organizations are active in advancing the discipline of systems engineering. Among these are IEEE, Institute of Industrial and Systems Engineers (IISE), International Council on Systems Engineering (INCOSE), Korean Council on Systems Engineering (KCOSE), National Defense Industrial Association (NDIA), Object Management Group (OMG), and SAE International. The membership of INCOSE, the leading society devoted entirely to systems engineering, is dwarfed in numbers by the memberships of IEEE and SAE, and yet, the constituency of INCOSE, engineers who would benefit from systems engineering, and their employers, is hugely greater than the IEEE and SAE memberships combined.

The INCOSE website has reflected a slow shift in orientation from systems engineers to systems engineering over the years. Whilst there is absolutely nothing wrong with the term “systems engineer” as a title for a specific job – it can be accurate and carries some justified prestige – the term when used as a synonym for someone who practices systems engineering – a race apart – disenfranchises the millions of engineers who could and should be practicing systems engineering. That cannot be good for attracting new members to systems engineering societies such as INCOSE. I believe that any orientation of professional society publicity towards systems engineers rather than systems engineering to be detrimental to attraction of new members. This is an issue for the Institute for Industrial and Systems Engineers, as well as INCOSE. On a positive note, I was encouraged to see INCOSE actively discourage the use in education resources of the term “systems engineer”, by way of instructions to content contributors in drafting of the INCOSE Systems Engineering Handbook 5th edition, now the current handbook version. The pointer is shifting.

Conclusions

A case has been made for the applicability of systems engineering to all possible subjects of engineering, based on a mapping of the applicability of two sets of principles of systems engineering to the development of:

1. Socio-technical systems.
2. Large, complex technology products and systems.
3. Small, simple technology products and systems.
4. Software-only systems.
5. Non-systems – engineered unitary objects not meaningfully viewable as being constructed of interacting elements.

The case has been strengthened by study data that demonstrates the value of systems engineering for the engineering of small, simple systems as well as larger, more complex systems, the principal difference in application being the degree of benefit.

This demonstration of general applicability has led to a conclusion that systems engineering could be, and should be, practiced by all engineers regardless of job title, not only by “a race apart” small minority. A corollary is the need for systems engineering content in every undergraduate engineering degree worldwide, a pursuit that is gaining momentum.

The term “systems engineer” is widely used and is an accurate job title for many jobs. The term is appropriate for this use as a job title, often carrying justified prestige. However, the term is problematic when used in explaining and promoting systems engineering. The terms “systems engineering” and “systems engineering practitioners” are recommended for use in describing, promoting, explaining, advocating and communicating about systems engineering. In doing so, the present disenfranchisement of the millions of engineers without “systems engineer” in their job title who would benefit from the practice of systems engineering can be avoided. Based on the data presented in this paper, we conclude that society will benefit accordingly from better project outcomes.

References

[1] International Council on Systems Engineering Website, https://www.incose.org/systems-engineering, 1 November 2020.

[2] NASA Systems Engineering Handbook, 1995.

[3] ISO/IEC/IEEE 15288:2015 Systems Engineering – System life cycle processes.

[4] Halligan, Robert John, Published training courseware, 2002.

[5] Halligan, Robert John, Published training courseware, 2012.

[6] Watson, Michael D., “Systems Engineering Principles and Hypotheses”, INSIGHT, May 2O19 Volume 22 / Issue 1.

[7] Elm, Joseph P., and Dennis R. Goldenson, “The Business Case for Systems Engineering Study: Results of the Systems Engineering Effectiveness Survey”, Carnegie Mellon University, Software Engineering Institute, Special Report CMU/SEI-2012-SR-009, 2012.

[8] Freeman, Linton C., “Elementary Applied Statistics: For Students in Behavior Science”. J. Wiley & Sons, 1965.

[9] Davies, Paul, personal correspondence, 2020.

Bibliography

Armstrong, James R., “Implementing Systems Engineering”, Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 2000.

Brill, James H., “Systems Engineering – A Retrospective View”, The Journal of The International Council on Systems Engineering, Vol. 1, No. 4, pp. 258-266, 1998.

Bruno, Michelle E. and Brian W. Mar, “Implementation of the Systems Engineering Discipline”, Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 1997.

Carlson, Harry, “Systems Engineering Training Program”, Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 1991.

Elm, J.; Goldenson, D.; El Emam, K.; Donatelli, & N.; Neisa, A. “A Survey of Systems Engineering Effectiveness – Initial Results” (CMU/SEI-2008-SR-034). Software Engineering Institute, Carnegie Mellon University, 2008. https://www.sei.cmu.edu/library/a-survey-of-systems-engineering-effectiveness-initial-results-2/

Elm, J. “The Business Case for Systems Engineering Study: Assessing Project Performance from Sparse Data”. (CMU/SEI-2012-SR-010). Software Engineering Institute, Carnegie Mellon University, 2012. https://www.sei.cmu.edu/library/the-business-case-for-systems-engineering-study-assessing-project-performance-from-sparse-data/

Elm, J. & Goldenson, D. The Business Case for Systems Engineering Study: Detailed Response Data. (CMU/SEI-2012-SR-011). Software Engineering Institute, Carnegie Mellon University, 2013. https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=73582

Fogle, Frank R., and Don L. Woodruff, “Definition of and Training in the Systems Engineering Process – A NASA Perspective”, Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 1992.

Frank, Moti, “Cognitive and Personality Characteristics of Successful Systems Engineers”, Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 2000.

Grady, Jeffrey O., “Local System Engineering Education Program Ignition and Sustenance”, Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 1991.

Harris, Michael B., “Development of Systems Engineers: A Structured Approach Based Upon International Experience”, Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 2000.

Kaufman, Mark S., “Internal Development of Systems Engineers”, Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 1997.

Lentz, V.A., “Five Realities for Systems Engineering in Commercial Enterprises”, Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 2000.

McAuley, James E., “What Should Systems Engineers Know and When Should They Know It?” Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 1992.

Mengot, Roy, “Building Systems Engineering Training In Industry”, Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 1992.

Might, Robert J. and Robert A. Foster, “Educating System Engineers: What Industry Needs and Expects Universities or Training Programs to Teach”, Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 1993.

Newbern, David and Jerome Nolte, “Systems Engineering Process Implementation in the Real World (Or Where the Theory Gets Tested)”, Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 2000.

Sheard, Sarah A., “Twelve Systems Engineering Roles”, Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 1996.

Sheard, Sarah A., “Three Types of Systems Engineering Implementation”, Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 2000.

Watts, Jerry G., and Brian W. Mar, “Important Skills and Knowledge to Include in Corporate Systems Engineering Training Programs”, Seminar Proceedings of the Annual International Council on Systems Engineering (INCOSE) on CD-ROM, 1997.