Welcome to this presentation on three tools that can change a company; a company that is reliant on successful engineering to survive and prosper.

My name is Robert Halligan. I am the Founder and Managing Director of Project Performance International and a person who has lived and breathed the pursuit of high-performance engineering since 1985, the year in which I realised how incompetent I was as an engineer.

I went looking for better ways of engineering things and discovered a substantial body of knowledge that was not being taught to universities, but was extremely helpful in achieving excellent results. I’m referring to the principles and the processes of engineering as contrasted with the emphasis in university undergraduate education on technology, physics and math. These two aspects together constitute the discipline of engineering – neither is sufficient alone. I embraced these newly discovered principles and methods body of knowledge with a vengeance, immediately producing much better results, and I’d like to think that I’ve subsequently contributed to the body of knowledge.

I share with you today three process tools (amongst many others that exist) that I’ve developed and used extensively over the years, and that can really make a difference.

Before moving to the tools, I would like to put what follows in further context.

PPI Training and Consulting in SE worldwide

Project Performance International (PPI) is a company of like-minded leaders who thrive on helping others succeed. We consult and train worldwide in the discipline broadly called ‘systems engineering’, which with some license can be defined as “an approach to engineering that embraces fields like systems thinking, design thinking, product development and engineering management, all of which are concerned with problem definition and problem-solving”.

PPI, which I formed in 1992, has so far consulted or trained physically in 41 countries across six continents to over 17,000 alumni, mainly engineering professionals but proponents of many other disciplines also. You may see from the map PPI’s staff and professional delivery footprint to date, and growing steadily.

Tool 1: PPI Requirement Writing Template

Parsing Template

Now, let’s look at these process tools. Two of the tools are requirements related the third is a project and engineering management tool. The first tool is a unique requirement authoring pattern or template. Most attempts at authoring patterns have been subject-matter specific and, as such, have met with limited success. This template, which I will call the ‘Parsing’ template, is based on different concepts, is entirely general in its application, and is extraordinarily effective.

SEI/AESS/NDIA 2012 Study Results: Requirements

Before going into the pattern, let me place it in context by drawing your attention to this summary of a comprehensive scientific quality study by Carnegie Mellon University on the correlation between process related activities and project performance. You can see the various activities looked at in this study. For smaller, simpler projects, the single most beneficial activity was shown to be requirements development and management, the focus of this tool that we’re about to discuss and the next.

For larger, more complex projects, project planning was the most beneficial activity, that is, with the largest correlation coefficient, a very strong positive correlation. This is the focus of the third of our tools. As you’ll see also, the requirements development and management activity also exhibited a very strong positive correlation for the larger, more complex projects, compelling evidence of further benefit.

Requirements: Greatest Cause of Project Problems

The role of requirements quality or the impact of lack thereof is also evidenced by numerous other studies, of which one is summarised here. And we’ve seen just the tip of the iceberg as far as evidence.

The Template

Which brings us on to look thoroughly at the Parsing template, a case where one size does, in fact, fit all and with no compromise.

Let me walk you through the template, which is based on the concept that within the requirement, there are information pieces that must be present and information pieces that may need to be present, and you see the pieces listed there. Every requirement must have within an actor, and the actor is grammatically the subject of the requirement. I’ll assume natural language expression of requirements, so in that case, I’ll say sentence – although the template, in fact, can be used for any language, natural or otherwise. So every requirement must have within it an actor, so “the message switch”, for example, shall switch the message, “the message switch” would be the actor. Every requirement must also have within it an action, “shall switch messages”, shall switch is the action.

Most function requirements need an object of action, “messages” is the object of action. “Shall switch” – what shall be switched? – messages.

All that requirement says is that over the whole life of the message switch, it has to switch at least two messages of arbitrary type under arbitrary conditions, and of course, there’s more to it than that.

So let’s add the remaining elements progressively. This needs to be done fairly quickly, of course, but I think you’ll get the idea. “The message switch”. Conditions for action: “when in message switching mode, upon receipt of a message”. Action: “shall switch”. Object of action, “that message”, huge improvement, but vague. Let’s reduce the vagueness by using constraint of action 2. So, “shall switch that message within ten milliseconds of receipt”, but we still have problems. Let us reduce the problems by adding a refinement of object: “for messages in ACP128 format having a valid routing indicator”. Let us then add constraints of Action 3: “from the message input port, to a message output port, corresponding to the routing indicator in the message”. We have three constraints of action, three phrases and then let’s add an exception to action, “unless the message is of FLASH priority”.

The “Other” field is not used in writing requirements, it is the proverbial garbage can. It is used in another application of this template, which we’ll go into a little later.

Example Requirement

You see the message now written in its full form. The message switch, when in message switching mode, upon receipt of a message, shall switch that message in accordance with IEEE 802.11g, within ten milliseconds of receipt, for messages in ACP128 format having a valid routing indicator, from the message input port, to a message output port corresponding to the routing indicator in the message, unless the message is of FLASH priority. That is a singular requirement, single actor, single action and single objective action.

Tool 2: PPI Requirements Capture and Validation Methodology

Our next tool is an overall very efficient, very effective methodology for capturing and validating requirements, or to put it more broadly, to achieve an adequate problem definition.

As you can see from the concept diagram, it is an analysis-based approach to that transformation, from information from stakeholders into the problem definition, adequate to drive the completion of development in an upfront approach or in an evolutionary fashion depending on circumstances driving development strategy. The low cost of this activity, typically 0.1 percent, up to 2 percent of total development cost, compares very favourably with the historical costs of requirements problems that this technique largely avoids. Those historical costs commonly reported being 10, 20, 50, 80 percent cost overruns and total project failures.

The Methodology

The methodology is illustrated here, starting with identification of stakeholders, and, as you would expect, reading, evaluating, assessing what we’re dealing with, identifying and resolving issues that affect our planning, and potentially measuring requirements quality.

I think it was Drucker who said, “You can’t control what you can’t measure”, and I believe that to be pretty much the case. Measurement of requirements quality is telling us in an objective, repeatable and auditable way, whether or not we have a requirements problem, and feeding into an estimation of how big a problem we have. I’ll come back to that; it’s feeding into our planning. We plan the analysis like we plan any engineering work or other work. We see on the diagram a set of individual activities that integrate to provide this comprehensive, effective methodology.

In planning, we may tailor the methodology down the closer the originating requirements are to already being good enough. In the extreme, we might select just one of these activities to get us from nearly good enough to good enough. Or if where we start is far removed from good enough, we may do all of these things.

Measuring Requirements Quality

We see here the concept of measuring requirements quality. The recommended metric, which I developed and have been using successfully for decades, has a number range of zero to one. Zero is total garbage, and one is perfection. We can’t get perfection, nor is perfection an objective.

Any set of requirements has a quality value on the objective, repeatable, technical scale. We see in the diagram a measured figure of 0.5. In a given set of circumstances, the metric (that figure) aligns to a level of risk due to defects in requirements. The risk calibration that you see here is fairly typical for non-trivial development of things of non-trivial importance.

A level of requirements quality corresponding to a medium level of risk due to defects to the requirements is illustrated, which says we need a higher standard. History tells us at which point that standard should be, to get the risk due to requirements defects within the low risk band It is the work of system requirements analysis that takes us from what we have to what we need. We estimate how much work, and that, of course, is a part of our planning of the analysis. The drivers to that estimation are listed on the diagram. The estimation itself of the quality is a relatively short activity, typically an hour and a half to three and a half hours to make one of these quality measurements, excluding report writing time.

Context Analysis

The first analysis that we will almost always plan to do in a requirements analysis is a context analysis, on a life cycle basis, meaning we are placing the system in the context of things with which it will interact. These include man-made things, naturally occurring things, purposeful interactions, interactions that are not purposeful but may still be relevant to requirements on a whole of life basis, setting a whole of life foundation for everything that follows.

Context analysis captures and validates mainly external interface requirements and environmental requirements, with dribs and drabs of other requirements types captured, validated and specified.

Design Requirements Analysis

We move on to our design requirements analysis, which is done on existing specified requirements looking for existing design requirements. We’re working with design requirements analysis in a similar time frame to the context analysis.

A design requirement is a requirement that directs that the system is built internally in a particular way. This analysis often results in the identification of invalid design requirements, about 90 percent of the design requirements being invalid is a common figure. Many of the invalid design requirements are replaced by requirements of other types, that accurately state what is needed.

Often design requirements are there as a perception of the solution, leaving unstated what is actually needed, so the real requirements replace them. Some design requirements disappear without a trace, and some turn out to be valid.

States and Modes Analysis

We then move on to States and Modes Analysis, which is the first point at which we start looking at the behaviour required of our system, using the concepts and language of states and modes. This is a very high return on investment analysis, a fairly short analysis, but if there are major requirements issues, it is likely to identify them. States and modes analysis uses a form of state transition diagram, not the old Ward and Mellor or Hatley-Pirbhai diagrams, which are limited in expressiveness, but using a suitably expressive notation which you see here, giving us the state mode requirements, requirements to have a state, requirements to have a mode and requirements on relationships between those states and other states, modes and other modes, mode in one state, mode in another state and defaults.

Parsing Analysis

The Parsing template reappears, but now in an analysis role used on existing specified requirements to evaluate their adequacy individually. We parse an existing specified requirement into a blank template, we look at the pieces, and we look at the adequacy of each piece against quality criteria. If something’s missing, but should be there, we fix that. If something is okay, we simply validate it. If something is there but not okay, we fix it. It’s a very powerful technique.

We see here the originating requirement, which is incomplete and unverifiable, and we see here a credible, sufficiently complete and certainly verifiable after-version.

Functional analysis

Another enormously powerful tool, a very widespread tool, but the widespread use of function analysis is generally in fairly basic forms, which are limited in their effectiveness. User stories and Jacobson use cases are what I’m talking about. They can be adequate in certain circumstances, but often are not. So we need also at our disposal a more robust approach, elements of which are illustrated here. This approach uses a suitable modelling notation that meets certain criteria. This approach allows us to determine what the system is required to do, using what inputs, from what sources, over what interfaces, delivering what outputs, to what destinations, over what interfaces and with what measures and values of performance, for functional and performance requirements on a life cycle basis. The approach provides a great deal of not just capture, but also validation. Neither user stories nor traditional Jacobson use cases provide this. So it’s a very powerful tool.

Entity Relationship Attribute (ERA) Analysis

ERA analysis generally reduces to data modelling and builds a data model where the entities are mainly or entirely inputs and outputs, the attributes are the attributes of an input or output (specification essentially), and the relationships are relationships that are relevant to requirements – relationships to be preserved, relationships to be verified as being correct, relationships to be changed or relationships that simply contribute to the understanding of the problem. If we output from the model a list of entities and their attributes, we have a data dictionary. If we extend that to non-data entities, we get an item dictionary, and that is a very useful tool in most requirement specifications for reducing the amount of work in the development of the requirement spec. and increasing the ease of use of the requirement specification. When I use the term requirement specification, I mean a set of requirements in some form. That can be a database, or it could be in a document form, it could be digital, it could be screen based, it can be hard copy, it can be whatever form we like as long as we can view the information, and the information relationships, that we need.

Stakeholder Value Analysis

Value Modelling. We have value modelling in the form of value of requirements (we’ll come back to that subject briefly later), we also have value modelling in the form of building a model of value added beyond just meeting requirements –  model that contains measures of effectiveness, which are any characteristics of a system or a product or service that add value beyond the threshold of acceptability defined by requirements. Associated with each measure of effectiveness is the threshold of acceptability which normally corresponds to a requirement, and a target or goal or objective, which is under the label ‘Best’ on the value model you see in the example. The Priority and Points columns are not necessarily used in communicating the model and using the model but may be used in building the model, or they may serve other purposes. The Weight column contains linear relative value relationships between the improvements over the desired ranges, adding to a 100 percent, giving contributions to an overall trade space. You see value functions in the table. The term ‘UF’ stands for Utility Function, which is another term used by academics for value function. The value function shows how the value changes across the range from a threshold of acceptability to the target, objective or goal figure. That information is brought together in a data file, accessible by modelling software. The model is very valuable for the insights that are achieved and the rapport that is developed with stakeholders in building the model. It’s even more valuable in carrying out trade-off studies in an objective and auditable way to gain insights into the merits of alternatives and to inform good decision-making.

Verification Requirements Development

Associated with system requirements are verification requirements. Their development is, or should be, an activity within system requirements analysis. It is not a large activity but a very valuable activity, typically costing about seven percent of the total requirements analysis cost.

The cost is seven percent of an already small figure that adds huge value in avoiding under verification and over verification, and also avoiding disputes over the adequacy of verification. A verification requirement states the characteristics of evidence required that a system requirement has been met. It drives verification design. With no verification requirements, it is anybody’s guess what verification design should be. A verification requirement can be as limited as just a directed verification method such as test, demonstration, analysis, simulation, inspection, analogy, certification. It is more likely to be more than that. It could be, for example, a confidence level, it could direct a number of test cases. What you see in the example is also quite representative. So verification requirements development is another activity within our system requirements analysis activity.

Clean-Up of Specified Requirements

Clean-up is a sort of small V verification activity. It may or may not be done with independence. The activity is normally performed on the output of the analysed set of requirements, searching against a set of words, parts of words and phrases that historically have been problematic in requirements. The words are not terms not to be used, but they are terms to use carefully, and also to double-check the usage of. That’s what this clean-up activity is doing. We do a search on the list, eyeball the hits and make sure that, where the term is used, its use is okay, not problematic. If it is problematic, we fix it.

That was our requirements analysis overall methodology.

Tool 3: PPI PBS/WBS Development

The last of our tools is Project Breakdown Structure development, or what is sometimes referred to as Work Breakdown Structure development, even though Work Breakdown Structure (WBS) should not be a breakdown of work. If it is, we lose most of the value of it.

How to develop a great PBS/WBS and what you get out of it

We have an illustration here of a well-developed Project Breakdown Structure. We’ll walk through how to construct a great PBS and what we get out of it.

The top box is the overall project, defined by cost, schedule, deliverables defined by requirements, and other information, the usual project definition.

Level two is constructed by firstly asking the question, ‘What products is the project to deliver?’. One of the answers here has been ten aircraft, another answer has been a flight simulator, as well as deliverable data and some integrated logistics support or integrated product support deliverables.

Next question, ‘What services is the project to deliver?’. The answer could be none, or in this case, the answer is “aircrew training”.

Third question, ‘What services are to be performed internal to the project, that are not required uniquely to create just one of the products we have already defined, or perform just one of the services we have already defined?’. There is always at least one answer, and it’s always the same answer, and it is project management. In this example, a second answer is insurance.

Then the fourth question, ‘What products, if any, are required that involve resources that are required to create more than just one of these products or perform more than one of these services that we have defined already?’. That is, that transcend two or more of the elements we have captured so far. The answer could be nothing, or in this case, the answer is “a new factory”.

This is a project-oriented project, so we’ll start with a product, ten aircraft, in developing level 3. What products are to be created to result in the creation of the ten aircraft? Answer, a non-deliverable, first article aircraft and ten production aircraft. What services in addition to these products need to be performed to result in the ten aircraft, as defined? The answer is delivery. What products, in addition to the deliverable products that we haven’t picked up already need to be brought to bear? Same story, same criteria. The answer is none, there’s nothing extra.

First article aircraft: what products? We don’t know, we haven’t designed it yet. Stop, this is an evolving structure. Now, if the situation was as I described, eventually, that question can be answered, which may be done progressively, or it may be as a single activity. Here the answers are shown as the airframe, propulsion system, flight control system, environmental control system, weapon system for military aircraft, electronic warfare self-protection system and fit out. What services? Aircraft system requirements analysis, aircraft system design at this physical level, aircraft integration and assembly at this physical level, flight test, qualification test, and the incremental efforts to achieve aircraft certification, are the answers. Then a third question: What products that are not to be a part of the aircraft but still involve resources are to be brought to bear, the same qualification regarding transcending two or more elements. One answer is a wind tunnel and a second answer is a new avionics laboratory. And so we go on. You see their project management and some entries below it, what services are to be performed to result in the performance of the project management service? The answers are project planning, project administration and project control. What products are to be brought to bear that involve resources that are not just required to perform just one of these services, that is, transcend two or more of the elements we have so far? One answer is a project management information system and the second answer is a new Mercedes for the project manager.

Now guys, what we get from this structure is a framework for costing. Every dollar in the project in the level two elements, every dollar in the ten aircraft is in the level three elements, and every dollar in the first article aircraft is in the level four elements.

We get a framework for scheduling. Each of these elements has a start, a duration and a finish for its creation or performance. There are dependencies that can be modelled to give those frameworks for scheduling.

We have a framework for definition, through requirements on the products and requirements on the services.

We have a framework for risk analysis. Everything that can go wrong in the project can go wrong by a product not being created correctly, a service not being performed correctly for internal reasons or through failure to get some necessary input from the outside world. We have a framework for risk analysis.

We have a framework for measurement and assignment of responsibility.

As soon as we have measurement and assignment of responsibility, we have a framework for reporting and a framework for organisational design.

Here, teams shown. Teams are responsible for PBS elements identified in the diagram. We see further aspects of team design down the bottom of the example. Subordinate team leaders are members of the superior team, and we also see horizontal relationships of cross team membership to give representation on one team of another team, and vice versa. A framework for organizational design.

We would lose most of this if we used a functional structure, a breakdown of work into subtasks. PBS is an enormously powerful tool!

Some Other High ROI Techniquesmodelled

Many other high return on investment techniques exist that I will identify, but I won’t go into. The use of Compromise Impact Values as an objective way of valuing requirements is a very sound and useful tool. The use of decision and event trees for supporting decisions involving what I call ‘event-based uncertainty’ (something will or will not be available by a certain date or we will or will not get some approval or license) invaluable and provides insight insight into the merits of decision alternatives.

Performance Thread Analysis is a terrific tool for iterative performance optimisation, reduction of risk arising for the possibility of failing to meet performance and avoiding excessive design margins. Requirements Quality Measurement is something we’ve talked about as providing an objective criterion for ‘good enough’ requirements, which generally means low risk due to defects requirements, and can be used for many things (I illustrated using requirements quality measurement for estimating and planning, but there are many other uses of that quality measurement). System effectiveness modelling implements the very simple principle of identifying feasible solution alternatives (can meet requirements), then pick the best. The effectiveness modelling gives us the reference for picking the best and for optimisation. We’ve already mentioned verification requirements providing an objective basis for verification design.

Free PPI Help to Clients

PPI provides many resources to PPI clients, and many of them are available to the engineering community in general, a way of giving back to the community. We see here a list of resources, including data item descriptions. It’s a very comprehensive set and I believe of very high quality, higher quality than available in any of the public domain resources that I know of.

PPI Data Item Descriptions:

• Project Plan (PP)
• Task Specification (TS)
• Statement of Work (SOW)
• Systems Engineering Plan (SEP)
• Operational Concept Description (OCD)
• System Requirements Specification (SyRS)
• Software Requirements Specification (SRS)
• Verification Requirements Specification (VRS)
• Interface Requirements Specifications (IRS)
• Interface Design Description (IDD)
• System/Subsystem Design Description (SSDD)
• Concept of Operations (CONOPS) – Operational Solution Description (OSD)
• Operational and Support Concept Description (O&SCD)
• Concept of Employment (CONEMP)
• Capability Systems Requirements Specification (CAPSYRS)
• Requirements Traceability Report – Requirements Analysis (RTR-RA)
• Requirements Traceability Report in System Design (RTR-SD)

More free PPI help to clients

Here is an example document set. We have data item descriptions or templates for key information types, and here we have examples of these key information sets as well as a coordinated set, starting with a capability system through capability engineering to identify technology items that are to form a part of the solution products and services and engineering artifacts associated with that whole development of a solution.

PPI Example Documents:

• Concept of Employment (CONEMP)
• Concept of Use (CONUSE or OCD)
• Capability System Requirements Specification (CapSyRS)
• Capability System Value Model
• Interface Requirements Specification (IRS)
• Operational Solution Description (OSD or CONOPS)
• Concept of Use (CONUSE – OCD) for a technology item
• Systems Requirements Specification (SyRS) for a technology item
• System Effectiveness Model for a technology item
• Statement of Work (SOW)
• Verification Requirements Specification (VRS) for a technology item.

Even more free PPI help to clients

PPI also publishes application guides on the various systems entering standards intended for general application and also one domain-specific standard, which is the European Space Agency ECSS10 Systems Engineering Standard. Watch this space for an application guide on the next version of ISO/IEC/IEEE 15288, which is not that far away, it’s in a committee draft at the time of recording this presentation.

PPI Application Guides to Systems Engineering Standards:

• EIA-632: 2003
• IEEE 1220
• ECSS-E-ST-10C
• ISO/IEC 15288:2008
• ISO/IEC/IEEE 15288:2015
• ISO/IEC/IEEE 15288:201X (when released)

And more again:

Three practice guides:

• Requirements Capture and Validation Guide
• Requirement Specification Development Guide (which has detail and content that I believe you won’t find elsewhere)
• Military Capability Development Guide

All substantial in volume and content and professionally produced.

The Systems Engineering Goldmine

PPI alumni, clients and guests also have access to the PPI’s Systems Engineering Goldmine, which provides access to about four gigabytes of curated systems engineering documents and a searchable definitions database of over 7900 terms used in systems engineering or in related practices.

The SEG and PPI-INCOSE SETDB

System Entry Tools Database is a jointly developed and operated database of systems engineering software tools and cloud services, jointly developed by PPI and INCOSE.

The SETDB provides search capabilities for tools and tool vendors. It also provides mapping to PPI’s systems engineering process elements, which collectively comprise the totality of systems engineering practice. It also provides mapping to the systems entering processes and also topics defined by the INCOSE Systems Engineering Handbook. So it’s a very substantial resource. I think you’ll get some idea of what’s there from just the home page, as well as some example pages containing, for example, synthetic tool data. Access to the Systems Engineering Tools Database is for INCOSE members and PPI account holders.

PPI SyEN – Systems Engineering Newsjournal

PPI also publishes a monthly systems engineering newsjournal, PPI SyEN, which is available to our alumni, clients and also available generally on the PPI website. PPI SyEN is a substantial publication, with 30 to 60 pages monthly full of systems engineering content – articles, knowledge pieces, news pieces and other content.

Contact PPI for SE Training and Consulting

Here are some contact details for our locations around the world, and that’s it folks. Thank you for your interest. If you have any questions, I’d be happy to take questions by email, or you could track me down on LinkedIn. I’m happy to chat with you on LinkedIn.