Page ii Foreword from a systems perspective, starting at mission needs and
conceptual studies through operations and disposal. In an attempt to demonstrate the potential
dangers of relying on purely ''cookbook" logical While it would be impossible to thank all of the people directly involved, it is essential to note the efforts of thinking, the mathematician/philosopher Carl Hempel Dr. Robert Shishko of the Jet Propulsion Laboratory. Bob posed a paradox. If we want to prove the hypothesis was largely responsible for ensuring the completion of this “AII ravens are black," we can look for many ravens effort. His technical expertise and nonstop determination and determine if they all meet our criteria. Hempel were critical factors to ensure the success of this project. suggested changing the hy –pothesis to its logical
contrapositive (a rewording with identical meaning) Mihaly Csikzenthmihali defined an optimal would be easier. The new hypothesis becomes: "All experience as one where there is "a sense of exhilaration, a nonblack things are nonravens." This transformation, deep sense of enjoyment that is long cherished and becomes a landmark in memory of what life should be like." I supported by the laws of logical thinking, makes it am not quite sure if the experience which produced this hand much easier to test, but unfortunately is ridiculous. –book can be described exactly this way, yet the sentiment Hempel's raven paradox points out the importance of seems reasonably close. common sense and proper background exploration,
even to subjects as intricate as systems engineering. —Dr. Edward J. Hoffman
Program Manager, NASA Headquarters In 1989, when the initial work on the NASA Systems Spring 199 Engineering Handbook was started, there were many
who were concerned about the dangers of a document that purported to teach a generic NASA approach to systems engineering. Like Hempel's raven, there were concerns over the potential of producing a "cookbook" which of fered the illusion of logic while ignoring experience and common sense. From the tremendous response to the initial (September 1992) draft of the handbook (in terms of both requests for copies and praise for the product), it seems early concerns were largely unfounded and that there is a strong need for this handbook.
The document you are holding represents what was deemed best in the original draft and updates information necessary in light of recommendations and changes within NASA. This handbook represents some of the best thinking from across NASA. Many experts influenced its outcome, and consideration was given to each idea and criticism. It truly represents a NASA-wide product and one which furnishes a good overview of NASA systems engineering.
The handbook is intended to be an educational guide written from a NASA perspective. Individuals who take systems engineering courses are the primary audience for this work. Working professionals who require a guidebook to NASA systems engineering represent a secondary audience.
It was discovered during the review of the draft document that interest in this work goes far beyond NASA. Requests for translating this work have come from international sources, and we have been told that the draft hand book is being used in university courses on the subject. All of this may help explain why copies of the original draft handbook have been in short supply.
The main purposes of the NASA Systems Engineering Handbook are to provide: 1) useful information to system engineers and project managers, 2) a generic description of NASA systems engineering which can be supported by center-specific documents, 3) a common language and perspective of the systems engineering process, and 4) a reference work which is consistent with NMI 7120.4/NHB 7120.5. The handbook approaches systems engineering
NASA Systems Engineering Handbook
Foreword to the September 1992 Draft As such, this present document is to be
considered a next-to-final draft. Your comments, When NASA began to sponsor agency – corrections and suggestions are welcomed, valued wide classes in systems engineering, it was to a and appreciated. Please send your remarks directly to doubting audience. Top management was quick to Robert Shishko, NASA Systems express concern. As a former Deputy Administrator Engineering Working Group. NASA/Jet Propulsion stated "How can you teach an agency-wide systems Laboratory, California Institute of Technology, engineering class when we cannot even agree on 4800 Oak Grove Drive, Pasadena, CA how to define it?" Good question, and one I must 91109-8099. admit caused us considerable concern at that time. —Francis T. Hoban The same doubt continued up until the publication of Program Manager, NASA Headquarters this handbook.
The initial systems engineering education conference was held in January 1989 at the Johnson Space Center. A number of representatives from other Centers at tended this meeting and it was decided then that we needed to form a working group to support the development of appropriate and tailored systems engineering courses. At this meeting the representatives from Marshal1 Space Flight Center (MSFC) expressed a strong desire to docu ment their own historic systems engineering process before any more of the key players left the Center. Other Centers also expressed a desire, if not as urgent as MSFC, to document their processes.
It was thought that the best way to reflect the totality of the NASA systems engineering process and to aid in developing the needed training was to prepare a top level (Level 0) document that would contain a broad definition of systems engineering, a broad process outline, and typi cal tools and procedures. In general, we wanted a top level overview of NASA systems engineering. To this document would be appended each Center's unique systems engineering manual. The group was well aware of the diversity each Center may have, but agreed that this approach would be quite acceptable.
The next step and the most difficult in this arduous process was to find someone to head this yet-to-be-formed working group. Fortunately for NASA, Donna [Pivirotto] Shirley of the Jet Propulsion Laboratory stepped up to the challenge. Today, through her efforts, those of the working group, and the skilled and dedicated authors, we have a unique and possibly a historic document.
During the development of the manual we decided to put in much more than may be appropriate for a Level 0 document with the idea that we could always refine the document later. It was more important to capture the knowledge when we could in order to better position our selves for later dissemination. If there is any criticism, it may be the level of detail contained in the manual, but this detail is necessary for young engineers. The present document does appear to serve as a good instructional guide, although it does go well beyond its original intent.
NASA Systems Engineering Handbook
Page 5 Preface JSC-49040. The SEPIT project life cycle is
intentionally consistent with that in NMI 7120.4/NHB This handbook was written to bring the 7120.5 (Management of Major System Programs and fundamental concepts and techniques of systems Projects), but provides more detail on its systems engineering to NASA personnel in a way that engineering aspects. recognizes the nature of NASA systems and the
NASA environment. The authors readily acknowledge This handbook consists of five core that this goal will not be easily realized. One reason is chapters: (1) systems engineering's intellectual that not everyone agrees on what systems process, (2) the NASA project life cycle, (3) engineering is, nor on how to do it. There are management issues in systems engineering, (4) legitimate differences of opinion on basic definitions, systems analysis and modeling issues, and (5) content, and techniques. Systems engineering itself is engineering specialty integration. These core a broad subject, with many different aspects. This chapters are supplemented by appendices, which can initial handbook does not (and cannot) cover all of be expanded to accommodate any number of them. templates and examples to illustrate topics in the core
chapters. The handbook makes extensive use of The content and style of this handbook show sidebars to define, refine, illustrate, and extend a teaching orientation. This handbook was meant to concepts in the core chapters without diverting the accompany formal NASA training courses on systems reader from the main argument. There are no engineering, not to be a stand-alone, comprehensive footnotes; sidebars are used instead. The structure of view of NASA systems engineering. Systems the handbook also allows for additional sections and engineering, in the authors' opinions, cannot be chapters to be added at a later date. learned simply by starting at a well-defined beginning Finally, the handbook should be considered only a and proceeding seamlessly from one topic to another. starting point. Both NASA as a systems engineering Rather, it is a field that draws from many engineering organization, and systems engineering as a discipline, disciplines and other intellectual domains. The are undergoing rapid evolution. Over the next five boundaries are not always clear, and there are many years, many changes will no doubt occur, and some interesting intellectual offshoots. Consequently, this are already in progress. NASA, for instance, is handbook was designed to be a top-level overview of moving toward implementation of the standards in the systems engineering as a discipline; brevity of International Standards Organization (ISO) 9000 exposition and the provision of pointers to other books family, which will affect many aspects of systems and documents for details were considered important engineering. In systems engineering as a discipline, guidelines. efforts are underway to merge existing systems
engineering standards into a common American The material for this handbook was drawn National Standard on the Engineering of Systems, from many different sources, including field center and then ultimately into an international standard. systems engineering handbooks, NASA management These factors should be kept in mind when using this instructions (NMls) and NASA handbooks (NHBs), handbook field center briefings on systems engineering
processes, non-NASA systems engineering textbooks
and guides, and three independent systems engineering courses taught to NASA audiences. The handbook uses this material to provide only top-level information and suggestions for good systems engineering practices; it is not intended in any way to be a directive.
By design, the handbook covers some topics that are also taught in Project Management/Program Control (PM/PC) courses, reflecting the unavoidable connectedness of these three domains. The material on the NASA project life cycle is drawn from the work of the NASA- wide Systems Engineering Working Group (SEWG), which met periodically in 1991 and 1992, and its successor, the Systems Engineering Process Improvement Task (SEPIT) team, which met in 1993 And 1994. This handbook's project life cycle is identical to that promulgated in the SEPIT report, NASA Systems Engineering Process for Programs and Projects,
NASA Systems Engineering Handbook
Page 6 Acknowledgements Mr. Don Woodruff, NASA/Marshall Space Flight This work was conducted under the overall direction Center of Mr. Frank Hoban, and then Dr. Ed Hoffman, NASA
Headquarters/Code FT, under the NASA Pro- Other Sources: gram/Project Management Initiative (PPMI). Dr. Mr. Dave Austin, NASA Headquarters/Code DSS Shahid Habib, NASA Headquarters/Code QW, Mr. Phillip R. Barela, NASA/Jet Propulsion Laboratory provided additional financial support to the NASA- Mr. J.W. Bott, NASA/Jet Propulsion Laboratory wide Systems Engineering Working Group and Dr. Steven L. Cornford, NASA/Jet Propulsion Systems Engineering Process Improvement Task Laboratory team. Many individuals, both in and out of NASA, Ms. Sandra Dawson, NASA/Jet Propulsion Laboratory contributed material, reviewed various drafts, or Dr. James W. Doane, NASA/Jet Propulsion otherwise provided valuable input to this handbook. Laboratory Unfortunately, the lists below cannot include everyone Mr. William Edminston, NASA/Jet Propulsion who shared their ideas and experience. Laboratory
Mr. Charles C. Gonzales, NASA/Jet Propulsion Principal Contributors: Laboratory Dr. Robert Shishko, NASA/Jet Propulsion Laboratory Dr. Jairus Hihn, NASA/Jet Propulsion Laboratory Mr. Robert G. Chamberlain, P.E., NASA/Jet Dr. Ed Jorgenson, NASA/Jet Propulsion Laboratory Propulsion Laboratory Mr. Richard V. Morris, NASA/Jet Propulsion
Laboratory Other Contributors: Mr. Tom Weber, Rockwell International/Space Mr. Robert Aster, NASA/Jet Propulsion Laboratory Systems Ms. Beth Bain, Lockheed Missiles and Space
Company Reviewers: Mr. Guy Beutelschies, NASA/Jet Propulsion Mr. Robert C. Baumann, NASA/Goddard Space Flight Laboratory Center Dr. Vincent Bilardo, NASA/Ames Research Center Mr. Chris Carl, NASA/Jet Propulsion Laboratory Ms. Renee I. Cox, NASA/Marshall Space Flight Dr. David S.C. Chu, Assistant Secretary of Center Defense/Pro-gram Dr. Kevin Forsberg, Center for Systems Management Analysis and Evaluation Dr. Walter E. Hammond, Sverdrup Techology, Inc. Mr. M.J. Cork, NASA/Jet Propulsion Laboratory Mr. Patrick McDuffee, NASA/Marshall Space Flight Mr. James R. French, JRF Engineering Services Center Mr. John L. Gasery, Jr., NASA/Stennis Space Center Mr. Harold Mooz, Center for Systems Management Mr. Frederick D. Gregory, NASA Headquarters/Code Ms. Mary Beth Murrill, NASA/Jet Propulsion Q Laboratory Mr. Don Hedgepeth, NASA/Langley Research Center Dr. Les Pieniazek Lockheed Engineering and Mr. Jim Hines, Rockwell International/Space Systems ScienceS Mr. Keith L. Hudkins, NASA Headquarters/Code MZ Co. Mr. Lou Polaski, Center for Systems Management Dr. Jerry Lake, Defense Systems Management Mr. Neil Rainwater, NASA/Marshall Space Flight College Center Mr. T.J. Lee, NASA/Marshall Space Flight Center Mr. Tom Rowell, NASA/Marshall Space Flight Center Mr. Jim Lloyd, NASA Headquarters/Code QS Ms. Donna Shirley, NASA/Jet Propulsion Laboratory Mr. Woody Lovelace, NASA/Langley Research Mr. Mark Sluka, Lockheed Engineering and Sciences Center Co. Mr. Ron Wade, Center for Systems Management Dr. Brian Mar, Department of Civil Engineering, SEPIT team members (not already mentioned): University Mr. Randy Fleming, Lockheed Missiles and Space of Washington Co. Dr. Ralph F. Miles, Jr., Center for Safety and System Dr. Frank Fogle, NASA/Marshall Space Flight Center Management, University of Southern California Mr. Tony Fragomeni, NASA/Goddard Space Flight Dr. Richard A. Montgomery, Montgomery and Center Dr. Shahid Habib, NASA Headquarters/Code Associates QW Mr. Bernard G. Morais, Synergistic Applications, Inc. Mr. Henning Krome, NASA/Marshall Space Flight Mr. Ron Moyer, NASA Headquarters/Code QW Center Mr. Rudy Naranjo, NASA Headquarters/Code MZ Mr. Ray Lugo, NASA/Kennedy Space Center Mr. Raymond L. Nieder, NASA/Johnson Space Mr. William C. Morgan, NASA/Johnson Space Center Center Dr. Mike Ryschkewitsch, NASA/Goddard Space Flight Mr. James E. Pearson, United Technologies Center Research Mr. Gerry Sadler, NASA/Lewis Research Center Center Mr. Dick Smart, Sverdrup Technology, Inc. Mr. Leo Perez, NASA Headquarters/Code QP Dr. James Wade, NASA/Johnson Space Center Mr. David Pine, NASA Headquarters/Code B Mr. Milam Walters, NASA/Langley Research Center
NASA Systems Engineering Handbook
Page 7 Mr. Glen D. Ritter, NASA/Marshall Space Flight Center Dr. Arnold Ruskin, NASA/Jet Propulsion Laboratory and University of California at Los Angeles Mr. John G. Shirlaw, Massachusetts Institute of Technology Mr. Richard E. Snyder, NASAlLangley Research Center Mr. Don Sova, NASA Headquarters/Code AE Mr. Marvin A. Stibich, USAF/Aeronautical Systems Center Mr. Lanny Taliaferro, NASAtMarshall Space Flight Center Editorial and graphics support: Mr. Stephen Brewster, NASA/Jet Propulsion Laboratory Mr. Randy Cassingham, NASA/Jet Propulsion Laboratory Mr. John Matlock, NASA/Jet Propulsion Laboratory —Dr. Robert Shishko Task Manager, NASA/Jet Propulsion Laboratory
NASA Systems Engineering Handbook
Page 1 1 Introduction The subject matter of systems engineering is 1.1 Purpose very broad. The coverage in this handbook is limited
to general concepts and generic descriptions of This handbook is intended to provide processes, tools, and techniques. It provides information on systems engineering that will be useful information on good systems engineering practices, to NASA sys-tem engineers, especially new ones. Its and pitfalls to avoid. primary objective is to provide a generic description of There are many textbooks that can be consulted for systems engineering as it should be applied in-depth tutorials. throughout NASA. Field centers' handbooks are
encouraged to provide center-specific details of This handbook describes systems implementation. engineering as it should be applied to the
development of major NASA systems. Systems For NASA system engineers to choose to engineering deals both with the system being keep a copy of this handbook at their elbows, it must developed (the product system) and the system that provide answers that cannot be easily found does the developing (the producing system). elsewhere. Conse-quently, it provides NASA-relevant Consequently, the handbook's scope properly perspectives and NASA-particular data. NASA includes systems engineering functions regardless of management instructions (NMIs) are referenced when whether they are performed by an in-house systems applicable. engineering organization, a program/project office, or
a system contractor. This handbook's secondary objective is to
serve as a useful companion to all of the various
While many of the producing system's courses in systems engineering that are being offered design features may be implied by the nature of the under NASA's auspices. tools and techniques of systems engineering, it does
not follow that institutional procedures for their 1.2 Scope and Depth application must be uniform from one NASA field center to another.
Selected Systems Engineering Reading
See the Bibliography for full refer
ence data and further reading suggestions.
undamentals of Systems Engineering
Systems Engineering and Analysis (2nd ed.), B.S. Blanchard and W. J. Fabrycky
Systems Engineering, Andrew P. Sage
n Introduction to Systems Engineering, J.E. Armstrong and Andrew P. Sage
Management Issues in Systems Engineering
stems Engineering, EIA/IS-632
IEEE Trial-Use Standard for application and Management of t
he Systems Engineering Process, IEEE Std 1220- 1994
Systems Engineering Management Guide, Defense Sy
stems Management College
System Engineering Management, B.S. Blanchard
Systems Engineering Methods, Harold Chestnut
Systems Concepts, Ralph Miles, Jr. (editor)
Successful Systems Engineering for Engineers and Managers,
Norman B. Reilly
stems Analysis and Modeling
Systems Engineering Tools, Harold Chestnut
Systems Analysis for Engineers and Managers, R.
de Neufville and J.H. Stafford
Cost Considerations in Systems Analysis, Gene H. Fisher
Space Systems Design and Operations
Space Vehicle Design, Michael D. Griffin and James R. French
Space Mission Analysis and Design (2nd ed.), Wiley J. Larson and
James R. Wertz (editors) D i f G h S ft B ij N A l
NASA Systems Engineering Handbook
Page 2 2 Fundamentals of Systems Engineering The NASA management instruction for the
acquisition of “major" systems (NMI 7120.4) defines a 2.1 Systems, Supersystems, and program as “a related series of undertakings that Subsystems continue over a period of time (normally years), which are designed to pursue, or are in support of, a
focused scientific or technical goal, and which are A system is a set of interrelated components which characterized by: design, development, and interact with one another in an organized fashion operations of systems." Programs are managed by toward a common purpose. The components of a NASA Headquarters, and may encompass several system may be quite diverse, consisting of persons, projects. organizations, procedures, software, equipment, end In the NASA context, a project encompasses 'or facilities. The purpose of a system may be as the design, development, and operation of one or humble as distributing electrical power within a more sys-tems, and is generally managed by a NASA spacecraft or as grand as exploring the surface of field center. Mars. Headquarters' management concerns
include not only the engineering of the systems, but Every system exists in the context of a all of the other activities required to achieve the broader supersystem, i.e., a collection of related desired end. These other activities include explaining systems. It is in that context that the system must be the value of programs and projects to Congress and enlisting international cooperation. The term mission A Hierarchical System Terminology is often used for a program project's purpose; its
connotations of fervor make it particu-larly suitable for The following hierarchical sequence of terms for such political activities, where the emo-tional content suc-cessively finer resolution was adopted by the of the term is a desirable factor. In everyday NASA –wide Systems Engineering Working conversation, the terms "project," "mission," and Group (SEWG) and its successor, the Systems "system" are often used interchangeably; while Engineering Process Im-provement Task (SEPIT) imprecise, this rarely causes difficulty. team:
System The Technical Sophistication Required to do Segment Systems Engineering Depends on the Project Element
Subsystem • The system's goals may be simple and easy to Assembly identify and measure—or they may be technically Subassembly complicated, requiring a great deal of insight Part about the environment or technology within or with
which the system must operate. Particular projects may need a different sequence
of layers— an instrument may not need as many • The system may have a single goal—or multiple layers, while a broad initiative may need to goals. There are techniques available for distinguish more layers. Projects should establish determining the relative values of multiple goals judged. Thus, managers in the supersystem set — but sometimes goals are truly incommensurate system policies, establish system objectives, and unquantifiable. determine system constraints, and define what costs
are relevant. They often have oversight authority over • The system may have users representing system design and operations decisions. factions with conflicting objectives. When there
are conflicting objectives, negotiated Most NASA systems are sufficiently complex compromises will be required. that their components are subsystems, which must
function in a coordinated way for the system to • Alternative system design concepts may be accomplish its goals. From the point of view of abundant—or they may require creative genius to systems engineering, each subsystem is a system in develop. its own right—that is, policies, requirements,
objectives, and which costs are relevant are • A "back-of-the-envelope" computation may be established at the next level up in the hierarchy. satisfactory for prediction of how well the Spacecraft systems often have such subsystems as alternative design concepts would do in propulsion, attitude control telecommunications, and achievement of the goals—or credibility may power. In a large project, the subsystems are likely to depend upon construction and testing of hardware be called "systems". The word system is also used or software models. within NASA generically, as defined in the first paragraph above. In this handbook, system" is generally used in its generic form.
NASA Systems Engineering Handbook
Page 3 2.2 Definition of Systems Engineering system must provide the most effectiveness for the
resources expended or, equivalently, it must be the Systems engineering is a robust approach to the least expensive for the effectiveness it provides. This design, creation, and operation of systems. In simple condition is a weak one because there are usually terms, the approach consists of identification and many designs that meet the condition. Think of each quantification of system goals, creation of alternative possible design as a point in the tradeoff space system design concepts, performance of design Cost trades, selection and implementation of the best
The cost of a system is the foregone value of the Systems Engineering per ElA/IS-632
re-sources needed to design, build, and operate it. Be-cause resources come in many forms— work Systems engineering is "an interdisciplinary approach per-formed by NASA personnel and contractors, encompassing the entire technical effort to evolve and verify an integrated and life-cycle balanced set of sys- materials, energy, and the use of facilities and tem people, product, and process solutions that satisfy equipment such as wind tunnels, factories, offices, customer needs. Systems engineering encompasses (a) and computers—it is of en convenient to express the technical efforts related to the development, these values in common terms by using monetary manufacturing, verification, deployment, operations, units (such as dollars). support) disposal of, and user training for, system prod-
ucts and processes; (b) the definition and management Effectiveness of the system configuration; (c) the translation of the
system definition into work breakdown structures; and The effectiveness of a system is a quantitative (d) development of information for management deci- sion making." measure of the degree to which the system's purpose is achieved. Effectiveness measures are usually very dependent upon system design, verification that the design is properly built performance. For example, launch vehicle and integrated, and post-implementation assessment effectiveness depends on the probability of of how well the system meets (or met) the goals. The successfully injecting a payload onto a usable approach is usually applied repeatedly and trajectory. The associated system performance recursively, with several increases in the resolution of attributes include the mass that can be put into a the system baselines (which contain requirements, specified nominal orbit, the trade between injected design details, verification procedures and standards, mass and launch velocity, and launch availability. cost and performance estimates, and so on).
Cost-Effectiveness Systems engineering is performed in concert
with system management. A major part of the system The cost-effectiveness of a system combines both engineer's role is to provide information that the the cost and the effectiveness of the system in the system manager can use to make the right decisions. context of its objectives. While it may be This includes identification of alternative design necessary to measure either or both of those in concepts and characterization of those concepts in ways that will help the system managers first discover between effectiveness and cost. A graph plotting the their preferences, then be able to apply them astutely. maximum achievable effectiveness of designs An important aspect of this role is the creation of available with current technology as a function of cost system models that facilitate assessment of the would in general yield a curved line such as the one alternatives in various dimensions such as cost, shown in Figure 1. (In the figure, all the dimensions of performance, and risk. effectiveness are represented by the ordinate and all
the dimensions of cost by the abscissa.) In other Application of this approach includes words, the curved line represents the envelope of the performance of some delegated management duties, currently available technology in terms of cost - such as maintaining control of the developing effectiveness. configuration and overseeing the integration of Points above the line cannot be achieved subsystems. with currently available technology e that is, they do
not represent feasible designs. (Some of those points 2.3 Objective of Systems Engineering may be feasible in the future when further
technological advances have been made.) Points inside the envelope are feasible, but are dominated The objective of systems engineering is to by designs whose combined cost and effectiveness see to it that the system is designed, built, and lie on the envelope. Designs represented by points on operated so that it accomplishes its purpose in the the envelope are called cost-effective (or efficient or most cost-effective way possible, considering non-dominated) solutions. performance, cost, schedule, and risk. Design trade studies, an important part of
the systems engineering process, often attempt to A cost-effective system must provide a particular kind find designs that provide a better combination of the of balance between effectiveness and cost: the
NASA Systems Engineering Handbook
Page 4 various dimensions of cost and effectiveness. When likely point, as is shown for design concept A in the the starting point for a design trade study is inside the figure. Distributions resulting from designs which have envelope, there are alternatives that reduce costs little uncertainty are dense and highly compact, as is without decreasing any aspect of effectiveness. or shown for concept B. Distributions associated with increase some aspect of effectiveness with risky designs may have significant probabilities of
producing highly undesirable outcomes, as is suggested by the presence of an additional low effectiveness/high cost cloud for concept C. (Of course, the envelope of such clouds cannot be a sharp line such as is shown in the figures, but must itself be rather fuzzy. The line can now be thought of as representing the envelope at some fixed confidence level -- that is, a probability of x of achieving that effectiveness.)
Both effectiveness and cost may require several descriptors. Even the Echo balloons obtained scientific data on the electromagnetic environment and atmospheric drag, in addition to their primary mission as communications satellites. Furthermore, Echo was the first satellite visible to the naked eye, an
unquantified -- but not unrecognized —aspect of its Figure 1 -- The Enveloping Surface of Non-dominated effectiveness. Costs, the expenditure of limited Designs. resources, may be measured in the several
dimensions of funding, personnel, use of facilities, out decreasing others and without increasing costs. and so on. Schedule may appear as an attribute of Then, the system manager's or system engineer's effectiveness or cost, or as a constraint. Sputnik, for decision is easy. Other than in the sizing of example, drew much of its effectiveness from the fact subsystems, such "win-win" design trades are that it was a "first"; a mission to Mars that misses its uncommon, but by no means rare. When the launch window has to wait about two years for alternatives in a design trade study, however, require another opportunity—a clear schedule constraint. trading cost for effectiveness, or even one dimension Risk results from uncertainties in realized of effectiveness for another at the same cost, the effectiveness, costs, timeliness, and budgets. decisions become harder. Sometimes, the systems that provide the
highest ratio of effectiveness to cost are the most desirable. However, this ratio is likely to be meaningless or—worse—misleading. To be useful and meaningful, that ratio must be uniquely determined and independent of the system cost. Further, there must be but a single measure of effectiveness and a single measure of cost. If the numerical values of those metrics are obscured by probability distributions, the ratios become uncertain as well; then any usefulness the simple, single ratio of two numbers might have had disappears. In some contexts, it is appropriate to seek the most effectiveness possible within a fixed budget; in other contexts, it is more appropriate to seek the least cost possible with specified effectiveness. In
these cases, there is the question of what level of Figure 2--Estimates of Outcomes to be Obtained from effectiveness to specify or of what level of costs to fix. Several Design Concepts Including Uncertainty. In practice, these may be mandated in the form of
performance or cost requirements; it then becomes The process of finding the most cost- appropriate to ask whether a slight relaxation of effective design is further complicated by uncertainty, requirements could produce a significantly cheaper which is shown in Figure 2 as a modification of Figure sys-tem or whether a few more resources could 1. Exactly what outcomes will be realized by a produce a significantly more effective system. particular system design cannot be known in advance Usually, the system manager must choose with certainty, so the projected cost and effectiveness among designs that differ in terms of numerous of a design are better described by a probability attributes. A variety of methods have been developed distribution than by a point. This distribution can be that can be used to help managers uncover their thought of as a cloud which is thickest at the most preferences between attributes and to quantify their likely value and thinner farther away from the most
NASA Systems Engineering Handbook
Page 5 analytic methods. These specialty engineering areas The System Engineer's Dilemma typically include reliability, maintainability, logistics,
test, production, transportation, human factors, quality At each cost-effective solution: assurance, and safety engineering. Specialty • To reduce cost at constant risk, performance engineers contribute throughout the systems must be reduced. engineering process; part of the system engineer's job • To reduce risk at constant cost, performance is to see that these functions are coherently must be reduced. integrated into the project at the right times and that • To reduce cost at constant performance, higher they address the relevant issues. One of the risks must be accepted. objectives for Chapter 6 is to develop an • To reduce risk at constant performance, higher understanding of how these specialty engineers costs must be accepted. contribute to the objective of systems engineering.
In both systems analysis and systems
In this context, time in the schedule is often a engineering, the amounts and kinds of resources to critical resource, so that schedule behaves like a be made available for the creation of the system are kind assumed to be among the decisions to be made. f t subjective assessments of relative value. When this Systems engineering concentrates on the creation of can be done, trades between attributes can be hardware and software architectures and on the assessed quantitatively. Often, however, the development and management of the interfaces attributes seem to be truly incommensurate; between subsystems, while relying on systems managers must make their decisions in spite of this analysis to construct the mathematical models and multiplicity. analyze the data to evaluate alternative designs and
to perform the actual design trade studies. Systems 2.4 Disciplines Related to Systems analysis often requires the use of tools from Engineering operations research, economics, or other decision
sciences, and systems analysis curricula generally The definition of systems engineering given include extensive study of such topics as probability, in Section 2.2 could apply to the design task facing a statistics, decision theory, queueing theory, game bridge designer, a radio engineer, or even a theory, linear and non-linear programming, and so on. committee chair. The systems engineering process In practice, many system engineers' academic can be a part of all of these. It cannot be the whole of background is richer in the engineering disciplines the job—the bridge designer must know the than in the decision sciences. As a consequence, the properties of concrete and steel, the radio engineer system engineer is often a consumer of systems must apply Maxwell's equations, and a committee analysis products, rather than a producer of them. chair must understand the personalities of the One of the major objectives for Chapter 5 is to members of the committee. In fact, the optimization of develop an understanding and appreciation of the systems requires collaboration with experts in a state of that art. variety of disciplines, some of which are compared to Operations research and operations systems engineering in the remainder of this section. engineering confine their attention to systems whose The role of systems engineering differs from components are assumed to be more or less that of system management in that engineering is an immutable. That is, it is assumed that the resources analytical, advisory and planning function, while with which the system operates cannot be changed, management is the decision-making function. Very but that the way in which they are used is amenable often, the distinction is irrelevant, as the same to optimization. Operations research techniques often individuals may perform both roles. When no factors provide powerful tools for the optimization of system enter the decision-making process other than those designs. that are covered by the analyses, system Within NASA, terms such as mission management may delegate some of the management analysis and engineering are often used to describe responsibility to the systems engineering function. all study and design efforts that relate to Systems engineering differs from what might determination of what the project's mission should be be called design engineering in that systems and how it should be carried out. Sometimes the engineering deals with the relationships of the thing scope is limited to the study of future projects. being designed to its supersystem (environment) and Sometimes the charters of organizations with such subsystems, rather than with the internal details of names include monitoring the capabilities of systems, how it is to accomplish its objectives. The systems ensuring that important considerations have not been viewpoint is broad, rather than deep: it encompasses overlooked, and overseeing trades between major the system functionally from end to end and systems— thereby encompassing operations temporally from conception to disposal. research, systems analysis, and systems engineering System engineers must also rely on activities. contributions from the specialty engineering Total quality management (TQM) is the disciplines, in addition to the traditional design application of systems engineering to the work disciplines, for functional expertise and specialized environment. That is, part of the total quality
NASA Systems Engineering Handbook
Page 6 management paradigm is the realization that an operating organization is a particular kind of system and should be engineered as one. A variety of specialized tools have been developed for this application area; many of them can be recognized as established systems engineering tools, but with different names. The injunction to focus on the satisfaction of customer needs, for example, is even expressed in similar terms. The use of statistical process control is akin to the use of technical performance and earned value measurements. Another method, qualify function deployment (QFD), is a technique of requirements analysis often used in systems engineering. The systems approach is common to all of these related fields. Essential to the systems approach is the recognition that a system exists, that it is embedded in a supersystem on which it has an impact, that it may contain subsystems, and that the system's objectives must be understood preferably explicitly identified.
2.5 The Doctrine of Successive Refinement
The realization of a system over its life cycle results from a succession of decisions among integration of lower-level elements is a part of the alternative courses of action. If the alternatives are systems engineering process. In fact, there is always precisely enough defined and thoroughly enough a danger that the top-down process cannot keep up understood to be well differentiated in the cost- with the bottom-up process. effectiveness space, then the system manager can There is often an early need to resolve the make choices among them with confidence. issues (such as the system architecture) enough so The systems engineering process can be that the system can be modeled with sufficient realism thought of as the pursuit of definition and to do reliable trade studies. understanding of design alternatives to support those When resources are expended toward the decisions, coupled with the overseeing of their implementation of one of several design options, the implementation. To obtain assessments that are crisp resources required to complete the implementation of enough to facilitate good decisions, it is often that design decrease (of course), while there is necessary to delve more deeply into the space of usually little or no change in the resources that would possible designs than has yet been done, as is be required by unselected alternatives. Selected illustrated in Figure 3. alternatives thereby become even more attractive
It should be realized, however, that this than those that were not selected. spiral represents neither the project life cycle, which Consequently, it is reasonable to expect the encompasses the system from inception through system to be defined with increasingly better disposal, nor the product development process by resolution as time passes. This tendency is formalized which the system design is developed and at some point (in Phase B) by defining a baseline implemented, which occurs in Phases C and D (see system definition. Usually, the goals, objectives, and Chapter 3) of the project life cycle. Rather, as the constraints are baselined as the requirements portion intellectual process of systems engineering, it is of the baseline. The entire baseline is then subjected inevitably reflected in both of them. to configuration control in an attempt to ensure that
successive changes are indeed improvements.
As the system is realized, its particulars Figure 3—The Doctrine of Successive Refinement become clearer—but also harder to change. As stated
above, the purpose of systems engineering is to make
sure that the development process happens in a way Figure 3 is really a double helix—each that leads to the most cost-effective final system. The create concepts step at the level of design basic idea is that before those decisions that are hard engineering initiates a ca-pabilities definition spiral to undo are made, the alternatives should be carefully moving in the opposite direction. The concepts can assessed. never be created from whole cloth. Rather, they result The systems engineering process is applied from the synthesis of potential capabilities offered by again and again as the system is developed. As the the continually changing state of technology. This system is realized, the issues addressed evolve and process of design concept development by the the particulars of the activity change.
NASA Systems Engineering Handbook
Page 7 As an Example of the Process of Successive but its first cause. It could be argued that recognition Refinement, Consider the Choice of Altitude for a of the need or opportunity for a new system is an Space Station such as Alpha entrepreneurial activity, rather than an engineering
one. • The first issue is selection of the general The end result of this step is the discovery location. Alternatives include Earth orbit, one of and delineation of the system's goals, which generally the Earth-Moon Lagrange points, or a solar orbit. express the desires and requirements of the eventual At the current state of technology, cost and risk users of the system. In the NASA context, the considerations made selection of Earth orbit an system's goals should also represent the long term easy choice for Alpha. • Having chosen Earth interests of the taxpaying public. orbit, it is necessary to select an orbit region.
Alternatives include low Earth orbit (LEO), high Identify and Quantify Goals. Before it is possible to Earth orbit and geosynchronous orbit; orbital compare the cost-effectiveness of alternative system inclination and eccentricity must also be chosen. design concepts, the mission to be performed by the One of many criteria considered in choosing LEO system must be delineated. The goals that are for Alpha was the design complexity associated developed should cover all relevant aspects of with passage through the Van Allen radiation effectiveness, cost, schedule, and risk, and should be belts. traceable to the goals of the supersystem. To make it • System design choices proceed to the selection easier to choose among alternatives, the goals should of an altitude maintenance strategy—rules that be stated in quantifiable, verifiable terms, insofar as implicitly determine when, where, and why to re- that is possible and meaningful to do. boost, such as "maintain altitude such that there It is also desirable to assess the constraints are always at least TBD days to reentry," "collision that may apply. Some constraints are imposed by the avoidance maneuvers shall always increase the state of technology at the time of creating or altitude," "reboost only after resupply flights that modifying system design concepts. Others may have brought fuel," "rotate the crew every TBD appear to be inviolate, but can be changed by higher days." levels of management. The assumptions and other • A next step is to write altitude specifications. relevant information that underlie constraints should These choices might consist of replacing the always be recorded so that it is possible to estimate TBDs (values to be determined) in the altitude the benefits that could be obtained from their strategy with explicit numbers. relaxation. • Monthly operations plans are eventually part of At each turn of the spiral, higher-level goals the complete system design. These would include are analyzed. The analysis should identify the scheduled reboost burns based on predictions of subordinate enabling goals in a way that makes them the accumulated effect of drag and the details of traceable to the next higher level. As the systems on-board microgravity experiments. engineering process continues, these are • documented as functional requirements (what must Actual firing decisions are based on determinations of be done to achieve the next-higher-level goals) and the orbit which results from the momentum actually added by previous firings, the atmospheric density as performance requirements (quantitative variations actually encountered, and so on. descriptions of how well the functional requirements
must be done). A clear operations concept often helps Note that decisions at every step require that to focus the require-ments analysis so that both the capabilities offered by available technology be functional and performance requirements are considered—often at levels of design that are more ultimately related to the original need or opportunity. detailed than seems necessary at first In later turns of the spiral, further elaborations may become documented as detailed functional and Most of the major system decisions (goals, performance specifications. architecture, acceptable life-cycle cost, etc.) are made
during the early phases of the project, so the turns of Create Alternative Design Concepts. Once it is the spiral (that is, the successive refinements) do not under-stood what the system is to accomplish, it is correspond precisely to the phases of the system life possible to devise a variety of ways that those goals cycle. Much of the system architecture can be ' seen" can be met. Sometimes, that comes about as a even at the outset, so the turns of the spiral do not consequence of considering alternative functional correspond exactly to development of the allocations and integrating available subsystem architectural hierarchy, either. Rather, they design options. Ideally, as wide a range of plausible correspond to the successively greater resolution by alternatives as is consistent with the design which the system is defined. organization's charter should be defined, keeping in Each of the steps in the systems engineering mind the current stage in the process of successive process is discussed below. refinement. When the bottom-up process is operating,
a problem for the system engineer is that the Recognize Need/Opportunity. This step is shown in designers tend to become fond of the designs they Figure 3 only once, as it is not really part of the spiral create, so they lose their objectivity; the system
NASA Systems Engineering Handbook
Page 8 engineer often must stay an "outsider" so that there is associated with the continually improving resolution more objectivity. reduces the spread between upper and lower bounds On the first turn of the spiral in Figure 3, the as the process proceeds. sub-ject is often general approaches or strategies,
sometimes architectural concepts. On the next, it is Select Concept. Selection among the alternative likely to be functional design, then detailed design, design concepts is a task for the system manager, and so on. who must take into account the subjective factors that The reason for avoiding a premature focus the system engineer was unable to quantify, in on a single design is to permit discovery of the truly addition to the estimates of how well the alternatives best design. Part of the system engineer's job is to meet the quantified goals (and any effectiveness, ensure that the design concepts to be compared take cost, schedule, risk, or other constraints). into account all interface requirements. "Did you When it is possible, it is usually well worth include the cabling?" is a characteristic question. the trouble to develop a mathematical expression, When possible, each design concept should be called an objective function, that expresses the values described in terms of controllable design parameters of combinations of possible outcomes as a single so that each represents as wide a class of designs as measure of cost-effectiveness, as is illustrated in is reasonable. In doing so, the system engineer Figure 4, even if both cost and effectiveness must be should keep in mind that the potentials for change described by more than one measure. When may include organizational structure, schedules, achievement of the goals can be quantitatively procedures, and any of the other things that make up expressed by such an objective function, designs can a system. When possible, constraints should also be be compared in terms of their value. Risks associated described by parameters. with design concepts can cause these evaluations to Owen Morris, former Manager of the Apollo be somewhat nebulous (because they are uncertain Spacecraft Program and Manager of Space Shuttle and are best described by probability distributions). In Systems and Engineering, has pointed out that it is this illustration, the risks are relatively high for design often useful to define design reference missions concept A. There is little risk in either effectiveness or which stress all of the system's capabilities to a cost for concept B. while the risk of an expensive significant extent and which al1 designs will have to failure is high for concept C, as is shown by be able to accomplish. The purpose of such missions is to keep the design space open. Consequently, it can be very dangerous to write them into the system specifications, as they can have just the opposite effect.
Do Trade Studies. Trade studies begin with an assess-ment of how well each of the design alternatives meets the system goals (effectiveness, cost, schedule, and risk, both quantified and otherwise). The ability to perform these studies is enhanced by the development of system models that relate the design parameters to those assessments— but it does not depend upon them. Controlled modification and development of design concepts, together with such system models, often permits the use of formal optimization techniques to find regions of the design space that warrant further
investigation— those that are closer to the optimum Figure 4—A Quantitative Objective Function, De" surface indicated in Figure 1. pendent on Life-Cycle Cost and All Aspects of Whether system models are used or not, the Effec-tiveness. design concepts are developed, modified,
reassessed, and compared against competing the cloud of probability near the x axis with a high cost alternatives in a closed-loop process that seeks the and essentially no effectiveness. Schedule factors best choices for further development. System and may affect the effectiveness values, the cost values, subsystem sizes are often determined during the and the risk distributions. trade studies. The end result is the determination of The mission success criteria for systems bounds on the relative cost-effectivenesses of the differ significantly. In some cases, effectiveness goals design alternatives, measured in terms of the may be much more important than all others. Other quantified system goals. (Only bounds, rather than projects may demand low costs, have an immutable final values, are possible because determination of schedule, or require minimization of some kinds of the final details of the design is intentionally deferred. risks. Rarely (if ever) is it possible to produce a The bounds, in turn, may be derived from the combined quantitative measure that relates all of the probability density functions.) Increasing detail important factors, even if it is expressed as a vector
NASA Systems Engineering Handbook
Page 9 with several components. Even when that can be often desirable. The accounting system should follow done, it is essential that the underlying factors and (not lead) the system architecture. In terms of relationships be thoroughly revealed to and breadth, partitioning should be done essentially all at understood by the system manager. The system once. As with system design choices, alternative parti manager must weigh the importance of the -tioning plans should be considered and compared unquantifiable factors along with the quantitative data before implementation. provided by the system engineer. If a requirements-driven design paradigm is used for Technical reviews of the data and analyses tile development of the system architecture, it must be are an important part of the decision support applied with care, for the use of "shells" creates a packages prepared for the system manager. The tendency for the requirements to be treated as decisions that are made are generally entered into the inviolable con straints rather than as agents of the configuration management system as changes to (or objectives. A goal, ob -jective or desire should never elaborations of) the system baseline. The supporting be made a requirement until its costs. are understood trade studies are archived for future use. An essential and the buyer is willing to pay for it. The capability to feature of the systems engineering process is that compute the effects of lower -level decisions on the trade studies are performed before decisions are quantified goals should be maintained throughout the made. They can then be baselined with much more partitioning process. That is, there should be a goals confidence. flowdown embedded in the requirements allocation At this point in the systems engineering process. process, there is a logical branch point. For those The process continues with creation of a issues for which the process of successive refinement variety of alternative design concepts at the next level has proceeded far of resolution, construction of models that permit prediction of how well those alternatives will satisfy Simple Interfaces are Preferred the quantified goals, and so on. It is imperative that According to Morris, NASA's former Acting plans for subsequent integration be laid throughout Administrator George Low, in a 1971 paper titled the partitioning. Integration plans include verification "What Made Apollo a Success," noted that only and validation activities as a matter of course. 100 wires were needed to link the Apollo
spacecraft to the Saturn launch vehicle. He Implement the Selected Design Decisions. When emphasized the point that a single person could the process of successive refinement has proceeded fully understand the interface and cope with all the far enough, the next step is to reverse the partitioning enough, the next step is to implement the decisions at process. When applied to the system architecture, that level of resolution (that is, unwind the recursive this "unwinding" of the process is called system process). For those issues that are still insufficiently integration. Conceptual system integration takes resolved, the next step is to refine the development place in all phases of the project life cycle. That is, further. when a design approach has been selected, the
approach is verified by "unwinding the process" to test Increase the Resolution of the Design. One of the whether the concept at each physical level meets the first issues to be addressed is how the system should expectations and requirements. Physical integration is be subdivided into subsystems. (Once that has been accomplished during Phase D. At the finer levels of done, the focus changes and the subsystems become resolution, pieces must be tested, assembled and/or systems -- from the point of view of a system integrated, and tested again. The system engineer's engineer. The partitioning process stops when the role includes the performance of the delegated subsystems are simple enough to be managed management du ties, such as configuration control holistically.) As noted by Morris, "the divi- be and overseeing the integration, verification, and managed holistically.) As noted by Morris, "the validation process. division of program activities to minimize the number The purpose of verification of subsystem and complexity of interfaces has a strong influence on integration is to ensure that the subsystems conform the overall program cost and the ability of the program to what was designed and interface with each other to meet schedules." as expected in all re spects that are important: Charles Leising and Arnold Ruskin have mechanical connections, effects on center of mass (separately) pointed out that partitioning is more art and products of inertia, electromagnetic interference, than science, but that there are guidelines available: connector impedance and voltage, power con To make interfaces clean and simple, similar sumption, data flow, and so on. Validation consists of functions, designs and tech nologies should be ensuring that the interfaced subsystems achieve their grouped. Each portion of work should be verifiable. intended results. While validation is even more Pieces should map conveniently onto the important than verification, it is usually much more organizational structure. Some of the functions that difficult to accomplish. are needed throughout the design (such as electrical
power) or throughout the organization (such as Perform the Mission. Eventually, the system is purchasing) can be centralized. Standardization—of called upon to meet the need or seize the opportunity such things as parts lists or reporting formats—is for which it was designed and built.
NASA Systems Engineering Handbook
Page 10 The system engineer continues to perform a variety of supporting functions, depending on the nature and duration of the mission. On a large project such as Space Station Alpha, some of these continuing functions include the validation of system effectiveness at the operational site, overseeing the maintenance of configuration and logistics documentation, overseeing sustaining engineering activities, compiling development and operations "lessons reamed" documents, and, with the help of the specialty engineering disciplines, identifying product improvement opportunities. On smaller systems, such as a Spacelab payload, only the last two may be needed.
NASA Systems Engineering Handbook
Page 11 3 The Project Life Cycle for Major in government. In general, the engineering NASA Systems development life cycle is dependent on the technical
nature of what's being developed, and the project life One of the fundamental concepts used cycle may need to be tailored accordingly. Barry W. within NASA for the management of major systems is Boehm described how several contemporary software the pro-gram/ project life cycle, which consists of a development processes work; in some of these categorization of everything that should be done to processes, the development and construction accomplish a project into distinct phases, separated activities proceed in parallel, so that attempting to by control gates. Phase boundaries are defined so separate the associated phases on a time line is that they provide more-or-less natural points for undesirable. Boehm describes a spiral, which reflects go/no-go decisions. Decisions to proceed may be the doctrine of successive refinement depicted in qualified by liens that must be removed within a Figure 3, but Boehm's spiral describes the software reasonable time. A project that fails to pass a control product development process in particular. His gate and has enough resources may be allowed to discussion applies as well to the development of “go back to the drawing board"—or it may be hardware products as it does to software. Other terminated. exam-ples of alternative processes are the rapid All systems start with the recognition of a prototyping and rapid development approaches. need or the discovery of an opportunity and proceed Selection of a product development process paradigm through various stages of development to a final must be a case-dependent decision, based on the disposition. While the most dramatic impacts of the system engineer's judgment and experience. analysis and optimization activities associated with Sometimes, it is appropriate to perform some systems engineering are obtained in the early stages, long-lead-time activities ahead of the time they would decisions that affect millions of dollars of value or cost nominally be done. Long-lead-time activities might continue to be amenable to the systems approach consist of technology developments, prototype even as the end of the system lifetime approaches. construction and testing, or even fabrication of difficult Decomposing the project life cycle into components. Doing things out of their usual sequence phases or-ganizes the entire process into more increases risk in that those activities could wind up manageable pieces. The project life cycle should having been either unnecessary or improperly provide managers with incremental visibility into the specified. On the other hand, overall risk can progress being made at points in time that fit with the sometimes be reduced by removal of such activities management and budgetary environments. NASA from the critical path. documents governing the acquisition of major Figure 5 (foldout, next page) details the systems (NMI 7120.4 and NHB 7120.5) define the resulting management and major systems phases of the project life cycle as: engineering products and control gates that
characterize the phases in NMI 7120.4 and NHB • 7120.5. Sections 3.1 to 3.6 contain narrative Pre-Phase A—Advanced Studies ("find a suitable descriptions of the purposes, major activities, project") • products, and control gates of the NASA project life Phase A—Preliminary Analysis ("make sure the cycle phases. Section 3.7 provides a more project is worthwhile") concentrated discussion of the role of systems • Phase B—Definition ("define the project and engineering in the process. Section 3.8 describes the establish a preliminary design") NASA budget cycle within which program/project • Phase C—Design ("complete the system design") managers and system engineers must operate. Phase D — Development ("build, integrate, and
verify the system, and prepare for operations") 3.1 Pre-Phase A—Advanced Studies • Phase E—Operations ("operate the system and
dispose of it properly"). The purpose of this activity, which is usually per-
formed more or less continually by "Advanced Phase A efforts are conducted by NASA field Projects" groups, is to uncover, invent, create, cen-ters; such efforts may rely, however, on pre- concoct and/or devise a broad spectrum of ideas and Phase A in-house and contracted advanced studies. alternatives for missions from which new projects The majority of Phase B efforts are normally (programs) can be selected. Typically, this activity accomplished by industry under NASA contract, but consists of loosely structured ex-aminations of new NASA field centers typically conduct parallel in-house ideas, usually without central control and mostly studies in order to validate the contracted effort and oriented toward small studies. Its major product is a remain an informed buyer. NASA usually chooses to stream of suggested projects, based on the contract with industry for Phases C and D, and often identification of needs and the discovery of does so for Phase E. Phase C is nominally combined opportunities that are potentially consistent with with Phase D, but when large production quantities NASA's mission. capabilities, priorities, and are planned, these are treated separately. resources. Alternatives to the project phases described above can easily be found in industry and elsewhere
NASA Systems Engineering Handbook
Page 12 system before seeking significant funding. According Pre-Phase A—Advanced Studies
to NHB 7120.5, the major products of this phase are a Purpose: To produce a broad spectrum of ideas formal Mission Needs Statement (MNS) and one or and alternatives for missions from which new pro- more credible, feasible designs and operations grams/ projects can be selected. concepts. John Hodge describes this phase as "a Major Activities and their Products: Identify structured version of the previous phase." missions consistent with charter Identify and Phase A—Preliminary Analysis involve users
Perform preliminary evaluations of possible Purpose: To determine the feasibility and missions Prepare program/project proposals, desirability of a suggested new major system and which include: its compatibility with NASA's strategic plans. • Mission justification and objectives Major Activities and their Products: • Possible operations concepts Prepare Mission Needs Statement • Possible system architectures Develop top-level requirements • Cost, schedule, and risk estimates. Develop corresponding evaluation criteria/metrics Develop master plans for existing program areas Identify alternative operations and logistics Information Baselined: concepts (nothing) Identify project constraints and system boundaries Control Gates: Consider alternative design concepts, including: Mission Concept Review feasibility and risk studies, cost and schedule Informal proposal reviews estimates, and advanced technology requirements Demonstrate that credible, feasible design(s) exist In the NASA environment, demands for new Acquire systems engineering tools and models systems derive from several sources. A major one is Initiate environmental impact studies the opportunity to solve terrestrial problems that may Prepare Project Definition Plan for Phase B be ad-dressed by putting instruments and other Information Baselined: devices into space. Two examples are weather (nothing) prediction and communications by satellite. General Control Gates: improvements in technology for use in space will Mission Definition Review continue to open new possibilities. Such opportunities Preliminary Non-Advocate Review are rapidly perceived as needs once the magnitude of Preliminary Program/Project Approval Review their value is understood. Technological progress makes possible missions that were previously impossible. Manned
trips to the moon and the taking of high resolution In Phase A, a larger team, often associated pictures of planets and other objects in the universe with an ad hoc program or project office, readdresses illustrate past responses to this kind of opportunity. the mission concept to ensure that the project New opportunities will continue to become available justification and practicality are sufficient to warrant a as our technological capabilities grow. place in NASA's budget. The team's effort focuses on Scientific progress also generates needs for analyzing mission requirements and establishing a NASA systems. As our understanding of the universe mission architecture. Activities become formal, and around us continues to grow, we are able to ask new the emphasis shifts toward establishing optimality and more precise questions. The ability to answer rather than feasibility. The effort addresses more these questions often depends upon the changing depth and considers many alternatives. Goals and state of technology. objectives are solidified, and the project develops Advanced studies may extend for several more definition in the system requirements, top-level years, and may be a sequence of papers that are only system architecture, and operations concept. loosely connected. These studies typically focus on Conceptual designs are developed and exhibit more establishing mission goals and formulating top-level engineering detail than in advanced studies. system requirements and operations concepts. Technical risks are identified in more detail and Conceptual designs are often offered to demonstrate technology development needs become focused. feasibility and support programmatic estimates. The The Mission Needs Statement is not shown emphasis is on establishing feasibility and desirability in the sidebar as being baselined, as it is not under rather than optimality. Analyses and designs are configuration control by the project. It may be under accordingly limited in both depth and number of configuration control at the program level, as may the options. program requirements documents and the Preliminary
Program Plan. 3.2 Phase A—Preliminary Analysis
The purpose of this phase is to further examine the feasibility and desirability of a suggested new major
NASA Systems Engineering Handbook
seek out more cost-effective designs. (Trade studies
should precede—rather than follow—system design Figure 6 – Overruns are Very Likely if Phases A decisions. Chamberlain, Fox, and Duquette describe and B are Underfunded. Phase B -- Definition
Purpose: To define the project in enough detail to establish an initial baseline capable of meeting mission needs. Major Activities and their Products: Prepare a Systems Engineering Management Plan Prepare a Risk Management Plan Initiate configuration management Prepare engineering specialty program plans Develop system-level cost-effectiveness model Restate mission needs as functional requirements Identify science payloads Establish the initial system requirements and verification requirements matrix Perform and archive trade studies Select a baseline design solution and a concept of operations Define internal and external interface 3.3 Phase B -- Definition requirements
(Repeat the process of successive refinement to The purpose of this phase is to establish an get "design-to" specifications and drawings, initial project baseline, which (according to NHB verifications plans, and interface documents to 7120.5) includes "a formal flowdown of the project- lower levels as appropriate) level performance requirements to a complete set of Define the work breakdown structure system and subsystem design specifications for both Define verification approach end policies flight and ground elements" and "corresponding Identify integrated logistics support requirements preliminary designs." The technical requirements Establish technical resource estimates and firm should be sufficiently detailed to establish firm life-cy-cle cost estimates schedule and cost estimates for the project. Develop statement(s) of work A Credible, Feasible Design Initiate advanced technology developments
Revise and publish a Project Plan A feasible system design is one that can be Reaffirm the Mission Needs Statement implemented as designed and can then Prepare a Program Commitment Agreement accomplish the system's goals within the Information Baselined: constraints imposed by the fiscal and operating System requirements and verification environment. To be credible, a design must not requirements matrix depend on the occurrence of unforeseen System architecture and work breakdown breakthroughs in the state of the art. While a structure credible design may assume likely improvements Concept of operations in the state of the art, it is nonetheless riskier than “Design-to” specifications at all levels Project plans, including schedule, resources, Actually, "the" Phase B baseline consists of acquisition strategies and risk management a collection of evolving baselines covering technical and business aspects of the project: system (and a decentralized process for ensuring that such trades subsystem) requirements and specifications, designs, lead efficiently to an optimum system design.) Major verification and operations plans, and so on in the products to this point include an accepted "functional" technical portion of the baseline, and schedules, cost baseline and preliminary "design-to" baseline for the projections, and management plans in the business system and its major end items. The effort also portion. Establishment of baselines implies the produces various engineering and management plans implementation of configuration management to prepare for managing the project's downstream procedures. (See Section 4.7.) processes, such as verification and operations, and Early in Phase B, the effort focuses on for implementing engineering specialty programs. allocating functions to particular items of hardware, Along the way to these products, projects software, personnel, etc. System functional and are subjected to a Non-Advocate Review, or NAR. performance requirements along with architectures This activity seeks to assess the state of project and designs become firm as system trades and definition in terms of its clarity of objectives and the subsystem trades iterate back and forth in the effort to thoroughness of technical and management plans,
NASA Systems Engineering Handbook
Page 15 technical documentation, alternatives explored, and of the system hierarchy. The CDR is held prior to the trade studies performed. The NAR also seeks to start of fabrication/production of end items for evaluate the cost and schedule estimates, and the hardware and prior to the start of coding of deliverable contingency reserve in these estimates. The timing of software products. Typically, the sequence of CDRs this review is often driven by the Federal budget reflects the integration process that will occur in the cycle, which requires at least 16 months between next phase— that is, from lower-level CDRs to the NASA's budget preparation for submission to the system-level CDR. Projects, however, should tailor President's Office of Management and Budget, and the sequencing of the reviews to meet their individual the Congressional funding for a new project start. needs. The final product of this phase is a "build-to" (See Section 3.8.) There is thus a natural tension baseline in sufficient detail that actual production can between the desire to have maturity in the project at proceed. the time of the NAR and the desire to progress Phase C—Design efficiently to final design and development. Later in Phase B, the effort shifts to
establishing a functionally complete design solution Purpose: To complete the detailed design of the (i.e., a "design-to" baseline) that meets mission goals sys-tem (and its associated subsystems, including and objectives. Trade studies continue. Interfaces its operations systems). among the major end items are defined. Engineering Major Activities and their Products: test items may be developed and used to derive data Add remaining lower-level design specifications to for further design work, and project risks are reduced the system architecture by successful technology developments and Refine requirements documents demonstrations. Phase B culminates in a series of Refine verification plans preliminary design reviews (PDRs), containing the Prepare interface documents system-level PDR and PDRs for lower-level end items (Repeat the process of successive refinement to as appropriate. The PDRs reflect the successive get "build-to" specifications and drawings, refinement of requirements into designs. Design verification plans, and interface documents at all issues uncovered in the PDRs should be resolved so levels) that final design can begin with unambiguous "design- Augment baselined documents to reflect the to" specifications. From this point on, almost all growing maturity of the system: system changes to the baseline are expected to represent architecture, verification requirements matrix, successive refinements, not fundamental changes. work breakdown structure, project plans Prior to baselining, the system architecture, Monitor project progress against project plans preliminary design, and operations concept must have Develop the system integration plan and the been validated by enough technical analysis and system operation plan design work to establish a credible, feasible design at Perform and archive trade studies a lower level of detail than was sufficient for Phase A. Complete manufacturing plan
Develop the end-to-end information system design 3.4 Phase C—Design Refine Integrated Logistics Support Plan
Identify opportunities for pre-planned product The purpose of this phase is to establish a improvement complete design (“build-to" baseline) that is ready to Confirm science payload selection fabricate (or code), integrate, and verify. Trade Information Baselined: studies continue. Engineering test units more closely All remaining lower-level requirements and resembling actual hardware are built and tested so as to establish confidence that the design will function in
the expected environments. Engineering specialty 3.5 Phase D—Development analysis results are integrated into the design, and the
manufacturing process and controls are defined and The purpose of this phase is to build and validated. Configuration management continues to verify the system designed in the previous phase, track and control design changes as detailed deploy it, and prepare for operations. Activities interfaces are defined. At each step in the successive include fabrication of hardware and coding of refinement of the final design, corresponding software, integration, and verification of the system. integration and verification activities are planned in Other activities include the initial training of operating greater detail. During this phase, technical personnel and implementation of the Integrated parameters, schedules, and budgets are closely Logistics Support Plan. For flight projects, the focus of tracked to ensure that undesirable trends (such as an activities then shifts to pre-launch integration and unexpected growth in spacecraft mass or increase in launch. For large flight projects, there may be an its cost) are recognized early enough to take extended period of orbit insertion, assembly, and corrective action. (See Section 4.9.) initial shake-down operations. The major product is a Phase C culminates in a series of critical system that has been shown to be capable of design reviews (CDRs) containing the system-level accomplishing the purpose for which it was created. CDR and CDRs corresponding to the different levels
NASA Systems Engineering Handbook
involve major changes to the system architecture;
changes of that scope constitute new "needs," and
the project life cycle starts over.
3.6 Phase E—Operations
Phase D—Development Phase E—Operations
Purpose: To build the subsystems (including the Purpose: To actually meet the initially identified op-erations system) and integrate them to create need or to grasp the opportunity, then to the system, meanwhile developing confidence dispose of the system in a responsible manner. that it will be able to meet the system Major Activities and their Products: requirements, then to deploy the system and Train replacement operators and maintainers ensure that it is ready for operations. Conduct the mission(s) Major Activities and their Products Maintain and upgrade the system Fabricate (or code) the parts (i.e., the lowest-level Dispose of the system and supporting processes items in the system architecture) Document Lessons Learned Integrate those items according to the integration Information Baselined: plan and perform verifications, yielding verified Mission outcomes, such as: components and subsystems • (Repeat the process of successive integration to Engineering data on system, subsystem and get a verified system) materials Develop verification procedures at all levels performance Perform system qualification verification(s) • Science data returned Perform system acceptance verification(s) • High resolution photos from orbit Monitor project progress against project plans • Accomplishment records ("firsts") Archive documentation for verifications performed • Discovery of the Van Allen belts Audit "as-built" configurations • Discovery of volcanoes on lo. Document Lessons Learned Operations and maintenance logs Prepare operator's manuals Problem/failure reports Prepare maintenance manuals Control Gates: Train initial system operators and maintainers Regular system operations readiness reviews Finalize and implement Integrated Logistics System upgrade reviews Support Plan Safety reviews Integrate with launch vehicle(s) and launch, Decommissioning Review perform orbit insertion, etc., to achieve a deployed system
Perform operational verification(s) Phase E encompasses the problem of Information Baselined: dealing with the system when it has completed its "As-built" and "as-deployed" configuration data mission; the time at which this occurs depends on Integrated Logistics Support Plan many factors. For a flight system with a short mission Command sequences for end-to-end command duration, such as a Spacelab payload, disposal may and telemetry validation and ground data require little more than de-integration of the hardware processing and its return to its owner. On large flight projects of Operator's manuals long duration, disposal may proceed according to Maintenance manuals long-established plans, or may begin as a result of Control Gates: unplanned events, such as accidents. Alternatively, Test Readiness Reviews (at all levels) technological advances may make it uneconomic to System Acceptance Review continue operating the system either in its current System functional and physical configuration configuration or an improved one. audits In addition to uncertainty as to when this part Flight Readiness Review(s) of the phase begins, the activities associated with Operational Readiness Review safely decommissioning and disposing of a system Safety reviews may be long and complex. Consequently, the costs and risks associated with different designs should be
considered during the project's earlier phases. The purpose of this phase is to meet the initially
identified need or to grasp the initially identified 3.7 Role of Systems Engineering in the opportunity. The products of the phase are the results Project Life Cycle of the mission. This phase encompasses evolution of
the system only insofar as that evolution does not
NASA Systems Engineering Handbook
Page 17 This section presents two "idealized" As product development progresses, a descriptions of the systems engineering activities series of baselines is progressively established, each within the project life cycle. The first is the Forsberg of which is put under formal configuration and Mooz "vee" chart, which is taught at the NASA management at the time it is approved. Among the program/project management course. me second is fundamental purposes of configuration management the NASA program/project life cycle process flow is to prevent requirements from "creeping." developed by the NASA-wide Systems Engineering The left side of the core of the vee is similar Process Improvement Task team, in 1993/94. to the so-called "waterfall" or "requirements-driven
design" model of the product development process. 3.7.1 The "Vee" Chart The control gates define significant decision points in
the process. Work should not progress beyond a Forsberg and Mooz describe what they call decision point until the project manager is ready to "the technical aspect of the project cycle" by a vee- publish and control the documents containing the shaped chart, starting with user needs on the upper decisions that have been agreed upon at that point. left and ending with a user-validated system on the However, there is no prohibition against doing upper right. Figure 7 provides a summary level detailed work early in the process. In fact, detailed overview of those activities. On the left side of the hardware and/or software models may be required at vee, decomposition and definition activities resolve the very earliest stages to clarify user needs or to the system architecture, creating the details of the establish credibility for the claim of feasibility. Early design. Integration and verification flow up and to the application of involved technical and support right as successively higher levels of subsystems are disciplines is an essential part of this process; this is verified, culminating at the system level. This in fact implementation of concurrent engineering. summary chart follows the basic outline of the vee At each level of the vee, systems chart developed by NASA as part of the Software engineering activities include off-core processes: Management and Assurance Program. ("CIs'' in the system design, advanced technology development, figure refer to the hardware and software trade studies, risk management, specialty engineering configuration items, which are controlled by the analysis and modeling. This is shown on the chart as configuration management system.) an orthagonal process in Figure 7(b). These activities
are performed at each level and may be repeated Decomposition and Definition. Although not shown many times within a phase. While many kinds of in Figure 7, each box in the vee represents a number studies and decisions are associated with the off-core of parallel boxes suggesting that there may be many activities, only decisions at the core level are put subsystems that make up the system at that level of under configuration management at the various decomposition. For the top left box, the various control gates. Off-core activities, analyses, and parallel boxes represent the alternative design models are used to substantiate the core decisions concepts that are initially evaluated. and to ensure that the risks have been mitigated or Figure 7—Overview of the Technical kAspect of the NASA Project Cycle.
NASA Systems Engineering Handbook
Page 18 determined to be acceptable. The off-core work is not and performance. This is a common approach in formally controlled, but the analyses, data and results software development. should be archived to facilitate replication at the The incremental development approach is appropriate times and levels of detail to support easy to describe in terms of the vee chart: all introduction into the baseline. increments have a common heritage down to the first There can, and should, be sufficient iteration PDR. The balance of the product development downward to establish feasibility and to identify and process has a series of displaced and overlapping quantify risks. Upward iteration with the requirements vees, one for each release. statements (and with the intermediate products as
well) is permitted, but should be kept to a minimum 3.7.2 The NASA Program/Project Life Cycle unless the user is still generating (or changing) Process Flow requirements. In software projects, upward
confirmation of solutions with the users is often
Another idealized description of the technical necessary because user requirements cannot be activities that occur during the NASA project life cycle adequately defined at the inception of the project. is illustrated in Figure 8 (foldout, next page). In the Even for software projects, however, iteration with figure, the NASA project life cycle is partitioned into user requirements should be stopped at the PDR, or ten process flow blocks, which are called stages in cost and schedule are likely to get out of control. this handbook. The stages reflect the changing nature Modification of user requirements after PDR should of the work that needs to be performed as the system be held for the next model or release of the product. If matures. These stages are related both temporally significant changes to user requirements are made and logically. Successive stages mark increasing after PDR, the project should be stopped and system refinement and maturity, and require the restarted with a new vee, reinitiating the entire products of previous stages as inputs. A transition to process. The repeat of the process may be quicker a new stage entails a major shift in the nature or because of the lessons learned the first time through, extent of technical activities. Control gates assess the but all of the steps must be redone. wisdom of progressing from one stage to another. Time and project maturity flow from left to (See Section 4.8.3 for success criteria for specific right on the vee. Once a control gate is passed, reviews.) From the perspective of the system backward iteration is not possible. Iteration with the engineer, who must oversee and monitor the user requirements, for example, is possible only technical progress on the system, Figure 8 provides a vertically, as is illustrated on the vee. more complete description of the actual work needed
through the NASA project life cycle. Integration and Verification. Ascending the right In practice, the stages do not always occur side of the vee is the process of integration and sequentially. Unfolding events may invalidate or verification. At each level, there is a direct modify goals and assumptions. This may neccessitate correspondence between activities on the left and revisiting or modifying the results of a previous stage. right sides of the vee. This is deliberate. The method The end items comprising the system often have of verification must be determined as the different development schedules and constraints. This requirements are developed and documented at each is especially evident in Phases C and D where some level. This minimizes the chances that requirements subsystems may be in final design while others are in are specified in a way that cannot be measured or fabrication and integration. verified. The products of the technical activities Even at the highest levels, as user support the systems engineering effort (e.g., requirements are translated into system requirements, requirements and specifications, trade studies, the system verification approach, which will prove that specialty engineering analyses, verification results), the system does what is required, must be and serve as inputs to the various control gates. For a determined. The technical demands of the verification detailed systems engineering product database, process, represented as an orthagonal process in database dictionary, and maturity guidelines, see Figure 7(c), can drive cost and schedule, and may in JSC-49040, NASA Systems Engineering Process for fact be a discriminator between alternative concepts. Programs and Projects. For example, if engineering models are to be used for Several topics suggested by Figures 7 and 8 verification or validation, they must be specified and merit special emphasis. These are concurrent costed, their characteristics must be defined, and their engineering, technology insertion, and the distinction development time must be incorporated into the between verification and validation. schedule from the beginning.
Concurrent Engineering. If the project passes early Incremental Development. If the user requirements control gates prematurely, it is likely to result in a are too vague to permit final definition at PDR, one need for significant iteration of requirements and approach is to develop the project in predetermined designs late in the development process. One way incremental releases. The first release is focused on this can happen is by failing to involve the appropriate meeting a minimum set of user requirements, with technical experts at early stages, thereby resulting in subsequent releases providing added functionality the acceptance of requirements that cannot be met
NASA Systems Engineering Handbook
Page 21 and the selection of design concepts that cannot be approach that is not dependent on the development of built, tested, maintained, and/or operated. new technology must be carried unless high risk is Concurrent engineering is the simultaneous acceptable. The technology development activity consideration of product and process downstream should be managed by the project manager and requirements by multidisciplinary teams. Specialty system engineer as a critical activity. engineers from all disciplines (reliability,
maintainability, human factors, safety, logistics, etc.) Verification vs. Validation. The distinction between whose expertise will eventually be represented in the verification and validation is significant: verification product have important contributions throughout the consists of proof of compliance with specifications, system life cycle. The system engineer is responsible and may be determined by test, analysis, for ensuring that these personnel are part of the demonstration, inspection, etc. (see Section 6.6). project team at each stage. In large projects, many Validation consists of proof that the system integrated product development teams (PDTs) may accomplishes (or, more weakly, can accomplish) its be required. Each of these, in turn, would be purpose. It is usually much more difficult (and much represented on a PDT for the next higher level in the more important) to validate a system than to verify it. project. In small projects, however, a small team is Strictly speaking, validation can be accomplished only often sufficient as long as the system engineer can at the system level, while verification must be augment it as needed with experts in the required accomplished throughout the entire system technical and business disciplines. architectural hierarchy. The informational requirements of doing concurrent engineering are demanding. One way concurrent engineering experts believe it can be made less burdensome is by an automated environment. In such an environment, systems engineering, design and analysis tools can easily Integrated Product Development Teams The detailed evaluation of product and process feasibility and the identification of significant uncertainties (system risks) must be done by experts from a variety of disciplines. An approach that has been found effective is to establish teams for the development of the product with representatives from all of the disciplines and processes that will eventually be involved. These integrated product development teams often have multidisciplinary (technical and business) members. Technical personnel are needed to ensure that issues such as producibility, verifiability, deployability, supportability, trainability, operability, and disposability are all considered in the design. In addition, business (e.g., procurement! representatives are added to the team as the need arises. Continuity of support from these specialty discipline organizations throughout the system life-cycle is highly desirable, though team composition and leadership can be expected to change as the system progresses from phase to phase. Figure 9—Typical NASA Budget Cycle.
exchange data, computing environments are interoperable, and product data are readily accessible
and accurate. For more on the characteristics of 3.8 Funding: The Budget Cycle automated environments, see for example Carter and
Baker, Concurrent Engineering, 1992. NASA operates with annual funding from Congress.
This funding results, however, from a three-year Technology Insertion. Projects are sometimes rolling process of budget formulation, budget initiated with known technology shortfalls, or with enactment, and finally, budget execution. A highly areas for which new technology will result in simplified representation of the typical budget cycle is substantial product improvement. Technology shown in Figure 9. development can be done in parallel with the project NASA starts developing its budget each evolution and inserted as late as the PDR. A parallel January with economic forecasts and general
NASA Systems Engineering Handbook
Page 22 guidelines being provided by the Executive Branch's Office of Management and Budget (OMB). In early May, NASA conducts its Program Operating Plan (POP) and Institutional Operating Plan (IOP) exercises in preparation for submittal of a preliminary NASA budget to the OMB. A final NASA budget is submitted to the OMB in September for incorporation into the President's budget transmittal to Congress, which generally occurs in January. This proposed budget is then subjected to Congressional review and approval, culminating in the passage of bills authorizing NASA to obligate funds in accordance with Congressional stipulations and appropriating those funds. The Congressional process generally lasts through the summer. In recent years, however, final bills have often been delayed past the start of the fiscal year on October 1. In those years, NASA has operated on continuing resolutions by Congress. With annual funding, there is an implicit funding control gate at the beginning of every fiscal year. While these gates place planning requirements on the project and can make significant replanning necessary, they are not part of an orderly systems engineering process. Rather, they constitute one of the sources of uncertainty that affect project risks and should be consided in project planning.
NASA Systems Engineering Handbook
Page 23 4 Management Issues in Systems work necessary to complete the project. Each task in Engineering the WBS should be traceable to one or more of the
system requirements. Schedules, which are This chapter provides more specific structured as networks, describe the time-phased information on the systems engineering products and activities that result in the product system in the WBS. approaches used in the project life cycle just The cost account structure needs to be directly linked described. These products and approaches are the to the work in the WBS and the schedules by which system engineer's contribution to project that work is done. (See Sections 4.3 through 4.5.) management, and are designed to foster structured The project's organization structure ways of managing a complex set of activities. describes the clusters of personnel assigned to
perform the work. These organizational structures are 4.1 Harmony of Goals, Work Products, and usually trees. Sometimes they are represented as a Organizations matrix of two interlaced trees, one for line
responsibilities, the other for project responsibilities. When applied to a system, the doctrine of In any case, the organizational structure should allow successive refinement is a "divide-and-conquer" identification of responsibility for each WBS task. strategy. Complex systems are successively divided Project documentation is the product of into pieces that are less complex, until they are simple particular WBS tasks. There are two fundamental enough to be conquered. This decomposition results categories of project documentation: baselines and in several structures for describing the product system archives. Each category contains information about and the producing system ("the system that produces both the product system and the producing system. the system"). These structures play important roles in The baseline, once established, contains information systems engineering and project management. Many describing the current state of the product system and of the remaining sections in this chapter are devoted producing system resulting from all decisions that to describing some of these key structures. have been made. It is usually organized as a Structures that describe the product system collection of hierarchical tree structures, and should include, but are not limited to, the requirements tree, exhibit a significant amount of cross-reference linking. system architecture, and certain symbolic information The archives contain all of the rest of the project's such as system drawings, schematics, and information that is worth remembering, even if only databases. The structures that describe the producing temporarily. The archives should contain all system include the project's work breakdown, assumptions, data, and supporting analyses that are schedules, cost accounts, and organization. These relevant to past, present, and future decisions. structures provide different perspectives on their Inevitably, the structure (and control) of the archives common raison d'etre: the desired product system. is much looser than that of the baseline, though cross Creating a fundamental harmony among these references should be maintained where feasible. (See structures is essential for successful systems Section 4.7.) engineering and project management; this harmony The structure of reviews (and their needs to be established in some cases by one-to-one associated control gates) reflect the time-phased correspondence between two structures, and in other activities associated with the realization of the product cases, by traceable links across several structures. It system from its product breakdown. The status is useful, at this point, to give some illustrations of this reporting and assessment structure provides key principle. information on the progress of those same activities. System requirements serve two purposes in On the financial side, the status reporting and the systems engineering process: first, they represent assessment structure should be directly linked to the a hierarchical description of the buyer's desired WBS, schedules, and cost accounts. On the technical product system as understood by the product side, it should be linked to the product breakdown development team (PDT). The interaction between and/or requirements tree. (See Sections 4.8 and 4.9.) the buyer and system engineer to develop these
requirements is one way the "voice of the buyer" is 4.2 Managing the Systems Engineering heard. Determining the right requirements— that is, Process: only those that the informed buyer is willing to pay The Systems Engineering Management Plan for—is an important part of the system engineer's job. Systems engineering management is a technical Second, system requirements also communicate to function and discipline that ensures that systems the design engineers what to design and build (or engineering and all other technical functions are code). As these requirements are allocated, they properly applied. become inexorably linked to the system architecture Each project should be managed in and product breakdown, which consists of the accordance with a project life cycle that is carefully hierarchy of system, segments, elements, tailored to the project's risks. While the project subsystems, etc. (See the sidebar on system termi- manager concentrates on managing the overall nology on page 3.) project life cycle, the project-level or lead system The Work Breakdown Structure (WBS) is engineer concentrates on managing its technical also a tree-like structure that contains the pieces of aspect (see Figure 7 or 8). This requires that the
NASA Systems Engineering Handbook
Page 24 system engineer perform or cause to be performed Plan, depend on the SEMP and must comply with it. the necessary multiple layers of decomposition, The SEMP should be comprehensive and describe definition? integration, verification and validation of how a fully integrated engineering effort will be the system, while orchestrating and incorporating the managed and conducted. appropriate concurrent engineering. Each one of
these systems engineering functions re-quires 4.2.2 Contents of the SEMP application of technical analysis skills and tech-
niques. Since the SEMP describes the project's The techniques used in systems engineering technical management approach, which is driven by management include work breakdown structures, the type of project, the phase in the project life cycle, network scheduling, risk management, requirements and the technical development risks. it must be traceability and reviews, baselines, configuration specifically written for each project to address these management, data management, specialty situations and issues. While the specific content of the engineering program planning, definition and SEMP is tailored to the project, the recommended readiness reviews, audits, design certification, and content is listed below. status reporting and assessment.
The Project Plan defines how the project will Part I—Technical Project Planning and Control. be managed to achieve its goals and objectives within This section should identify organizational defined programmatic constraints. The Systems responsibilities and authority for systems engineering Engineering Management Plan (SEMP) is the management, including control of contracted subordinate document that defines to all project engineering; levels of control established for participants how the project will be technically performance and design requirements, and the managed within the constraints established by the control method used; technical progress assurance Project Plan. The SEMP communicates to all methods; plans and schedules for design and participants how they must respond to pre-established technical program/project reviews; and control of management practices. For instance, the SEMP documentation. This section should describe: should describe the means for both internal and
external (to the project) interface control. The SEMP • The role of the project office The role of the user also communicates how the systems engineering The role of the Contracting Office Technical management techniques noted above should be Representative (COTR) applied. • The role of systems engineering The role of
design engineering 4.2.1 Role of the SEMP • The role of specialty engineering
• Applicable standards The SEMP is the rule book that describes to • Applicable procedures and training all participants how the project will be technically • Baseline control process managed. The responsible NASA field center should • Change control process have a SEMP to describe how it will conduct its • Interface control process technical management, and each contractor should • have a SEMP to describe how it will manage in Control of contracted (or subcontracted) accordance with both its contract and NASA's engineering technical management practices. Since the SEMP is • Data control process Make-or-buy control project- and contract-unique, it must be updated for process each significant programmatic change or it will • Parts, materials, and process control become outmoded and unused, and the project could • Quality control slide into an uncontrolled state. The NASA field center • Safety control should have its SEMP developed before attempting to • Contamination control prepare an initial cost estimate, since activities that • Electromagnetic interference and incur cost, such as tech-nical risk reduction, need to electromagnetic compatibility (EMI/EMC) be identified and described beforehand. The • Technical performance measurement process contractor should have its SEMP developed during • Control gates the proposal process (prior to costing and pricing) • Internal technical reviews because the SEMP describes the technical content of • Integration control the project, the potentially costly risk management • Verification control activities, and the verification and validation • Validation control. techniques to be used, all of which must be included
in the preparation of project cost estimates. Part II—Systems Engineering Process. This The project SEMP is the senior technical section should contain a detailed description of the management document for the project; all other process to be used, including the specific tailoring of technical control documents, such as the Interface the process to the requirements of the system and Control Plan, Change Control Plan. Make-or-Buy project; the procedures to be used in implementing Control Plan, Design Review Plan, Technical Audit the process; in-house documentation; the trade study
NASA Systems Engineering Handbook
Page 25 methodology; the types of mathematical and or aspect of the project life cycle, are developed. This simulation models to be used for system cost- becomes the keel of the project that ultimately effectiveness evaluations; and the generation of determines the project's length and cost. The specifications. development of the programmatic and technical This section should describe the: management approaches requires that the key project
personnel develop an understanding of the work to be • System decomposition process performed and the relationships among the various • System decomposition format parts of that work. (See Sections 4.3 and 4.4 on Work • System definition process Breakdown Structures and network schedules, • System analysis and design process respectively.) The SEMP's development requires • Requirements allocation process contributions from knowledgeable programmatic and • Trade study process technical experts from all areas of the project that can • System integration process significantly influence the project's outcome. The • System verification process involvement of recognized experts is needed to • establish a SEMP that is credible to the project System qualification process manager and to secure the full commitment of the • System acceptance process project team. • System validation process
• Risk management process 4.2.4 Managing the Systems Engineering • Life-cycle cost management process Process: Summary • Specification and drawing structure •
Configuration management process • The systems engineering organization, and Data management process • specifically the project-level system engineer, is Use of mathematical models responsible for managing the project through the • Use of simulations technical aspect of the project life cycle. This • Τools to be used. responsibility includes management of the
decomposition and definition sequence, and Part III—Engineering Specialty Integration. This management of the integration, verification, and sec-tion of the SEMP should describe the integration validation sequence. Attendant with this management and coordination of the efforts of the specialty is the requirement to control the technical baselines of engineering disciplines into the systems engineering the project. Typically, these baselines are the: process during each iteration of that process. Where “functional," ''design to,” "build-to'' (or "code-to"), "as- there is potential for overlap of specialty efforts, the built" (or "as-coded"), and ''as-deployed." Systems SEMP should define the relative responsibilities and engineering must ensure an efficient and logical authorities of each. This section should contain, as progression through these baselines. needed, the project's approach to: Systems engineering is responsible for
system decomposition and design until the "design-to" • Concurrent engineering specifications of all lower-level configuration items • The activity phasing of specialty disciplines have been produced. Design engineering is then • The participation of specialty disciplines responsible for developing the ''build-to" and "code-to" • The involvement of specialty disciplines documentation that complies with the approved • The role and responsibility of specialty disciplines "design-to" baseline. Systems engineering audits the • The participation of specialty disciplines in SEMP Lessons Learned from DoD Experience system decomposition and definition • A well-managed project requires a coordinated • The role of specialty disciplines in verification and Systems Engineering Management Plan that is validation used through the project cycle. • Reliability • A SEMP is a living document that must be up- • Maintainability dated as the project changes and kept consistent • Quality assurance with the Project Plan. • A meaningful SEMP must • Integrated logistics be the product of ex-perts from all areas of the • Human engineering project. • Safety • Projects with little or insufficient systems engi- • Producibility neering discipline generally have major problems. • Survivability/vulnerability • Weak systems engineering, or systems engi- • Environmental assessment neering placed too low in the organization, cannot • Launch approval. perform the functions as required.
• The systems engineering effort must be skillfully 4.2.3 Development of the SEMP managed and well communicated to all project The SEMP must be developed concurrently with the participants. Project Plan. In developing the SEMP, the technical • The systems engineering effort must be respon- approach to the project, and hence the technical sive to both the customer and the contractor in- terests.
NASA Systems Engineering Handbook
Page 26 design and coding process and the design • Identifies the higher level element into which the engineering solutions for compliance to all higher WBS element will be integrated level baselines. In performing this responsibility, • Shows the cost account number of the element. systems engineering must ensure and document
requirements traceability. A WBS should also have a companion WBS Systems engineering is also responsible for dictionary that contains each element's title, the overall management of the integration, identification number, objective, description, and any verification, and validation process. In this role, systems engineering conducts Test Readiness Reviews and ensures that only verified configuration items are integrated into the next higher assembly for further verification. Verification is continued to the system level, after which system validation is conducted to prove compliance with user requirements. Systems engineering also ensures that concurrent engineering is properly applied through the project life cycle by involving the required specialty engineering disciplines. The SEMP is the guiding document for these activities.
4.3 The Work Breakdown Structure
A Work Breakdown Structure (WBS) is a hierarchical breakdown of the work necessary to complete a project. The WBS should be a product- based, hierarchical division of deliverable items and associated services. As such, it should contain the project's Product Breakdown Structure (PBS), with the specified prime product(s) at the top, and the systems, segments, subsystems, etc. at successive lower levels. At the lowest level are products such as hardware items, software items, and information items (documents, databases, etc.) for which there is a cognizant engineer or manager. Branch points in the hierarchy should show how the PBS elements are to be integrated. The WBS is built from the PBS by adding, at each branch point of the PBS, any necessary service elements such as management, systems engineering, integration and verification (I&V), and integrated logistics support (ILS). If several WBS elements require similar equipment or software, then a higher level WBS element might be defined to perform a block buy or a development activity (e.g., "System Support Equipment"). Figure 10 shows the relationship between a .system. a PBS, and a WBS. A project WBS should be carried down to the cost account level appropriate to the risks to be managed. The appropriate level of detail for a cost account is determined by management's desire to have visibility into costs, balanced against the cost of planning and reporting. Contractors may have a Contract WBS (CWBS), which is appropriate to the dependencies (e.g., receivables) on other WBS contractor's needs to control costs. A summary elements. This dictionary provides a structured project CWBS, consisting of the upper levels of the full description that is valuable for Figure 10 -- The CWBS, is usually included in the project WBS to Relationship Between a System, a Product report costs to the contracting organization. Breakdown Structure, and a Work Breakdown WBS elements should be identified by title Structure. orienting project members and other and by a numbering system that performs the interested parties. It fully describes the products following functions: and/or services expected from each WBS element.
This section provides some techniques for developing • Identifies the level of the WBS element a WBS, and points out some mistakes to avoid.
NASA Systems Engineering Handbook
Page 27 Appendix B.2 provides an example of a WBS for an Developing a successful project WBS is airborne telescope that follows the principles of likely to require several iterations through the project product-based WBS development. life cycle since it is not always obvious at the outset
what the full extent of the work may be. Prior to 4.3.1 Role of the WBS developing a preliminary WBS, there should be some
development of the system architecture to the point A product-based WBS is the organizing where a preliminary PBS can be created. The PBS structure for: and associated WBS can then be developed level by
level from the top down. In this approach, a project- • Project and technical planning and scheduling level system engineer finalizes the PBS at the project • Cost estimation and budget formulation. (In level, and provides a draft PBS for the next lower particular, costs collected in a product-based level. The WBS is then derived by adding appropriate WBS can be compared to historical data. This is services such as management and systems identified as a primary objective by DoD engineering to that lower level. This process is standards for WBSs.) repeated recursively until a WBS exists down to the • Defining the scope of statements of work and desired cost account level. specifications for contract efforts An alternative approach is to define all levels • Project status reporting, including schedule, cost, of a complete PBS in one design activity, and then workforce, technical performance, and integrated develop the complete WBS. When this approach is cost/schedule data (such as Earned Value and taken, it is necessary to take great care to develop the estimated cost at completion) PBS so that all products are included, and all • Plans, such as the SEMP, and other assembly/integration and verification branches are documentation products, such as specifications correct. The involvement of people who will be and drawings. responsible for the lower level WBS elements is
recommended. It provides a logical outline and vocabulary that
describes the entire project, and integrates A WBS for a Multiple Delivery Project. There are information in a consistent way. If there is a schedule several terms for projects that provide multiple slip in one element of a WBS, an observer can deliveries, such as: rapid development, rapid determine which other WBS elements are most likely prototyping, and incremental delivery. Such projects to be affected. Cost impacts are more accurately should also have a product-based WBS, but there will estimated. If there is a design change in one element be one extra level in the WBS hierarchy, immediately of the WBS, an observer can determine which other under the final prime product(s), which identifies each WBS elements will most likely be affected, and these delivery. At any one point in time there will be both elements can be consulted for potential adverse active and inactive elements in the WBS. impacts.
A WBS for an Operational Facility. A WBS for 4.3.2 Techniques for Developing the WBS managing an operational facility such as a flight operations center is analogous to a WBS for
developing a system. The difference is that the
NASA Systems Engineering Handbook
Page 28 products in the PBS are not necessarily completed Scheduling is an essential component of once and then integrated, but are produced on a planning and managing the activities of a project. The routine basis. A PBS for an operational facility might process of creating a network schedule can lead to a consist largely of information products or service much better understanding of what needs to be done, products provided to external customers. However, how long it will take, and how each element of the the general concept of a hierarchical breakdown of project WBS might affect other elements. A complete products and/or services would still apply. network schedule can be used to calculate how long it The rules that apply to a development WBS will take to complete a project, which activities also apply to a WBS for an operational facility. The determine that duration (i.e., critical path activities), techniques for developing a WBS for an operational and how much spare time (i.e., float) exists for all the facility are the same, except that services such as other activities of the project. (See sidebar on critical maintenance and user support are added to the PBS, path and float calculation). An understanding of the and services such as systems engineering, project's schedule is a prerequisite for accurate integration, and verification may not be needed. project budgeting.
Keeping track of schedule progress is an 4.3.3 Common Errors in Developing a WBS essential part of controlling the project, because cost
and technical problems often show up first as There are three common errors found in WBSs:\ schedule problems. Because network schedules
show how each activity affects other activities, they • ·Error 1: The WBS describes functions, not prod- are essential for predicting the consequences of ucts. This makes the project manager the only schedule slips or accelerations of an activity on the one formally responsible for products. entire project. Network scheduling systems also help • ·Error 2: The WBS has branch points that are not managers accurately assess the impact of both consistent with how the WBS elements will be technical and resource changes on the cost and integrated. For instance, in a flight operations schedule of a project. system with a distributed architecture, there is
typically software associated with hardware items 4.4.2 Network Schedule Data and Graphical that will be integrated and verified at lower levels Formats of a WBS. It would then be inappropriate to
separate hardware and software as if they were Network schedule data consist of: separate systems to be integrated at the system • Activities level. This would make it difficult to assign • Dependencies between activities (e.g., where an accountability for integration and to identify the activity depends upon another activity for a costs of integrating and testing components of a receivable) system. • Products or milestones that occur as a result of • ·Error 3: The WBS is inconsistent with the PBS. one or more activities This makes it possible that the PBS will not be • Duration of each activity. fully implemented, and generally complicates the
management process. A work flow diagram (WFD) is a graphical display of Some examples of these errors are shown in the first three data items above. A network schedule Figure 11. Each one prevents the WBS from contains all four data items. When creating a network successfully performing its roles in project planning schedule, graphical formats of these data are very and organizing. These errors are avoided by using the useful. Two general types of graphical formats, shown WBS development techniques described above. in Figure 12, are used. One has activities-on-arrows,
with products and dependencies at the beginning and 4.4 Scheduling end of the arrow. This is the typical format of the
Program Evaluation and Review Technique (PERT) Products described in the WBS are the result chart. The second, called precedence diagrams, has of activities that take time to complete. An orderly and boxes that represent activities; dependencies are then efficient systems engineering process requires that shown by arrows. Due to its simpler visual format and these activities take place in a way that respects the reduced requirements on computer resources, the underlying time precedence relationships among precedence diagram has become more common in them. This is accomplished by creating a network recent years. schedule, which explicitly take s into account the The precedence diagram format allows for dependencies of each activity on other activities and simple depiction of the following logical relationships: receivables from outside sources. This section • Activity B begins when Activity A begins (Start- discusses the role of scheduling and the techniques Start, or SS) for building a complete network schedule. • Activity B begins only after Activity A ends (Fin-
ish- Start, or FS) 4.4.1 Role of Scheduling • Activity B ends when Activity A ends (Finish-
Finish, or FF).
Critical Path and Float Calculation The critical path is the sequence of activities that will take the longest to accomplish. Activities that NASA Systems Engineering Handbook
Page are not on the critical path have a certain amoun 29 t of time that they can be delayed until they, too are on a critical path. This time is called float. There
are two types of float, path float and free float. Each of these three activity relationships may be Path float is where a sequence of activities modified by attaching a lag (+ or-) to the relationship, collectively have float. If there is a delay in an as shown in Figure 12. activity in this sequence, then the path float for all It is possible to summarize a number of low-level subsequent activities is reduced by that amount. activities in a precedence diagram with a single Free float exists when a delay in an activity will activity. This is commonly referred to as hammocking. have no effect on any other activity. For example, One takes the initial low-level activity, and attaches a if activity A can be finished in 2 days, and activity summary activity to it using the first relationship B requires 5 days, and activity C requires described above. The summary activity is then completion of both A and B. then A would have 3 attached to the final low-level activity using the third days of free float. relationship described above. Unless one is Float is valuable. Path float should be hammocking, the most common relationship used in conserved where possible, so that a reserve precedence diagrams is the second one mentioned exists for future activities. Conservation is much above. The activity-on-arrow format can represent the less important for free float. identical time-precedence logic as a precedence To determine the critical path, there is first a diagram by creating artificial events and activities as "forward pass" where the earliest start time of needed. each activity is calculated. The time when the last
activity can be completed becomes the end point 4.4.3 Establishing a Network Schedule for that schedule. Then there is a "backward
pass", where the latest possible start point of each Scheduling begins with project-level activity is calculated, assuming that the last schedule objectives for delivering the products activity ends at the end point previously described in the upper levels of the WBS. To develop calculated. Float is the time difference between network schedules that are consistent with the the earliest start time and the latest start time of project's objectives, the following six steps are applied an activity. Whenever this is zero, that activity is to each cost account at the lowest available level of on a critical path. the WBS.
Step 1: Identify activities and dependencies needed to complete each WBS element. Enough • Ensuring that the cost account WBS is extended activities should be identified to show exact schedule downward to describe all significant products!, dependencies between activities and other WBS including documents, reports, hardware and elements. It is not uncommon to have about 100 software items activities identified for the first year of a WBS element • For each product, listing the steps required for its that will require 10 work-years per year. Typically, generation and drawing the process as a work there is more schedule detail for the current year, and flow diagram much less detail for subsequent years. Each year, • Indicating the dependencies among the products, schedules are updated with additional detail for the and any integration and verification steps within the work package.
Step 2: Identify and negotiate external dependencies. External dependencies are any receivables from outside of the cost account, and any deliverables that go outside of the cost account. Informal negotiations should occur to ensure that there is agreement with respect to the content, format, and labeling of products that move across cost account boundaries. This step is designed to ensure that lower level schedules can be integrated. Step 3: Estimate durations of all activities. As-sumptions behind these estimates (workforce, availability of facilities, etc.) should be written down for future reference. Step 4: Enter the schedule data for the WBS element into a suitable computer program to obtain a network schedule and an estimate of the critical path for that element. (There are many commercially available software packages for this function.) This step enables the cognizant engineer, team leader, Figure 12—A current y ctivity ear. Th -on-Arrow and Preced is first step is ence most easily and/or system engineer to review the schedule logic. Diagrams for Ne accomplished by tw : ork Schedules. It is not unusual at this point for some iteration of
NASA Systems Engineering Handbook
Page 30 steps 1 to 4 to be required in order to obtain a Good scheduling systems provide satisfactory schedule. Often too, reserve will be capabilities to show resource requirements over time, added to critical path activities, often in the form of a and to make adjustments so that the schedule is dummy activity, to ensure that schedule commitments feasible with respect to resource constraints over can be met for this WBS element. time. Resources may include workforce level, funding Step 5: Integrate schedules of lower level profiles, important facilities, etc. Figure 14 shows an WBS elements, using suitable software, so that all example of an unleveled resource profile. The dependencies between WBS elements are correctly objective is to move the start dates of tasks that have included in a project network. It is important to include float to points where the resource profile is feasible. If the impacts of holidays, weekends, etc. by this point. that is not sufficient, then the assumed task durations The critical path for the project is discovered at this for resource-intensive activities should be reexamined step in the process. and, accordingly, the resource levels changed. Step 6: Review the workforce level and
funding profile over time, and make a final set of 4.5 Budgeting and Resource Planning adjustments to logic and durations so that workforce
levels and funding levels are reasonable. Adjustments Budgeting and resource planning involves to the logic and the durations of activities may be the establishment of a reasonable project baseline needed to converge to the schedule targets budget, and the capability to analyze changes to that established at the project level. This may include baseline resulting from technical and/or schedule adding more activities to some WBS element, deleting changes. The project's WBS, baseline schedule, and redundant activities, increasing the workforce for budget should be viewed by the system engineer as some activities that are on the critical path, or finding mutually dependent, reflecting the technical content, ways to do more activities in parallel, rather than in time, and cost of meeting the project's goals and series. If necessary, the project level targets may objectives. need to be adjusted, or the scope of the project may The budgeting process needs to take into need to be reviewed. Again, it is good practice to account whether a fixed cost cap or cost profile exists. have some schedule reserve, or float, as part of a risk When no such cap or profile exists, a baseline budget mitigation strategy. is developed from the WBS and network schedule. The product of these last steps is a feasible This specifically involves combining the project's baseline schedule for each WBS element that is workforce and other resource needs with the consistent with the activities of all other WBS appropriate workforce rates and other financial and elements, and the sum of all these schedules is programmatic factors to obtain cost element consistent with both the technical scope and the estimates. These elements of cost include: schedule goals for the project. There should be
enough float in this integrated master schedule so that schedule and associated cost risk are acceptable to the project and to the project's customer. Even when this is done, time estimates for many WBS elements will have been underestimated, or work on some WBS elements will not start as early as had been originally assumed due to late arrival of receivables. Consequently, replanning is almost always needed to meet the project's goals.
4.4.4 Reporting Techniques
Summary data about a schedule is usually described in Gantt charts. A good example of a Gantt chart is shown in Figure 13. (See sidebar on Gantt chart features.) Another type of output format is a table that shows the float and recent changes in float of key activities. For example, a project manager may wish to know precisely how much schedule reserve has been consumed by critical path activities, and whether reserves are being consumed or are being preserved in the latest reporting period. This table provides information on the rate of change of Figure 16 – Characterizing Risks by Likelihood schedule re serve. and Severity.
4.4.5 Resource Leveling
NASA Systems Engineering Handbook
Desirable Features in Gantt Charts T
he Gantt chart shown in Figure 13 (below) illustrates the followi
ng desirable features:
A heading that describes the WBS el
ement, the responsible manager, the date of the baseline used, and the
date that status was reported.
A milestone section in the main body (lines 1 and 2)
An activity section in the main body. Activity data shown includes:
a. WBS elements (lines 3, 5, 8, 12, 16, and 20)
b. Activities (indented from WBS elements)
c. Current plan (shown as thick bars)
d. Baseline plan (same as current plan, or if diffe
rent, represented by thin bars under the thick bars)
e. Status line at the appropriate date
f. Slack for each activity
(dashed lines above the current plan bars)
g. Schedule slips from the baseline (dashed lines below
the milestone on line 12) •
A note section, where the symbols in the main body
can be explained. •
is Gantt chart shows only 23 lines, which is a summary
of the activities currently being worked for this WBS
element. It is appropriate to tailor the amount of detail reported to
those items most pertinent at the time of status
Figure 13 – An Example of a Gantt Chart.
NASA Systems Engineering Handbook
level, budgeting and resource planning must also
ensure that an adequate level of contingency funds • Direct labor costs Overhead costs Other direct are included to deal with unforeseen events. Some costs (travel, data processing, etc.) Subcontract risk management methods are discussed in Section costs 4.6. • Material costs
• General and administrative costs 4.6 Risk Management • Cost of money (i.e., interest payments, if
applicable) Risk management comprises purposeful • Fee (if applicable) thought to the sources, magnitude, and mitigation of • Contingency. risk, and actions directed toward its balanced
reduction. As such, risk management is an integral When there is a cost cap or a fixed cost part of project management, and contributes directly profile, there are additional logic gates that must be to the objectives of systems engineering. NASA policy satisfied before the system engineer can complete the objectives with regard to project risks are expressed budgeting and planning process. A determination in NMI 8070.4A, Risk Management Policy. These are needs to be made whether the WBS and network to: schedule are feasible with respect to mandated cost
caps and/or cost profiles. If not, the system engineer
needs to recommend the best approaches for either • Provide a disciplined and documented approach to stretching out a project (usually at an increase in the risk management throughout the project life cycle total cost), or descoping the project's goals and • Support management decision making by providing objectives, requirements, design, and/or integrated risk assessments (i.e., taking into account implementation approach. (See sidebar on schedule cost, schedule, performance, and safety concerns) slippage.) • Communicate to NASA management the Whether a cost cap or fixed cost profile significance of assessed risk levels and the decisions exists, it is important to control costs after they have made with respect to them. been baselined. An important aspect of cost control is
project cost and schedule status reporting and There are a number of actions the system assessment, methods for which are discussed in engineer can take to effect these objectives. Principal among them is planning and completing a well- Assessing the Effect of Schedule Slippage
conceived risk management program. Such a program encompasses several related activities Certain elements of cost, called fixed costs, are during the systems engineering process. The mainly time related, while others, called variable structure of these activities is shown in Figure 15. costs, are mainly product related. If a project's
schedule is slipped, then the fixed costs of completing it increase. The variable costs remain Risk the same in total (excluding inflation adjustments), The term risk has different meanings depending but are deferred downstream, as in the figure on the context. Sometimes it simply indicates the below. To quickly assess the effect of a simple degree of l variability in the outcome or result of a schedule slippage: particular action. In the context of risk
management during the systems engineering • Convert baseline budget plan from nominal process, the term denotes a combination of both (real-year) dollars to constant dollars the likelihood of various outcomes and their • distinct consequences. The focus, moreover, is Divide baseline budget plan into fixed and generally on undesired or unfavorable outcomes variable costs • such as the risk of a technical failure, or the risk of Enter schedule slip implementation exceeding a cost target. • Compute new variable costs including any work-free disruption costs • Repeat last two steps until an acceptable The first is planning the risk management imple-mentation is achieved program, which should be documented in a risk • Compute new fixed costs management program plan. That plan, which • Sum new fixed and variable costs elaborates on the SEMP, contains: • Convert from constant dollars to nominal
(real-year) dollars. • The project's overall risk policy and objectives • The programmatic aspects, of the risk management activities (i.e., responsibilities, resources, schedules and milestones, etc.) Section 4.9.1 of this handbook. Another is cost and • A description of the methodologies, processes, schedule risk planning, such as developing risk and tools to be used for risk identification and avoidance and work-around strategies. At the project
NASA Systems Engineering Handbook
Page 33 qualitative; hence in systems engineering literature, Figure 15 -- Risk Management Structure this step is sometimes called qualitative risk Diagram. assessment. The output of this step is a list of significant risks (by phase) to be given specific characterization, risk analysis, and risk mitigation management attention. and tracking In some projects, qualitative methods are • A description of the role of risk management with adequate for making risk management decisions; in respect to reliability analyses, formal reviews, others, these methods are not precise enough to and status reporting and assessment understand the magnitude of the problem, or to • Documentation requirements for each risk allocate scarce risk reduction resources. Risk analysis management product and action. is the process of quantifying both the likelihood of
occurrence and consequences of potential future The level of risk management activities events (or "states of nature" in some texts). The should be consistent with the project's overall risk system engineer needs to decide whether risk policy established in conjunction with its NASA identification and characterization are adequate, or Headquarters program office. At present, formal whether the increased precision of risk analysis is guidelines for the classification of projects with needed for some uncertainties. In making that respect to overall risk policy do not exist; such determination, the system engineer needs to balance guidelines exist only for NASA payloads. These are the (usually) higher cost of risk analysis against the promulgated in NMI 8010.1A, Classification of NASA value of the additional information. Pay-loads, Attachment A, which is reproduced as
Appendix B.3. Risk mitigation is the formulation, selection, With the addition of data tables containing and execution of strategies designed to economically the results of the risk management activities, the risk reduce risk. When a specific risk is believed to be management program plan grows into the project's intolerable, risk analysis and mitigation are often Risk Management Plan (RMP). These data tables performed iteratively, so that the effects of alternative should contain the project's identified significant risks. mitigation strategies can be actively explored before For each such risk, these data tables should also one is chosen. Tracking the effectivity of these contain the relevant characterization and analysis strategies is closely allied with risk mitigation. Risk results, and descriptions of the related mitigation and tracking plans (including any descope options and/or required technology developments). A sample RMP outline is shown as Appendix B.4. The technical portion of risk management begins with the process of identifying and characterizing the project's risks. The objective of this step is to understand what uncertainties the project faces, and which among them should be given greater attention. This is accomplished by categorizing (in a consistent manner) uncertainties by their likelihood of occurrence (e.g., high, medium, or low), and separately, according to the severity of their consequences. This categorization forms the basis for ranking uncertainties by their relative riskiness. Uncertainties with both high likelihood and severely adverse consequences are ranked higher than those without these characteristics, as Figure 16 suggests. The primary methods used in this process are
NASA Systems Engineering Handbook
Page 34 Management Guide wraps "funding, schedule, contract relations, and political risks" into the broader category of programmatic risks. While these terms are useful in informal discussions, there appears to be no formal taxonomy free of ambiguities. One reason, mentioned above, is that often one type of risk can be exchanged for another. A second reason is that some of these categories move together, as for example, cost risk and political risk (e.g., the risk of project cancellation). Another way some have categorized risk is by the degree of mathematical predictability in its underlying uncertainty. The distinction has been made between an uncertainty that has a known probability mitigation is often a challenge because efforts and distribution, with known or estimated parameters, and expenditures to reduce one type of risk may increase one in which the underlying probability distribution is another type. (Some have called this the systems either not known, or its parameters cannot be engineering equivalent of the Heisenberg Uncertainty objectively quantified. Principle in quantum mechanics.) The ability (or An example of the first kind of uncertainty necessity) to trade one type of risk for another means occurs in the unpredictability of the spares upmass that the project manager and the system engineer requirement for alternative Space Station Alpha need to understand the system-wide effects of various designs. While the requirement is stochastic in any strategies in order to make a rational allocation of particular logistics cycle, the probability distribution resources. can be estimated for each design from reliability Several techniques have been developed for theory and empirical data. Examples of the second each of these risk management activities. The kind of uncertainty occur in trying to predict whether a principal ones, which are shown in Table 1, are Shuttle accident will make resupply of Alpha discussed in Sections 4.6.2 through 4.6.4. The impossible for a period of time greater than x months, system engineer needs to choose the techniques that or whether life on Mars exists. Modem subjectivist best fit the unique requirements of each project. (also known as Bayesian) probability theory holds that A risk management program is needed the probability of an event is the degree of belief that throughout the project life cycle. In keeping with the a person has that it will occur, given his/her state of doctrine of successive refinement, its focus, however, information. As that information improves (e.g., moves from the "big picture" in the early phases of the through the acquisition of data or experience), the project life cycle (Phases A and B) to more specific subjectivist's estimate of a probability should issues during design and development (Phases C and converge to that estimated as if the probability D). During operations (Phase E), the focus changes distribution were known. In the examples of the again. A good risk management program is always previous paragraph, the only difference is the forward-looking. In other words, a risk management probability estimator's perceived state of information. program should address the project's on-going risk Consequently, subjectivists find the distinction issues and future uncertainties. As such, it is a natural between the two kinds of uncertainty of little or no part of concurrent engineering. The RMP should be practical significance. The implication of the updated throughout the project life cycle. subjectivist's view for risk management is that, even
with little or no data, the system engineer's subjective 4.6.1 Types of Risks probability estimates form a valid basis for risk
decision making. There are several ways to describe the
various types of risk a project manager/system 4.6.2 Risk Identification and Characterization engineer faces. Traditionally, project managers and Techniques system engineers have attempted to divide risks into
three or four broad categories — namely, cost, A variety of techniques are available for risk schedule, technical, and, sometimes, safety (and/or identification and characterization. The thoroughness hazard) risks. More recently, others have entered the with which this step is accomplished is an important lexicon, including the categories of organizational, determinant of the risk management program's management, acquisition, supportability, political, and success. programmatic risks. These newer categories reflect
the expanded set of concerns of project managers Expert Interviews. When properly conducted, expert and system engineers who must operate in the in-terviews can be a major source of insight and current NASA environment. Some of these newer information on the project's risks in the expert's area categories also represent supersets of other of knowledge One key to a successful interview is in categories. For example, the Defense Systems identifying an ex pert who is close enough to a risk Management College (DSMC) Systems Engineering issue to understand it thoroughly, and at the same
NASA Systems Engineering Handbook
Page 35 time, able (and willing) to step back and take an FMECAs, FMEAs, Digraphs, and Fault Trees. objective view of the probabilities and consequences. Failure Modes, Effects, and Criticality Analysis A second key to success is advanced preparation on (FMECA), Failure Modes and Effects Analysis the part of the interviewer. This means having a list of (FMEA), digraphs, and fault trees are specialized risk issues to be covered in the interview, developing techniques for safety (and/or hazard) risk a working knowledge of these issues as they apply to identification and characterization. These techniques the project, and developing methods for capturing the focus on the hardware components that make up the information acquired during the interview. system. According to MIL-STD-1629A, FMECA is "an Initial interviews may yield only qualitative ongoing procedure by which each potential failure in a infor-mation, which should be verified in follow-up system is analyzed to determine the results or effects rounds. Expert interviews are also used to solicit thereof on the system, and to classify each potential quantitative data and information for those risk issues failure mode according to its severity." Failures are that qualitatively rank high. These interviews are often generally classified into four seventy categories: the major source of inputs to risk analysis models built
using the techniques described in Section 4.6.3. • Category I—Catastrophic failure (possible death
or system loss) Independent Assessment. This technique can take • Category II—Critical failure (possible major in- several forms. In one form, it can be a review of jury or system damage) project documentation, such as Statements of Work, • Category III—Major failure (possible minor injury acquisition plans, verification plans, manufacturing or mission effectiveness degradation) plans, and the SEMP. In another form, it can be an • Category IV — Minor failure (requires system evaluation of the WBS for completeness and maintenance, but does not pose a hazard to consistency with the project's schedules. In a third personnel or mission effectiveness). form, an independent assessment can be an
independent cost (and/or schedule) estimate from an A complete FMECA also includes an outside organization. estimate of the probability of each potential failure.
These prob-abilities are usually based, at first, on Risk Templates. This technique consists of subjective judgment or experience factors from similar examining and then applying a series of previously kinds of hardware components, but may be refined developed risk templates to a current project. Each from reliability data as the system development template generally covers a particular risk issue, and progresses. An FMEA is similar to an FMECA, but then describes methods for avoiding or reducing that typically there is less emphasis on the severity risk. The most-widely recognized series of templates classification portion of the analysis. Digraph analysis appears in DoD 4245.7-M, Transition from is an aid in determining fault tolerance, propagation, Development to Production ...Solving the Risk and reliability in large, interconnected systems. Equation. Many of the risks and risk responses Digraphs exhibit a network structure and resemble a described are based on lessons reamed from DoD schematic diagram. The digraph technique permits programs, but are general enough to be useful to the integration of data from a number of individual NASA projects. As a general caution, risk templates FMECAs/FMEAs, and can be translated into fault cannot provide an exhaustive list of risk issues for trees, described in Section 6.2, if quantitative every project, but they are a useful input to risk probability estimates am needed. identification.
4.6.3 Risk Analysis Techniques Lessons Learned. A review of the lessons learned
files, data, and reports from previous similar projects The tools and techniques of risk analysis rely can produce insights and information for risk heavily on the concept and "laws" (actually, axioms identification on a new project. For technical risk and theorems) of probability. The system engineer identification, as an example, it makes sense to needs to be familiar with these in order to appreciate examine previous projects of similar function, the full power and limitations of these techniques. The architecture, or technological approach. The lessons products of risk analyses are generally quantitative learned from the Infrared Astronomical Satellite probability and consequence estimates for various (IRAS) project might be useful to the Space Infrared outcomes, more detailed understanding of the Telescope Facility (SIRTF) project, even though the dominant risks, and improved capability for allocating latter's degree of complexity is significantly greater. risk reduction resources. The key to ap-plying this technique is in recognizing
what aspects are analogous in two projects, and what Decision Analysis. Decision analysis is one data are relevant to the new project. Even if the technique to help the individual decision maker deal documented lessons learned from previous projects with a complex set of uncertainties. Using the divide- are not applicable at the system level, there may be and-conquer approach common to much of systems valuable data applicable at the subsystem or engineering, a complex uncertainty is decomposed component level. into simpler ones, which are then treated separately.
The decomposition continues until it reaches a level
NASA Systems Engineering Handbook
Page 36 at which either hard information can be brought to Probabilistic Risk Assessment Pitfalls bear, or intuition can function effectively. The
decomposition can be graphically represented as a Risk is generally defined in a probabilistic risk decision tree. The branch points, called nodes, in a assess-ment (PRA) as the expected value of a decision tree represent either decision points or consequence function—that is: chance events. Endpoints of the tree are the potential R = Σ PS CS S outcomes. (See the sidebar on a decision tree where PS is the probability of outcome s, and CS is example for Mars exploration.) the consequence of outcome s. To attach In most applications of decision analysis, probabilities to outcomes, event trees and fault these outcomes are generally assigned dollar values. trees are developed. These techniques have been From the probabilities assigned at each chance node used since 1953, but by the late 1970s, they were and the dollar value of each outcome, the distribution under attack by PRA practitioners. The reasons of dollar values (i.e., consequences) can be derived include the following: for each set of decisions. Even large complex
decision trees can be represented in currently • Fault trees are limiting because a complete available decision analysis software. This software set of failures is not definable. can also calculate a variety of risk measures. • Common cause failures could not be In brief, decision analysis is a technique that captured properly. An example of a common allows: cause fail-ure is one where all the valves in a
system have a defect so that their failures are • A systematic enumeration of uncertainties and not truly inde-pendent. encoding of their probabilities and outcomes • PRA results are sometimes sensitive to • An explicit characterization of the decision simple changes in event tree assumptions maker's attitude toward risk, expressed in terms • Stated criteria for accepting different kinds of of his, her risk aversion risks are often inconsistent, and therefore not • A calculation of the value of "perfect information," appropriate for allocating risk reduction re- thus setting a normative upper bound on sources. information-gathering expenditures • Many risk-related decisions are driven by per- • Sensitivity testing on probability estimates and ceptions, not necessarily objective risk as outcome dollar values. defined by the above equation. Perceptions
of consequences tend to grow faster than the Probabilistic Risk Assessment (PRA). A PRA con-sequences themselves—that is, several seeks to measure the risk inherent in a system's small accidents are not perceived as strongly design and operation by quantifying both the as one large one, even if fatalities are likelihood of various possible accident sequences and identical. their consequences. A typical PRA application is to • There are difficulties in dealing with determine the risk associated with a specific nuclear incommen-surables, as for example, lives vs. power plant. Within NASA, PRAs are used to dollars. demonstrate, for example, the relative safety of launching spacecraft containing RTGs (Radioisotope Thermoelectric Generators). American Nuclear Society and Institute of Electrical The search for accident sequences is and Electronic Engineers (IEEE). facilitated by event trees, which depict initiating
events and combinations of system successes and Probabilistic Network Schedules. Probabilistic failures, and fault trees, which depict ways in which network schedules, such as PERT (Program the system failures represented in an event tree can Evaluation and Review Technique), permit the occur. When integrated, an event tree and its duration of each activity to be treated as a random associated fault tree(s) can be used to calculate the variable. By supplying PERT with the minimum, probability of each accident sequence. The structure maximum, and most likely duration for each activity, a and mathematics of these trees is similar to that for probability distribution can be computed for project decision trees. The consequences of each accident completion time. This can then be used to determine, sequence are generally measured both in terms of for example, the chances that a project (or any set of direct economic losses and in public health effects. tasks in the network) will be completed by a given (See sidebar on PRA pitfalls.) date. In this probabilistic setting, however, a unique Doing a PRA is itself a major effort, requiring critical path may not exist. Some practitioners have a number of specialized skills other than those also cited difficulties in obtaining meaningful input provided by reliability engineers and human factors data for probabilistic network schedules. A simpler engineers. PRAs also require large amounts of alternative to a full probabilistic network schedule is to system design data at the component level, and perform a Monte Carlo simulation of activity durations operational procedures data. For additional along the project's critical path. (See Section 5.4.2.) information on PRAs, the system engineer can
reference the PRA Procedures Guide (1983) by the
NASA Systems Engineering Handbook
Page 37 Probabilistic Cost and Effectiveness Models. further risk information gathering and assessments.) These models offer a probabilistic view of a project's Second, a risk can sometimes be shared with a co- cost and effectiveness outcomes. (Recall Figure 2.) participant—that is, with a international partner or a This approach explicitly recognizes that single point contractor. In this situation, the goal is to reduce values for these variables do not adequately NASA's risk independent of what happens to total represent the risk conditions inherent in a project. risk, which may go up or down. There are many ways These kinds of models are discussed more to share risks, particularly cost risks, with contractors. completely in Section 5.4. These include various incentive contracts and
warranties. The third and fourth responses require 4.6.4 Risk Mitigation and Tracking that additional specific planning and actions be Techniques undertaken.
Typical technical risk mitigation actions Risk identification and characterization and include additional (and usually costly) testing of risk analysis provide a list of significant project risks subsystems and systems, designing in redundancy, that re-quire further management attention and/or and building a full engineering model. Typical cost risk action. Because risk mitigation actions are generally mitigation actions include using off-the-shelf hardware not costless, the system engineer, in making and, according to Figure 6, providing sufficient recommendations to the project manager, must funding during Phases A and B. Major supportability balance the cost (in resources and time) of such risk mitigation actions include providing sufficient actions against their value to the project. Four initial spares to meet the system's availability goal and responses to a specific risk are usually available: (1) a robust resupply capability (when transportation is a deliberately do nothing, and accept the risk, (2) share significant factor). For those risks that cannot be the risk with a co-participant, (3) take preventive mitigated by a design or management approach, the action to avoid or reduce the risk, and (4) plan for system engineer should recommend the contingent action. establishment of reasonable financial and schedule The first response is to accept a specific risk contingencies, and technical margins. consciously. (This response can be accompanied by Whatever strategy is selected for a specific risk, it and its underlying rationale should be documented in a
NASA Systems Engineering Handbook
Page 38 risk mitigation plan, and its effectivity should be combined effects of likelihood and consequences tracked through the project life cycle, as required by are significant, should be given specific NMI 8070.4A. The techniques for choosing a management attention. Reviews conducted (preferred) risk mitigation strategy are discussed in throughout in the project life cycle should help to Chapter 5, which deals with the larger role of trade force out risk issues. studies and system modeling in general. Some • Apply qualitative and quantitative techniques to techniques for planning and tracking are briefly understand the dominant risks and to improve the mentioned here. allocation of risk reduction resources; this may
include the development of project-specific risk Watchlists and Milestones. A watchlist is a analysis models such as decision trees and compilation of specific risks, their projected PRAs. consequences, and early indicators of the start of the • Formulate and execute a strategy to handle each problem. The risks on the watchlist are those that risk, including establishment, where appropriate, were selected for management attention as a result of of reasonable financial and schedule completed risk management activities. A typical contingencies and technical margins watchlist also shows for each specific risk a triggering • Track the effectivity of each risk mitigation strat- event or missed milestone (for example, a delay in the egy. delivery of long lead items), the related area of impact
(production schedule), and the risk mitigation Good risk management requires a team effort - strategy, to be used in response. The watchlist is that is, system engineers and managers at all levels periodically reevaluated and items are added, of the project need to be involved. However, risk modified, or deleted as appropriate. Should the management responsibilities must be assigned to triggering event occur, the projected consequences specific individuals. Successful risk management should be updated and the risk mitigation strategy practices often evolve into in stitutional policy. revised as needed.
4.7 Configuration Management Contingency Planning, Descope Planning, and
Parallel Development. These techniques are Configuration management is the discipline generally used in conjunction with a watchlist. The of identifying and formalizing the functional and focus is on developing credible hedges and work- physical characteristics of a configuration item at arounds, which are activated upon a triggering event. discrete points in the product evolution for the To be credible, hedges often require that additional purpose of maintaining the integrity of the product resources be expended, which provide a return only if system and controlling changes to the baseline. The the triggering event occurs. In this sense, these baseline for a project contains all of the technical techniques and resources act as a form of project requirements and related cost and schedule insurance. (The term contingency here should not be requirements that are sufficiently mature to be confused with the use within NASA of the same term accepted and placed under change control by the for project-held reserves.) NASA project manager. The project baseline consists
of two parts: the technical baseline and the business Critical Items/Issues Lists. A Critical Items/Issues baseline. The system engineer is responsible for List (CIL) is similar to a watchlist, and has been managing the technical baseline and ensuring that it extensively used on the Shuttle program to track is consistent with the costs and schedules in the items with significant system safety consequences. business baseline. Typically, the project control office An example is shown as Appendix B.5. manages the business baseline. Configuration
management requires the formal agreement of both C/SCS and TPM Tracking. Two very important risk the buyer and the seller to proceed according to the tracking techniques—cost and schedule control up-to-date, documented project requirements (as they systems (C/SCS) and Technical Performance exist at that phase in the project life cycle), and to Measure (TPM) tracking—are discussed in Sections change the baseline requirements only by a formal 4.9.1 and 4.9.2, respectively. configuration control process. The buyer might be a
NASA pro gram office or an external funding agency. 4.6.5 Risk Management: Summary For example, the buyer for the GOES project is
NOAA, and the seller is the NASA GOES project Uncertainty is a fact of life in systems engineering. To office. management must be enforced at all levels; in deal with it effectively, the risk manager needs a the next level for this same example, the NASA disciplined approach. In a project setting, a good- GOES project office is the buyer and the seller is the practice approach includes efforts to: contractor, the Loral GOES project office.
Configuration management is established through • Plan, document, and complete a risk program/project requirements documentation and, management program where applicable, through the contract Statement of • Identify and characterize risks for each phase of Work. the project; high risks, those for which the
NASA Systems Engineering Handbook
Page 39 Configuration management is essential to sound. These early detailed studies and tests must be conduct an orderly development process, to enable documented and retained in the project archives, but the modification of an existing design, and to provide they are not part of the technical baseline. for later replication of an existing design.
Configuration management often provides the 4.7.2 Techniques of Configuration information needed to track the technical progress of Management the project since it manages the project's
configuration documentation. (See Section 4.9.2 on The techniques of configuration management include Technical Performance Measures.) The project's configuration (or baseline) identification, configuration approach to configuration management and the control, configuration verification, and configuration methods to be used should be documented in the accounting (see Figure 18). project's Configuration Management Plan. A sample outline for this plan is illustrated in Appendix B.6. The plan should be tailored to each project's specific needs and resources, and kept current for the entire project life cycle.
4.7.1 Baseline Evolution
The project-level system engineer is responsible for ensuring the completeness and technical integrity of the technical baseline. The technical baseline includes:
• Functional and performance requirements (or specifications) for hardware, software, information items, and processes • Interface requirements • Specialty engineering requirements • • Verification requirements • Data packages, documentation, and drawing trees Applicable engineering standards. The project baseline evolves in discrete steps through the project life cycle. An initial baseline may be established when the top-level user requirements expressed in the Mission Needs Statement are placed under configuration control. At each interphase control gate, increased technical detail is added to the maturing baseline. For a typical project, there are five sequential technical
• Functional baseline at System Requirements Configuration Identification. Configuration Review (SRR) identification of a baseline is accomplished by • "Design-to" baseline at Preliminary Design creating and formally releasing documentation that Review (PDR) describes the baseline to be used, and how changes • "Build-to" (or "code-to") baseline at the Critical to that baseline will be accounted for, controlled, and Design Review (CDR) • "As-built" (or ''as-coded,,) released. Such documentation includes requirements baseline at the System Acceptance Review (product, process, and material), specifications, (SAR) drawings, and code listings. Configuration • "As-deployed" baseline at Operational Readiness documentation is not formally considered part of the Review (ORR). technical baseline until approved by control gate
action of the buyer. The evolution of the five baselines is An important part of configuration illustrated in Figure 17. As discussed in Section 3.7.1, identification is the physical identification of individual only decisions made along the core of the "vee" in configuration items using part numbers, serial Figure 7 are put under configuration control and numbers, lot numbers, version numbers, document included in the approved baseline. Systems analysis, control numbers, etc. risk management, and development test activities (off
the core of the vee) must begin early and continue Configuration Control. Configuration control is the throughout the decomposition process of the project process of controlling changes to any approved life cycle to prove that the core-level decisions are baseline by formal action of a configuration control board (CCB). This area of configuration management
NASA Systems Engineering Handbook
Page 40 is usually the most visible to the system engineer. In The system engineer should also ensure that the large programs/projects, configuration control is active approved baseline is communicated in a timely accomplished by a hierarchy of configuration control manner to all those relying on it. This communication boards, reflecting multiple levels of control. Each keeps project teams apprised as to the distinction configuration control board has its own areas of between what is frozen under formal change control control and responsibilities, which are specified in the and what can still be decided without configuration Configuration Management Plan. control board approval. Typically, a configuration control board Configuration control is essential at both the meets to consider change requests to the business or contractor and NASA field center levels. Changes technical baseline of the program/project. The determined to be Class I to the contractor must be program/project manager is usually the board chair, referred to the NASA project manager for resolution. who is the sole decision maker. The configuration This process is described in Figure 19. The use of a manager acts as the board secretary, who skillfully preliminary Engineering Change Proposal (ECP) to guides the process and records the official events of forewarn of an impending change provides the project the process. In a configuration control board forum, a manager with sufficient preliminary information to number of issues should be addressed: determine whether the contractor should spend NASA
contract funds on a formal ECP. This technique is • What is the proposed change? designed to save significant contract dollars. • What is the reason for the change? Class 1 changes affect the approved baseline • What is the design impact? and hence the product version identification. Class 2 • What is the effectiveness or performance impact? changes are editorial changes or internal changes not • What is the schedule impact? "visible" to the external interfaces. Class 2 changes • at is the risk of making the change? are dispositioned by the contractor's CCB and do not • What is the impact on operations? require the NASA project manager's approval. • What is the impact to support equipment and Overly formalized systems can become so services? burdensome that members of the project team may • What is the impact on spares requirements? • What is the effectivity of the change? Configuration Control Board Conduct • What documentation is affected by the change?
• Objective: To review evaluations, and then Is the buyer supportive of the change? approve or disapprove proposed changes to the
project's technical or business baselines. A review of this information should lead to a well- Participants: Project manager (chair), project- informed decision. When this information is not level system engineer, managers of each affected available to the configuration control board, organization, configuration manager (secretary), unfounded decisions are made, often with negative presenters. consequences to the program or project. Format: Presenter covers recommended change Once a baseline is placed under configuration and discusses related system impact. The control, any change requires the approval of the presentation is reviewed by the system engineer configuration control board. The project manager for completeness prior to presentation. chairs the configuration control board, while the Decision: The CCB members discuss the system engineer or configuration manager is Change Request (CR) and formulate a decision. responsible for reviewing all material for Project manager agrees or overrides. The completeness before it is presented to the board, and secretary prepares a CCB directive, which records for ensuring that all affected organizations are and directs the CR's disposition. represented in the configuration control board forum.
NASA Systems Engineering Handbook
Page 41 try to circumvent the process. It is essential that the on two important examples: the Physical formality of the change process be appropriately Configuration Audit and the Design Certification tailored to the needs of each project. However, there Review.) Each of these serves to review and must always be effective configuration control on challenge the data presented for conformance to the every project. previously approved baseline. For software projects, it is routine to use version Configuration Accounting. Configuration accounting control for both pre-release and post-release (sometimes called configuration status accounting) is deliverable systems. It is equally important to maintain the task of maintaining, correlating, releasing, version con-for hardware-only systems. reporting, and storing configuration data. Essentially a Approved changes on a development project that data management function, configuration accounting has only one deliverable obviously are only applicable ensures that official baseline data is retained, to that one deliverable item. However, for projects that available, and distribution-controlled for project use. It have multiple deliverables of "identical" design, also performs the important function of tracking the changes may become effective on the second or status of each change from inception through subsequent production articles. In such a situation, implementation. A project's change status system the configuration control board must decide the should be capable of identifying each change by its effectivity of the change, and the configuration control unique change identification number (e.g., ECRs, system must maintain version control and CRs, RlDs, waivers, deviations, modification kits) and identification of the "as-built" configuration for each report its current status. article. Incremental implementation of changes is
common in projects that have a deliberate policy of The Role of the Configuration Manager. The introducing product or process improvements. As an configuration manager is responsible for the example, the original 1972 plan held that each of the application of these techniques. In doing so, the Space Shuttle orbiters would be identical. In reality, configuration manager performs the following each of the orbiters is different, driven primarily by the functions: desire to achieve the original payload requirement of
65,000 pounds. Proper version control documentation • Conceives and manages the configuration has been essential to the sparing, fielding, and management system, and documents it in the maintenance of the operational fleet. Configuration Management Plan Configuration Verification. Configuration verification • Acts as secretary of the configuration control is the process of verifying that resulting products (e.g., board (controls the change approval process) hardware and software items) conform to the • Controls changes to baseline documentation intentions of the designers and to the standards Controls release of baseline documentation established by preceding approved baselines, and • Initiates configuration verification audits. that baseline documentation is current and accurate.
Configuration verification is accomplished by two 4.7.3 Data Management types of control gate activity: audits and technical
reviews. (See Section 4.8.4 for additional information
NASA Systems Engineering Handbook
Page 42 For any project, proper data management is scheduled to communicate an approach, demonstrate essential for successful configuration management. an ability to meet requirements, or establish status. Before a project team can produce a tangible product, Reviews help to develop a better understanding it must produce descriptions of the system using among task or project participants, open words, drawings, schematics, and numbers (i.e., communication channels, alert participants and symbolic information). There are several vital management to problems, and open avenues for characteristics the symbolic information must have. solutions. First the information must be shareable. Whether it is The purpose of an audit is to provide NASA in electronic or paper form, the data must be readily management and its contractors a thorough available, in the most recently approved version, to all examination of adherence to program/project policies, members of the project team. plans, requirements, and specifications. Audits are Second, symbolic information must be the systematic examination of tangible evidence to durable. This means that it must be recalled determine adequacy, validity, and effectiveness of the accurately every time and represent the most current activity or documentation under review. An audit may version of the baseline. The baseline information examine documentation of policies and procedures, cannot change or degrade with repeated access of as well as verify adherence to them. the database or paper files, and cannot degrade with The purpose of a control gate is to provide a time. This is a non-trivial statement, since poor data scheduled event (either a review or an audit) that management practices (e.g., allowing someone to NASA management will use to make program or borrow the only copy of a document or drawing) can project go/no-go decisions. A control gate is a allow controlled information to become lost. Also, the material must be retained for the life of the Project Termination program/project (and possibly beyond), and a It should be noted that project termination, while complete set of documentation for each baseline usually disappointing to project personnel, may be change must be retained. a proper reaction to changes in external Third, the symbolic information must be conditions or to an improved understanding of the traceable upward and downward. A database must be system's projected cost-effectiveness. developed and maintained to show the parentage of any requirement. The database must also be able to management event in the project life cycle that is of display all children derived from a given requirement. sufficient importance to be identified, defined, and Finally, traceability must be provided to reports that included in the project schedule. It re-quires formal document trade study results and other decisions that examination to evaluate project status and to obtain played a key role in the flowdown of requirements. approval to proceed to the next management event The data management function therefore according to the Program/Project Plan. - encompasses managing and archiving supporting
analyses and trade study data, and keeping them 4.8.2 General Principles for Reviews convenient for configuration management and general
project use. Review Boards. The convening authority, which
super-vises the manager of the activity being 4.8 Reviews, Audits, and Control Gates reviewed, normally appoints the review board chair.
Unless there are compelling technical reasons to the The intent and policy for reviews, audits, and control contrary, the chair should not be directly associated gates should be developed during Phase A and with the project or task under review. The convening defined in the Program/Project Plan. The specific authority also names the review board members. The implementation of these activities should be majority of the members should not be directly consistent with the types of reviews and audits associated with the program or project under review. described in this section, and with the NASA
Program/Project Life Cycle chart (see Figure 5) and Internal Reviews. During the course of a project or the NASA Program/Project Life Cycle Process Flow task, it is necessary to conduct internal reviews that chart (see Figure 8). However, the timing of reviews, present technical approaches, trade studies, audits, and control gates should be tailored to each analyses, and problem areas to a peer group for specific project. evaluation and comment. The timing, participants,
and content of these reviews is normally defined by 4.8.1 Purpose and Definitions the project manager or the manager of the performing
organization. Internal reviews are also held prior to The purpose of a review is to furnish the forum and participation in a formal control gate review. process to provide NASA management and their Internal reviews provide an excellent means contractors assurance that the most satisfactory for controlling the technical progress of the project. approach, plan or design has been selected, that a They also should be used to ensure that all interested configuration item has been produced to meet the parties are involved in the design and development specified requirements, or that a configuration item is early on and throughout the process. Thus, ready. Reviews (technical or management) are representatives from areas such as manufacturing
NASA Systems Engineering Handbook
Page 43 and quality assurance should attend the internal chair submits, on a timely basis, a written report, reviews as active participants. They can then, for including recommendations for action, to the example, ensure that the design is producible and convening authority with copies to the cognizant that quality is managed through the project life cycle. managers. In addition, some organizations utilize a Red
Team. This is an internal, independent, peer-level Standing Review Boards. Standing review boards review conducted to identify any deficiencies in are selected for projects or tasks that have a high requests for proposals, proposal responses, level of activity, visibility, and/or resource documentation, or presentation material prior to its requirements. Selection of board members by the release. The project or task manager is responsible convening authority is generally made from senior for establishing the Red Team membership and for field center technical and management staff. deciding which of their recommendations are to be Supporting members or advisors may be added to the implemented. board as required by circumstances. If the review
board is to function over the life of a project, it is Review Presentation Material. Presentations using advisable to select extra board members and rotate existing documentation such as specifications, active assignments to cover needs. drawings, analyses, and reports may be adequate.
Copies of any prepared materials (such as 4.8.3 Major Control Gates viewgraphs) should be provided to the review board
and meeting attendees. Background information and This section describes the purpose, timing, review presentation material of use to board members objectives, success criteria, and results of the major should be distributed to the members early enough to control gates in the NASA project life cycle. This enable them to examine it prior to the review. For information is intended to provide guidance to project major reviews, this time may be as long as 30 managers and system engineers, and to illustrate the calendar days. progressive maturation of review activities and
systems engineering products. The checklists Review Conduct. All reviews should consist of oral provided below aid in the preparation of specific presentations of the applicable project requirements review entry and exit criteria, but do not take their and the approaches, plans, or designs that satisfy place. To minimize extra work, review material should those requirements. These presentations normally are be keyed to project documentation. given by the cognizant design engineer or his/her
immediate supervisor. Mission Concept Review. It is highly recommended that in addition to Purpose—The Mission Concept Review the review board, the review audience include project (MCR) affirms the mission need, and examines the personnel (NASA and contractor) not directly proposed mission's objectives and the concept for associated with the design being reviewed. This is meeting those objectives. It is an internal review that required to utilize their cross-discipline expertise to usually occurs at the cognizant NASA field center. identify any design shortfalls or recommend design Timing—Near the completion of a mission improvements. The review audience should also feasibility study. include non-project specialists in the area under Objectives—The objectives of the review review, and specialists in production/fabrication, are to: testing, quality assurance, reliability, and safety.
Some reviews may also require the presence of both • Demonstrate that mission objectives are the contractor's and NASA's contracting officers. complete and understandable Prior to and during the review, board • Confirm that the mission concepts demonstrate members and review attendees may submit requests technical and programmatic feasibility of meeting for action or engineering change requests (ECRs) that the mission objectives document a concern, deficiency, or recommended • Confirm that the customer's mission need is clear improvement in the presented approach, plan, or and achievable design. Following the review, these are screened by • Ensure that prioritized evaluation criteria are the review board to consolidate them, and to ensure provided for subsequent mission analysis. that the chair and cognizant manager(s) understand
the intent of the requests. It is the responsibility of the Criteria for Successful Completion —The review board to ensure that adequate closure following items compose a checklist to aid in responses for each of the action requests are determining readiness of MCR product preparation: obtained.
• Are the mission objectives clearly defined and Post Review Report. The review board chair has the stated? Are they unambiguous and internally responsibility to develop, where necessary, a consistent? consensus of the findings of the board, including an • Will satisfaction of the preliminary set of assessment of the risks associated with problem requirements provide a system which will meet areas, and develop recommendations for action. The mission objectives?
NASA Systems Engineering Handbook
Page 44 • Is the mission feasible? Has there been a the remaining trades been identified to select the solution identified which is technically feasible? Is final system design? the rough cost estimate within an acceptable cost • Are the upper levels of the system PBS range? completely defined? • Have the concept evaluation criteria to be used in • Are the decisions made as a result of the trades candidate system evaluation been identified and consistent with the evaluation criteria established prioritized? at the MCR? • Has the need for the mission been clearly identi- • Has an optimal final design converged to a few fied? alternatives? • Are the cost and schedule estimates credible? • Have technology risks been identified and have • Was a technology search done to identify existing mitigation plans been developed? assets or products that could satisfy the mission
or parts of the mission? Results of Review—A successful MDR supports
the decision to further develop the system Results of Review—A successful MCR supports architecture/design and any technology needed to the determination that the proposed mission meets accomplish the mission. The results reinforce the the customer need, and has sufficient quality and mission's merit and provide a basis for the system merit to support a field center management decision acquisition strategy. to propose further study to the cognizant NASA
Program Associate Administrator (PAA) as a System Definition Review. candidate Phase A effort. Purpose—The System Definition Review
(SDR) examines the proposed system Mission Definition Review. architecture/design and the flowdown to all functional Purpose—The Mission Definition Review (MDR) elements of the system. examines the functional and performance Timing—Near the completion of the system requirements defined for the system and the definition stage. It represents the culmination of preliminary program/project plan, and assures that the efforts in system requirements analysis and requirements and the selected architecture/design will allocation. satisfy the mission. Objectives—The objectives of the SDR are Timing—Near the completion of the mission to: definition stage.
Objectives—The objectives of the review are to: • Demonstrate that the architecture/design is
acceptable. that requirements allocation is • Establish that the allocation of the functional complete, and that a system that fulfills the system requirements is optimal for mission mission objectives can be built within the satisfaction with respect to requirements trades constraints posed and evaluation criteria that were internally • Ensure that a verification concept and preliminary established at MCR verification program are defined • Validate that system requirements meet mission • Establish end item acceptance criteria objectives • Identify technology risks and the • Ensure that adequate detailed information exists plans to mitigate those risks to support initiation of further development or • Present refined cost, schedule, and personnel acquisition efforts. resource estimates.
Criteria for Successful Completion —The fol- Criteria for Successful Completion —The fol- lowing items compose a checklist to aid in lowing items compose a checklist to aid in determining readiness of SDR project preparation: determining readiness of MDR product preparation:
• Will the top-level system design selected meet • Do the defined system requirements meet the the system requirements, satisfy the mission mission objectives expressed at the start of the objectives, and address operational needs? program/project? • Can the top-level system design selected be built • Are the system-level requirements complete, within cost constraints and in a timely manner? consistent, and verifiable? Have preliminary Are the cost and schedule estimates valid in view allocations been made to lower levels? of the system requirements and selected • Have the requirements trades converged on an architecture? Have all the system-level optimal set of system requirements? Do the requirements been allocated to one or more trades address program/project cost and lower levels? schedule constraints as wel1 as mission • Have the major design issues for the elements technical needs? Do the trades cover a broad and subsystems been identified? Have major risk spectrum of options? Have the trades identified areas been identified with mitigation plans? for this set of activities been completed? Have
NASA Systems Engineering Handbook
Page 45 • Have plans to control the development and • Are all Cl "design-to" specifications complete and design process been completed? ready for formal approval and release? • Is a development verification/test plan in place to • Has an acceptable operations concept been provide data for making informed design developed? decisions? Is the minimum end item product • Does the proposed design satisfy requirements performance documented in the acceptance critical to human safety and mission success? criteria? • Do the human factors considerations of the pro- • Is there sufficient information to support proposal posed design support the intended end users' efforts? Is there a complete validated set of ability to operate the system and perform the requirements with sufficient system definition to mission effectively? support the cost and schedule estimates? • Have the production, verification, operations, and
other specialty engineering organizations Results of Review—As a result of successful reviewed the design? completion of the SDR, the system and its operation • Is the proposed design producible? Have long are well enough understood to warrant design and lead items been considered? acquisition of the end items. Approved specifications • Do the specialty engineering program plans and for the system, its segments, and preliminary design specifications provide sufficient guidance, specifications for the design of appropriate functional constraints, and system requirements for the elements may be released. A configuration design engineers to execute the design? management plan is established to control design and • Is the reliability analysis based on a sound requirement changes. Plans to control and integrate methodology, and does it allow for realistic the expanded technical process are in place. logistics planning and life-cycle cost analysis?
• Are sufficient project reserves and schedule slack Preliminary Design Review. The Preliminary Design available to proceed further? Review (PDR) is not a single review but a number of
reviews that includes the system PDR and PDRs Results of Review — As a result of conducted on specific Configuration Items ( CIs). successful completion of the PDR, the "design-to" Purpose—The PDR demonstrates that the baseline is ap-proved. It also authorizes the project to preliminary design meets all system requirements proceed to final design. with acceptable risk. It shows that the correct design
option has been selected, interfaces identified, and Critical Design Review. The Critical Design Review verification methods have been satisfactorily (CDR) is not a single review but a number of reviews described. It also establishes the basis for proceeding that start with specific Cls and end with the system with detailed design. CDR. Timing—After completing a full functional Purpose—The CDR discloses the complete implementation. sys-tem design in full detail, ascertains that technical Objectives—The objectives of the PDR are problems and design anomalies have been resolved, to: and ensures that the design maturity justifies the • Ensure that all system requirements have been decision to initiate fabrication/manufacturing, allocated, the requirements are complete, and integration, and verification of mission hardware and the flowdown is adequate to verify system software. performance Timing—Near the completion of the final • Show that the proposed design is expected to design stage. meet the functional and performance Objectives—The objectives of the CDR are requirements at the Cl level to: • Show sufficient maturity in the proposed design
approach to proceed to final design • Ensure that the "build-to" baseline contains de- • Show that the design is verifiable and that the tailed hardware and software specifications that risks have been identified, characterized, and can meet functional and performance mitigated where appropriate. requirements
• Ensure that the design has been satisfactorily Criteria for Successful Completion —The audited by production, verification, operations, fol-lowing items compose a checklist to aid in and other specialty engineering organizations determining readiness of PDR product preparation: • Ensure that the production processes and
controls are sufficient to proceed to the • Can the proposed preliminary design be fabrication stage expected to meet all the requirements within the • Establish that planned Quality Assurance (QA) planned cost and schedule? activities will establish perceptive verification and • Have all external interfaces been identified? screening processes for producing a quality • Have all the system and segment requirements product been allocated down to the CI level?
NASA Systems Engineering Handbook
Page 46 • Verify that the final design fulfills the • Ensure that the system meets acceptance criteria specifications established at PDR. that were established at SDR
• Establish that the system meets requirements Criteria for Successful Completion —The and will function properly in the expected following items compose a checklist to aid in operational environments as reflected in the test determining readiness of CDR product preparation: data, demonstrations, and analyses
• Establish an understanding of the capabilities • Can the proposed final design be expected to and operational constraints of the ''as-built'' meet all the requirements within the planned cost system, and that the documentation delivered and schedule? with the system is complete and current. • Is the design complete? Are drawings ready to
begin production? Is software product definition Criteria for Successful Completion —The sufficiently mature to start coding? fol-lowing items compose a checklist to aid in • Is the "build-to" baseline sufficiently traceable to determining readiness of SAR product preparation: assure that no orphan requirements exist?
• Do the design qualification results from software • Are tests and analyses complete? Do they prototyping and engineering item testing, indicate that the system will function properly in simulation, and analysis support the conclusion the expected operational environments? that the system will meet requirements? • Does the system meet the criteria described in • Are all internal interfaces completely defined and the acceptance plans? compatible? Are external interfaces current? • Is the system ready to be delivered (flight items • Are integrated safety analyses complete? Do to the launch site and non-flight items to the they show that identified hazards have been intended operational facility for installation)? controlled, or have those remaining risks which • Is the system documentation complete and cannot be controlled been waived by the accurate? appropriate authority? • Is it clear what is being bought? • Are production plans in place and reasonable?
• Are there adequate quality checks in the Results of Review — As a result of production process? successful completion of the SAR, the system is • Are the logistics support analyses adequate to accepted by the buyer, and authorization is given to identify integrated logistics support resource ship the hardware to the launch site or operational requirements? facility, and to install soft-ware and hardware for • Are comprehensive system integration and operational use. verifica tion plans complete?
Flight Readiness Review. Results of Review — As a result of successful Purpose —The Flight Readiness Review completion of the CDR, the "build-to" baseline, (FRR) examines tests, demonstrations, analyses, and produc-tion, and verification plans are approved. audits that determine the system's readiness for a Approved drawings are released and authorized for safe and successful launch and for subsequent flight fabrication. It also authorizes coding of deliverable operations. It also ensures that all flight and ground software (according to the "build-to" baseline and hardware, software, personnel, and procedures are coding standards presented in the review), and operationally ready. system qualification testing and integration. All open Timing—After the system has been issues should be resolved with closure actions and configured for launch. schedules. Objectives—The objectives of the FRR are
to: System Acceptance Review.
Purpose—The System Acceptance Review • Receive certification that flight operations can (SAR) examines the system, its end items and safely proceed with acceptable risk documentation, and test data and analyses that • Confirm that the system and support elements support verification. It also ensures that the system are properly configured and ready for launch has sufficient technical maturity to authorize its • Establish that all interfaces are compatible and shipment to and installation at the launch site or the function as expected intended operational facility. • Can the proposed final design be expected to Timing—Near the completion of the system meet all the requirements within the planned cost fabrication and integration stage. and schedule? Objectives—The objectives of the SAR are • Is the design complete? Are drawings ready to to: begin production? Is software product definition
sufficiently mature to start coding? • Establish that the system is ready to be delivered • Is the "build-to" baseline sufficiently traceable to and accepted under DD-250 assure that no orphan requirements exist?
NASA Systems Engineering Handbook
Page 47 • Do the design qualification results from software • Are tests and analyses complete? Do they prototyping and engineering item testing, indicate that the system will function properly in simulation, and analysis support the conclusion the expected operational environments? that the system will meet requirements? • Does the system meet the criteria described in • Are all internal interfaces completely defined and the acceptance plans? compatible? Are external interfaces current? • Is the system ready to be delivered (flight items • Are integrated safety analyses complete? Do to the launch site and non-flight items to the they show that identified hazards have been intended operational facility for installation)? controlled, or have those remaining risks which • Is the system documentation complete and cannot be controlled been waived by the accurate? appropriate authority? • Is it clear what is being bought? • Are production plans in place and reasonable?
• Are there adequate quality checks in the Results of Review — As a result of successful production process? completion of the SAR, the system is accepted by the • Are the logistics support analyses adequate to buyer, and authorization is given to ship the hardware identify integrated logistics support resource to the launch site or operational facility, and to install requirements? soft-ware and hardware for operational use. • Are comprehensive system integration and
Purpose — The Flight Readiness Review Results of Review — As a result of successful (FRR) examines tests, demonstrations, analyses, and completion of the CDR, the "build-to" baseline, audits that determine the system's readiness for a production, and verification plans are approved. safe and successful launch and for subsequent flight Approved drawings are released and authorized for operations. It also ensures that all flight and ground fabrication. It also authorizes coding of deliverable hardware, software, personnel, and procedures are software (according to the "build-to" baseline and operationally ready. coding standards presented in the review), and Timing—After the system has been system qualification testing and integration. All open configured for launch. issues should be resolved with closure actions and Objectives—The objectives of the FRR are schedules. to:
System Acceptance Review. • Receive certification that flight operations can Purpose—The System Acceptance Review safely proceed with acceptable risk (SAR) examines the system, its end items and • Confirm that the system and support elements documentation, and test data and analyses that are properly configured and ready for launch support verification. It also ensures that the system • Establish that all interfaces are compatible and has sufficient technical maturity to authorize its function as expected shipment to and installation at the launch site or the • Establish that the system state supports a launch intended operational facility. ' go" decision based on go/no-go criteria. Timing—Near the completion of the system
fabrication and integration stage. Criteria for Successful Completion —The Objectives—The objectives of the SAR are following items compose a checklist to aid in to: determining readiness of FRR product preparation:
• Establish that the system is ready to be delivered • Ιs the launch vehicle ready for launch? • Is the and accepted under DD-250 space vehicle hardware ready for safe launch • Ensure that the system meets acceptance criteria and subsequent flight with a high probability for that were established at SDR achieving mission success? • Establish that the system meets requirements • Are all flight and ground software elements ready and will function properly in the expected to support launch and flight operations? operational environments as reflected in the test • Are all interfaces checked out and found to be data, demonstrations, and analyses functional? • Establish an understanding of the capabilities • Have all open items and waivers been examined and operational constraints of the ''as-built'' and found to be acceptable? system, and that the documentation delivered • Are the launch and recovery environmental with the system is complete and current. factors within constraints?
Criteria for Successful Completion —The Results of Review — As a result of fol-lowing items compose a checklist to aid in successful FRR completion, technical and procedural determining readiness of SAR product preparation: maturity exists for system launch and flight
NASA Systems Engineering Handbook
Page 48 authorization. and in some cases initiation of system Timing—When major items within the operations. system are no longer needed to complete the
mission. Operational Readiness Review. Objectives—The objectives of the DR are Purpose — The Operational Readiness to: Review (ORR) examines the actual system
characteristics and the procedures used in its • Establish that the state of the mission and or operation, and ensures that all flight and ground system requires decommissioning/disposal. hardware, software, personnel, procedures, and user Possibilities include no further mission need, documentation reflect the deployed state of the broken degraded system elements, or phase out system accurately. of existing system assets due to a pending Timing—When the system and its upgrade operational and support equipment and personnel are • Demonstrate that the plans for decommissioning, ready to undertake the mission. disposal, and any transition are correct, current Objectives—The objectives of the ORR are and appropriate for current environmental to: constraints and system configuration
• Establish that resources are in place to support • Establish that the system is ready to transition disposal plans into an operational mode through examination of • Ensure that archival plans have been completed available ground and flight test results, analyses, for essential mission and project data. and operational demonstrations
• Confirm that the system is operationally and
logistically supported in a satisfactory manner Criteria for Successful Completion —The fol- considering all modes of operation and support lowing items compose a checklist to aid in (normal, contingency, and unplanned) determining readiness of DR product preparation: • Establish that operational documentation is
complete and represents the system • Are reasons for decommissioning/disposal well configuration and its planned modes of operation documented? • Establish that the training function is in place and • Is the disposal plan completed and compliant has demonstrated capability to support all with local, state, and federal environmental aspects of system maintenance, preparation, regulations? operation, and recovery. • Does the disposal plan address the disposition of
existing hardware, software, facilities, and Criteria for Successful Completion —The processes? following items compose a checklist to aid in • Have disposal risks been addressed? determining readiness of ORR product preparation: • Have data archival plans been defined? • Are the system hardware, software, personnel, • Are sufficient resources available to complete the and procedures in place to support operation? disposal plan? • Have all anomalies detected during prelaunch, • Is a personnel transition plan in place? launch, and orbital flight been resolved, docu-
mented, and incorporated into existing Results of Review—A successful DR operational support data? completion assures that the decommissioning and • Are the changes necessary to transition the disposal of system items and processes are system from flight test to an operational appropriate and effective. configuration ready to be made?
• Are all waivers closed? 4.8.4 Interim Reviews • Are the resources in place, or financially planned
and approved to support the system during its Interim reviews are driven by programmatic operational lifetime? and/or NASA Headquarters milestones that are not
necessarily supported by the major reviews. They are Results of Review — As a result of often multiple review processes that provide important successful ORR completion, the system is ready to information for major NASA reviews, programmatic assume normal operations and any potential hazards decisions, and commitments. Program/project due to launch or flight operations have been resolved tailoring dictates the need for and scheduling of these through use of redundant design or changes in reviews. operational procedures.
Requirements Reviews. Prior to the PDR, the Decommissioning Review. mission and system requirements must be thoroughly Purpose — The Decommissioning Review analyzed, allocated, and validated to assure that the (DR) confirms that the reasons for decommissioning project can effectively understand and satisfy the are valid and appropriate, and examines the current system status and plans for disposal.
NASA Systems Engineering Handbook
Page 49 mission need. Specifically, these interim requirements Objectives—The objectives of the review reviews confirm whether: are to:
• Confirm that the mission concept satisfies the • The proposed project supports a specific NASA customer's needs program deficiency • Confirm that the mission requirements support • In-house or industry-initiated efforts should be identification of external and long-lead support employed in the program realization requirements (e.g., DoD, international, facility • The proposed requirements meet objectives resources) • The requirements will lead to a reasonable • Determine the adequacy of the analysis products solution to support development of the preliminary Phase • The conceptual approach and architecture are B approval package. credibly feasible and affordable.
Criteria for Successful Completion—The These issues, as well as requirements ambiguities, following items compose a checklist to aid in are resolved or resolution actions are assigned. determining readiness of MRR product preparation: Interim requirements reviews alleviate the risk of Are the top-level mission requirements sufficiently excess design and analysis burdens too far into the defined to describe objectives in measurable life cycle. parameters? Are assumptions and constraints defined
and quantified? Safety Reviews. Safety reviews are conducted to • Is the mission and operations concept adequate ensure compliance with NHB 1700.1B, NASA Safety to support preliminary program/project Policy and Requirements Document, and are documentation development, including the approved by the program/project manager at the Engineering Master Plan/Schedule, Phase B recommendation of the system safety manager. Their Project Definition Plan, technology assessment, purpose, objectives, and general schedule are initial Phase B/C/D resource requirements, and contained in appropriate safety management plans. acquisition strategy development? Are evaluation Safety reviews address possible hazards associated criteria sufficiently defined? with system assembly, test, operation, and support. • Are measures of effectiveness established? Special consideration is given to possible operational • Are development and life-cycle cost estimates and environmental hazards related to the use of realistic? nuclear and other toxic materials. (See Section 6.8.) • Have specific requirements been identified that Early reviews with field center safety personnel are high risk/high cost drivers, and have options should be held to identify and understand any been described to relieve or mitigate them? problems areas, and to specify the requirements to
control them. Results of Review—Successful completion of
the MRR provides confidence to submit information Software Reviews. Software reviews are scheduled for the Preliminary Non-Advocate Review and by the program/project manager for the purpose of subsequent submission of the Mission Needs ensuring that software specifications and associated Statement for approval. products are well understood by both program/project
and user personnel. Throughout the development System Requirements Review. cycle, the pedigree, maturity, limitations, and Purpose — The System Requirements schedules of delivered preproduction items, as well as Review (SRR) demonstrates that the product the Computer Software Configuration Items (CSCI), development team understands the mission (i.e., are of critical importance to the project's engineering, project-level) and system-level requirements. operations, and verification organizations. Timing—Occurs (as required) following the
formation of the team. Readiness Reviews. Readiness reviews are Objectives—The objectives of the review conducted prior to commencement of major events are to: that commit and expose critical program/project
resources to risk. These reviews define the risk • Confirm that the system-level requirements meet environment and address the capability to the mission objectives satisfactorily operate in that environment. • Confirm that the system-level specifications of
the system are sufficient to meet the project Mission Requirements Review. objectives. Purpose — The Mission Requirements
Review (MRR) examines and substantiates top-level Criteria for Successful Completion —The requirements analysis products and assesses their following items compose a checklist to aid in readiness for external review. determining readiness of SRR project preparation: Timing—Occurs (as required) following the
maturation of the mission requirements in the mission definition stage.
NASA Systems Engineering Handbook
Page 50 • Are the allocations contained in the system Purpose — The Software Specification specifications sufficient to meet mission Review (SoSR) ensures that the software objectives? specification set is sufficiently mature to support • Are the evaluation criteria established and preliminary design efforts. realistic? Timing—Occurs shortly after the start of • Are measures of effectiveness established and preliminary design. realistic? Objectives—The review objectives are to: • Are cost estimates established and realistic?
• Has a system verification concept been • Verify that all software requirements from the identified? system specification have been allocated to • Are appropriate plans being initiated to support CSCls and documented in the appropriate projected system development milestones? software specifications • Have the technology development issues been • Verify that a complete set of functional, identified along with approaches to their solution? performance, interface, and verification
requirements for each CSCI has been developed Results of Review—Successful completion • Ensure that the software requirement set is both of the SRR freezes program/project requirements and complete and understandable. leads to a formal decision by the cognizant Program
Associate Administrator (PAA) to proceed with Criteria for Successful Completion —The proposal request preparations for project fol-lowing items comprise a checklist to aid in implementation. determining the readiness of SoSR product
preparation: System Safety Review.
Purpose—System Safety Review(s) (SSR) • Are functional CSCI descriptions complete and pro-vides early identification of safety hazards, and clear? ensures that measures to eliminate, reduce, or control • Are the software requirements traceable to the the risk associated with the hazard are identified and system specification? executed in a timely, cost-effective manner. • Are CSCI performance requirements complete Timing—Occurs (as needed) in multiple and unambiguous? Are execution time and phases of the project cycle. storage requirements realistic? Objectives—The objectives of the reviews • Is control and data flow between CSCIs defined? are to: • Are all software-to-software and software-to-
hardware interfaces defined? • Identify those items considered as critical from a • Are the mission requirements of the system and safety viewpoint associated operational and support environments • Assess alternatives and recommendations to defined? Are milestone schedules and special mitigate or eliminate risks and hazards delivery requirements negotiated and complete? • Ensure that mitigation/elimination methods can • Are the CSCI specifications complete with be verified. respect to design constraints, standards, quality
assurance, testability, and delivery preparation? Criteria for Successful Completion —The
following items comprise a checklist to aid in Results of Review—Successful completion determining readiness of SSR product preparation: of the SoSR results in release of the software
specifications based upon their development • Have the risks been identified, characterized, and requirements and guidelines, and the start of quantified if needed? preliminary design activities. • Have design/procedural options been analyzed,
and quantified if needed to mitigate significant Test Readiness Review. Purpose—The Test risks? Readiness Review (TRR) ensures that the test article • Have verification methods been identified for hardware/software, test facility, ground support candidate options? personnel, and test procedures are ready for testing,
and data acquisition, reduction, and control. Result of Review—A successful SSR Timing—Held prior to the start of a formal results in the identification of hazards and their test. The TRR establishes a decision point to proceed causes in the pro-posed design and operational with planned verification (qualification and/or modes, and specific means of eliminating, reducing, acceptance) testing of CIs, subsystems, and/or or controlling the hazards. The methods of safety systems. verification will also be identified prior to PDR. At Objectives—The objectives of the review CDR, a safety baseline is developed. are to:
Software Specification Review.
NASA Systems Engineering Handbook
Page 51 • Confirm that in-place test plans meet verification • Is the design certified? Have incomplete design requirements and specifications elements been identified? • Confirm that sufficient resources are allocated to • Have risks been identified and characterized. and the test effort mitigation efforts defined? • Examine detailed test procedures for • Has the bill of materials been reviewed and completeness and safety during test operations critical parts been identified? • Determine that critical test personnel are test-and • Have delivery schedules been verified? safety-certified • Have altemative sources been identified? • Confirm that test support software is adequate, • Have adequate spares been planned and pertinent, and verified. budgeted?
• Are the facilities and tools sufficient for end item Criteria for Successful Completion —The production? Are special tools and test equipment fol-lowing items comprise a checklist to aid in specified in proper quantities? determining the readiness of TRR product • Are personnel qualified? preparation: • Are drawings certified?
• Is production engineering and planning mature • Have the test cases been reviewed and analyzed for cost-effective production? for expected results? Are results consistent with • Are production processes and methods test plans and objectives? consistent with quality requirements? Are they • Have the test procedures been "dry run"? Do compliant with occupational safety, they indicate satisfactory operation? environmental, and energy conservation • Have test personnel received training in test regulations? operations and safety procedures? Are they
certified? Results of Review—A successful ProRR • Are resources available to adequately support results in certification of production readiness by the the planned tests as well as contingencies, project manager and involved specialty engineering including failed hardware replacement? organizations. All open issues should be resolved with • Has the test support software been demonstrated closure actions and schedules. to handle test configuration assignments, and
data acquisition, reduction, control, and Design Certification Review. archiving? Purpose — The Design Certification Review
(DCR) ensures that the qualification verifications Results of Review—A successful TRR demonstrated design compliance with functional and signifies that test and safety engineers have certified performance requirements. that preparations are complete, and that the project Timing — Follows the system CDR, and manager has authorized formal test initiation. after qualification tests and all modifications needed
to implement qualification-caused corrective actions Production Readiness Review. have been completed. Purpose — The Production Readiness Objectives—The objectives of the review Review (ProRR) ensures that production plans, are to: facilities, and personnel are in place and ready to
begin production. • Confirm that the verification results met functional Timing—After design certification and prior and performance requirements, and that test to the start of production. plans and procedures were executed correctly in Objectives—The objectives of the review the specified environments are to: • Certify that traceability between test article and
production article is correct, including name, • Ascertain that all significant production identification number, and current listing of all engineering problems encountered during waivers development are resolved • Identify any incremental tests required or • Ensure that the design documentation is conducted due to design or requirements adequate to support manufacturing/fabrication changes made since test initiation, and resolve • Ensure that production plans and preparations issues regarding their results. are adequate to begin manufacturing/fabrication
• Establish that adequate resources have been Criteria for Successful Completion —The allocated to support end item production. fol-lowing items comprise a checklist to aid in
determining the readiness of DCR product Criteria for Successful Completion —The preparation: following items comprise a checklist to aid in
determining the readiness of ProRR product • Are the pedigrees of the test articles directly preparation: traceable to the production units?
NASA Systems Engineering Handbook
Page 52 • Is the verification plan used for this article current and approved? • Do the test procedures and environments used comply with those specified in the plan? • Are there any changes in the test article configuration or design resulting from the as-run tests? Do they require design or specification This loop is applicable at each level of the changes, and/or retests? project hierarchy. Planning data, status reporting • Have design and specification documents been data, and assessments flow up the hierarchy with audited? appropriate aggregation at each level; decisions • Do the verification results satisfy functional and cause actions to be taken down the hierarchy. performance requirements? Managers at each level determine (consistent with • Do the verification, design, and specification policies established at the next higher level of the documentation correlate? project hierarchy) how often, and in what form,
reporting data and assessments should be made. In Results of Review—As a result of a establishing these status reporting and assessment successful DCR, the end item design is approved for requirements, some principles of good practice are: \ production. All open issues should be resolved with
closure actions and schedules. • Use an agreed-upon set of well-defined status
reporting variables • Report these core variables Functional and Physical Configuration Audits. The in a consistent format at all project levels Physical Configuration Audit (also known as a • Maintain historical data for both trend configuration inspection) verifies that the physical identification and cross-project analyses configuration of the product corresponds to the "build- • Encourage a logical process of rolling up status to" (or ''code-to") documentation previously approved reporting variables, (e.g., use the WBS for at the CDR. The Functional Configuration Audit obligations/costs status reporting and PBS for verifies that the acceptance test results are consistent mass status reporting) with the test requirements previously approved at the • Support assessments with quantitative risk PDR and CDR. It ensures that the test results indicate measures performance requirements were met, and test plans • Summarize the condition of the project by using and procedures were executed correctly. It should color-coded (red, yellow, and green) alert zones also document differences between the test unit and for all core reporting variables. production unit, including any waivers.
Regular, periodic (e.g., monthly) tracking of 4.9 Status Reporting and Assessment the core status reporting variables is recommended,
through some status reporting variables should be An important part of systems engineering planning is tracked more often when there is rapid change or determining what is needed in time, resources, and cause for concern. Key reviews, such as PDRs and people to realize the system that meets the desired CDRs, are points at which status reporting measures goals and objectives. Planning functions, such as and their trends should be carefully scrutinized for WBS preparation, scheduling, and fiscal resource early warning signs of potential problems. Should requirements planning, were discussed in Sections there be indications that existing trends, if allowed to 4.3 through 4.5. Project management, however, does continue, will yield an unfavorable outcome, not end with planning; project managers need visibility replanning should begin as soon as practical. into the progress of those plans in order to exercise This section provides additional information proper management control. This is the purpose of on status reporting and assessment techniques for the status reporting and assessing processes. Status costs and schedules, technical performance, and reporting is the process of determining where the systems engineering process metrics. project stands in dimensions of interest such as cost,
schedule, and technical performance. Assessing is 4.9.1 Cost and Schedule Control Measures the analytical process that converts the output of the
reporting process into a more useful form for the Status reporting and assessment on costs project manager -- namely, what are the future and schedules provides the project manager and implications of current trends? Lastly, the manager system engineer visibility into how well the project is must decide whether that future is acceptable, and tracking against its planned cost and schedule what changes, if any, in current plans are needed. targets. From a management point of view, achieving Planning, status reporting, and assessing are systems these targets is on a par with meeting the technical engineering and/or program control functions; performance requirements of the system. It is useful decision making is a management one. to think of cost and schedule status reporting and These processes together form the feedback assessment as measuring the performance of the loop depicted in Figure 20. This loop takes place on a "system that produces the system." continual basis throughout the project life cycle.
NASA Systems Engineering Handbook
Page 53 NHB 9501.2B, Procedures for Contractor difference, BCWPt -BCWSt, is called the schedule Reporting of Correlated Cost and Performance Data, variance at time t. provides specific requirements for cost and schedule The Actual Cost of Work Performed (ACWPt) status reporting and assessment based on a project's is a third statistic representing the funds that have dollar value and period of performance Generally, the been expended up to time t on those WBS elements. NASA Form 533 series of reports is applicable to The difference between the budgeted and actual NASA cost-type (i.e., cost reimbursement and fixed- costs, BCWPt ACWPt, is called the cost variance at price incentive) contracts. However, on larger time t. Such variances may indicate that the cost contracts (>$25M), which require Form 533P, NHB Estimate at Completion (EACt) of the project is 9501.2B allows contractors to use their own reporting different from the budgeted cost. These types of systems in lieu of 533P reporting. The project variances enable a program analyst to estimate the manager/system engineer may choose to evaluate EAC at any point in the project life cycle. (See sidebar the completeness and quality of these reporting on computing EAC.) systems against criteria established by the project If the cost and schedule baselines and the manager/system engineer's own field center, or technical scope of the work are not fully integrated, against the DoD's Cost/Schedule Cost System then cost and schedule variances can still be Criteria (C/SCSC). The latter are widely accepted by calculated, but the incomplete linkage between cost industry and government, and a variety of tools exist data and schedule data makes it very difficult (or for their implementation. impossible) to estimate the current cost EAC of the project. Control of Variances and the Role of the System Engineer. When negative variances are large enough to represent a significant erosion of reserves, then management attention is needed to either correct the variance, or to replan the project. It is important to establish levels of variance at which action is to be taken. These levels are generally lower when cost and schedule baselines do not support Earned Value calculations. The first action taken to control an excessive negative variance is to have the cognizant manager or system engineer investigate the problem, determine its cause, and recommend a solution. Computing the Estimate at Completion
EAC can be estimated at any point in the project. The appropriate formula depends upon the reasons associated for any variances that may exist. If a variance exists due to a one-time event, such as an accident, then EAC = BUDGET +
ACWP -BCWP where BUDGET is original Assessment Methods. The traditional method of planned cost at completion. If a variance exists for cost and schedule control is to compare baselined systemic reasons, such as a general cost and schedule plans against their actual values. In underestimate of schedule durations, or a steady program control terminology, a difference between redefinition of requirements, then the variance is actual performance and planned costs or schedule assumed to continue to grow over time, and the status is called a variance. equation is: EAC = BUDGET x (ACWP / BCWP). Figure 21 illustrates two kinds of variances If there is a growing number of liens, and some related concepts. A properly constructed action items, or significant problems that will Work Breakdown Structure (WBS) divides the project increase the difficulty of future work, the EAC work into discrete tasks and products. Associated with might grow at a greater rate than estimated by the each task and product (at any level in the WBS) is a above equation. Such factors could be addressed schedule and a budgeted (i.e., planned) cost. The using risk management methods described in Budgeted Cost of Work Scheduled (BCWSt) for any Section 4.6. set of WBS elements is the budgeted cost of all work In a large project, a good EAC is the on tasks and products in those elements scheduled to result of a variance analysis that may use of a be completed by time t. The Budgeted Cost of Work combination of these estimation methods on Performed (BCWPt) is a statistic representing actual different parts of the WBS. A rote formula should performance. BCWPt, also called Earned Value (EVt), not be used as a substitute for understanding the is the budgeted cost for tasks and products that have underlying causes of variances. actually been produced (completed or in progress) at time t in the schedule for those WBS elements. The
NASA Systems Engineering Handbook
Page 54 There are a number of possible reasons why variance only to specific PBS elements). The system engineer problems occur: needs to decide which generic and unique TPMs are
worth tracking at each level of the PBS. The system • A receivable was late or was unsatisfactory for engineer should track the measure of system some reason effectiveness (when the project maintains such a • A task is technically very difficult and requires measure) and the principal performance or technical more resources than originally planned attributes that determine it, as top-level TPMs. At • Unforeseeable (and unlikely to repeat) events lower levels of the PBS, TPMs worth tracking can be occurred, such as illness, fire, or other calamity. identified through the functional and performance
requirements levied on each individual system, Although the identification of variances is largely segment, etc. (See sidebar on high-level TPMs.) a program control function, there is an important In selecting TPMs, the system engineer systems engineering role in their control. That role should focus on those that can be objectively arises because the correct assessment of why a measured during the project life cycle. This negative variance is occurring greatly increases the measurement can be done directly by testing, or chances of successful control actions. This indirectly by a combination of testing and analysis. assessment often requires an understanding of the Analyses are often the only means available to cost, schedule, and technical situation that can only determine some high-level TPMs such as system be provided by the system engineer. reliability, but the data used in such analyses should
be based on demonstrated values to the maximum 4.9.2 Technical Performance Measures practical extent. These analyses can be performed
using the same measurement methods or models Status reporting and assessment of the system's used during trade studies. In TPM tracking, however, technical performance measures (TPMs) instead of using estimated (or desired) performance complements cost and schedule control. By tracking or technical attributes, the models are exercised using the system's TPMs, the project manager gains demonstrated values. As the project life cycle visibility into whether the delivered system will actually proceeds through Phases C and D, the measurement meet its performance specifications (requirements). of TPMs should become increasingly more accurate Beyond that, tracking TPMs ties together a number of because of the availability of more "actual" data about basic systems engineering activities—that is, a TPM the system. tracking program forges a relationship among Examples of High-Level TPMs for Planetary systems analysis, functional and performance Spacecraft and Launch Vehicles requirements definition, and verification and validation
TPMs) for planetary spacecraft include: • Systems analysis activities identify the key • End-of-mission (EOM:) dry mass performance or technical attributes that • Injected mass (includes EOM dry mass, determine system effectiveness; trade studies baseline mission plus reserve propellant, performed in systems analysis help quantify the other consumables and upper stage adaptor system's performance requirements. mass) • Functional and performance requirements • Consumables at EOM definition activities help identify verification and • Power demand (relative to supply) validation requirements. • Onboard data processing memory demand • Verification and validation activities result in • Onboard data processing throughput time quantitative evaluation of TPMs. Onboard data bus capacity • "Out-of-bounds" TPMs are signals to replan • Total pointing error. Mass and power fiscal, schedule, and people resources; demands by spacecraft subsystems and sometimes new systems analysis activities need science instruments may be tracked to be initiated. separately as well. For launch vehicles, high -
level TPMs include: Tracking TPMs can begin as soon as a baseline design has been established, which can • Total vehicle mass at launch occur early in Phase B. A TPM tracking program • Payload mass (at nominal altitude or orbit) should begin not later than the start of Phase C. Data • Payload volume to support the full set of selected TPMs may, • Injection accuracy however, not be available until later in the project life • Launch reliability cycle. • In-flight reliability
• For reusable vehicles, percent of value Selecting TPMs. In general, TPMs can be generic recovered For expendable vehicles, unit (attributes that are meaningful to each Product production cost at the n th unit. (See sidebar Breakdown Structure (PBS) element, like mass or on Learning Curve Theory.) reliability) or unique (attributes that are meaningful
NASA Systems Engineering Handbook
Page 55 Lastly, the system engineer should select those TPMs that must fall within well-defined (quantitative) limits for reasons of system effectiveness or mission feasibility. Usually these limits represent either a firm upper or lower bound constraint. A typical example of such a TPM for a spacecraft is its injected mass, which must not exceed the capability of the selected launch vehicle. Tracking injected mass as a high-level TPM is meant to ensure that this does not happen.
Assessment Methods. The traditional method of assessing a TPM is to establish a time-phased planned profile for it, and then to compare the demonstrated value against that profile. The planned profile represents a nominal "trajectory" for that TPM taking into account a number of factors. These factors include the technological maturity of the system, the planned schedule of tests and demonstrations, and any historical experience with similar or related systems. As an example, spacecraft dry mass tends to grow during Phases C and D by as much as 25 to 30 percent. A planned profile for spacecraft dry mass may try to compensate for this growth with a lower initial value. The final value in the planned profile usually either intersects or is asymptotic to an allocated requirement (or specification). The planned profile method is the technical performance measurement counterpart to the Earned Value method for cost and schedule control described earlier. A closely related method of assessing a TPM relies on establishing a time-phased margin requirement for it, and comparing the actual margin against that requirement. The margin is generally defined as the difference between a TPM's demonstrated value and its allocated requirement. The margin requirement may be expressed as a percentage of the allocated requirement. The margin requirement generally declines through Phases C and D, reaching or approaching zero at their completion. Depending on which method is chosen, the system engineer's role is to propose reasonable planned profiles or margin requirements for approval by the cognizant manager. The value of either of these methods is that they allow management by exception -- that is, only deviations from planned profiles or margins below requirements signal potential future problems requiring replanning. If this occurs, then new cost, schedule, and/or technical changes should be proposed. Technical changes may imply some new planned profiles. This is illustrated for a hypothetical TPM in Figure 22(a). In this example, a The margin management method of significant demonstrated variance (i.e., unanticipated assessing is illustrated for the same example in growth) in the TPM during design and development of Figure 22(b). The replanning at time t occurred when the system resulted in replanning at time t. The the TPM fell significantly below the margin replanning took the form of an increase in the allowed requirement. The new higher allocation for the TPM final value of the TPM (the "allocation"). A new resulted in a higher margin requirement, but it also planned profile was then established to track the TPM immediately placed the margin in excess of that over the remaining time of the TPM tracking program. requirement. Both of these methods recognize that the final value of the TPM being tracked is uncertain
NASA Systems Engineering Handbook
Page 56 distribution for the final TPM value. (See sidebar on An Example of the Risk Management Method for tracking spacecraft mass.) Early in the TPM tracking Tracking Spacecraft Mass program, when the demonstrated value is based on
indirect means of estimation, this distribution typically During Phases C and D, a spacecraft's injected has a larger statistical variance than later, when it is mass can be considered an uncertain quantity. based on measured data, such as a test result. When Estimates of each subsystem's and each a TPM stays along its planned profile (or equivalently, instrument's mass fare' however, made when its margin remains above the corresponding periodically by the design engineers. These margin requirement), the narrowing of the statistical estimates change and become more accurate as distribution should allow the TPM to remain in the actual parts and components are built and green alert zone (in Figure 22(c)) despite its growth. integrated into subsystems and instruments. The three methods represent different ways to assess Injected mass can also change during Phases C TPMs and communicate that information to and D as the quantity of propellant is fine-tuned to management, but whichever is chosen, the pattern of meet the mission design requirements. Thus at success or failure should be the same for all three. each point during development, the spacecraft's
injected mass is better represented as a Relationship of TPM Tracking Program to the probability distribution rather than as a single SEMP . The SEMP is the usual document for point. describing the project's TPM tracking program. This The mechanics of obtaining a probability description should include a master list of those TPMs distribution for injected mass typically involve to be tracked, and the measurement and assessment making estimates of three points -- the lower and methods to be employed. If analytical methods and upper bounds and the most likely injected mass models are used to measure certain high-level TPMs, value. These three values can be combined into then these need to be identified. The reporting parameters that completely define a probability frequency and timing of assessments should be distribution like the one shown in the figure below specified as well. In determining these, the system The launch vehicle's "guaranteed" engineer must balance the project's needs for payload capability, designated the "LV accurate, timely, and effective TPM tracking against Specification," is shown as a bold vertical line. the cost of the TPM The area under the probability curve to the left of tracking program. The TPM tracking program plan, which elaborates on the the bold vertical line represents the probability that SEMP, should specify each TPM's allocation, time- the spacecraft's injected mass will be less than or phased planned profile or margin requirement, and equal to the launch vehicle's payload capability. If alert zones, as appropriate to the selected injected mass is a TPM being tracked using the assessment method. risk management method, this probability could be
plotted in a display similar to Figure 22(c). If this probability were nearly one, then 4.9.3 Systems Engineering Process Metrics the project manager might consider adding more
objectives to the mission in order to take Status reporting and assessment of systems advantage of the "large margin" that appears to engineering process metrics provides additional exist. In the above figure, however, the probability visibility into the performance of the "system that is significantly less than one. Here, the project produces the system." As such, these metrics manager might consider descoping the project, for supplement the cost and schedule control measures example by removing an instrument or otherwise discussed in Section 4.9.1. changing mission objectives. The project manager Systems engineering process metrics try to could also solve the problem by requesting a quantify the effectiveness and productivity of the larger launch vehicle! systems engineering process and organization. Within a single project, tracking these metrics allows the throughout most of Phases C and D. The margin system engineer to better understand the health and management method attempts to deal with this progress of that project. Across projects (and over implicitly by establishing a margin requirement that time), the tracking of systems engineering process reduces the chances of the final value exceeding its metrics allows for better estimation of the cost and allocation to a low number, for example five percent time of performing systems engineering functions. It or less. A third method of reporting and assessing also allows the systems engineering organization to deals with this risk explicitly. demonstrate its commitment to the TQM principle of The risk management method is illustrated continuous improvement. for the same example in Figure 22(c). The replanning
at time t occurred when the probability of the final Selecting Systems Engineering Process Metrics. TPM value being less than the allocation fell Generally, systems engineering process metrics fall precipitously into the red alert zone. The new higher into three categories -- those that measure the allocation for the TPM resulted in a substantial progress of the systems engineering effort, those that improvement in that probability. The risk management measure the quality of that process, and those that method requires an estimate of the probability measure its productivity. Different levels of systems
NASA Systems Engineering Handbook
Page 57 engineering management are generally interested in exist, the most common is the number of systems different metrics. For example, a project manager or engineering hours dedicated to a particular function or lead system engineer may focus on metrics dealing activity. Because not all systems engineering hours with systems engineering staffing, project risk cost the same, an appropriate weighing scheme management progress, and major trade study should be developed to ensure comparability of hours progress. A subsystem system engineer may focus across systems engineering personnel. on subsystem requirements and interface definition Displaying schedule-related metrics can be progress and verification procedures progress. It is accomplished in a table or graph of planned quantities useful for each system engineer to focus on just a few vs. actuals. With quality- and productivity-related process metrics. Which metrics should be tracked metrics, trends are generally more important than depends on the system engineer's role in the total isolated snapshots. The most useful kind of systems engineering effort. The systems engineering assessment method allows comparisons of the trend process metrics worth tracking also change as the on a current project with that for a successfully project moves through its life cycle. completed project of the same type. The latter Collecting and maintaining data on the provides a benchmark against which the system systems engineering process is not without cost. engineer can judge his/her own efforts. Status reporting and assessment of systems engineering process metrics divert time and effort from the process itself. The system engineer must balance the value of each systems engineering process metric against its collection cost. The value of these metrics arises from the insights they provide into the process that cannot be obtained from cost and schedule control measures alone. Over time, these metrics can also be a source of hard productivity data, which are invaluable in demonstrating the potential returns from investment in systems engineering tools and training.
Examples and Assessment Methods. Table 2 lists some systems engineering process metrics to be considered. This list is not intended to be exhaustive. Because some of these metrics allow for different interpretations, each NASA field center needs to define them in a common-sense way that fits its own processes. For example, each field center needs to determine what it meant by a completed versus an approved requirement, or whether these terms are even relevant. As part of this definition, it is important to recognize that not all requirements, for example, need be lumped together. It may be more useful to track the same metric separately for each of several different types of requirements. Quality-related metrics should serve to indicate when a part of the systems engineering process is overloaded and/or breaking down. These metrics can be defined and tracked in several different ways. For example, requirements volatility
can be quantified as the number of newly identified
requirements, or as the number of changes to already-approved requirements. As another example, Engineering Change Request (ECR) processing could be tracked by comparing cumulative ECRs opened versus cumulative ECRs closed, or by plotting the age profile of open ECRs, or by examining the number of ECRs opened last month versus the total number open. The system engineer should apply his/her own judgment in picking the status reporting and assessment method. Productivity-related metrics provide an indication of systems engineering output per unit of input. Although more sophisticated measures of input
NASA Systems Engineering Handbook
systems engineering spiral described in Chapter 2. 5 Systems Analysis and Modeling Issues This section discusses the steps of the process in The role of systems analysis and modeling is to greater detail. Trade studies help to define the produce rigorous and consistent evaluations so as to emerging system at each level of resolution. One key foster better decisions in the systems engineering message of this section is that to be effective, the process. By helping to progress the system design process requires the participation of many skills and a toward an optimum, systems analysis and modeling unity of effort to move toward an optimum system contribute to the objective of systems engineering. design. This is accomplished primarily by performing trade Figure 23 shows the trade study process in studies of plausible alternatives. The purpose of this simplest terms, beginning with the step of defining the chapter is to describe the trade study process, the system's goals and objectives, and identifying the methods used in trade studies to quantify system constraints it must meet. In the early phases of the effectiveness and cost, and the pitfalls to avoid. project life cycle, the goals, objectives, and
constraints are usually stated in general operational terms. In later phases of the project life cycle, when Systems Analysis the architecture and, perhaps, some aspects of the
design have already been decided, the goals and Gene Fisher defines systems analysis as "inquiry objectives may be stated as performance to assist decision makers in choosing preferred requirements that a segment or subsystem must future courses of action by (1) systematically meet. examining and reexamining the relevant At each level of system resolution, the objectives, and alternative policies and strategies system engineer needs to understand the full for achieving them; and (2) comparing implications of the goals, objectives, and constraints quantitatively where possible the economic costs, in order to formulate an appropriate system solution. effectiveness, and risks of the alternaternatives.” This step is accomplished by performing a functional analysis. Functional analysis is the systematic process of identifying, describing, and relating the
functions a system must perform in order to fulfil1 its
goals and objectives. In the early phases of the 5.1 The Trade Study Process project life cycle, the functional analysis deals with the
top-level functions that need to be performed by the The trade study process is a critical part of the
NASA Systems Engineering Handbook
Page 59 system, where they need to be performed, how often, measurement of system effectiveness, system under what operational concept and environmental performance or technical attributes, and system cost. conditions, and so on. The functional analysis needs Many different selection rules are possible. only to proceed to a level of decomposition that The selection rule in a particular trade study may enables the trade study to define the system depend on the context in which the trade study is architecture. In later phases of the project life cycle, being conducted—in particular, what level of system the functional analysis proceeds to whatever level of design resolution is being addressed. At each level of decomposition is needed to fully define the system the system design, the selection rule generally should design and interfaces. (See sidebar on functional be chosen only after some guidance from the next analysis techniques.) higher level. The selection rule for trade studies at Closely related to defining the goals and lower levels of the system design should be in objectives, and performing a functional analysis, is the consonance with the higher level selection rule. step of defining the measures and measurement Defining plausible alternatives is the step of methods for system effectiveness (when this is creating some alternatives that can potentially practical), system performance or technical attributes, achieve the goals and objectives of the system. This and system cost. (These variables are collectively step depends on understanding (to an appropriately called outcome variables, in keeping with the detailed level) the system's functional requirements discussion in Section 2.3. Some systems engineering and operational concept. Running an alternative books refer to these variables as decision criteria, but through an operational time line or reference mission this term should not be confused with selection rule, Functional Analysis Techniques described below. Sections 5.2 and 5.3 discuss the
concepts of system cost and system effectiveness, • Functional analysis is the process of respectively, in greater detail.) This step begins the identifying, de-scribing, and relating the analytical portion of the trade study process, since it functions a system must per-form in order to suggests the involvement of those familiar with fulfill its goals and objectives. Functional quantitative methods. analysis is logically structured as a top-down For each measure, it is important to address hierarchical decomposition of those functions, the question of how that quantitative measure will be and serves several important roles in the com-puted— that is, which measurement method is to systems engineering process: be used. One reason for doing this is that this step • To draw out all the requirements the system then explicitly identifies those variables that are must meet important in meeting the system's goals and • To help identify measures for system objectives. effectiveness and its underlying performance Evaluating the likely outcomes of various or technical attributes at all levels alternatives in terms of system effectiveness, the • To weed out from further consideration in underlying performance or technical attributes, and trade studies those alternatives that cannot cost before actual fabrication and/or programming meet the system's goals and objectives usually requires the use of a mathematical model or • To provide insights to the system-level (and series of models of the system. So a second reason below) model builders, whose mathematical for specifying the measurement methods is that the models will be used in trade studies to necessary models can be identified. evaluate the alternatives. Sometimes these models are already
available from previous projects of a similar nature; Several techniques are available to do other times, they need to be developed. In the latter functional analysis. The primary functional case, defining the measurement methods should analysis technique is the Functional Flow Block trigger the necessary system modeling activities. Diagram (FFBD). These diagrams show the Since the development of new models can take a network of actions that lead to the fulfillment of a considerable amount of time and effort, early function. Although the FFBD network shows the identification is needed to ensure they will be ready logical sequence of “what" must happen, it does for formal use in trade studies. not ascribe a time duration to functions or Defining the selection rule is the step of between functions. To understand time-critical explicitly determining how the outcome variables will requirements, a Time Line Analysis (TLA) is used. be used to make a (tentative) selection of the A TLA can be applied to such diverse operational preferred alternative. As an example, a selection rule functions as spacecraft command sequencing and may be to choose the alternative with the highest launch vehicle processing. A third technique is the estimated system effectiveness that costs less than x N 2 diagram, which is a matrix display of functional dollars (with some given probability), meets safety interactions, or data flows, at a particular requirements, and possibly meets other political or hierarchical level. Appendix B.7 provides further schedule constraints. Defining the selection rule is discussion and examples of each of these essentially deciding how the selection is to be made. techniques. This step is independent from the actual
NASA Systems Engineering Handbook
Page 60 is a useful way of determining whether it can plausibly In an ideal world, all input values would be fulfill these requirements. (Sometimes it is necessary pre-cisely known, and models would perfectly predict to create separate behavioral models to determine outcome variables. This not being the case, the how the system reacts when a certain stimulus or system engineer should supplement point estimates control is applied, or a certain environment is of the outcome variables for each alternative with encountered. This provides insights into whether it computed or estimated uncertainty ranges. For each can plausibly fulfill time-critical and safety uncertain key input, a range of values should be requirements.) Defining plausible alternatives also estimated. Using this range of input values, the requires an understanding of the technologies sensitivity of the outcome variables can be gauged, available, or potentially available, at the time the and their uncertainty ranges calculated. The system system is needed. Each plausible alternative should engineer may be able to obtain meaningful probability be documented qualitatively in a description sheet. distributions for the outcome variables using Monte The format of the description sheet should, at a Carlo simulation (see Section 5.4.2), but when this is minimum, clarify the allocation of required system not feasible, the system engineer must be content functions to that alternative's lower-level architectural with only ranges and sensitivities. or design components (e.g.. subsystems). This essentially completes the analytical One way to represent the trade study portion of the trade study process. The next steps can alternatives under consideration is by a trade tree. be described as the judgmental portion. Combining During Phase A trade studies, the trade tree should the selection rule with the results of the analytical contain a number of alternative high-level system activity should enable the system engineer to array architectures to avoid a premature focus on a single the alternatives from most preferred to least, in one. As the systems engineering process proceeds, essence making a tentative selection. branches of the trade tree containing unattractive This tentative selection should not be alternatives will be "pruned," and greater detail in accepted blindly. In most trade studies, there is a terms of system design will be added to those need to subject the results to a "reality check" by branches that merit further attention. The process of considering a number of questions. Have the goals, pruning unat-tractive early alternatives is sometimes objectives, and constraints truly been met? Is the known as doing "killer trades." (See sidebar on trade tentative selection heavily dependent on a particular trees.) set of input values to the measurement methods, or Given a set of plausible alternatives, the next does it hold up under a range of reasonable input step is to collect data on each to support the values? (In the latter case, the tentative selection is evaluation of the measures by the selected said to be robust.) Are there sufficient data to back up measurement methods. If models are to be used to the tentative selection? Are the measurement calculate some of these measures, then obtaining the methods sufficiently discriminating to be sure that the model inputs provides some impetus and direction to tentative selection is really better than other the data collection activity. By providing data, alternatives? Have the subjective aspects of the engineers in such disciplines as reliability, problem been fully addressed? maintainability, producibility, integrated logistics, If the answers support the tentative software, testing, operations, and costing have an selection, then the system engineer can have greater important supporting role in trade studies. The data confidence in a recommendation to proceed to a collection activity, however, should be orchestrated by further resolution of the system design, or to the the system engineer. The results of this step should implementation of that design. The estimates of be a quantitative description of each alternative to system effectiveness, its underlying performance or accompany the qualitative. technical attributes, and system cost generated during Test results on each alternative can be the trade study process serve as inputs to that further especially useful. Early in the systems engineering resolution. The analytical portion of the trade study process, performance and technical attributes are process often provide the means to quantify the generally uncertain and must be estimated. Data from performance or technical (and cost) attributes that the breadboard and brassboard testbeds can provide system's lower levels must meet. These can be additional confidence that the range of values used as formalized as performance requirements. model inputs is correct. Such confidence is also If the reality check is not met, the trade study enhanced by drawing on data collected on related process returns to one or more earlier steps. This pre-viously developed systems. iteration may result in a change in the goals, The next step in the trade study process is to objectives, and constraints, a new alternative, or a quantify the outcome variables by computing change in the selection rule, based on the new estimates of system effectiveness, its underlying information generated during the trade study. The system performance or technical attributes, and reality check may, at times, lead instead to a decision system cost. If the needed data have been collected, to first improve the measures and measurement and the measurement methods (for example, models) methods (e.g., models) used in evaluating the are in place, then this step is, in theory, mechanical. alternatives, and then to repeat the analytical portion In practice, considerable skill is often needed to get of the trade study process. meaningful results.
NASA Systems Engineering Handbook
Page 61 5.1.1 Controlling the Trade Study Process resources available to do the study because the work
required in defining additional alternatives and There are a number of mechanisms for obtaining the necessary data on them can be controlling the trade study process. The most considerable. However, focusing on too few or too important one is the Systems Engineering similar alternatives defeats the purpose of the trade Management Plan (SEMP). The SEMP specifies the study process. major trade studies that are to be performed during A fourth mechanism for controlling the trade each phase of the project life cycle. It should also study process can be exercised through the use (and spell out the general contents of trade study reports, misuse) of models. Lastly, the choice of the selection which form part of the decision support packages (i.e., rule exerts a considerable influence on the results of documentation submitted in conjunction with formal the trade study process. These last two issues are reviews and change requests). discussed in Sections 5.1.2 and 5.1.3, respectively. A second mechanism for controlling the
trade study process is the selection of the study team 5.1.2 Using Models leaders and members. Because doing trade studies is
part art and part science, the composition and Models play important and diverse roles in systems experience of the teams is an important determinant engineering. A model can be defined in several ways, of the study's ultimate usefulness. A useful technique including: to avoid premature focus on a specific technical
designs is to include in the study team individuals with • An abstraction of reality designed to answer differing technology backgrounds. specific questions about the real world Another mechanism is limiting the number of • An imitation, analogue, or representation of a real alternatives that are to be carried through the study. world process or structure; or This number is usually determined by the time and • A conceptual, mathematical, or physical tool to An Example of a Trade Tree for a Mars Rover
The figure below shows part of a trade tree for a robotic Mars rover system, whose goal is to find a suitable manned landing site. Each layer represents some aspect of the system that needs to be treated in a trade study to determine the best alternative. Some alternatives have been eliminated a priori because of technical feasibility, launch vehicle constraints, etc. The total number of alternatives is given by the number of end points of the tree. Even with just a few layers, the number of alternatives can increase quickly. (This tree has already been pruned to eliminate low-autonomy, large rovers.) As the systems engineering process proceeds, branches of the tree with unfavorable trade study outcomes are discarded. The remaining branches are further developed by identifying more detailed trade studies that need to be made. A whole family of (implicit) alternatives can be represented in a trade tree by a continuous variable. In this example, rover speed or range might be so represented. By treating a variable this way, mathematical optimization techniques can be applied. Note that a trade tree is, in essence, a decision tree without chance nodes. (See the sidebar on decision trees.)
NASA Systems Engineering Handbook
Page 62 Trade Study Reports assist a decision maker. Trade study reports should be prepared for each
trade study. At a minimum, each trade study Together, these definitions are broad enough report should identify: to encompass physical engineering models used in
the verification of a system design, as well as • The system issue under analysis schematic models like a functional flow block diagram • System goals and objectives (or and mathematical (i.e., quantitative) models used in requirements, as appropriate to the level of the trade study process. This section focuses on the resolution), and constraints last. • The measures and measurement methods The main reason for using mathematical (models) used models in trade studies is to provide estimates of • All data sources used system effectiveness, performance or technical • The alternatives chosen for analysis attributes, and cost from a set of known or estimable • The computational results, including quantities. Typically, a collection of separate models uncertainty ranges and sensitivity analyses is needed to provide all of these outcome variables. performed The heart of any mathematical model is a set of • The selection rule used meaningful quantitative relationships among its inputs • The recommended alternative. and outputs. These relationships can be as simple as
adding up constituent quantities to obtain a total, or as Trade study reports should be maintained as complex as a set of differential equations describing part of the system archives so as to ensure the trajectory of a spacecraft in a gravitational field. traceability of decisions made through the Ideally, the relationships express causality, not just systems engineering process. Using a generally correlation. consistent format for these reports also makes it
easier to review and assimilate them into the Types of Models. There are a number of ways formal change control process. mathematical models can be usefully categorized. One way is according to its purpose in the trade study process—that is, what system issue and what level of (usually the system's behavior under different detail the model addresses, and with which outcome assumptions) using the sheer computational power of variable or variables the model primarily deals. Other current computers. commonly used ways of categorizing mathematical Linear, nonlinear, integer and dynamic program- models focus on specific model attributes such as ming models are another important class of models in whether a model is: trade studies because they can optimize an objective
function representing an important outcome variable • Static or dynamic (for example, system effectiveness) for a whole class • Deterministic or probabilistic (also called of implied alternatives. Their power is best applied in stochastic) situations where the system's objective function and • Descriptive or optimizing. constraints are well understood, and these constraints
can be written as a set of equalities and inequalities. These terms allow model builders and model
users to enter into a dialogue with each other about Pitfalls in Using Models. Models always embody as- the type of model used in a particular analysis or sumptions about the real world they purport to trade study. No hierarchy is implied in the above list; represent, and they always leave something out. none of the above dichotomous categorizations Moreover, they are usually capable of producing stands above the others. highly accurate results only when they are addressing Another taxonomy can be based on the degree of rigorously quantifiable questions in which the analytic tractability. At one extreme on this scale, an "physics" is well understood as, for example, a load "analytic" model allows a closed-form solution for a dynamics analysis or a circuit analysis. out-come variable of interest as a function of the In dealing with system issues at the top model inputs. At the other extreme, quantification of a level, however, this is seldom the case. There is often outcome variable of interest is at best ordinal, while in a significant difference between the substantive the middle are many forms of mathematical simulation system cost-effectiveness issues and questions, and models. the questions that are mathematically tractable from a Mathematical simulations are a particularly useful modeling perspective. For example, the type of model in trade studies. These kinds of models program/project manager may ask: "What's the best have been successfully used in dealing quantitatively space station we can build in the current budgetary with large complex systems problems in environment?" The system engineer may try to deal manufacturing, transportation, and logistics. with that question by translating it into: "For a few Simulation models are used for these problems plausible station designs, what does each provide its because it is not possible to "solve" the system's users, and how much does each cost?" When the equations analytically to obtain a closed-form solution, system engineer then turns to a model (or models) for yet it is relatively easy to obtain the desired results answers, the results may only be some approximate
NASA Systems Engineering Handbook
Page 63 costs and some user resource measures based on a predictions. A history of successful predictions lends few engineering relationships. The model has failed to credibility to a model, but full validation—proof that adequately address even the system engineer's more the model's prediction is in accord with reality—is very limited question, much less the program/project difficult to attain since observational evidence on manager's. Compounding this sense of model those predictions is generally very scarce. While it is incompleteness is the recognition that the model's certainly advantageous to use tried-and-true models relationships are often chosen for their mathematical that are often left as the legacy of previous projects, convenience, rather than a demonstrated empirical this is not always possible. Systems that address new validity. Under this situation, the model may produce problems often require that new models be developed insights, but it cannot provide definitive answers to the for their trade studies. In that case, full validation is substantive questions on its own. Often too, the out of the question, and the system engineer must be system engineer must make an engineering content with models that have logical consistency and interpretation of model results and convey them to the some limited form of outside, independent cor- project manager or other decision maker in a way that roboration. captures the essence of the original question. Responsiveness of a model is a measure of As mentioned earlier, large complex its power to distinguish among the different problems often require multiple models to deal with alternatives being considered in a trade study. A different aspects of evaluating alternative system responsive lunar base cost model, for example, architectures (and designs). It is not unusual to have should be able to distinguish the costs associated separate models to deal with costs and effectiveness, with different system architectures or designs, or to have a hierarchy of models—i.e., models to deal operations concepts, or logistics strategies. with lower level engineering issues that provide useful Another desirable model characteristic is results to system-level mathematical models. This transparency, which occurs when the model's situation itself can have built-in pitfalls. mathematical relationships, algorithms, parameters, One such pitfall is that there is no guarantee supporting data, and inner workings are open to the that all of the models work together the way the user. The benefit of this visibility is in the traceability system engineer intends or needs. One submodel's of the model's results. Not everyone may agree with specialized assumptions may not be consistent with the results, but at least they know how they were the larger model it feeds. Optimization at the derived. Transparency also aids in the acceptance subsystem level may not be consistent with system- process. It is easier for a model to be accepted when level optimization. Another such pitfall occurs when a its documentation is complete and open for comment. key effectiveness variable is not represented in the Proprietary models often suffer from a lack of ac- cost models. For example, if spacecraft reliability is a ceptance because of a lack of transparency. key variable in the system effectiveness equation, and Upfront user friendliness is related to the if that reliability does not appear as a variable in the ease with which the system engineer can learn to use spacecraft cost model, then there is an important the model and prepare the inputs to it. Backend user disconnect. This is because the models allow the friendliness is related to the effort needed to interpret spacecraft designer to be-lieve it is possible to boost the model's results and to prepare trade study reports the effectiveness with increased reliability without for the tentative selection using the selection rule. paying any apparent cost penalty. When the models
fail to treat such important interactions, the system 5.1.3 Selecting the Selection Rule engineer must ensure that others do not reach false
conclusions regarding costs and effectiveness. The analytical portion of the trade study
process serves to produce specific information on Characteristics of a Good Model. In choosing a system effectiveness, its underlying performance or model (or models) for a trade study, it is important to technical attributes, and cost (along with uncertainty recognize those characteristics that a good model ranges) for a few alternative system architectures has. This list in-cludes: (and later, system designs). These data need to be
brought together so that one alternative may be • Relevance to the trade study being performed selected. This step is accomplished by applying the • Credibility in the eye of the decision maker selection rule to the data so that the alternatives may • Responsiveness be ranked in order of preference. • Transparency The structure and complexity of real world • User friendliness. decisions in systems engineering often make this
ranking a difficult task. For one, securing higher Both relevance and credibility are crucial to effectiveness almost always means incurring higher the acceptance of a model for use in trade studies. costs and/or facing greater uncertainties. In order to Relevance is determined by how well a model choose among alternatives with different levels of addresses the substantive cost-effectiveness issues effectiveness and costs, the system engineer must in the trade study. A model's credibility results from understand how much of one is worth in terms of the the logical consistency of its mathematical other. An explicit cost-effectiveness objective function relationships, and a history of successful (i.e., correct) is seldom available to help guide the selection
NASA Systems Engineering Handbook
Page 64 general conditions under which each is applicable; definitive guidance on which to use in each and every case has not been attempted. Table 3 first divides selection rules according to the importance of uncertainty in the trade study. This division is reflective of two different classes of decision problems —decisions to be made under conditions of certainty, and decisions to be made decision, as any system engineer who has had to under conditions of uncertainty. Uncertainty is an make a budget-induced system descope decision will inherent part of systems engineering, but the attest. distinction may be best explained by reference to A second, and major, problem is that an Figure 2, which is repeated here as Figure 24. In the expression or measurement method for system former class, the measures of system effectiveness, effectiveness may not be possible to construct, even performance or technical attributes, and system cost though its underlying performance and technical for the alternatives in the trade study look like those attributes are easily quantified. These underlying for alternative B. In the latter class, they look like attributes are often the same as the technical those for alternative C. When they look like those for performance measures (TPMs) that are tracked alternative A, conditions of uncertainty should apply, during the product development process to gauge but often are not treated that way. whether the system design will meet its performance The table further divides each of the above requirements. In this case, system effectiveness may, classes of decision problems into two further at best, have several irreducible dimensions. categories: those that apply when cost and What selection rule should be used has been effectiveness measures are scalar quantities, and the subject of many books and articles in the decision thus suffice to guide the system engineer to the best sciences —management science, operations alternative, and those that apply when cost and research and economics. A number of selection rules effectiveness cannot be represented as scalar are applicable to NASA trade studies. Which one quantities. should be used in a particular trade study depends on
a number of factors: Selection Rules When Uncertainty Is Subordinate,
or Not Considered. Selecting the alternative that • The level of resolution in the system design The maximizes net benefits (benefits minus costs) is the phase of the project life cycle rule used in most cost-benefit analyses. Cost-benefit • Whether the project maintains an overall system analysis applies, however, only when the return on a effectiveness model project can be measured in the same units as the • How much less-quantifiable, subjective factors costs, as, for example, in its classical application of contribute to the selection evaluating water resource projects. • Whether uncertainty is paramount, or can effec-
tively be treated as a subordinate issue Another selection rule is to choose the • Whether the alternatives consist of a few alternative that maximizes effectiveness for a given qualitatively different architectures designs, or level of cost. This rule is applicable when system many similar ones that differ only in some effectiveness and system cost can be unambiguously quantitative dimensions. measured, and the appropriate level of cost is known.
Since the purpose of the selection rule is to compare This handbook can only suggest some and rank the alternatives, practical application selection rules for NASA trade studies, and some requires that each of the alternatives be placed on an
NASA Systems Engineering Handbook
Page 65 equal cost basis. For certain types of trade studies, this does not present a problem. For example, The Analytic Hierarchy Process changing system size or output, or the number of
platforms or instruments, may suffice. In other types AHP is a decision technique in which a figure of of trade studies, this may not be possible. merit is determined for each of several A related selection rule is to choose the alternatives through a series of pair-wise alternative that minimizes cost for a given level of comparisons. AHP is normally done in six steps: effectiveness. This rule presupposes that system (1) Describe in summary form the alternatives effectiveness and system cost can be unambiguously under consideration. measured, and the appropriate level of effectiveness (2) Develop a set of high-level evaluation is known. Again, practical application requires that objectives; for example, science data return, each of the alternatives be put on an equal national prestige, technology advancement, effectiveness basis. This rule is dual to the one above etc. in the following sense: For a given level of cost, the (3) Decompose each hi-level evaluation same alternative would be chosen by both rules; objective into a hierarchy of evaluation similarly, for a given level of effectiveness, the same attributes that clarify the meaning of the alternative would be chosen by both rules. objective. When it is not practical to equalize the cost (4) Determine, generally by conducting or the effectiveness of competing alternatives, and structured interviews with selected cost caps or effectiveness floors do not rule out all individuals (“experts”) or by having them fill alternatives save one, then it is necessary to form, out structured questionnaires, the relative either explicitly or implicitly, a cost-effectiveness importance of the evaluation objectives and objective function like the one shown in Figure 4 attributes through pair-wise comparisons. (Section 2.5). The cost-effectiveness objective (5) Have each evaluator make separate pair- function provides a single measure of worth for all wise comparisons of the alternatives with combinations of cost and effectiveness. When this respect to each evaluation attribute. These selection rule is applied, the alternative with the subjective evaluations are the raw data highest value of the cost-effectiveness objective inputs to a separately developed AHP function is chosen. program , which produces a single figure of Another group of selection rules is needed merit for each alternative. This figure of merit when cost and/or effectiveness cannot be is based on relative weight determined by represented as scalar quantities. To choose the best the evaluators themselves. alternative, a multi-objective selection rule is needed. (6) Iterate the questionnaire and AHP evaluation A multi-objective rule seeks to select the alternative process until a consensus ranking of the that, in some sense, represents the best balance alternative is achieved among competing objectives. To accomplish this,
each alternative is measured (by some quantitative With AHP, sometimes consensus is achieved method) in terms of how well it achieves each quickly; other times, several feedback rounds are objective. For example, the objectives might be re-quired. The, feed back consists of reporting the national prestige, upgrade or expansion potential, com-puted values (for each evaluator and for the science data return, low cost, and potential for group) for each option, reasons for differences in international partnerships. Each alternative's "scores" evaluation, and identified areas of contention against the objectives are then combined in a value and/or inconsistency. Individual evaluators may function to yield an overall figure of merit for the choose to change their subjective judgments on alternative. The way the scores are combined should both attribute weights and preferences. At this reflect the decision maker's preference structure. The point, inconsistent and divergent preferences can alternative that maximizes the value function (i.e., with be targeted for more detailed study. the highest figure of merit) is then selected. In AHP assumes the existence of an underlying essence, this selection rule recasts a multi-objective preference “Vector” (with magnitudes and decision problem into one involving a single, directions) that is revealed through the pair-wise measurable objective. comparisons. This is a powerful assumption, One way, but not the only way, of forming which may at best hold only for the participating the figure of merit for each alternative is to linearly evaluators. The figure of merit produced for each combine its scores computed for each of the alternative is the result of the group’s subjective objectives—that is, compute a weighted sum of the judgments and is not necessarily a reproducible scores. MSFC-HDBK-1912, Systems Engineering result. For information on AHP, see Thomas L. (Volume 2) recommends this selection rule. The Saaty, The Analytic Hierarchy Process, 1980. weights used in computing the figure of merit can be assigned a priori or determined using Multi Attribute software packages are available to automate either Utility Theory (MAUT). Another technique of forming a MAUT or AHP. If the wrong weights, objectives, or figure of merit is the Analytic Hierarachy Process attributes are chosen in either technique, the entire (AHP). Several microcomputer-based commercial process may obscure the best alternative. Also, with
NASA Systems Engineering Handbook
Page 66 Multi-Attribute Utility Theory MAUT is a decision technique in which a figure of merit (or utility) is determined for each of several alternatives through a series of preference-revealing comparisons of simple lotteries. An abbreviated MAUT decision mechanism can be described in six steps: (1) Choose a set of descriptive, but quantifiable, attributes designed to characterize each alternative. (2) For each alternative under consideration, generate values for each attribute in the set; these may be point estimates, or probability distributions, if the uncertainty in attribute values warrants explicit treatment. (3) (3) Develop an attribute utility function for each attribute in the set. Attribute utility functions range from 0 to 1; the least desirable value, xi 0 , of an attribute (over its range of plausible values) is assigned a utility value of 0, and the most desirable, xi*, is assigned a utility value of 1. That is, ui(xi 0 ) = 0 and ui(xi*) = 1. The utility value of an attribute value, xi, intermediate between the least desirable and most desirable is assessed by finding the value xi such that the decision maker is indifferent between receiving xi for sure, or, a lottery that yields xi 0 with probability pi or xi* with probability 1 - pi. From the mathematics of MAUT, ui(xi) = pi ui(xi 0 ) + (1 - pi) ui(xi*) = 1 - pi. (4) Repeat the process of indifference revealing until there are enough discrete points to approximate a continuous attribute utility function. (5) Combine the individual attribute utility functions to form a multiattribute utility function. This is also done using simple lotteries to reveal indifference between receiving a particular set of attribute values with certainty, or, a lottery of attribute values. In its simplest form, the resultant multiattribute utility function is a weighted sum of the individual attribute utility functions. (6) Evaluate each alternative using the multiattribute utility function.
The most difficult problem with MAUT is getting the decision makers or evaluators to think in terms of lotteries. This can often be overcome by an experienced interviewer. MAUT is based on a set of mathematical axioms about the way individuals should behave when confronted by uncertainty. Logical consistency in ranking alternatives is assured so long as evaluators adhere to the axioms; no guarantee can be made that this will always be the case. An extended discussion of MAUT is given in Keeney and Raiffa, Decisions with Multiple Objectives: Preferences and Value Tradeoffs, 1976. A textbook application of MAUT to a NASA problem can be found in Jeffrey H. Smith, et al., An Application of Multiaffribute Decision Analysis to the Space Station Freedom Program, Case Study: Automation and Robotics Technol ogy Evaluation, 1990. either technique, the individual evaluators may tend to cannot be described by a set of continuous reflect the institutional biases and preferences of their mathematical relationships. respective organizations. The results, therefore, may
depend on the mix of evaluators. (See sidebars on Selection Rules When Uncertainty Predominates. AHP and MAUT.) When the measures of system effectiveness, Another multi-objective selection rule is to performance or technical attributes, and system cost choose the alternative with the highest figure of merit for the alternatives in the trade study look like those from among those that meet specified individual for alternative C in Figure 22, the selection of the best objectives. This selection rule is used extensively by alternative may need to be handled differently. This is Source Evaluation Boards (SEBs) in the NASA because of the general propensity of decision makers procurement process. Each proposal, from among to show risk-averse behavior when dealing with large those meeting specific technical objectives variations in cost and/or effectiveness outcomes. In (requirements), is scored on such attributes as such cases, the expected value (i.e., the mean) of technical design, price, systems engineering process some stochastic outcome variable is not a satisfactory quality, etc. In applying this rule, the attributes being point measure of that variable. scored by the SEB are known to the bidders, but their To handle this class of decision problem, the weighing may not be. (See NHB 5103.6B.) system engineer may wish to invoke a von Neumann- In trade studies where no measure of system Morgenstem selection rule. In this case, alternatives effectiveness can be constructed, but performance or are treated as "gambles" (or lotteries). The probability technical attributes can be quantified, a possible of each outcome is also known or can be subjectively selection rule is to choose the alternative that estimated, usually by creating a decision tree. The minimizes cost for given levels of performance or von Neumann-Morgenstem selection rule applies a technical attributes. This rule presupposes that separately developed utility function to each outcome, system cost can be unambiguously measured, and is and chooses the alternative that maximizes the related to the all of the quantified performance or expected utility. This selection rule is easy to apply technical attributes that are considered constraints. when the lottery outcomes can be measured in Practical application again requires that all of the dollars. Although multi-attribute cases are more alternatives be put on an equal basis with respect to complex, the principle remains the same. the performance or technical attributes. This may not The basis for the von Neumann-Morgenstem be practical for trade studies in which the alternatives selection rule is a set of mathematical axioms about
NASA Systems Engineering Handbook
Page 67 how individuals should behave when confronted by
uncertainty. Practical application of this rule requires This section deals with the role of costs in an ability to enumerate each "state of nature" the systems analysis and engineering process, how to (hereafter, simply called "state"), knowledge of the measure it, how to control it, and how to obtain outcome associated with each enumerated state for estimates of it. The reason costs and their estimates each alternative, the probabilities for the various are of great importance in systems engineering goes states, and a mathematical expression for the back to the principal objective of systems engineering: decision maker's utility function. This selection rule fulfilling the system's goals in the most cost-effective has also found use in the evaluation of system manner. The cost of each alternative should be one of procurement alternatives. See Section 4.6.3 for a the most important outcome variables in trade studies discussion of some related topics, including decision performed during the systems engineering process. analysis, decision trees, and probabilistic risk One role, then, for cost estimates is in assessment. helping to choose rationally among alternatives. Another selection rule for this class of Another is as a control mechanism during the project decision problem is called the minimax rule. To apply life cycle. Cost measures produced for project life it, the sys-tem engineer computes a loss function for cycle reviews are important in determining whether each enumerated state for each alternative. This rule the system goals and objectives are still deemed valid chooses the alternative that minimizes the maximum and achievable, and whether constraints and loss. Practical application requires an ability to boundaries are worth maintaining. These measures enumerate each state and define the loss function. are also useful in determining whether system goals Because of its "worst case" feature, this rule has and objectives have properly flowed down through to found some application in military systems. the various subsystems.
As system designs and operational concepts 5.1.4 Trade Study Process: Summary mature, cost estimates should mature as well. At each
review, cost estimates need to be presented and System architecture and design decisions will be compared to the funds likely to be available to made. The purpose of the trade study process is to complete the project. The cost estimates presented at ensure that they move the design toward an optimum. early reviews must be given special attention since The basic steps in that process are: they usually form the basis under which authority to
proceed with the project is given. The system Understand what the system's goals, objectives, and engineer must be able to provide realistic cost constraints are, and what the system must do to meet estimates to the project manager. In the absence of them—that is, understand the functional requirements such estimates, overruns are likely to occur, and the in the operating environment. credibility of the entire system development process, Devise some alternative means to meet the functional both internal and exter-nal, is threatened. requirements. In the early phases of the project life
cycle, this means focusing on system architectures; in 5.2.1 Life-Cycle Cost and Other Cost later phases, emphasis is given to system designs. Measures Evaluate these alternatives in terms of the outcome
variables (system effectiveness, its underlying A number of questions need to be addressed so that performance or technical attributes, and system cost). costs are properly treated in systems analysis and Mathematical models are useful in this step not only engineering. These questions include: for forcing recognition of the relationships among the
outcome variables, but also for helping to determine • What costs should be counted? what the performance requirements must be • How should costs occurring at different times be quantitatively. treated? Rank the alternatives according to an appropriate • What about costs that cannot easily be measured selection rule. in dollars? Drop less-promising alternatives and proceed to next
level of resolution, if needed. What Costs Should be Counted. The most
comprehensive measure of the cost of an alternative This process cannot be done as an isolated is its life-cycle cost. According to NHB 7120.5, a activity. To make it work effectively, individuals with system's life-cycle cost is "the total of the direct, different skills—system engineers, design engineers, indirect, recurring, nonrecurring, and other related specialty engineers, program analysts, decision costs incurred, or estimated to be incurred in the scientists, and project managers—must cooperate. design, development, production, operation, The right quantitative methods and selection rule maintenance, support, and retirement [of it] over its must be used. Trade study assumptions, models, and planned life span." A less formal definition of a results must be documented as part of the project system's life-cycle cost is the total cost of acquiring, archives. owning, and disposing of it over its entire lifetime.
System life-cycle cost should be estimated and used 5.2 Cost Definition and Modeling in the evaluation of alternatives during trade studies.
NASA Systems Engineering Handbook
Page 68 The system engineer should include in the life-cycle regarding the way future resources are spent can be cost those resources, like civil service work-years, made. Sunk costs may alter the cost of continuing that may not require explicit project expenditures. A with a particular alternative relative to others, but system's life-cycle cost, when properly computed, is when choosing among alternatives, only those costs the best measure of its cost to NASA. that remain should be counted. Life-cycle cost has several components, as At the end of its useful life, a system may shown in Figure 25. Applying the informal definition have a positive residual or salvage value. This value above, life-cycle cost consists of (a) the costs of exists if the system can be sold, bartered, or used by acquiring a usable system, (b) the costs of operating another system. This value needs to be counted in and supporting it over its useful life, and (c) the cost of life-cycle cost, and is generally treated as a negative disposing of it at the end of its useful life. cost. The system acquisition cost includes more
than the DDT&E and procurement of the hardware Costs Occurring Over Time. The life-cycle cost and software; it also includes the other start-up costs combines costs that typically occur over a period of resulting from the need for initial training of personnel, several years. Costs incurred in different years cannot initial spares, the system's technical documentation, be treated the same because they, in fact, represent support equipment, facilities, and any launch services different resources to society. A dollar wisely invested needed to place the system at its intended operational today will return somewhat more than a dollar next site. year. Treating a dollar today the same as a dollar next The costs of operating and supporting the year ignores this potential trade. system include, but are not limited to, operations
personnel and supporting activities, ongoing
integrated logistics support, and pre-planned product
improvement. For a major system, these costs are Discounting future costs is a way of making often substantial on an annual basis, and when costs occurring in different years commensurable. accumulated over years of operations can constitute When applied to a stream of future costs, the the majority of life -cycle cost. discounting procedure yields the present discounted At the start of the project life cycle, all of value (PDV) of that stream. The effect of discounting these costs lie in the future. At any point in the project is to reduce the contribution of costs incurred in the life cycle, some costs will have been expended. future relative to costs incurred in the near term. These expended resources are known as sunk costs. Discounting should be performed whether or not there For the purpose of doing trade studies, the sunk costs is inflation, though care must be taken to ensure the of any alternative under consideration are irrelevant, right discount rate is used. (See sidebar on PDV.) no matter how large. The only costs relevant to In trade studies, different alternatives often current design trades are those that lie in the future. have cost streams that differ with respect to time. One The logic is straightforward: the way resources were alternative with higher acquisition costs than another spent in the past cannot be changed. Only decisions may offer lower operations and support costs. Without
Calculating Present Discounted Value
Calculating the PDV is a way of reducing a stream NASA Systems Engineering Handbo of costs to a single number ok so that alternative Page 69 streams can be compared unambiguously. Several formulas for PDV are used, depending on whether time is to be treated as a discrete or a continuous variable, and whether the project's According to NHB 7120.5, every NASA time horizon is finite or not. The following equation program/project must: is useful for evaluating system alternatives when
costs have been estimated as yearly amounts, • Develop and maintain an effective capability to and the project's anticipated useful life is T years. estimate, assess, monitor, and control its life- For alternative i, cycle cost throughout the project life cycle
• Relate life-cycle cost estimates to a well-defined T PDV technical baseline, detailed project schedule, and i = Σ Cit (1 + r) -t t=0
set of cost-estimating assumptions where r is the annual discount rate and C • Identify the life-cycle costs of alternative levels of it is the esti-mated cost of alternative i in year t. system requirements and capability Once the yearly costs have been estimated, the • Report time-phased acquisition cost and techni- choice of the discount rate is crucial to the cal parameters to NASA Headquarters. evaluation since it ultimately affects how much or
how little runout costs contribute to the PDV. There are a number of actions the system While calculating the PDV is generally accepted engineer can take to effect these objectives. Early as the way to clear with costs occurring over a decisions in the systems engineering process tend to period of years, there is much disagreement and have the greatest effect on the resultant system life- confusion over the appropriate discount rate to cycle cost. Typically, by the time the preferred system apply in systems engineering trade studies. The architecture is selected, between 50 and 70 percent Office of Management and Budget (OMB) has of the system's life-cycle cost has been "locked in." By mandated the use of a rate of seven percent for the time a preliminary system design is selected, this NASA systems when constant dollars (dollars figure may be as high as 90 percent. This presents a adjusted to the price level as of some fixed point major dilemma to the system engineer, who must lead in time) are used in the equation. When nominal this selection process. Just at the time when dollars (sometimes called then-year, runout or decisions are most critical, the state of information real-year dollars) are used, the OMB-mandated about the alternatives is least certain. Uncertainty annual rate should be increased by the inflation about costs is a fact of systems engineering. rate assumed for that year. Either approach yields This suggests that efforts to acquire better essentially the same PDV. For details, see OMB information about the life-cycle cost of each Circular A-94, Guidelines and Discount Rates for alternative early in the project life-cycle (Phases A Benefit Cost Analysis of Federal Programs, and B) potentially have very high payoffs. The system October 1992 engineer needs to understand what the principal life- cycle cost drivers are. Some major questions to discounting, it would be difficult to know which stream consider are: How much does each already on well- truly represents the lower life-cycle cost. Trade understood technology? Can the system be studies should report the PDV of life-cycle cost for manufactured using routine processes or are higher each alternative as an outcome variable. precision processes required? What tests are needed
to verify and validate each alternative system design, Difficult-To-Measure Costs. In practice, some costs and how costly are they? What reliability levels are pose special problems. These special problems, needed by each alternative? What environmental and which are not unique to NASA systems, usually occur safety requirements must be satisfied? in two areas: (a) when alternatives have differences in For a system whose operational life is the irreducible chances of loss of life and (b) when expected to be long and to involve complex activities, externalities are present. Two examples of the life-cycle cost is likely to be far greater than the externalities that impose costs are pollution caused by acquisition costs alone. Consequently, it is particularly some launch systems and the creation of orbital important with such a system to bring in the specialty debris. Because it is difficult to place a dollar figure on engineering disciplines such as reliability, these resource uses, they are generally called maintainability, supportability, and operations incommensurable costs. The general treatment of engineering early in the systems engineering process, these types of costs in trade studies is not to ignore as they are essential to proper life-cycle cost them, but instead to keep track of them along with estimation. dollar costs. Another way of acquiring better information
on the cost of alternatives is for the project to have 5.2.2 Controlling Life-Cycle Costs independent cost estimates prepared for comparison
purposes. The project manager/system engineer must Another mechanism for controlling life-cycle ensure that the system life-cycle cost (established at cost is to establish a life-cycle cost management the end of Phase A) is initially compatible with NASA's program as part of the project's management budget and strategic priorities and that it approach. (Life-cycle cost management has demonstratively remains so over the project life cycle. traditionally been called design-to-life-cycle cost.) Such a program establishes life-cycle cost as a
NASA Systems Engineering Handbook
Page 70 design goal, perhaps with sub-goals for acquisition state of information available to the cost analyst at costs or operations and support costs. More each point in the project life cycle. Table 4 shows this specifically, the objectives of a life-cycle cost dependence. management program are to:
• Identify a common set of ground rules and assumptions for life-cycle cost estimation • Ensure that best-practice methods, tools, and models are used for life-cycle cost analysis • Track the estimated life-cycle cost throughout the project life cycle, and, most important • Integrate life-cycle cost considerations into the Parametric (or "top-down") cost models are design and development process via trade most useful when only a few key variables are known studies and formal change request assessments. or can be estimated. The most common example of a
parametric model is the statistical Cost Estimating Trade studies and formal change request Relationship (CER). A single equation (or set of assessments provide the means to balance the equations) is derived from a set of historical data effectiveness and life-cycle cost of the system. The relating one or more of a system's characteristics to complexity of integrating life-cycle cost considerations its cost using well-established statistical methods. A into the design and development process should not number of statistical CERs have been developed to be underestimated, but neither should the benefits, Statistical Cost Estimating Relationships: Example which can be measured in terms of greater cost- and Pitfalls effectiveness. The existence of a rich set of potential
life-cycle cost trades makes this complexity even One model familiar to most cost analysts is the greater. historically based CER. In its usual form, this The Space Station Alpha Program provides model is a linear expression with cost (the many examples of such potential trades. As one dependent variable) as a function of one or more example, consider the life-cycle cost effect of descriptive characteristics. The coefficients of the increasing the mean time between failures (MTBF) of linear expression are estimated by fitting historical Alpha's Orbital Replacement Units (ORUs). This is data from previously completed projects of a likely to increase the acquisition cost, and may similar nature using statistical regression increase the mass of the station. However, annual techniques. This type of model is analytic and maintenance hours and the weight of annual deterministic. An example of this type of model for replacement spares will decline. The same station estimating the first unit cost, C, of a space- availability may be achieved with fewer on-orbit qualified Earth-orbiting receiver/exciter is: spares, thus saving precious internal volume used for
spares storage. For ORUs external to the station, the In C = 3.822 + 1.110 In W + 0.436 In z amount of extravehicular activity, with its associated
logistics support, will also decline. With such complex where W is the receiver/exciter's weight, and z is interactions, it is difficult to know what the optimum the number of receiver boxes; In is the natural point is. At a minimum, the system engineer must logarithm function. (Source: U.S. Air Force have the capability to assess the life-cycle cost of Systems Command-Space Division, Unmanned each alternative. (See Appendix B.8 on the operations Space Vehicle Cost Model, Seventh Edition, and operations cost effects of ORU MTBF and August 1994.) CERs are used extensively in Section 6.5 on Integrated Logistics Support.) advanced technology systems, and have been
challenged on both theoretical and practical 5.2.3 Cost Estimating grounds. One challenge can be mounted on the
basis of the assumption of an unchanging The techniques used to estimate each life- relationship between cost and the independent cycle cost component usually change as the project variables. Others have questioned the validity of life cycle proceeds. Methods and tools used to CERs based on weight, a common indevariable in support budget estimates and life-cycle cost trades in many models, in light of advances in electronic Phase A may not be sufficiently detailed to support packaging and composite materials. Objections to those activities during Phase C/D. Further, as the using statistical CERs also include problems of project life cycle proceeds, the requirements and the input accuracy, low statistical significance due to system design mature as well, revealing greater detail limited data points, ignoring the statistical in the Work Breakdown Structure (WBS). This should confidence bands, and, lastly, biases in the enable the application of cost estimating techniques underlying data. at a greater resolution. Three techniques are described below -- parametric cost models, analogy, and grass-roots. Typically, the choice of technique depends on the
NASA Systems Engineering Handbook
Page 71 estimate a spacecraft's hardware acquisition cost. Grass-roots (or "bottom-up") estimates are These typically use an estimate of its weight and the result of rolling up the costs estimated by each other characteristics, such as design complexity and organization performing work described in the WBS. inheritance, to obtain an estimate of cost. Similarly, Properly done, grass-roots estimates can be quite software CERs have been developed as well, relying accurate, but each time a `'what if" question is raised, on judgments about source lines of code and other a new estimate needs to be made. Each change of factors to obtain development costs. (See sidebar on assumptions voids at least part of the old estimate. statistical CERs.) Because the process of obtaining grassroots Another type of parametric model relies on estimates is typically time-consuming and labor- accepted relationships. One common example can be intensive, the number of such estimates that can be found in the application of logistics relationships to the prepared during trade studies is in reality severely estimation of repair costs and initial and recurring limited. spares costs. The validity of these cost estimates also Whatever technique is used, the direct cost depends on the quality of the input parameters. of each hardware and software element often needs The principal advantages of parametric cost to be "wrapped" (multiplied by a factor greater than models are that the results are reproducible, are more one) to cover the costs of integration and test, easily documented than other methods, and often can program management systems engineering, etc. be produced with the least amount of time and effort. These additional costs are called system-level costs, This makes a properly constructed performance- and are often calculated as percentages of the direct based parametric cost model useful in early trade costs. studies.
Analogy is another way of estimating costs. Using Parametric Cost Models. A number of When a new system or component has functional and parametric cost models are available for costing performance characteristics similar to an existing one NASA systems. Some of these are shown in Table 5. whose cost is known, the known cost can be adjusted Unfortunately, none alone is sufficient to estimate life- to reflect engineering judgments about differences. cycle cost. Assembling an estimate of life-cycle cost often requires that several different models (along with the other two techniques) be used together. To integrate the costs being estimated by these different models, the system engineer should ensure that the inputs to and assumptions of the models are consistent, that all relevant life-cycle cost components are covered, and that the timing of costs is correct.
The system engineer may sometimes find it necessary to make some adjustments to model results to achieve a life-cycle cost estimate. One such situation occurs when the results of different models, whose estimates are expressed in different year constant dollars, must be combined. In that case, an appropriate inflation factor must be applied. Another such situation arises when a model produces a cost estimate for the first unit of a hardware item, but the project requires multiple units. In that case, a learning curve can be applied to the first unit cost to obtain the required multiple-unit estimate. (See sidebar on learning curve theory.) A third situation requiring additional calculation occurs when a model provides a cost estimate of the total
NASA Systems Engineering Handbook
Learning Curve Theory
The learning curve (also known as the progress or
experience curve) is the time-honored way of
dealing with the empirical observation that the unit
of fabricating multiple units of complex systems
like aircraft and spacecraft tends to decline as the
number increases. In its usual form, the theory
states that as the total quantity produced doubles,
the cost per unit decreases by a constant
percentage. The cost per unit may be either the
average cost over the number produced, or the cost of the last unit produced. In the first case, the An Example of a Cost Spreader Function: The curve is generally known as the cumulative Beta Curve average learning curve; in the second case, it is
known as the unit learning curve. Both One technique for spreading estimated acquisition formulations have essentially the same rate of costs over time is to apply the beta curve. This learning. fifth-degree polynomial, which was developed at Let C(1) be the unit cost of the first JSC in the late 1960s, expresses the cumulative production unit, and C(Q) be the unit cost of the Q cost fraction as a function of the cumulative time fraction, T: th production unit, then learning curve theory states there is a number, b, such that
C(Q) = C(1) Q Cum Cost Fraction = 10T 2 (1 - T)2 (A + BT) + T 4 b
(5 - 4T) for 0 ≤T ≤1. The number b is specified by the rate of learning.
A 90 percent learning rate means that the unit A and B are parameters (with 0 ≤A + B ≤1 ) that cost of the second production unit is 90 percent of determine the shape of the beta curve. In the first production unit cost; the unit cost of the particular, these parameters control what fraction fourth is 90 percent of the unit cost of the second, of the cumulative cost has been expended when and so on. In general, the ratio of C(2Q) to C(Q) is 50 percent of the cumulative time has been the learning rate, LR, expressed as a decimal; reached. The figure below shows three examples: using the above equation, b = In (LR)/ln 2, where with A = 1 and B = 0 as in curve (1), 81 percent of In is the natural logarithm. the costs have been expended at 50 percent of Learning curve theory may not always be the cumulative time; with A = 0 and B = 1 as in applicable because, for example, the time rate of curve (2), 50 percent of the costs have been production has no effect on the basic equation. expended at 50 percent of the cumulative time; in For more detail on learning curves, including curve (3) with A = B = 0, it's 19 percent empirical studies and tables for various learning rates, see Harold Asher, Cost -Quantity Relationships in the Airframe Industry, R-291, The Rand Corporation, 1956. .
Typically, JSC uses a 50 percent profile
with A = 0 and B = 1, or a 60 percent profile with A
= 0.32 and B = 0.68, based on data from previous
NASA Systems Engineering Handbook
Page 73 acquisition effort, but doesn't take into account the underlying system performance or technical attributes multi-year nature of that effort. The system engineer that mathematically determine system effectiveness. can use a set of "annual cost spreaders" based on the (Note that each of the above facets may have several typical ramping-up and subsequent ramping-down of dimensions. If this is the case, then each dimension acquisition costs for that type of project. (See sidebar can be considered a function of the underlying system on beta curves.) Although some general parametric performance or technical attributes.) Ideally, there is a cost models for space systems are already available, strong connection between the system functional their proper use usually requires a considerable analysis, system effectiveness measure, and the investment in reaming time. For projects outside of functional and performance requirements. The same the domains of these existing cost models, new cost functional analysis that results in the functional re- models may be needed to support trade studies. quirements flowdown also yields the system Efforts to develop these need to begin early in the effectiveness and performance measures that are project life cycle to ensure their timely application optimized (through trade studies) to produce the during the systems engineering process. Whether system performance requirements. existing models or newly created ones are used, the An effectiveness measurement method or SEMP and its associated life-cycle cost management model should provide trustworthy relationships plan should identify which (and how) models are to be between these underlying performance or technical used during each phase of the project life cycle. attributes and the measure of system effectiveness.
Early in the project life cycle, the effectiveness model 5.3 Effectiveness Definition and Modeling may embody simple parametric relationships among
the high-level performance and technical attributes The concept of system effectiveness is more and the measure of system effectiveness. In the later elusive than that of cost. Yet, it is also one of the most phases of the project life cycle, the effectiveness important factors to consider in trade studies. In model may use more complex relationships requiring selecting among alternatives, the system engineer more detailed, specific data on operational scenarios must take into account system effectiveness, even and on each of the alternatives. In other words, early when it is difficult to define and measure reliably. effectiveness modeling during architecture trade A measure of system effectiveness studies may take a functional view, while later describes the accomplishment of the system's goals modeling during design trade studies may shift to a and objectives quantitatively. Each system (or family product view. This is not unlike the progression of the of systems with identical goals and objectives) has its cost modeling from simple parametrics to more own measure of system effectiveness. There is no detailed grass-roots estimates. universal measure of effectiveness for NASA The system engineer must tailor the effectiveness systems, and no natural units with which to express measure and its measurement method to the effectiveness. Further, effectiveness is dependent on resolution of operational concept mature, the context (i.e., project or supersystem) in which the system is being operated, and any measure of it must Practical Pitfalls in Using Effectiveness Measures in take this into account. The system engineer can, Trade Studies however, exploit a tew basic, common features of
system effectiveness in developing strategies for Obtaining trustworthy relationships among the measuring it. system performance or technical attributes and
system effec-tiveness is often difficult, Purported 5.3.1 Strategies for Measuring System effectiveness mod -els often only treat one or two Effectiveness of the facets described in the text. Supporting
models may not have been properly integrated. System effectiveness is almost always Data are often incomplete or unreliable. Under multifaceted, and is typically the result of the these conditions, reported system effectiveness combined effects of: results for different alternatives in a trade study
may show only the relative effectiveness of the • System output quality • Size or quantity of system alternatives within the context of the trade study. output The system engineer must recognize the practical • System coverage or comprehensiveness pitfalls of using such results. • System output timeliness • System availability.
effectiveness estimates should mature as well. The A measure of effectiveness and its system engineer must be able to provide realistic measurement method (i.e., model) should focus on estimates of system effectiveness and its underlying the critical facet (or facets) of effectiveness for the performance and technical attributes not only for trade trade study issue under consideration. Which facets studies, but for project management through the are critical can often be deduced from the tracking of TPMs. accompanying functional analysis. The functional This discussion so far has been predicated analysis is also very useful in helping to identify the on one accepted measure of system effectiveness.
NASA Systems Engineering Handbook
Page 74 The job of computing system effectiveness is flight systems. No attempt has been made to considerably easier when the system engineer has a enumerate all possible performance or technical single measure and measurement method (model). attributes, or to fill in each possible entry in the table; But, as with costs, a single measure may not be its purpose is illustrative only. possible. When it does not exist, the system engineer For many of the systems shown in the table, must fall back to computing the critical high-level, but system effectiveness is largely driven by continual (or nevertheless still underlying, system performance or continuous) operations at some level of output over a technical attributes. In effect, these high-level period of years. This is in contradistinction to an performance or technical attributes are elevated to the Apollo-type project, in which the effectiveness is status of measures of (system) effectiveness (MoEs) largely determined by the successful completion of a for trade study purposes, even though they do not single flight within a clearly specified time horizon. represent a truly comprehensive measure of system The measures of effectiveness in these two cases are effectiveness. correspondingly different. In the former case (with its These high-level performance or technical lengthy operational phase and continual output), attributes might represent one of the facets described system effectiveness measures need to incorporate above, or they may be only components of one. They quantitative measures of availability. The system are likely to re quire knowledge or estimates of lower- engineer accomplishes that through the involvement order performance or technical attributes. Figure 26 of the specialty engineers and the application of shows how system effectiveness might look in an specialized models described in the next section. hierarchical tree structure. This figure corresponds, in
some sense, to Figure 25 on life-cycle cost, though 5.3.3 Availability and Logistics Supportability rolling up by simple addition obviously does not apply Modeling to system effectiveness.
Lastly, it must be recognized that system One reason for emphasizing availability and effectiveness, like system cost, is uncertain. This fact logistics supportability in this chapter is that future is given a fuller treatment in Section 5.4. NASA systems are less likely to be of the "launch-
and-logistically forget" type. To the extent that logistic 5.3.2 NASA System Effectiveness Measures support considerations are major determinants of
system effectiveness during operations, it is essential The facets of system effectiveness in Figure that logistics support be thoroughly analyzed in trade 26 are generic. Not all must apply to a particular studies during the earlier phases of the project life system. The system engineer must determine which cycle. A second reason is that availability and logistics performance or technical attributes make up system supportability have been rich domains for effectiveness, and how they should be combined, on methodology and model development. The increasing a system-by-system basis. Table 6 provides sophistication of the methods and models has allowed examples of how each facet of system effectiveness the system-wide effects of different support could be interpreted for specific classes of NASA
NASA Systems Engineering Handbook
Page 75 alternatives to be more easily predicted. In turn, this inexact. Some logistics supportability models may means more opportunities to improve system deal with a single resource; others may deal with effectiveness (or to lower life-cycle cost) through the several resources simultaneously. They may take the integration of logistics considerations in the system form of a simple spreadsheet or a large computer design. simulation. Greater capability from these types of Availability models relate system design and models is gen-erally achieved only at greater expense integrated logistics support technical attributes to the in time and effort. The system engineer must availability component of the system effectiveness determine what availability and logistics supportability measure. This type of model predicts the resulting models are needed for each new system, taking into system availability as a function of the system account the unique operations and logistics concepts component failure and repair rates and the logistics and environment of that system. Generally both types support resources and policies. (See sidebar on of models are needed in the trade study process to measures of availability.) transform specialty engineering data into forms more Logistics supportability models relate system useful to the system engineer. Which availability and design and integrated logistics support technical logistics supportability models are used during each attributes to one or more "resource requirements" phase of the protect life cycle should be identified in needed to operate the system in the accomplishment the SEMP. of its goals and objectives. This type of model Another role for these models is to provide focuses, for example, on the system maintenance quantitative requirements for incorporation into the requirements, number and location of spares, system's formal Integrated Logistics Support (ILS) processing facility requirements, and even optimal Plan. Figure 27 shows the role of availability and inspection policies. In the past, logistics supportability logistics supportability models in the trade study models have typically been based on measures process. pertaining to that particular resource or function alone. Essential to obtaining useful products from For example, a system's desired inventory of spares any availability and/or logistics supportability model is was detemmined on the basis of meeting measures of the collection of high quality specialty engineering supply efficiency, such as percent of demands met. data for each alternative system design. (Some of This tended to lead to suboptimal resource these data are also used in probabilistic risk requirements from the system's point of view. More assessments performed in risk management modem models of logistics supportability base activities.) The system engineer must coordinate resource requirements on the system availability efforts to collect and maintain these data in a format effects. (See sidebar on logistics supportability suitable to the trade studies being performed. This models.) task is made considerably easier by using digital
databases in relational table formats such as the one Some availability models can be used to currently under development for MIL-STD- 1388-2B. determine a logistics resource requirement by Continuing availability and logistics computing the quantity of that resource needed to supportability modeling and data collection through achieve a particular level of availability, holding other the operations phase permits operations trend logistics resources fixed. The line between availability analysis and assessment on the system (e.g., is models and logistics supportability models can be system availability declining or improving?) In general,
NASA Systems Engineering Handbook
Page 76 Measures of Availability Availability can be calculated as the ratio of operating time to total time, where the denominator, total time, can be divided into operating time ("uptime") and "downtime." System availability depends on any factor that contributes to downtime. Underpinning system availability, then, are the reliability and maintainability attributes of the system design, but other logistics support factors can also play significant roles. If these attributes and support factors, and the operat-ing environment of the system are unchanging, then several measures of steady-state availability can be readily calcu-lated. (When steady-state conditions do not apply, availability can be calculated, but is made considerably more complex by the dynamic nature of the underlying conditions.) The system engineer should be familiar with the equations below describing three concepts of steady-state availability for systems that can be repaired.
• Inherent = MTTF / (MTTF + MTTR) • Achieved = MTTMA / (MTTMA + MMT) • Operational = MTTMA / (MTTMA + MMT + MLDT) = MTTMA / (MTTMA + MDT) where: MTTF = Mean time to failure MTTR = Mean time to repair (corrective) MTTMA = Mean time to a maintenance action (corrective and preventive) MMT = Mean (active) maintenance time (corrective and preventative) MLDT = Mean logistics delay time (includes downtime due to administrative delays, and waiting for spares, maintenance personnel, or supplies) MDT = Mean downtime (includes downtime due to (active) maintenance and logistics delays)
Availability measures can be also calculated at a point in time, or as an average over a period of time. A further, but manageable, complication in calculating availability takes into account degraded modes of operation for redundant systems. For systems that cannot be repaired, availability and reliability are equal. (See sidebar on page 92.) this kind of analysis and assessment is extremely effectiveness is needed when point estimates for useful in identifying potential areas for product these outcome variables do not "tell the whole improvement such as greater system reliability, lower story"—that is, when information about the variability cost logistics support, and better maintenance and in a system's projected cost and effectiveness is spares poli cies. (See Section 6.5 for more on relevant to making the right choices about that Integrated Logistics Support.) system. When these uncertainties have the potential
to drive a decision, the systems or program analyst 5.4 Probabilistic Treatment of Cost and must do more than just acknowledge that they exist. Effectiveness Some useful techniques for modeling the effects of
uncertainty are described below in Section 5.4.2. A probabilistic treatment of cost and These techniques can be applied to both cost models Logistics Supportability Models:
Two Examples Logistics supportability models utilize the reliability and maintainability attributes a particular system design, and other logistics system variables, to quantify the demands (i.e., requirements) for scarce logistics resources during operations. The models described here were both developed for Space Station Freedom. One is a stochastic simulation in which each run is a "trial" drawn from a population of outcomes. Multiple runs must be made to develop accurate estimates of means and variances for the variables of interest. The other is a deterministic analytic model. Logistic supportability models may be of either type. These two models deal with the unique logistics environment of Freedom. SIMSYLS is a comprehensive stochastic simulation of on-orbit maintenance and logistics resupply of Freedom. It provides estimates of the demand (means and variances) for maintenance resources such as EVA and IVA, as well as for logistics upmass and downmass resources. In addition to the effects of actual and false ORU failures, the effects of various other stochastic events such as launch vehicle and ground repair delays can be quantified. SIMSYLS also produces several measures of operational availability. The model can be used in its availability mode or in its resource requirements mode. M-SPARE is an availability-based optimal spares model. It determines the mix of ORU spares at any spares budget level that maximizes station availability, defined as the probability that no ORU had more demands during a resupply cycle than it had spares to satisfy those demands. Unlike SIMSYLS, M-SPARE's availability measure deals only with the effect of spares. M-SPARE starts with a target availability (or budget) and determines the optimal inventory, a capability not possessed by SIMSYLS. For more detail, see DeJulio, E., SIMSYLS User's Guide, Boeing Aerospace Operations, February 1990, and Kline, Robert, et al., The M-SPARE Model, LMI, NS901R1, March 1990.
NASA Systems Engineering Handbook
Page 77 and effectiveness models, though the majority of Another set of model uncertainties that examples given are for cost models. contribute to the uncertainty in the cost estimate
concerns the model coefficients that have been
estimated from historical data. Even in a well-behaved 5.4.1 Sources of Uncertainty in Models statistical regression equation, the estimated
coefficients could have resulted from chance alone, There are a number a sources of uncertainty and therefore cost predictions made with the model in the kinds of models used in systems analysis. have to be stated in probabilistic terms. (Fortunately, Briefly, these are: the upper and lower bounds on cost for any desired
level of confidence can be easily calculated. Uncertainty about the correctness of the model's Presenting this information along with the cost structural equations, in particular whether the estimate is strongly recommended.) functional form chosen by the modeler is the best The above uncertainties are present even if representation of the relationship between an the cost model inputs that describe a new system are equation's inputs and output precisely known in Phase A. This is rarely true; more Uncertainty in model parameters, which are, in a very often, model inputs are subject to considerable real sense, also chosen by the modeler; this guesswork early in the project life cycle. The uncertainty is evident for model coefficients derived uncertainty in a model input can be expressed by from statistical regression, but even known physical attributing a probability distribution to it. This applies constants are subject to some uncertainty due to whether the input is a physical measure such as experimental or measurement error weight, or a subjective measure such as a "complexity Uncertainty in the true value of model inputs (e.g., factor." Model input uncertainty can extend even to a estimated weight or thermal properties) that describe grass-roots cost model that might be used in Phases a new system. C and D. In that case, the source of uncertainty is the
failure to identify and capture the "unknown- As an example, consider a cost model unknowns." The model inputs -- the costs estimated consisting of one or more statistical CERs. In the by each performing organization -- can then be early phases of the project life cycle (Phases A and thought of as variables having various probability B), this kind of model is commonly used to provide a distributions. cost estimate for a new NASA system. The project
manager needs to understand what confidence 5.4.2 Modeling Techniques for Handling he/she can have in that estimate. Uncertainty One set of uncertainties concerns whether
the input variables (for example, weight) are the The effect of model uncertainties is to induce proper explanatory variables for cost, and whether a uncertainty in the model's output. Quantifying these linear or log-linear form is more appropriate. Model uncertainties involves producing an overall probability misspecification is by no means rare, even for strictly distribution for the output variable, either in terms of engineering relationships. its probability density function (or mass function for discrete output variables) or its cumulative distribution
NASA Systems Engineering Handbook
Page 78 function. (See sidebar on cost S-curves.) Some efficient in many situations than Monte Carlo techniques for this are: simulation, described next.
• Analytic solution Monte Carlo Simulation. This technique is often • Decision analysis used to calculate an approximate solution to a • Monte Carlo simulation. stochastic model that is too complicated to be solved by analytic methods alone. A Monte Carlo simulation The Cost S-Curve is a way of sampling input points from their respective
domains in order to estimate the probability The cost S-curve gives the probability of a distribution of the output variable. In a simple Monte project's cost not exceeding a given cost estimate. Carlo analysis, a value for each uncertain input is This probability is sometimes called the budget drawn at random from its probability distribution, confidence level. This curve aids in establishing which can be either discrete or continuous. This set of the amount of contingency and Allowance for random values, one for each input, is used to Program Adjustment (APA) funds to set aside as compute the corresponding output value, as shown in a reserve against risk. Figure 28. The entire process is then repeated k times. These k output values constitute a random sample from the probability distribution over the output variable induced by the input probability distributions. For an example of the usefulness of this technique, recall Figures 2 (in Chapter 2) and 24 (this chapter), which show the projected cost and effectiveness of three alternative design concepts as probability "clouds." These clouds may be reasonably interpreted as the result of three system-level Monte Carlo simulations. The information displayed by the clouds is far greater than that embodied in point estimates for each of the alternatives. An advantage of the Monte Carlo technique
is that standard statistical tests can be applied to In the S-curve shown above, the project's estimate the precision of the resulting probability cost commitment provides only a 40 percent level distribution. This permits a calculation of the number of confidence; with reserves, the level is increased of runs (samples) needed to obtain a given level of to 50 percent. The steepness of the S-curve tells precision. If computing time or costs are a significant the project manager how much the level of constraint, there are several ways of reducing them confidence improves when a small amount of through more deliberate sampling strategies. See reserves are added. Note that an Estimate at Completion (EAC) S-curve could be used in conjunction with the risk management approach described for TPMs (see Section 4.9.2), as another method of cost status reporting and assessment meet.
Analytic Solution. When the structure of a model and its uncertainties permit, a closed-form analytic solution for the required probability density (or cumulative distribution) function is sometimes feasible. Examples can be found in simple reliability models (see Figure 29).
Decision Analysis. This technique, which was discussed in Section 4.6, also can produce a cumulative distribution function, though it is necessary to descretize any continuous input probability distributions. The more probability intervals that are used, the greater the accuracy of the results, but the larger the decision tree. Furthermore, each uncertain model input adds more than linear computational complexity to that tree, making this technique less
NASA Systems Engineering Handbook
Page 79 MSFC-HDBK-1912, Systems Engineering (Volume 2), for a discussion of these strategies. Commercial software to perform Monte Carlo simulation is available. These include add-in packages for some of the popular spreadsheets, as well as packages that allow the systems or program analyst to build an entire Monte Carlo model from scratch on a personal computer. These packages generally perform the needed computations in an efficient manner and provide graphical displays of the results, which is very helpful in communicating probabilistic information. For large applications of Monte Carlo simulation, such as those used in addressing logistics supportability, custom software may be needed. (See the sidebar on logistics supportability models.) Monte Carlo simulation is a fairly easy technique to apply. Also, what a particular combination of uncertainties mean can often be communicated more clearly to managers. A powerful example of this technique applied to NASA flight readiness certification is found in Moore, Ebbeler, and Creager, who combine Monte Carlo simulation with traditional reliability and risk analysis techniques.
NASA Systems Engineering Handbook
Page 80 6 Integrating Engineering Specialties Into the plans. Systems Engineering Process As part of the technical effort, specialty
engineers often perform tasks that are common This chapter discusses the basic concepts, across disciplines. Foremost, they apply specialized tech-niques, and products of some of the specialty analytical techniques to create information needed by engineering disciplines, and how they fit into the the project manager and system engineer. They also systems engineering process. help define and write system requirements in their
areas of expertise, and they review data packages, 6.1 Role of the Engineering Specialties engineering change requests (ECRs), test results,
and documentation for major project reviews. The Specialty engineers support the systems project manager and/or system engineer need to engineering process by applying specific knowledge ensure that the information and products so and analytic methods from a variety of engineering generated add value to the project commensurate specialty disciplines to ensure that the resulting with their cost. system is actually able to perform its mission in its The specialty engineering technical effort operational environment. These specialty engineering should also be well integrated both in time and disciplines typically include reliability, maintainability, content, not separate organizations and disciplines integrated logistics, test, fabrication/production, operating in near isolation (i.e., more like a basketball human factors, quality assurance, and safety team, rather than a golf foursome). This means, as an engineering. One view of the role of the engineering example, that the reliability engineer's FMECA (or specialties, then, is mission assurance. Part of the equivalent analysis) results are passed at the right system engineer's job is to see that these mission time to the maintainability engineer, whose assurance functions are coherently integrated into the maintenance analysis is subsequently incorporated project at the right times and that they address the into the logistics support analysis (LSA). LSA results, relevant issues. in turn, are passed to the project-level system Another idea used to explain the role of the engineer in time to be combined with other cost and engineering specialties is the "Design-for-X" concept. effectiveness data for a major trade study. The X stands for any of the engineering "ilities" (e.g., Concurrently, the reliability engineer's FMECA results reliability, testability, producibility, supportability) that are also passed to the risk manager to incorporate the project level system engineer needs to consider to critical items into the Critical Items List (CIL) when meet the project's goals/objectives. While the relevant deemed necessary, and to alert the PDT to develop engineering specialties may vary on NASA projects appropriate design or operations mitigation strategies. by virtue of their diverse nature, some are always The quality assurance engineer's effort should be in- needed. It is the system engineer's job to identify the tegrated with the reliability engineer's so that, for particular engineering specialities needed for his/her example, component failure rate assumptions in the tailored Product Development Team (PDT). The latter's reliability model are achieved or bettered by selected organizational approach to integrating the the actual (flight) hardware. This kind of process engineering specialities into the systems engineering harmony and timeliness is not easily realized in a process and the technical effort to be made should be project; it nevertheless remains a goal of systems summarized in the SEMP (Part III). Depending on the engineering. nature and scope of the project, the technical effort
may also need more detailed documentation in the 6.2 Reliability form of individual specialty engineering program
Reliability Relationships The system engineer should be familiar with the following reliability parameters and mathematical relationships for continuously operated systems Many reliability analyses assume that failures are random so that λ(t) = λ and the failure probability density follows an exponential distribution. In that case, R(t) = exp (-λt), and the Mean Time To Failure (MTTF) = 1/λ. Another popular assumption that has been shown to apply to many systems is a failure probability density that follows a Weibull distribution; in that case, the hazard rate λ(t) satisfies a simple power law as a function of t. With the proper choice of Weibull parameters, the constant hazard rate can be recovered as a special case. While these (or similar) assumptions may be analytically convenient, a system's actual hazard rate may be less predictable. (Also see bathtub curve sidebar!)
NASA Systems Engineering Handbook
Page 81 Reliability can be defined as the probability The reliability engineer works with other that a device, product, or system will not fail for a specialty engineers (e.g., the quality assurance, given period of time under specified operating maintainability, verification, and producibility conditions. Reliability is an inherent system design engineers) on system reliability issues. On small characteristic. As a principal contributing factor in projects, the reliability engineer may perform some or operations and support costs and in system all of these other jobs as well. effectiveness (see Figure 26), reliability plays a key
role in determining the system's cost-effectiveness. 6.2.2 Reliability Program Planning
6.2.1 Role of the Reliability Engineer The reliability program for a project
describes what activities will be undertaken in support Reliability engineering is a major specialty of reliability engineering. The reliability engineer disci-pline that contributes to the goal of a cost- develops a reliability program considering its cost, effective system. This is primarily accomplished in the schedule, and risk implications. This planning should systems engineering process through an active role in begin during Phase A. The project manager/system implementing specific design features to ensure that engineer must work with the reliability engineer to the system can perform in the predicted physical develop an appropriate reliability program as many environments throughout the mission, and by making factors need to be considered in developing this independent predictions of system reliability for program. These factors include: design trades and for (test program, operations, and Lunar Excursion Module (LEM) Reliability integrated logistics support) planning.
The reliability engineer performs several Part of the reliability engineer's job is to develop tasks, which are explained in more detail in NHB an understanding of the underlying physical and 5300.4(1A -1), Reliability Program Requirements for human-induced causes of failures, rather than Aeronautical and Space System Contractors. In brief, assuming that all failures are random. According these tasks include: to Joseph Gavin, Director of the LEM Program at
Grumman, "after about 10 years of testing of • Developing and executing a reliability program individual [LEM] components and subsystems, plan [NASA] found something like 14,000 anomalies, • Developing and refining reliability prediction only 22 of which escaped definite understanding.” models, including associated environmental (e.g., vibration, acoustic, thermal, and EMI/EMC) models, and predictions of system reliability.
These models and predictions should reflect • NASA payload classification. The reliability applicable experience from previous projects. program's analytic content and its documentation • Establishing and allocating reliability goals and of problems and failures are generally more environmental design requirements extensive for a Class A payload than for a Class • Supporting design trade studies covering such D one. (See Appendix B.3 for classification issues as the degree of redundancy and reliability guidelines.) vs. maintainability • Mission environmental risks. Several mission • Supporting risk management by identifying environmental models may need to be design attributes likely to result in reliability developed. For flight projects, these include problems and recommending appropriate risk ground (transportation and handling), launch, on- mitigations orbit (Earth or other), and planetary • Developing reliability data for timely use in the environments. In addition, the reliability engineer project's maintainability and ILS programs must address design and verification • Developing environmental test requirements and requirements for each such environment. specifications for hardware qualification. The • Degree of design inheritance and reliability engineer may provide technical analysis hardware/software reuse. and justification for eliminating or relaxing
qualification test requirements. These activities The reliability engineer should document the are usually closely coordinated with the project's reliability program in a reliability program plan, which verification program. should be summarized in the SEMP (Part III) and • Performing analyses on qualification test data to updated as needed through the project life cycle; the verify reliability predictions and validate the summary may be sufficient for small projects. system reliability prediction models, and to understand and resolve anomalies • Collecting reliability data under actual operations conditions as a part of overall system validation.
NASA Systems Engineering Handbook
• Conduct formal inspections of manufacturing 6.2.3 Designing Reliable Space-Based facilities, processes, and documentation Systems • Perform acceptance testing or inspections on all
parts when possible.
Designing reliable space-based systems
has always been a goal for NASA, and many painful Fault Tolerance. Fault tolerance is a system design lessons have been reamed along the way. The characteristic associated with the ability of a system system engineer should be aware of some basic to continue operating after a component failure has design approaches for achieving reliability. These occurred. It is implemented by having design basic approaches include fault avoidance, fault redundancy and a fault detection and response tolerance, and functional redundancy. capability. Design redundancy can take several forms,
some of which are represented in Figure 29 along Fault Avoidance. Fault avoidance, a joint objective of with their reliability relationships. the reliability engineer and quality assurance engineer
(see Section 6.3), includes efforts to: Functional Redundancy. Functional redundancy is a
system design and operations characteristic that • Provide design margins, or use appropriate allows the system to respond to component failures in aerating guidelines, if available a way sufficient to meet mission requirements. This • Use high-quality parts where needed. (Failure usually involves operational work-arounds and the rates for Class S parts are typically one-fourth of use of components in ways that were not originally those procured to general military specifications.) intended. As an example, a repair of the damaged • Consider materials and electronics packaging Galileo high-gain antenna was impossible, but a work- carefully around was accomplished by software fixes that further compressed the science data and images;
NASA Systems Engineering Handbook
Page 83 these were then returned through the low-gain precursors to a full probabilistic risk assessment antenna, although at a severely reduced data rate. (PRA). For more on this technique, see the U.S. These three approaches have different costs Nuclear Regulatory Commission Fault Tree associated with their implementation: Class S parts Handbook. are typically more expensive, while redundancy adds
mass, volume, costs, and complexity to the system. Reliability Models. Reliability models are used to Different approaches to reliability may therefore be predict the reliability of alternative appropriate for different projects. In order to choose architectures/designs from the estimated reliability of the best balance among approaches, the system each component. For simple systems, reliability can engineer must understand the system- level effects often be calculated by applying the rules of probability and life-cycle cost of each approach. To achieve this, to the various components and "strings" identified in trade study methods of Section 5.1 should be used in the reliability block diagram. (See Figure 29.) For combination with reliability analysis tools and more complex systems, the method of minimal cut techniques. sets, which relies on the rules of Boolean algebra, is often used to evaluate a system's fault tree. When individual component reliability functions are themselves uncertain, Monte Carlo simulation methods may be appropriate. These methods are described in reliability engineering textbooks, and software for calculating reliability is widely available. For a compilation of models/software, see D. Kececioglu, Reliability, Availability, and Maintainability Software Handbook.
FMECAs and FMEAs. Failure Modes, Effects, and Criticality Analysis (FMECA) and Failure Modes and Effects Analysis (FMEA) are specialized techniques for hardware failure and safety risk identification and characterization. (Also see Section 4.6.2.)
Problem/Failure Reports (P/ FRs). The reliability engineer uses the Problem/Failure Reporting System (or an approved equivalent) to report reliability problems and nonconformances encountered during qualification and acceptance testing (Phase D) and operations (Phase E).
6.3 Quality Assurance
Even with the best of available designs, 6.2.4 Reliability Analysis Tools and hardware fabrication (and software coding) and Techniques testing are subject to the vagaries of Nature and
human beings. The system engineer needs to have Reliability Block Diagrams. Reliability block some confidence that the system actually produced diagrams are used to portray the manner in which the and delivered is in accordance with its functional, components of a complex system function together. performance, and design requirements. Quality These diagrams compactly describe how components Assurance (QA) provides an independent assessment are connected. Basic reliability block diagrams are to the project manager/system engineer of the items shown in Figure 29. produced and processes used during the project life
cycle. The quality assurance engineer typically acts Fault Trees and Fault Tree Analysis. A fault tree is as the system engineer's eyes and ears in this a graphical representation of the combination of faults context. The project manager/system engineer must that will result in the occurrence of some (undesired) work with the quality assurance engineer to develop a top event. It is usually constructed during a fault tree quality assurance program (the extent, responsibility, analysis, which is a qualitative technique to uncover and timing of QA activities) tailored to the project it credible ways the top event can occur. In the supports. As with the reliability program, this largely construction of a fault tree, successive subordinate depends on the NASA payload classification (see failure events are identified and logically linked to the Appendix B.3). top event. The linked events form a tree structure
connected by symbols called gates, some basic 6.3.1 Role of the Quality Assurance Engineer examples of which appear in the fault tree shown in
Figure 30. Fault trees and fault tree analysis are often
NASA Systems Engineering Handbook
Page 84 FRR) on issues of design, materials, workmanship, fabrication and verification processes, and other characteristics that could degrade product system quality.
6.3.2 Quality Assurance Tools and Techniques
PCA/FCA. The Physical Configuration Audit (PCA) veri-fies that the physical configuration of the system corre-sponds to the approved "build-to" (or "code-to") docu-mentation. The Functional Configuration Audit (FCA) verifies that the acceptance verification (usually, test) results are consistent with the approved verification requirements. (See Section 4.8.4.)
In-Process Inspections. The extent, timing, and location of in-process inspections are documented in the quality assurance program plan. These should be conducted in consonance with the manufacturing/fabrication and verification program plans. (See Sections 6.6 and 6.7.)
QA Survey. A QA survey examines the operations, The quality assurance engineer performs pro-cedures, and documentation used in the project, several tasks, which are explained in more detail in and evaluates them against established standards NHB 5300.4(1B), Quality Program Provisions for and benchmarks. Recommendations for corrective Aeronautical and Space System Contractors. In brief, actions are reported to the project manager. these tasks include:
Material Review Board. The Material Review Board • Developing and executing a quality assurance (MRB), normally established by the project manager program plan and chaired by the project-level quality assurance • Ensuring the completeness of configuration engineer, performs formal dispositions on management procedures and documentation, nonconformances. and monitoring the fate of ECRs/ECPs (see
Section 4.7) 6.4 Maintainability • Participating in the evaluation and selection of
procurement sources Maintainability is a system design characteristic • Inspecting items and facilities during associated with the ease and rapidity with which the manufacturing/fabrication, and items delivered to system can be retained in operational status, or safely NASA field centers and economically restored to operational status • Ensuring the adequacy of personnel training and following a failure. Often used (though imperfect) technical documentation to be used during measures of maintainability include mean manufacturing/fabrication maintenance downtime, maintenance effort (work • Ensuring verification requirements are properly hours) per operating hour, and annual maintenance specified, especially with respect to test cost. However measured, maintainability arises from environments, test configurations, and pass/fail many factors: the system hardware and software criteria design, the required skill levels of maintenance • Monitoring qualification and acceptance tests to personnel, adequacy of diagnostic and maintenance ensure compliance with verification requirements procedures, test equipment effectiveness, and the and test procedures, and to ensure that test data physical environment under which maintenance is are correct and complete performed. • Monitoring the resolution and close-out of
nonconformances and Problem/Failure Reports 6.4.1 Role of the Maintainability Engineer (P/FRs) • Verifying that the physical configuration
of the system conforms to the "build-to" (or Maintainability engineering is another major specialty "code-to") documentation approved at CDR discipline that contributes to the goal of a cost- • Collecting and maintaining QA data for effective system. This is primarily accomplished in the subsequent failure analyses. systems engineering process through an active role in
implementing specific design features to facilitate safe The quality assurance engineer also participates maintenance actions in the predicted physical in major reviews (primarily SRR, PDR, CDR, and environments, and through a central role in
NASA Systems Engineering Handbook
Page 85 developing the integrated logistics support (ILS) studies early in the project life cycle. These trade system. (See Section 6.5 on ILS.) studies should focus on the cost effectiveness of The maintainability engineer performs alternative maintenance concepts in the context of several tasks, which are explained in more detail in overall system optimization. NHB 5300.4(1E), Maintainability Program
Requirements for Space Systems. In brief, these Maintenance Levels for Space Station Alpha tasks include: • Developing and executing a
maintainability pro-gram plan. This is usually done in As with many complex systems, the maintenance conjunction with the ILS program plan. concept for Alpha calls for three maintenance
levels: organizational, intermediate, and depot (or • Developing and refining the system maintenance vendor). The system engineer should be familiar concept as a part of the ILS concept with these terms and the basic characteristics • Establishing and allocating maintainability associated with each level. As an example, requirements. These requirements should be consider Alpha consistent with the maintenance concept and
traceable to system-level availability objectives. Level Work Performed Spares • Performing an engineering design analysis to Organizational On-orbit crew performs ORU identify maintainability design deficiencies remove-and-replace, visual inspections, minor • Performing analyses to quantify the system's servicing and calibration. Few. maintenance resource requirements, and
documenting them in the Maintenance Plan Inter-mediate Depot/ Vendor KSC maintenance • Verifying that the system’s maintainability facility repairs ORUs, performs de-tailed requirements and maintenance-related aspects inspections, servicing, calibrations, and some of the ILS requirements are met modifications. • Collecting maintenance data under actual
operations conditions as part of ILS system Factory performs major overhauls, modifications, validation. and complex calibrations. needed. Extensive
More extensive, or fabricated as needed. Many of the analysis tasks above are accomplished as part of the Logistics Support Analysis (LSA), described in Section 6.5.3. The
maintainability engineer also participates in and The Maintenance Plan, which appears as a contributes to major project reviews on the above major technical section in the Integrated Logistics items as appropriate to the phase of the project. Support Plan (ILSP), documents the system
maintenance concept, its maintenance resource 6.4.2 The System Maintenance Concept and requirements, and supporting maintainability analyses. The Maintenance Plan provides other Maintenance Plan inputs to the ILSP in the areas of spares,
maintenance facilities, test and support equipment, As the system operations concept and user and, for each level of maintenance, it provides requirements evolve, so does the ILS concept. maintenance training programs, facilities, technical Central to the latter is the system maintenance data, and aids. The supporting analyses should concept. It serves as the basis for establishing the establish the feasibility and credibility of the system's maintainability design requirements and its Maintenance Plan with aggregate estimates of logistics support resource requirements (through the corrective and preventive maintenance workloads, LSA process). In developing the system maintenance initial and recurring spares provisioning requirements, concept, it is useful to consider the mission profile, and system availability. Aggregate estimates should how the system will be used, its operational be the result of using best practice maintainability availability goals, anticipated useful life, and physical analysis tools and detailed maintainability data environ ments. suitable for the LSA. (See Section 6.5.3.) Traditionally, a description of the system
maintenance concept is hardware-oriented, though 6.4.3 Designing Maintainable Space-Based this need not always be so. The system maintenance concept is typically described in terms of the Systems anticipated levels of maintenance (see sidebar on
maintenance levels), general repair policies regarding Designing NASA space-based systems for corrective and preventive maintenance, assumptions maintainability will be even more important in the about supply system responsiveness, the availability future. For that reason, the system engineer should of new or existing facilities, and the maintenance be aware of basic design features that facilitate IVA environment. Initially, the system maintenance and EVA maintenance. Some examples of good concept may be based on experience with similar practice include: systems, but it should not be exempt from trade
NASA Systems Engineering Handbook
Page 86 Use coarse and fine installation alignment guides as to ensure a minimum number of items necessary to assure ease of Orbital Replacement Unit • Select ORU fasteners to minimize accessibility (ORU) installation and removal time consistent with good design practice
• Design the ORU surface structure so that no • Have minimum sweep clearances between safety hazard is created during the removal, interface tools and hardware structures; include replacement, test, or checkout of any ORU during adequate clearance envelopes for those IVA or EVA maintenance; include maintenance activities where access to an cautions/warnings for mission or safety critical opening is required ORUs • Define reach envelopes, crew load/forces, and
general work constraints for IVA and EVA • Design software to facilitate modifications, maintenance tasks verifica tions, and expansions • Consider corrective and preventive maintenance • Allow replacement of software segments on-line task frequencies in the location of ORUs without disrupting mission or safety critical func- • Allow replacement of an ORU without removal of tions other ORUs • Allow on- or off-line software modification, re- • Choose a system thermal design that precludes placement, or verification without introducing degradation or damage during ORU replacement hazardous conditions. or maintenance to any other ORU
• Simplify ORU handling to reduce the likelihood of 6.4.4 Maintainability Analysis Tools and mishandling equipment or parts Techniques • Encourage commonality, standardization, and
interchangeability of tooling and hardware items Maintenance Functional Flow Block Diagrams Maintainability Lessons Learned from HST Repair (FFBDs). Maintenance FFBDs are used in the same (STS-61) way as system FFBDs, described in Appendix B.7.1.
At the top level, maintenance FFBDs supplement and When asked (for this handbook! what clarify the system maintenace concept; at lower maintainability lessons were learned from their levels, mission, the STS~61 crew responded with the
following: Maintenance Time Lines. Maintenance time line
analysis (see Appendix B.7.3) is performed when • The maintainability considerations designed time-to-restore is considered a critical factor for into the Hubble Space Telescope (HST) mission effectiveness and/or safety. (Such cases worked. might include EVA and emergency repair procedures.) A maintenance time line analysis may • For spacecraft in LEO, don’t preclude a be a simple spreadsheet or, at the other end, involve servicing option; this means, for example, extensive computer simulation and testing. including a grapple fixture even t trough it has
a cost and mass impact. FMECAs and FMEAs. Failure Modes, Effects, and • When servicing is part of the maintenance Criti-cality Analysis (FMECA) and Failure Modes and concept, make sure that it's applied Effects Analysis (FMEA) are specialized techniques throughout the space craft. (The HST Solar for hardware failure and safety risk identification and Array Electronics Box, for example, was not characterization. They are discussed in this handbook designed to be replaced, but had to be under risk management (see Section 4.6.2) and nevertheless!) reliability engineering (see Section 6.2.4). For the • Pay attention to details like correctly sizing maintainability engineer, the FMECA/FMEA needs to the hand holds, and using connectors and be augmented at the LRU/ORU level with failure fasteners designed for easy removal and prediction data (i.e., MTTF or MTBF), failure detection reattachment. means, and identification of corrective maintenance
actions (for the LSA task inventory). Other related advice:
Maintainability Models. Maintainability models are • Make sure ground-based mock-ups and used in assessing how well alternative designs meet draw-ings exactly represent the "as- maintainability requirements, and in quantifying the deployed" con-figuration. maintenance resource requirements. Modeling • Verify tool-to-system interfaces, especially approaches may range from spreadsheets that when new tools are involved. aggregate component data, to complex Markov Make provision in the maintainability program for models and stochastic simulations. They often use high-fidelity maintenance training. they provide a reliability and time-to-restore data at the LRU/ORU basis for the LSA's maintenance task inventory. level obtained from experience with similar
NASA Systems Engineering Handbook
Page 87 components in existing systems. Some typical uses to requirements necessary to ensure sustained which these models are put include: operation of the system
• Design Interface: the interaction and relationship • Annual maintenance hours and/or maintenance of logistics with the systems engineering process downtime estimates to ensure that supportability influences the • System MTTR and availability estimates (see definition and design of the system so as to sidebar on availability measures on page 86) reduce life-cycle cost • Trades between reliability and maintainability • Technical Data: the recorded scientific, • Optimum LRU/ORU repair level analysis (ORLA) engineering, technical, and cost information used • Optimum (reliability-centered) preventive to define, produce, test, evaluate, modify, deliver, maintenance analysis support, and operate the system • Spares requirements analysis • Training: the processes, procedures, devices, • Mass/volume estimates for (space-based) spares and equipment required to train personnel to • Repair vs. discard analysis. operate and support the system
• Supply Support: actions required to provide all LSA and LSAR. The Logistics Support Analysis the necessary material to ensure the system's (LSA) is the formal technical mechanism for supportability and usability objectives are met integrating supportability considerations into the • Test and Support Equipment: the equipment re- systems engineering process. Many of the above quired to facilitate development, production, and tools and techniques provide maintainability inputs to operation of the system the LSA, or are used to develop LSA outputs. Results • Transportation and Handling: the actions, re- of the LSA are captured in Logistics Support Analysis sources, and methods necessary to ensure the Record (LSAR) data tables, which formally document proper and safe movement, handling, packaging, the baselined ILS system. (See Section 6.5.3.) and storage of system items and materials
• Human Resources and Personnel Planning: Problem/Failure Reports (P/ FRs). The actions required to determine the best skills-mix, maintainability engineer uses the Problem/Failure considering current and future operator, Reporting System (or an approved equivalent) to maintenance, engineering, and administrative report maintainability problems and nonconformances personnel costs encountered during qualification and acceptance • System Facilities: real property assets required to testing (Phase D) and operations (Phase E). develop and operate a system.
6.5 Integrated Logistics Support 6.5.2 Planning for ILS
The objective of Integrated Logistics Support ILS planning should begin early in the project (ILS) activities within the systems engineering life cycle, and should be documented in an ILS process is to ensure that the product system is program plan. This plan describes what ILS activities supported during development (Phase D) and are planned, and how they will be conducted and operations (Phase E) in a cost-effective manner. This integrated into the systems engineering process. For is primarily accomplished by early, concurrent major projects, the ILS program plan may be a consideration of supportability characteristics, separate document because the ILS system (ILSS) performing trade studies on alternative system and may itself be a major system. For smaller projects, the ILS concepts, quantifying resource requirements for SEMP (Part III) is the logical place to document such each ILS element using best-practice techniques, and information. An important part of planning the ILS acquiring the support items associated with each ILS program concerns the strategy to be used in element. During operations, ILS activities support the performing the Logistics Support Analysis (LSA) since system while seeking improvements in its cost- it can involve a major commitment of logistics effectiveness by conducting analyses in response to engineering specialists. (See Section 6.5.3.) actual operational conditions. These analyses Documenting results of ILS activities through continually reshape the ILS system and its resources the project life cycle is generally done in the requirements. Neglecting ILS or poor ILS decisions Integrated Logistics Support Plan (ILSP). The ILSP is invariably have adverse effects on the life-cycle cost the senior ILS document used by the project. A of the resultant system. preliminary ILSP should be prepared by the
completion of Phase B and subsequently maintained. 6.5.1 ILS Elements This plan documents the project's logistics support
concept, responsibility for each ILS element by project According to NHB 7120.5, the scope of ILS phase, and LSA results, especially trade study in-cludes the following nine elements: results. For major systems, the ILSP should be a
distinct and separate part of the system • Maintenance: the process of planning and documentation. For smaller systems, the ILSP may executing life-cycle repair/services concepts and be integrated with other system documentation. The
NASA Systems Engineering Handbook
Page 88 ILSP generally contains the following technical through a well-conceived ILS program. In brief, a sections: good-practice approach to achieving cost-effective • Maintenance Plan—Developed from the system ILS includes efforts to: maintenance concept and refined during the
system design and LSA processes. (NMI • Develop an ILS program plan, and coordinate it 5350.1A, Maintainability and Maintenance with the SEMP (Part III) Planning Policy, and NHB 5300.4(1E), • Perform the technical portion of the plan, i.e., the Maintainability Program Re-quirements for Space Logistics Support Analysis, to select the best Systems, do not use the term ILS, but they combined system and LS alternative, and to nevertheless mandate almost all of the steps quantify the resulting logistics resource found in an LSA. See Section 6.4.2 for more requirements details on the maintenance plan.) • Document the selected ILS system and • Personnel and Training Plan—Identifies both summarize the logistics resource requirements in operator and maintenance training, including the ILSP descriptions of training programs, facilities, • Provide supportability inputs to the system equipment, technical data, and special training requirements and/or specifications aids. According to NMI 5350.1A/NHB 5300.4(1E), • Verify and validate the selected ILS system. the maintenance training element is part of the
maintenance plan. 6.5.3 ILS Tools and Techniques: The • Supply Support Plan—Covers required quantities Logistics Support Analysis of spares (reparable and expendable) and
consumables (identified through the LSA), and The Logistics Support Analysis (LSA) is the procedures for their procurement, packaging, formal technical mechanism for integrating handling, storage, and transportation. This plan supportability considerations into the systems should also cover such issues as inventory engineering process. The LSA is performed iteratively management, breakout screening, and demand over the project life cycle so that successive data collection and analysis. According to NMI refinements of the system design move toward the 5350.1A/NHB 5300.4(1E), the spares supportability objectives. To make this happen, the provisioning element is part of the maintenance ILS engineer identifies supportability and plan. supportability-related design factors that need to be • Test and Support Equipment Plan —Covers re- considered in trade studies during the systems quired types, geographical location, and engineering process. The project-level system quantities of test and support equipment engineer imports these considerations largely through (identified through the LSA). According to NMI their impact on projected system effectiveness and 5350.1A/NHB 5300.4(1E), it is part of the life-cycle cost. The ILS engineer also acts as a maintenance plan. system engineer (for the ILSS) by identifying ILSS • Technical Data Plan—Identifies procedures to functional requirements, performing trade studies on acquire and maintain all required technical data. the ILSS, documenting the logistics support resources According to NMI 5350.1A/NHB 5300.4(1E), that will be required, and overseeing the verification technical data for training is part of the and validation of the ILSS. maintenance plan. Transportation and Handling The LSA process found in MIL-STD-1388-1A Plan —Covers all equipment, containers, and can serve as a guideline, but its application in NASA supplies (identified through the LSA), and should be tailored to the project. Figures 31a and 31b procedures to support packaging, handling, show the LSA process in more detail as it proceeds storage, and transportation of system through the NASA project life cycle. Each iteration components uses more detailed inputs and provides more • Facilities Plan—Identifies all real property assets refinement in the output so that by the time operations required to develop, test, maintain, and operate begin (Phase E), the full complement of logistics the system, and identifies those requirements support resources has been identified and the ILSS that can be met by modifying existing facilities. It verified. The first step at each iteration is to should also provide cost and schedule understand the mission, the system projections for each new facility or modification. architecture/design, and the ILSS parameters. • Disposal Plan—Covers equipment, supplies, and Specifically, the first step encompasses the following procedures for the safe and economic disposal of activities: all items (e.g., condemned spares), including
ultimately the system itself. • Receiving (from the project-level system
engineer) factors related to the intended use of The cost of ILS (and hence the life-cycle cost of the system such as the operations concept, the system) is driven by the inherent reliability and mission duration, number of units, orbit maintainability characteristics of the system design. parameters, space transportation options, The project level system engineer must ensure that allocated supportability characteristics, etc. these considerations influence the design process
NASA Systems Engineering Handbook
Page 89 • Documenting existing logistics resource supportability factors associated with the various capabilities and/or assets that may be cost- system and operations concepts being effective to apply to or combine with the ILSS for considered. Such factors might include the system being developed operations team size, system RAM (reliability, • Identifying technological opportunities that can be availability and maintain ability) parameters, exploited. (This includes both new technologies estimated annual IVA/EVA maintenance hours in the system architecture/design that reduce and upmass requirements, etc. logistics support resource requirements as well • Using the above to assist the project-level system as new technologies within the ILSS that make it engineer in projecting system effectiveness and less expensive to meet any level of logistics life cycle cost, and establishing system support resource requirements.) availability and/or system supportability goals. • Documenting the ILS concept and initial (See NHB 7120.5, and this handbook, Section "strawman" ILSS, or updating (in later phases) 5.3.3.) the baseline ILSS. • Identifying and characterizing the system
supportability risks. (See NHB 7120.5, and this The ILS engineer uses the results of these handbook, Section 4.6.) activities to establish supportability and supportability- • Documenting supportability-related design con- related design factors, which are passed back to the straints. project-level system engineer. This means:
The heart of the LSA lies in the next group of • Identifying and estimating the magnitude of activities, during which systems engineering and analysis are applied to the ILSS itself. The ILS
NASA Systems Engineering Handbook
Page 90 engineer must first identify the functional MIL-STD-1388-1A/2B requirements for the ILSS. The functional analysis
process establishes the basis for a task inventory NHB 7120.5 suggests MIL-STD 1388 as a associated with the product system and, with the task guideline for doing an LSA. MIL-STD 1388-1A is inventory, aids in the identification of system design divided into five sections: deficiencies requiring redesign. The task inventory
generally includes corrective and preventive LSA Planning and Control (not shown in Figures maintenance tasks, and other operations and support 31a and 31b) tasks arising from the ILSS functional requirements. A Mission and Support System Definition (shown as principal input to the in-ventory of corrective and boxes A and B) preventive maintenance tasks, which is typically Preparation and Evaluation of Alternatives (shown constructed by the maintainability engi-neer, is the as boxes C, D, and E) FMECA/FMEA (or equivalent analysis). The Determination of Logistics Support Resource FMECA/FMEA itself is typically performed by the reli- Requirements (shown as box F) ability engineer. The entire task inventory is Supportability Assessment (shown as boxes G documented in Logistics Support Analysis Record and H. (LSAR) data tables. MIL-STD 1388-1A also provides useful The ILS engineer then creates plausible ILSS tips and encourages principles already alternatives, and conducts trade studies in the established in this handbook: functional analysis, manner described earlier in Section 5.1. The trade successive refinement of designs through trade studies focus on different issues depending on the studies, focus on system effectiveness and life- phase of the project. In Phases A and B, trade studies cycle cost, and appropriate models and selection focus on high-level issues such as whether a rules. spacecraft in LEO should be serviceable or not, what MIL-STD 1388-2B contains the LSAR relational mix of logistics modules seems best to support an data table formats and data dictionaries for inhabited space station, or what's the optimum documenting ILS information and LSA results in number of maintenance levels and locations. In machine-readable form. Phases C and D, the focus changes, for example to an individual end-item's op-timum repair level. In Phase E, when the system design and its logistics storage, and testing. For new access-to-space support requirements are essentially understood, systems, support may be needed during an extended trade studies often revisit issues in the light of period of developmental launches, and for inhabited operational data. These trade studies almost always space stations, during an extended period of on-orbit rely on techniques and models especially created for assembly operations. The ILS engineer also the purpose of doing a LSA. For a catalog of LSA contributes to risk management activities by techniques and models, the system engineer can considering the adequacy of spares provisioning, and consult the Logistics Support Analysis Techniques of logistics plans and processes. For example, spares Guide (1985), Army Materiel Command Pamphlet No. provisioning must take into account the possibility that 700-4. production lines will close during the anticipated By the end of Phase B, the results of the ILSS useful life of the system. functional analyses and trade studies should be As part of verification and validation activity, sufficiently refined and detailed to provide quantitative the ILS engineer performs supportability verification data on the logistics support resource requirements. planning and gathers supportability verification/test This is accomplished by doing a task analysis for data during Phase D. These data are used to identify each task in the task inventory. These requirements and correct deficiencies in the system design and are formally documented by amending the LSAR data ILSS, and to update the LSAR data tables. During tables. Together, ILSS trade studies, LSA models, Phase E, supportability testing and analyses are and LSAR data tables provide the project-level conducted under actual operational conditions. These system engineer with important life-cycle cost data data provide a useful legacy to product improvement and measures of (system) effectiveness (MoEs), efforts and future projects. which are successively refined through Phases C and
D as the product system becomes better defined and 6.5.4 Continuous Acquisition and Life-Cycle better data become available. The relationship Support between inputs (from the specialty engineering
disciplines) to the LSA process and its outputs can be LSA documentation and supporting LSAR seen in Figure 27 (see Section 5.3). data tables contain large quantities of data. Making In performing the LSA, the ILS engineer also use of these data in a timely manner is currently determines and documents (in the LSAR data tables) difficult because changes occur often and rapidly the logistics resource requirements for Phase D during definition, design, and development (Phases B system integration and verification, and deployment through D). Continuous Acquisition and Life-Cycle (e.g., launch). For most spacecraft, this support Support (CALS)—changed in 1993 from Computer- includes pre-launch transportation and handling, Aided Acquisition and Logistics Support—technology
NASA Systems Engineering Handbook
Page 91 can reduce this dilemma by improving the digital design stability is achieved), thus reducing the risk of exchange of data across NASA field centers and costly mistakes. Faster vendor response time also between NASA and its contractors. Initial CALS means reduced spares inventories during operations. efforts within the logistics engineering community
focused on developing CALS digital data exchange 6.6 Verification standards; current emphasis has shifted to database
integration and product definition standards, such as Verification is the process of confirming that STEP (Standard for the Exchange of Product) Model deliverable ground and flight hardware and software Data. are in compliance with functional, performance, and CALS represents a shift from a paper- (and design requirements. The verification process, which labor-) intensive environment to a highly automated includes planning, requirements definition, and and integrated one. Concommitant with that are compliance activities, begins early and continues expected benefits in reduced design and development throughout the project life cycle. These activities are time and costs, and in the improved quality improved an integral part of the systems engineering process. quality of ILS products and decisions. CALS cost At each stage of the process, the system engineer's savings accrue primarily in three areas: concurrent job is to understand and assess verification results, engineering, configuration control, and ILS functions. and to lead in the resolution of any anomolies. This In a concurrent engineering environment, NASA's section describes a generic NASA verification process multi-disciplinary PDTs (which may mirror and work that begins with a verification program concept and with those of a system contractor) can use CALS continues through operational and disposal technology to speed the exchange of and access to verification. Whatever process is chosen by the data among PDTs. Availability of data through CALS program/project should be documented in the SEMP. on parts and suppliers also permits improved parts
selection and acquisition. (See Section 3.7.2 for more The objective of the verification program is to on concurrent engineering.) ensure that all functional, performance, and design Configuration control also benefits from requirements (from Level I program/project CALS technology. Using CALS to submit, process, requirements) have been met. Each project develops and track ECRs/ECPs can reduce delays in approving a verification program considering its cost, schedule, or rejecting them, along with the indirect costs that and risk implications. No one program can be applied delays cause. Al-though concurrent engineering is to every project, and each verification activity and expected to reduce the number of ECRs/ECPs during product must be assessed as to its applicability to a design and development (Phases C and D), their specific project. The verification program requires timely disposition can produce significant cost considerable coordination by the verification engineer, savings. (See Section 4.7.2 for more on configuration as both system design and test organizations are control.) typically involved to some degree throughout. Lastly, CALS technology potentially enables
ILS functions such as supply support to be performed 6.6.1 Verification Process Overview simultaneously and with less manual effort than at
present. For example, procurement of design-stable Verification activities begin in Phase A of a components and spares can begin earlier (to allow project. During this phase, inputs to the project's earlier testing); at the same time, provisioning for integrated master schedule and cost estimates are other components can be further deferred (until made as the verification program concept takes Can NASA Benefit from CALS? shape. These planning activities increase in Phase B
with the refinement of requirements, costs, and The DoD CALS program was initiated in 1985, schedules. In addition, the system's requirements are since 1988, it has been required on new DoD assessed to determine preliminary methods of systems. Ac-cording to Clark, potential DOD-wide verification and to ensure that the requirements can savings from CALS exceeds $160M (FY92$). be verified. The outputs of Phase B are expanded in However, GAO studies have been critical of DoD's Phase C as more detailed plans and procedures are CALS’ implementation. These criticisms focused prepared. In Phase D, verification activities increase on CALS' limited ability to share information substantially; these activities normally include among users. qualification and acceptance verification, followed by For NASA field centers to realize savings from verification in preparation for deployment and CALS, new enabling investments in hardware, operational verification. Figures 32a and 32b show software, and training may be required. While this process through the NASA project life cycle. many of NASA's larger contractors have already (Safety reviews as applied to verification activities are installed CALS technology, the system engineer not shown as separate activities in the figures.) wishing to employ CALS must recognize that both
GALS and non-CALS approaches may be needed The Verification Program Concept. A verification to interact with small business suppliers, and that pro-gram should be tailored to the project it supports. proprietary contractor data, even when digitized, The project manager/system engineer must work with needs to be protected. the verification engineer to develop a verification
NASA Systems Engineering Handbook
Page 92 program concept. Many factors need to be considered the verification program. Trade studies should be in developing this concept and the subsequent performed to support decisions about verification verification program. These factors include: methods and re-quirements, and the selection of
facility types and locations. As an example, a • Project type, especially for flight projects. trade study might be made to decide between Verification methods and timing depend on the performing a test at a centralized facility or at type of flight article involved (e.g., an experiment, several decentralized locations. payload, or launch vehicle). • Risk implications. Risk management must be • NASA payload classification. The verification considered in the development of the verification activities and documentation required for a program. Qualitative risk assessments and specific flight article generally depend upon its quantitative risk analyses (e.g., a FMECA) often NASA payload classification. As expected, the identify new concerns that can be mitigated by verification program for a Class A payload is additional testing, thus increasing the extent of considerably more comprehensive than that for a verification activities. Other risk assessments Class D payload. (See Appendix B.3 for contribute to trade studies that determine the classification guidelines.) preferred methods of verification to be used and • Project cost and schedule implications. when those methods should be performed. As an Verification activities can be significant drivers of example, a trade might be made between a project's cost and schedule; these implications performing a modal test versus determining should be considered early in the development of modal characteristics by a less costly, but less revealing, analysis. The project manager/system
NASA Systems Engineering Handbook
Page 93 engineer must determine what risks are system performance prior to the next test/operation. acceptable in terms of the project's cost and Environmental testing is an individual test or series of schedule. tests conducted on flight or flight-configured hardware • Availability of verification facilities/sites and and/or software to assure it will perform satisfactorily transportation assets to move an article from one in its flight environment. Environmental tests include location to another (when needed). This requires vibration, acoustic, and thermal vacuum. coordination with the ILS engineer. Environmental testing may be combined with • Acquisition strategy (i.e., in-house development functional testing if test objectives warrant. or system contract). Often a NASA field center Verification by analysis is a process used in can shape a contractor's verification process lieu of (or in addition to) testing to verify compliance to through the project's Statement of Work (SoW). specifications/requirements. The selected techniques • Degree of design inheritance and may include systems engineering analysis, statistics hardware/software reuse. and qualitative analysis, computer and hardware
simulations, and com-puter modeling. Analysis may Verification Methods and Techniques. The system be used when it can be determined that: (1) rigorous engineer needs to understand what methods and and accurate analysis is possible; (2) testing is not techniques the verification engineer uses to verify feasible or cost-effective; (3) similarity is not compliance with requirements. In brief, these methods applicable; and/or (4) verification by inspection is not and techniques are: adequate.
Verification by demonstration is the use of • Test actual demonstration techniques in conjunction with • Analysis requirements such as maintainability and human • Demonstration engineering features. Verification by similarity is the • Similarity process of assessing by review of prior acceptance • Inspection data or hardware configuration and applications that • the article is similar or identical in design and Simulation manufacturing process to another article that has • Validation of records. previously been qualified to equivalent or more stringent specifications. Verification by inspection is Analyses and Models the physical evaluation of equipment and/or
documentation to verify design features. Inspection is Analyses based on models are used extensively used to verify construction features, workmanship, throughout a program/project to verify and and physical dimensions and condition (such as determine compliance to performance and design cleanliness, surface finish, and locking hardware). requirements. Most verification requirements that Verification by simulation is the process of verifying cannot be verified by a test activity are verified design features and performance using hardware or through analyses and modeling. The analysis and software other than flight items. Verification by modeling process begins early in the project life validation of records is the process of using cycle and continues through most of Phase D; manufacturing records at end-item acceptance to these analyses and models are updated verify construction features and processes for flight periodically as actual data that are used as inputs hardware. become available. Often, analyses and models
are validated or corroborated by the results of a Verification Stages. Verification stages are defined test activity. Any verification-related results should periods of verification activity when different be documented as part of the project's archives. verification goals are met. In this handbook, the following verification stages are used for flight systems:
Verification by test is the actual operation of • Development equipment during ambient conditions or when • subjected to specified environments to evaluate Qualification performance. Two subcategories can be defined: • Acceptance functional testing and environmental testing. • Preparation for deployment (also known as pre Functional testing is an individual test or series of launch) electrical or mechanical performance tests conducted • Operational (also known as on-orbit or in-flight) on flight or flight-configured hardware and/or software • Disposal (as needed). at conditions equal to or less than design
specifications. Its purpose is to establish that the The development stage is the period during system performs satisfactorily in accordance with which a new project or system is formulated and design and performance specifications. Functional implemented up to the manufacturing of qualification testing generally is performed at ambient conditions. or flight hardware. Verification activities during this Functional testing is performed before and after each stage (e.g., breadboard testing) provide confidence environmental test or major move in order to verify that the system can accomplish mission
NASA Systems Engineering Handbook
Page 94 goals/objectives. When tests are conducted during should be in accordance with project milestones, and this stage, they are usually performed by the design should be updated as verification activities are organization, or by the design and test organizations refined. together. Also, some program/project requirements Verification Reports may be verified or partially verified through the
activities of the PDR and CDR, both of which occur A verification report should be provided for each during this stage. Any development activity used to analysis and at a minimum, for each major test formally satisfy program/project requirements should activity, such as functional testing, environmental have quality assurance oversight. testing, and end-to-end compatibility testing The qualification stage is the period during which occurs over long periods of time or is separated the flight (protoflight approach) or flight-type hardware by other activities, verification reports may be is verified to meet functional, performance. and needed for each individual test activity, such as design requirements. Verifications during this stage functional testing, acoustic testing, vibration are conducted on flight-configured hardware at testing, and thermal vacuum/thermal balance conditions more severe than acceptance conditions to testing. Verification reports should be completed establish that the hardware wild perform satisfactorily within a few weeks following a test, and should in the flight environments with sufficient margin. The provide evidence of compliance with the acceptance stage is the period during which the verification requirements for which it was deliverable flight end-item is shown to meet conducted. The verification report should include functional, performance, and design requirements as appropriate: under conditions specified for the mission. The Verification objectives and degree to which they acceptance stage ends with shipment of the flight were met hardware to the launch site. Description of verification activity The preparation for deployment stage begins with Test configuration and differences from flight the arrival of the flight hardware and/or software at the configuration launch site and terminates at launch. Requirements Specific result of each test and each procedure verified during this stage are those that demand the including annotated tests • Specific result of each integrated vehicle and/or launch site facilities. The analysis operational verification stage begins at liftoff; during Test performance data tables, graphs, this stage, flight systems are verified to operate in illustrations, and pictures space environment conditions, and requirements Descriptions of deviations from nominal results, demanding space environments are verified. The problems/failures, approved anomaly corrective disposal stage is the period during which disposal actions, and re-test activity requirements are verified. Summary of non-conformance/discrepancy
reports including dispositions 6.6.2 Verification Program Planning Conclusion and recommendations relative to
success of verification activity Verification program planning is an Status of support equipment as affected by test interactive and lengthy process occurring during all Copy of as-run procedure phases of a project. but more heavily during Phase C. Authentication of test results and authorization of The verification engineer develops a preliminary acceptability. definition of verification requirements and activities based on the program/project and mission requirements. An effort should be made throughout a project's mission and system definition to phrase During planning, the verification engineer requirements in absolute terms in order to simplify also identifies the documentation necessary to their verification. As the system and interface support the verification program. This documentation requirements are established and refined, the normally includes: (1) a Verification Requirements verification engineer assesses them to determine the Matrix (VRM), (2) a Master Verification Plan (MVP), appropriate method of verification or combination (3) a Verification Requirements and Specifications thereof. These requirements and the method(s) of Document (VRSD), and (4) a Verification verification are then documented in the ap-propriate Requirements Compliance Document (VRCD). requirements document. Documentation for test procedures and reports may Using the methods of verification to be also be defined. Because the system engineer should performed for each verification stage, along with the be familiar with these basic elements of a verification levels (e.g., part, subsystem, system) at which the process, each of these is covered below. verifications are to be performed, and any
environmental controls (e.g., contamination) that must Verification Requirements Matrix. The Verification be maintained, the verification engineer outlines a Requirements Matrix (VRM) is that portion of a preliminary schedule of verification activities requirements document (generally a System associated with development, qualification, and Requirements Document or Cl specification) that acceptance of the system. This preliminary schedule defines how each functional, performance, and design
NASA Systems Engineering Handbook
Page 95 requirement is to be verified, the stage in which requirements for that verification activity only. The verification is to occur, and (sometimes) the verification engineer develops the VRSD from an applicable verification levels. The verification engineer understanding of the requirements, the verification develops the VRM in coordination with the design, program concept, and the flight article. The VRSD is systems engineering, and test organizations. VRM baselined prior to the start of the verification activity. contents are tailored to each project's requirements, The heart of the VRSD is a data table that includes and the level of detail in VRMs may vary. The VRM is the following fields: baselined as a result of the PDR, and essentially
establishes the basis for the verification program. A A numerical designator assigned to each requirement sample VRM for a CI specification is shown in A statement of the specific requirement to be verified Appendix B.9. The "pass/fail" criteria and tolerances for each
requirement Master Verification Plan. The Master Verification Any constraints that must be observed Plan (MVP) is the document that describes the overall Any remarks to aid in the understanding of the verification program. The MVP provides the content requirement and depth of detail necessary to provide full visibility Location where the requirement will be verified. of all verification activities. Each major activity is
defined and described in detail. The plan The VRSD, along with flight article drawings encompasses qualification, acceptance, pre-launch, and schematics, is the basis for the development of operational, and disposal verification activities for verification procedures, and is also used as one of the flight hardware and software. (Development stage bases for development of the Verification verification activities are not normally documented in Requirements Compliance Document (VRCD). the plan, but may be documented elsewhere.) The
plan provices a general schedule and sequence of Verification Requirements Compliance Document. events for major verification activities. It also The Verification Requirements Compliance Document describes test software, Ground Support Equipment (VRCD) provides the evidence of compliance to each (GSE), and facilities necessary to support the Level I through Level n design, performance, safety, verification activities. The verification engineer and interface requirement, and to each VRSD develops the plan through a thorough understanding requirement. The flowdown to VRSD requirements of the verification program concept, the requirements completes the full requirements traceability. in the Program (i.e., Level I) Requirements Document Compliance with all the requirements ensures that (PRD), System/Segment (i.e., Level II) Requirements Level I requirements have been met. Document (SRD), and/or the CI specification, and the The VRCD defines, for each requirement, methods identified in the VRM of those documents. the method(s) of verification and corresponding Again, the development of the plan requires that the compliance information for each method employed. verification engineer work closely with the design, The compliance information provides either the actual systems engineering, and test organizations. A data, or a reference to the location of the actual data sample outline for this plan is illustrated in Appendix that shows compliance with the requirement. (The B.10. document also shows any non-compliances by
referencing the related Non-Compliance Report Verification Requirements and Specifications (NCR) or Problem/Failure Report (P/FR); following Docu -ment. The Verification Requirements and resolution of the anomaly, the document specifies Specifications Document (VRSD) defines the detailed appropriate re-verification information.) The requirements and specifications for the verification of compliance information may reference a verification a flight article, including the ground system/segment. report, an automated test program, a verification The VRSD specifies requirements and specifications procedure, an analysis report, or a test. The inputting for activities covering qualification through operational of compliance information into the compliance verification. Requirements are also defined for flight document occurs over a lengthy period of time, and software verification after the software has been on large systems and payloads, the effort may be installed in the flight article. The VRSD should cover continuous. The information in the compliance verifications by all methods; some programs/projects, document must be up-to-date for the System however, use a document that defines only Acceptance Review(s) (SAR) and Flight Readiness requirements to be satisfied by test. The VRSD Review (FRR). The compliance document is not should include all requirements defined in Level I, II, baselined because compliance information is input to and III requirements documents plus derived the document throughout the entire project life cycle. requirements. The VRSD defines the acceptance It is, however, an extremely important part of the criteria and any constraints for each requirement. The project's archives. VRSD typical]y identifies the locations where The heart of the Verification Requirements requirements will be verified. On large Compliance Document is also a data table with links programs/projects, a VRSD is normally developed for to the corresponding requirements. The VRCD each verification activity/location (e.g., thermal- includes the following fields: vacuum testing), and is tailored to include
NASA Systems Engineering Handbook
Page 96 • A numerical designator assigned to each Layouts, schematics, or diagrams showing requirement identification, location, and interconnection of test • A numerical designator that defines the equipment, test articles, and measuring points document where the requirement is defined Identification of hazardous situations or operations • A statement of the specific requirement for which Precautions and safety instructions to ensure safety compliance is to be defined of personnel and prevent degradation of test articles • Verification method used to verify the and measuring equipment requirement Environmental and/or other conditions to be • Location of the data that show compliance with maintained with tolerances the requirement statement. This information Constraints on inspection or testing could be a test, report, procedure, analysis Special instructions for non-conformances and report, or other information that fully defines anomalous occurrences or results where the compliance data could be found. Specifications for facility, equipment maintenance, Retest information is also shown. housekeeping, certification inspection, and safety and • Any non-conformances that occurred during the handling requirements before, during, and after the verification activities total verification activity. • Any statements of compliance information as to any non-compliance or acceptance by means Test Readiness Reviews other than the method identified, such as a
waiver. A Test Readiness Review (TRR) is held prior to
each major test to ensure the readiness of all Verification Procedures. The verification procedures ground, flight, and operational systems to support are documents that provide step-by-step instructions the performance of the test. A review of the for performing a given verification activity. The detailed status of the facilities, Ground Support procedure is tailored to the verification activity that is Equipment (GSE), test design, software, to be performed to satisfy a requirement, and could procedures, and verification requirements is be a test, demonstration, or any other verification- made. The test activities and schedule are related activity. The procedure is written to satisfy outlined and personnel responsibilities are requirements defined by the VRSD, and is submitted identified. Verification emphasis is directed toward prior to the Test Readiness Review (TRR) or the start ensuring that all verification requirements that of the verification activity in which the procedure is have been identified for the test have been used. (See sidebar on TRRs.) included in the test design and procedures. Procedures are also used to verify the acceptance of facilities, electrical and mechanical ground support equipment, and special test
equipment. The information generally contained in a The procedure may provide blank spaces for procedure is as follows, but it may vary according to recording of results and narrative comments in order the activity and test article: that the completed procedure can serve as part of the
verification report. The as-run and certified copy of the Nomenclature and identification of the test article or procedure is maintained as part of the project's material archives. Identification of test configuration and any differ-ences
from flight configuration 6.6.3 Qualification Verification Identification of objectives and criteria established for
the test by the applicable verification specifica tion Qualification stage verification activities Characteristics and design criteria to be inspected or begin after completion of development of the flight tested, including values, with tolerances, for hardware designs, and include analyses and testing acceptance or rejection to ensure that the flight or flight-type hardware (and Description, in sequence, of steps and operations to software) will meet functional and performance be taken requirements in anticipated environmental conditions. Identification of computer software required Qualification tests generally are designed to subject Identification of measuring, test, and recording the hardware to worst case loads and environmental equipment to be used, specifying range, accuracy, stresses. Some of the verifications performed to and type ensure hardware compliance to worst case loads and Certification that required computer test pro-grams/ environments are vibration/acoustic, pressure limits, support equipment and software have been verified leak rates, thermal vacuum, thermal cycling, prior to use with flight hardware electromagnetic interference and electromagnetic Any special instructions for operating data recording compatibility (EMI/EMC), high and low voltage limits, equipment or other automated test equipment as and life time/cycling. During this stage, many applicable performance requirements are verified, while analyses and models are updated as test data are acquired. Safety requirements, defined by hazard
NASA Systems Engineering Handbook
Page 97 analysis reports, may also be satisfied by qualification Verifications performed during this stage testing. ensure that no visible damage to the system has Qualification usually occurs at the occurred during shipment and that the system component or subsystem level, but could occur at the continues to function properly. If system elements are system level as well. When a project decides against shipped separately and integrated at the launch site, building dedicated qualification hardware, and uses testing of the system and system interfaces is the flight hardware itself for qualification purposes, the generally required. If the system is integrated into a process is termed protoflight. Additional information carrier, the interface to the carrier must also be on protoflight testing is contained in MSFC-HDBK- verified. Other verifications include those that occur 670, General Environmental Test Guidelines (GETG) following integration into the launch vehicle and those for Protoflight Instruments and Experiments. that occur at the launch pad; these are intended to
ensure that the system is functioning and in its proper 6.6.4 Acceptance Verification launch configuration. Contingency verifications and
procedures are developed for any contingencies that The acceptance stage verification activities can be foreseen to occur during pre-launch and provide the assurance that the flight hardware and countdown. These contingency verifications and software are in compliance with all functional, procedures are critical in that some contingencies performance, and design requirements, and are ready may require a return of the launch vehicle or flight for shipment to the launch site. The acceptance stage article from the launch pad to a processing facility. begins with the acceptance of each individual
component or piece part for assembly into the flight Software IV&V article and continues through the SAR.
Some verifications cannot be performed after Some project managers/system engineers may a flight article, especially a large one, has been wish to add IV&V (Independent Verification and assembled and integrated (e.g., due to Validation) to the software verification program. inaccessability). When this occurs, these verifications IV&V is a process whereby the products of the are performed during fabrication and integration, and software development life cycle are independently are known as in-process tests. Acceptance testing, reviewed, verified, and validated by an then, begins with in-process testing and continues organization that is neither the developer nor the through functional testing, environmental testing, and acquirer of the software. The IV&V agent should end-to-end compatibility testing. Functional testing have no stake in the success or failure of the normally begins at the component level and continues software; the agent’s only interest should be to at the systems level, ending with all systems make sure that the software is thoroughly tested operating simultaneously. All tests are performed in against its requirements. accordance with requirements defined in the VRSD. IV&V activities duplicate the project’s V&V When flight hardware is unavailable, or its use is activities step-by-step during the life cycle, with inappropriate for a specific test, simulators may be the exception that the IV&V agent does no used to verify interfaces. Anomalies occurring during informal testing If IV&V is employed formal a test are documented on the appropriate reporting system (NCR or P/FR), and a proposed resolution 6.6.6 Operational and Disposal Verification should be defined before testing continues. Major
anomalies, or those that are not easily dispositioned, Operational verification provides the may require resolution by a collaborative effort of the assurance that the system functions properly in a system engineer, and the design, test, and other (near-) zero gravity and vacuum environment. These organizations. Where appropriate, analyses and verifications are performed through system activation models are validated and updated as test data are and operation, rather than through a verification acquired. activity. Systems that are assembled on-orbit must
have each interface verified, and must function 6.6.5 Preparation for Deployment Verification properly during end-to-end testing. Mechanical
interfaces that provide fluid and gas flow must be The pre-launch verification stage begins with verified to ensure no leakage occurs, and that the arrival of the flight article at the launch site and pressures and flow rates are within specification. concludes at liftoff. During this stage, the flight article Environmental systems must be verified. The is processed and integrated with the launch vehicle. requirements for all operational verification activities The launch vehicle could be the Shuttle, some other are defined in the VRSD. launch vehicle, or the flight article could be part of the Disposal verification provides the assurance launch vehicle. Verifications requirements for this that the safe deactivation and disposal of all system stage are defined in the VRSD. When the launch site products and processes has occurred. The disposal is the Kennedy Space Center, the Operations and stage begins in Phase E at the appropriate time (i.e., Maintenance Requirements and Specifications either as scheduled, or earlier in the event of Document (OMRSD) is used in lieu of the VRSD. premature failure or accident), and concludes when all mission data have been acquired and verifications
NASA Systems Engineering Handbook
Page 98 necessary to establish compliance with disposal
requirements are finished. Both operational and 6.7.2 Producibility Tools and Techniques disposal verification activities may also include
validation assess meets—that is, assessments of the Manufacturing Functional Flow Block Diagrams degree to which the system accomplished the desired (FFBDs). Manufacturing FFBDs are used in the same mission goals/objectives. way system FFBDs. described in Appendix B.7.1, are
used. At the top level, manufacturing FFBDs 6.7 Producibility supplement and clarify the system's manufacturing
sequence. Producibility is a system characteristic
associated with the ease and economy with which a Risk Management Templates. The risk management completed design can be transformed (i.e., fabricated, templates of DoD 4245.7M, Transition from manufactured, or coded) into a hardware and/or Development to Production ...Solving the Risk software realization. While major NASA systems tend Equation, are a widely recognized series of risks, risk to be produced in small quantities, a particular responses, and lessons reamed from DoD producibility feature can be critical to a system's cost- experience. These templates, which were designed to effectiveness, as experience with the Shuttle's reduce risks in production, can be tailored to thermal tiles has shown. individual NASA projects.
6.7.1 Role of the Production Engineer Producibility Assessment Worksheets. These
work-sheets, which were also developed for DoD, use The production engineer supports the a judg-ment- based scoring approach to help choose systems engineering process (as a part of the multi- among alternative production methods. See disciplinary PDT) through an active role in Producibility Measurement for DoD Contracts. implementing specific design features to enhance
producibility, and by performing the production Producibility Models. Producibility models are used engineering analyses needed by the project. These in addressing a variety of issues such as assessing tasks and analyses include: the feasibility of alternative manufacturing plans, and
estimating production costs as a part of life-cycle cost Performing the manufacturing/fabrication portion of management. Specific producibility models may the system risk management program (see Section include: 4.6). This is accomplished by conducting a rigorous
production risk assessment and by planning effective • Scheduling models for estimating production risk mitigation actions. output, and for integrating system enhancements Identifying system design features that enhance and/or spares production into the manufacturing producibility. Efforts usually focus on design sequence simplification, fabrication tolerances, and avoidance of • Manufacturing or assembly flow simulations, e.g., hazardous materials. discrete event simulations of factory activities Conducting producibility trade studies to determine • Production cost models that include learning and the most cost-effective fabrication/manufacturing production rate sensitivities. (See sidebar page process 82.) Assessing production feasibility within project
constraints. This may include assessing contractor Statistical Process Control/Design of and principal subcontractor production experience Experiments. These techniques, long applied in and capability, new fabrication technology, special manufacturing to identify the causes of unwanted tooling, and production personnel training variations in product quality and reduce their effects, requirements. Identifying long-lead items and critical have had a rebirth under TQM. A collection of materials currently popular techniques of this new quality Estimating production costs as a part of life-cycle cost engineering is known as Taguchi methods. For first- management hand information on Taguchi methods, see his book, Developing production schedules Quality Engineering in Production Systems, 1989. A Developing approaches and plans to validate handbook approach to to some of these techniques fabrication/manufacturing processes. can be found in the Navy's Producibility Measurement
Guidelines: Methodologies for Product Integrity. The results of these tasks and production
engineering analyses are documented in the 6.8 Social Acceptability Manufacturing Plan with a level of detail appropriate
to the phase of the project. The production engineer NASA systems must be acceptable to the also participates in and contributes to major project society that funds them. The system engineer takes reviews (primarily PDR and CDR) on the above items, this into account by integrating mandated social and to special interim reviews such as the Production concerns into the systems engineering process. For Readiness Review (ProRR). some systems, these concerns can result in
NASA Systems Engineering Handbook
Page 99 significant design and cost penalties. Even when What is NEPA? social concerns can be met, the planning and analysis
associated with doing so can be time-consuming The National Environmental Policy Act (NEPA) of (even to the extent of affecting the project's critical 1969 declares a national environmental policy and path), and use significant specialized engineering goals, and provides a method for accomplishing resources. The system engineer must include these those goals. NEPA requires an Environmental costs in high-level trade studies of alternative Impact Statement (EIS) for "major federal actions architectures/designs. significantly affecting the quality of the human
environment." Some environmental impact 6.8.1 Environmental Impact reference docu-ments include:
National Environmental Policy Act (NEPA) of NASA policy and federal law require all 1969, as amended (40 CFR 1500-1508) NASA actions that may impact the quality of the Procedures for Implementing the National environment be executed in accordance with the Environmental Policy Act (14 CFR 1216.3) policies and procedures of the National Environmental Implementing the Requirements of the National Policy Act (NEPA). For any NASA project or other Environmental Policy Act, NHB 8800.11 major NASA effort, this requires that studies and Executive Order 11514, Protection and En- analyses be produced explaining how and why the hancement of Environmental Quality, March 5, project is planned, and the nature and scope of its 1970, as amended by Executive Order 11991, potential environmental impact. These studies must May 24, 1977 be performed whether the project is conducted at Executive Order 12114, Environmental Effects NASA Headquarters, a field center, or a contractor Abroad of Major Federal Actions, January 4, facility, and must properly begin at the earliest period 1979. of project planning (i.e., not later than Phase A).
Findings, in the form of an Environmental Assessment (EA) and, if warranted, through the more thorough analyses of an Environmental Impact Statement Significant Impact (FONSI). The analyses performed (EIS), must be presented to the public for review and should identify the environmental effects of all comment. (See sidebar on NEPA.) reasonable alternative methods of achieving the At the outset, some NASA projects will be of project's goals/objectives so that they may be such a magnitude and nature that an EIS is clearly compared. The alternative of taking no action (i.e., not going to be required by NEPA, and some will clearly doing the project) should also be studied. Although not need an EIS. Most major NASA projects, there is no requirement that NASA select the however, fall in between, where the need for an EIS is alternative having the least environmental impact, a priori unclear, in such cases an EA is prepared to there must be sufficient information available to make determine whether an EIS is indeed required. NASA's clear what those impacts would be, and to describe experience since 1970 has been that projects in the reasoning behind NASA's preferred selection. The which there is the release—or potential release— of environmental analyses are an integral part of the large or hazardous quantities of pollutants (rocket project's systems engineering process. exhaust gases, exotic materials, or radioactive The EA is the responsibility of the NASA substances), require an EIS. For projects in this Head-quarters Program Associate Administrator category, an EA is not performed, and the project's (PAA) responsible for the proposed project or action. analyses should focus on and support the preparation The EA can be carried out at Headquarters or at a of an EIS. NASA field center. Approval of the EA is made by the The NEPA process is meant to ensure that responsible PAA. Most often, approval of the EA the project is planned and executed in a way that takes the form of a memorandum to the Associate meets the national environmental policy and goals. Administrator (AA) for Management Systems and First, the process helps the system engineer shape Facilities (Code J) stating either that the project the project by putting potential environmental requires an EIS, or that it does not. If an EIS is found concerns in the forefront during Phase A. Secondly, to be necessary, a Notice of Intent (NOI) to prepare the process provides the means for reporting to the an EIS is written; if an EIS is found to be public the project's rationale and implementation unnecessary, a Finding of No Significant Impact method. Finally, it allows public review of and (FONSI) is written instead. comment on the planned effort, and requires NASA to
consider and respond to those comments. The Finding of No Significant Impact (FONSI). A FONSI system engineer should be aware of the following should briefly present the reasons why the proposed NEPA process elements. project or action, as presented in the EA, has been
judged to have no significant effect on the human Environmental Assessment (EA). An EA is a environment, and does not therefore require the concise public document that serves to provide preparation of an EIS. The FONSI for projects and sufficient evidence and analyses for determining actions that are national in scope is published in the whether to prepare either an EIS or a Finding of No Federal Register, and is available for public review for
NASA Systems Engineering Handbook
Page 100 a 30-day period. During that time, any supporting External review of the draft EIS is also information is made readily available on request. managed by the Code J AA. A notice announcing the
release and availability of the draft EIS is published in Notice of Intent (NOI). A Notice of Intent to file an the Federal Register, and copies are distributed with a EIS should include a brief description of the proposed request for comments. Upon receipt of the draft, the project or action, possible alternatives, the primary Environmental Protection Agency (EPA) also places a environmental issues uncovered by the EA, and notice in the Federal Register, and the date of that NASA's proposed scoping procedure, including the publication is the date that all time limits related to the time and place of any scoping meetings. The NOI is draft's release begin. A minimum of 45 days must be prepared by the responsible Headquarters PAA and allowed for comments. Comments from external published in the Federal Register. It is also sent to reviewers received by the Code J AA will be sent to interested parties. the office responsible for preparing the EIS. Each
comment should be incorporated in the final EIS. Scoping. The responsible Headquarters PAA must The draft form of the final EIS, modified as con-duct an early and open process for determining required by the review process just described, should the scope of issues to be addressed in the EIS, and be forwarded to the Code J AA for a final review for identifying the significant environmental issues. before printing and distribution. The final version Scoping is also the responsibility of the Headquarters should include satisfactory responses to all PAA responsible for the proposed project or action; responsible comments. While NASA need not yield to however, the responsible Headquarters PAA often each and every opposing comment, NASA's position works closely with the Code J AA. Initially, scoping should be rational, logical, and based on data and must consider the full range of environmental arguments stronger than those cited by the parameters en route to identifying those that are commentors opposing the NASA views. significant enough to be addressed in the EIS. According to NHB 8800.11, Implementing Examples of the environmental categories and the Pro-visions of the National Environmental Policy questions that should be asked in the scoping Act (Section 309.b), "an important element in the EIS process are contained in NHB 8800.11, Implementing process is in-volvement of the public. Early the Provisions of the National En-vironmental Policy involvement can go a long way toward meeting Act, Section 307.d. complaints and objections regarding a proposed
action, and experience has taught that a fully Environmental Impact Statement (EIS). The EA and informed and involved public is considerably more scoping elements of the NEPA process provide the supportive of a proposed action. When a proposed responsible Headquarters PAA with an evaluation of action is believed likely to generate significant public significant environmental effects and issues that must concern, the public should be brought in for be covered in the EIS. Preparation of the EIS itself consultation in the early planning stages. If an EIS is may be carried out by NASA alone, or with the warranted, the public should be involved both in assistance or cooperation of other government scoping and in the EIS review. Early involvement can agencies and/or a contractor. If a contractor is used, help lead to selection of the best alternative and to the the contractor should execute a disclosure statement least public objection." prepared by NASA Headquarters indicating that the
contractor has no interest in the outcome of the Record of Decision (ROD). When the EIS process project. has been completed and public review periods have The section on environmental consequences elapsed, NASA is free to make and implement the is the analytic heart of the EIS, and provides the basis decision(s) regarding the proposed project or action. for the comparative evaluation of the alternatives. The At that time, a Record of Decision (ROD) is prepared analytic results for each alternative should be by the Headquarters PAA responsible for the project displayed in a way that highlights the choices offered or action. The ROD becomes the official public record the decision maker(s). An especially suitable form is a of the consideration of environmental factors in matrix showing the alternatives against the categories reaching the decision. The ROD is not published in of environmental impact (e.g., air pollution, water the Federal Register, but must be kept in the official pollution, endangered species). The matrix is filled in files of the program/project in question and made with (an estimate of) the magnitude of the available on request. environmental impact for each alternative and
category. The subsequent discussion of alternatives 6.8.2 Nuclear Safety Launch Approval is an extremely important part of the EIS, and should
be given commensurate attention. Presidential Directive/National Security NASA review of the draft EIS is managed by Council Memorandum-25 (PD/NSC-25) requires that the Code J AA. When submitted for NASA review, the flight projects calling for the use of radioactive draft EIS should be accompanied by a proposed list of sources follow a lengthy analysis and review process federal, state and local officials, and other interested in order to seek approval for launch. The nuclear parties. safety launch approval process is separate and distinct from the NEPA compliance process. While
NASA Systems Engineering Handbook
Page 101 there may be overlaps in the data-gathering for both, • Update and refine the project, flight system, the documentation required for NEPA and nuclear launch vehicle, and radioactive source safety launch approval fulfill separate federal and descriptions NASA requirements. While NEPA is to be done at the • Update and refine the radioactive source design earliest stages of the project, launch approval officially trade study developed during Phase A begins with Phase C/D. • Assist DOE where appropriate in conducting a Phase A/B activities are driven by the preliminary assessment of the mission's nuclear requirements of the EA/EIS. At the earliest possible risk and environmental hazards. time (not later than Phase A), the responsible
Headquarters PAA must undertake to develop the The launch approval engineer is also project EA/EIS and a Safety Analysis/Launch responsible for coordinating the activities, interfaces, Approval Plan in coordination with the nuclear power and record-keeping related to mission nuclear safety system integration engineer and/or the launch vehicle issues. The following tasks are managed by the integration engineer. A primary purpose of the EA/EIS launch approval engineer: is to ensure a comprehensive assessment of the • Develop the project EA/EIS and Safety Analy-sis/ rationale for choosing a radioactive source. In Launch Approval Plan addition, the EA/EIS illuminates the environmental • Maintain a database of documents related to effects of alternative mission designs, flight systems, EA/EIS and nuclear safety launch approval tasks. and launch vehicles, as well as the relative nuclear This database will help form and maintain the safety concerns of each alternative. The launch audit trail record of how and why technical approval engineer ensures that the following specific decisions and choices are made in the mission requirements are met during Phase A: development and planning process. Attention to
this activity early on saves time and expense • Conduct a radioactive source design trade study later in the launch approval process when the that includes the definition, spacecraft design project may be called upon to explain why a impact evaluation, and cost trades of all particular method or alternative was given greater reasonable alternatives weight in the planning process. • Identify the flight system requirements that are • Provide documentation and review support as specific to the radioactive source appropriate in the generation of mission data and • For nuclear power alternatives, identify flight trade studies required to support the EA/EIS and system power requirements and alternatives, and safety analyses define the operating and accident environments • Establish a project point-of-contact to the launch to allow DOE (U.S. Department of Energy) to vehicle integration engineer, DOE, and NASA assess the applicability of existing nuclear power Headquarters regarding support to the EA/EIS system design(s). and nuclear safety launch approval processes.
This includes responding to public and During Phase B, activities depend on the Congressional queries regarding radioactive specifics of the project's EA/EIS plan. The responsible source safety issues, and supporting proceedings Headquarters PAA determines whether the resulting from any litigation that may occur. preparation and writing of the EA/EIS will be done at a • Provide technical analysis support as required for NASA field center, at NASA Headquarters, or by a the generation of accident and/or command contractor, and what assistance will be required from destruct environment for the radioactive source other field centers, the launch facility, DOE, or other safety analysis. The usual technique for the agencies and organizations. The launch approval technical analysis is a probabilistic risk engineer ensures that the following specific assessment (PRA). See Section 4.6.3. requirements are met during Phase B:
6.8.3 Planetary Protection
NASA Systems Engineering Handbook
Page 102 The U.S. is a signatory to the United Nation's protection requirements. This document is typically Treaty of Principles Governing the Activities of States reported on by the NASA PPO at a meeting of the in the Exploration and Use of Outer Space, Including Committee on Space Research (COSPAR) to inform the Moon and Other Celestial Bodies. Known as the other spacefaring nations of NASA's degree of "Outer Space" treaty, it states in part (Article IX) that compliance with international planetary protection exploration of the Moon and other celestial bodies requirements. shall be conducted "so as to avoid their harmful
contamination and also adverse changes in the
environment of the Earth resulting from the introduction of extraterrestrial matter." NASA policy (NMI 8020.7D) specifies that the purpose of preserving solar system conditions is for future biological and organic constituent exploration. It also establishes the basic NASA policy for the protection of the Earth and its biosphere from planetary and other extraterrestrial sources of contamination. The general regulations to which NASA flight projects must adhere are set forth in NHB 8020.12B, Planetary Protection Provisions for Robotic Extraterrestrial Missions. Different requirements apply to different missions, depending on which solar system object is targeted and the spacecraft or mission type (flyby, orbiter, lander, sample-return, etc.). For some bodies (such as the Sun, Moon, Mercury), there are no outbound contamination requirements. Present requirements for the outbound phase of missions to Mars, however, are particularly rigorous. Planning for planetary protection begins in Phase A, during which feasibility of the mission is established. Prior to the end of Phase A, the project manager must send a letter to the Planetary Protection Officer (PPO) within the Office of the AA for Space Science stating the mission type and planetary targets, and requesting that the mission be assigned a planetary protection category. Table 7 shows the current planetary protection categories and a summary of their associated requirements. Prior to the Preliminary Design Review (PDR) at the end of Phase B. the project manager must submit to the NASA PPO a Planetary Protection Plan detailing the actions that will be taken to meet the requirements. The project's progress and completion of the requirements are reported in a Planetary Protection Pre-Launch Report submitted to the NASA PPO for approval. The approval of this report at the Flight Readiness Review (FRR) constitutes the final approval for the project and must be obtained for permission to launch. An update to this report, the Planetary Protection Post-Launch Report, is prepared to report any deviations from the planned mission due to actual launch or early mission events. For sample return missions only, additional reports and reviews are required: prior to launch toward the Earth, prior to commitment to Earth reentry, and prior to the release of any extraterrestrial sample to the scientific community for investigation. Finally, at the formally declared end-of-mission, a Planetary Protection End- of-Mission Report is prepared. This document reviews the entire history of the mission in comparison to the original Planetary Protection Plan, and documents the degree of compliance with NASA's planetary
NASA Systems Engineering Handbook
Page 103 Appendix A—Acronyms GOES Geosynchonous Orbiting Environmental Acronyms are useful because they provide a Satellite shorthand way to refer to an organization, a kind of GSE Ground Support Equipment document, an activity or idea, etc. within a generally HQ NASA Headquarters understood context. Their overuse, however, can HST Hubble Space Telescope interfere with communications. The NASA Lexicon I&V Integration and Verification contains the results of an attempt to provide a ILS Integrated Logistics Support comprehensive list of all acronyms used in NASA ILSP I ntegrated Logistics Support Plan systems engineering. This appendix contains two ILSS Integrated Logistics Support System lists: The acronyms used in this handbook and the IOP Institutional Operating Plan acronyms or some of the major NASA organizations. IRAS Infrared Astronomical Satellite
IV&V Independent Verification and Validation AA ssociate Administrator (NASA) IVA Intravehicular Activities APA llowance for Program Adjustment LEM Lunar Excursion Module (Apollo) ACWP ctual Cost of Work Performed LEO Low Earth Orbit AGE erospace Ground Equipment LMEPO Lunar/Mars Exploration Program Office AHP Analytic Hierarchy Process LMI Logistics Management Institute BCWP Budgeted Cost of Work Performed LOOS Launch and Orbital Operations Support BCWS Budgeted Cost of Work Scheduled LRU Line Replaceable Unit C/SCSC Cost/Schedule Control System LSA Logistics Support Analysis Criteria LSAR Logistics Support Analysis Record CALS Continuous Acquisition and Life-Cycle MDT Mean Downtime Support MCR Mission Concept Review CCB Configuration (or Change) Control Board MDR Mission Definition Review CDR Critical Design Review MESSOC Model for Estimating Space Station CER Cost Estimating Relationship Operations CI Configuration Item MICM Multi-variable Instrument Cost Model CIL Critical Items List MLDT Mean Logistics Delay Time CoF Construction of Facilities MMT Mean Maintenance Time COSPAR Committee on Space Research MNS Mission Needs Statement COTR Contracting Office Technical Representative MoE Measure of (system) Effectiveness CPM Critical Path Method MRB Material Review Board CR Change Request MRR Mission Requirements Review CSCI Computer Software Configuration Item MTBF Mean Time Between Failures CSM Center for Systems Management MTTF Mean Time To Failure CWBS Contract Work Breakdown Structure MTTMA Mean Time To a Maintenance Action DCR Design Certification Review MTTR Mean Time To Repair/Restore DDT&E Design, Development, Test and Evaluation NAR Non-Advocate Review DoD (U.S.) Department of Defense NCR Non-Compliance (or Non-Conformance) DOE (U.S.) Department of Energy Report DR Decommissioning Review NEPA National Environmental Policy Act DSMC Defense Systems Management College NHB NASA Handbook EA Environmental Assessment NMI NASA Management Instruction EAC Estimate at Completion NOAA (U.S.) National Oceanic and Atmospheric ECP Engineering Change Proposal Administration ECR Engineering Change Request NOI Notice of Intent EIS Environmental Impact Statement OMB Office of Management and Budget EMC Electromagnetic compatibility OMRSD Operations and Maintenance EMI Electromagnetic interference Requirements and Specifications Document (KSC) EOM End of Mission ORLA Optimum Repair Level Analysis EPA (U.S.) Environmental Protection Agency ORR Operational Readiness Review EVA Extravehicular Activities ORU Orbital Replacement Unit EVM Earned Value Measurement P/FR Problem Failure Report FCA Functional Configuration Audit PAA Program Associate Administrator (NASA) FFBD Functional Flow Block Diagram PAR Program/Project Approval Review FH Flight Hardware PBS Product Breakdown Structure FMEA Failure Modes and Effects Analysis PCA Physical Configuration Audit FMECA Failure Modes, Effects, and Criticality PDR Preliminary Design Review Analysis PDT Product Development Team FONSI Finding of No Significant Impact PDV Present Discounted Value FRR Flight Readiness Review PERT Program Evaluation and Review Technique GAO General Accounting Office POP Program Operating Plan
NASA Systems Engineering Handbook
Page 104 PPAR
Preliminary Program/Project Approval E. Broad St., Athens GA 30602 Review GISS Goddard Institute for Space Studies (GSFC), PPO Planetary Protection Officer 2880 Broadway, New York NY 10025 PRA Probabilistic Risk Assessment GSFC Goddard Space Flight Center, Greenbelt Rd., PRD Program Requirements Document Greenbelt MD 20771 ProRR Production Readiness Review HQ National Aeronautics and Space QA Quality Assurance Administration Headquarters, Washington DC QFD Quality Function Deployment 20546 RAM Reliability, Availability, and Maintainability JPL Jet Propulsion Laboratory, 4800 Oak Grove RAS Requirements Allocation Sheet Dr., Pasadena CA 91109 RID Review Item Discrepancy JSC Lyndon B. Jhonson Space Center, Houston RMP Risk Management Plan TX 77058 ROD Record of Decision KSC John F. Kennedy Space Center, Kennedy RTG Radioisotope Thermoelectric Generator Space Center FL 32899 SAR System Acceptance Review SCC Slidell Computer Complex, 1010 Gauss Blvd, SDR System Definition Review Slidell LA 70458 SEB Source Evaluation Board SSC John C. Stennis Space Center, Stennis Space SEMP Systems Engineering Management Plan Center MS 39529 SEPIT Systems Engineering Process Improvement STIF Scientific & Technical Information Facility, Task P.O. Box 8757, BWI Airport MD 21240 SEWG
Systems Engineering Working Group WFF Wallops Flight Facility (GSFC), Wallops (NASA)SI Le Systeme International d' Unites (the Island VA 23337 international [metric] system of units) WSTF White Sands Test Facility (JSC), P.O. Drawer SIRTF Space Infrared Telescope Facility MM, Las Cruces NM 88004 SOFIA Stratospheric Observatory for Infrared
SoSR Software Specification Review SoW Statement of Work SSR System Safety Review SRD System/Segment Requirements Document SRM&QA Safety, Reliability, Maintainability, and Quality Assurance SRR System Requirements Review STEP Standard for the Exchange of Product (model data) STS Space Transportation System SSA Space Station Alpha SSF Space Station Freedom TBD To Be Determined; To Be Done TDRS Tracking and Data Relay Satellite TLA Time Line Analysis TLS Time Line Sheet TPM Technical Performance Measure(ment) TQM Total Quality Management TRR Test Readiness Review V&V Verification and Validation VMP Verification Master Plan VRCD Verification Requirements Compliance Document VRM Verification Requirements Matrix VRSD Verification Requirements and Specifications Document WBS Work Breakdown Structure WFD Work Flow Diagram
ARC Ames Research Center, Moffett Field CA 94035 COSMIC Computer Software Management & Information Center, University of Georgia, 382
NASA Systems Engineering Handbook
Page 105 Appendix B—Systems Engineering Requirements/Plans Templates and Examples 3.2 Integration System Test Plans
3.3 Compatibility with Supporting Activities Appendix B.1—A Sample SEMP Outline 3.3.1 System Cost-Effectiveness 3.3.2 Value Engineering
3.3.3 TQM/Quality Assurance An outline recommended by the Defense 3.3.4 Materials and Processes Systems Management College for the Systems
Engineering Management Plan is shown below. This outline is a sample only, and should be tailored for the nature of the project and its inherent risks.
Systems Engineering Management Plan
Title Page Introduction
Part 1 - Technical Program Planning and Control 1.0 Responsibilities and Authority 1.1 Standards, Procedures, and Training 1.2 Program Risk Analysis 1.3 Work Breakdown Structures 1.4 Program Review 1.5 Technical Reviews 1.6 Technical Performance Measurements 1.7 Change Control Procedures 1.8 Engineering Program Integration 1.9 Interface Control 1.10 Milestones/Schedule 1.11 Other Plans and Controls
Part 2 - Systems Engineering Process 2.0 Mission and Requirements Analysis 2.1 Functional Analysis 2.2 Requirements Allocation 2.3 Trade Studies 2.4 Design Optimization/Effectiveness Compatibility 2.5 Synthesis 2.6 Technical Interface Compatibility 2.7 Logistic Support Analysis 2.8 Producibility Analysis 2.9 Specification Tree/Specifications 2.10 Documentation 2.11 Systems Engineering Tools
Part 3—Engineering Specialty/Integration Requirements 3.1 Integration Design/Plans 3.1.1 Reliability 3.1.2 Maintainability 3.1.3 Human Engineering 3.1.4 Safety 3.1.5 Standardization 3.1.6 Survivability/Vulnerability 3.1.7 Electromagnetic Compatibility/Interference 3.1.8 Electromagnetic Pulse Hardening 3.1.9 Integrated Logistics Support 3.1.10 Computer Resources Lifecycle Management Plan 3.1.1 1 Producibility 3.1.12 Other Engineering Specialty
NASA Systems Engineering Handbook
Page 106 Appendix B.2 -- A "Tailored" WBS for an
Figure B- I shows a partial Product
Breakdown Structure (PBS) for the proposed
Stratospheric Observatory for Infrared Astronomy
(SOFIA), a 747SP aircraft outfitted with a 2.5 to 3.0 m
telescope. The PBS has been elaborated for the
airborne facility's telescope element. The PBS level
names have been made consistent with the sidebar
on page 3 of this handbook.
Figures B-2 through B-5 show a
corresponding Work Breakdown Structures (WBSs)
based on the principles in Section 4.3 of this
handbook. At each level, the prime product
deliverables from the PBS are WBS elements. The
WBS is completed at each level by adding needed
service (i.e., functional) elements such as
management, systems engineering, integration and
test, etc. The integration and test WBS element at
each level refers to the activities of unifying prime
product deliverables at that level.
Although the SOFIA project is used as an
illustration in this appendix, the SOFIA WBS should
be tailored to fit actual conditions at the start of Phase
C/D as determined by the project manager. One
example of a condition that could substantially change
the WBS is international participation in the project.
NASA Systems Engineering Handbook
NASA Systems Engineering Handbook
NASA Systems Engineering Handbook
NASA Systems Engineering Handbook
Page 110 Appendix B.4—A Sample Risk Management Plan Outline
1.0 Introduction 1.1 Purpose and Scope of the RMP 1.2 Applicable Documents and Definitions 1.3 Program/Project (or System) Description 2.0 Risk Management Approach 2.1 Risk Management Philosophy/Overview 2.2 Management Organization and Responsibilities 2.3 Schedule, Milestones, and Reviews 2.4 Related Program Plans 2.5 Subcontractor Risk Management 2.6 Program/Project Risk Metrics 3.0 Risk Management Methodologies, Processes, and Tools 3.1 Risk Identification and Characterization 3.2 Risk Analysis 3.3 Risk Mitigation and Tracking 4.0 Significant Identified Risks* 4.1 Technical Risks 4.2 Programmatic Risks 4.3 Supportability Risks 4.4 Cost Risks 4.5 Schedule Risks Each subsection contains risk descriptions, charac - terizations, analysis results, mitigation actions, and reporting metrics .
NASA Systems Engineering Handbook
NASA Systems Engineering Handbook
Page 112 Appendix B.6—A Sample Configuration Management Plan Outline
1.0 Introduction 1.1 Description of the Cls 1.2 Program Phasing and Milestones 1.3 Special Features 2.0 Organization 2.1 Structure and Tools 2.2 Authority and Responsibility 2.3 Directives and Reference Documents 3.0 Configuration Identification 3.1 Baselines 3.2 Specifications 4.0 Configuration Control 4.1 Baseline Release 4.2 Procedures 4.3 CI Audits 5.0 Interface Management 5. 1 Documentation 5.2 Interface Control 6.0 Configuration Traceability 6.1 Nomenclature and Numbering 6.2 Hardware Identification 6.3 Software and Firmware Identification 7.0 Configuration Status Accounting and Communications 7.1 Data Bank Description 7.2 Data Bank Content 7.3 Reporting 8.0 Configuration Management Audits 9.0 Subcontractor/Vendor Control
NASA Systems Engineering Handbook
Page 113 Appendix B.7—Techniques of Functional TDRS continually to allow for the reception of Analysis emergency commands or transmission of emergency
data? The FFBD also incorporates alternate and Appendix B.7 is reproduced from the contingency operations, which improve the probability Defense Systems Management Guide, published of mission success. The flow diagram provides an January 1990 by the Defense Systems Management understanding of total operation of the system, serves College, Ft. Belvoir, VA. as a basis for development of operational and • • • contingency proce-dures, and pinpoints areas where changes in operational procedures could simplify the System requirements are analyzed to overall system operation. In certain cases, alternate identify those functions which must be performed to FFBDs may be used to represent various means of satisfy the objectives of each functional area. Each satisfying a particular function until data are acquired, function is identified and described in terms of inputs, which permits selection among the alternatives. outputs, and interface requirements from top down so
that subfunctions are recognized as part of larger functional areas. Functions are arranged in a logical B.7.2 N 2 Diagrams sequence so that any specified operational usage of
the system can be traced in an end-to-end path. The N 2 diagram has been used extensively Although there are many tools available, functional to develop data interfaces, primarily in the software identification is accomplished primarily through the areas. However, it can also be used to develop use of 1) functional flow block diagrams (FFBDs) to hardware inter-faces. The basic N 2 chart is shown in depict task sequences and relationships, 2) N Figure B-7. The system functions are placed on the 2 diagrams to develop data interfaces, and 3) time line diagonal; the remainder of the squares in the N x N analyses to depict the time sequence of time-critical matrix represent the interface inputs and outputs. functions. Where a blank appears, there is no interface between
the respective functions. Data flows in a clockwise B.7.1 Functional Flow Block Diagrams direction between functions (e.g., the symbol F1 F2 indicates data flowing from function F
1, to function F2). The data being transmitted can be defined in the The purpose of the FFBD is to indicate the appropriate squares. Alternatively, the use of circles sequential relationship of all functions that must be and numbers permits a separate listing of the data accomplished by a system. FFBDs depict the time interfaces as shown in Figure B-8. The clockwise flow sequence of functional events. That is, each function of data between functions that have a feedback loop (represented by a block) occurs following the can be illustrated by a larger circle called a control preceding function. Some functions may be loop. The identification of a critical function is also performed in parallel, or alternate paths may be shown in Figure B-8, where function F taken. The duration of the function and the time 4 has a number of inputs and outputs to all other functions in the between functions is not shown, and may vary from a upper module. A simple flow of interface data exists fraction of a second to many weeks. The FFBDs are between the upper and lower modules at functions F function oriented, not equipment oriented. In other 7 and F words, they identify "what" must happen and do not 8. The lower module has complex interaction among its functions. The N assume a particular answer to "how" a function will be 2 chart can be taken down into successively lower levels to the hardware and performed. software component functional levels. In addition to FFBDs are developed in a series of levels. defining the data that must be supplied across the FFBDs show the same tasks identified through interface, the N functional decomposition and display them in their 2 chart can pinpoint areas where conflicts could arise. logical, sequential relationship. For example, the
entire flight mission of a spacecraft can be defined in
a top level FFBD, as shown in Figure B-6. Each block in the first level diagram can then be expanded to a
series of functions, as shown in the second level
diagram for "perform mission operations." Note that
the diagram shows both input (transfer to operational
orbit) and output (transfer to space transportation
system orbit), thus initiating the interface identification and control process. Each block in the second level
diagram can be progressively developed into a series
of functions, as shown in the third level diagram on
Figure B-6. These diagrams are used both to develop
requirements and to identify profitable trade studies. For example, does the spacecraft antenna acquire the
tracking and data relay satellite (TDRS) only when the
payload data are to be transmitted, or does it track
NASA Systems Engineering Handbook
NASA Systems Engineering Handbook
NASA Systems Engineering Handbook
Page 116 defining subsystem/component time requirements, time line analysis can be used to develop trade studies in areas other than time consideration (e.g., should the spacecraft location be determined by the ground network or by onboard computation using navigation satellite inputs? Figure B-10 is an example of a maintenance TLS which illustrates that availability of an item (a distiller) is dependent upon the completion of numerous maintenance tasks accomplished concurrently. Furthermore, it illustrates the traceability to higher level requirements by referencing the appropriate FFBD and requirement allocation sheet (RAS).
B.7.3 Time Line Analysis
Time line analysis adds consideration of functional durations and is used to support the development of design requirements for operation, test and maintenance functions. The time line sheet (TLS) is used to perform and record the analysis of time critical functions and functional sequences. Additional tools such as mathematical models and computer simulations may be necessary. Time line analysis is performed on those areas where time is critical to the mission success, safety, utilization of resources, minimization of down time, and/or increasing availability. Not all functional sequences require time line analysis, only those in which time is a critical factor. The following areas are often categorized as time critical: 1) functions affecting system reaction time, 2) mission turnaround time, 3) time countdown activities, and 4) functions requiring time line analysis to determine optimum equipment and/or personnel utilization. An example of a high level TLS for a space program is shown in Figure B-9. For time critical function sequences, the time requirements are specified with associated tolerances. Time line analyses play an important role in the trade-off process between man and machine. The decisions between automatic and manual methods will be made and will determine what times are allocated to what subfunctions. In addition to
NASA Systems Engineering Handbook
NASA Systems Engineering Handbook
NASA Systems Engineering Handbook
Page 119 Appendix B.8 -- The Effect of Changes in ORU MTBF on Space Station Freedom Operations
The reliability of Space Station Freedom's (SSF) Orbital Replacement Units (ORUs) has a profound effect on its operations costs. This reliability is measured by the Mean Time Between Failures (MTBF). One study of the effects, by Dr. William F. Fisher and Charles Price, was SSF External Maintenance Task Team Final Report (JSC, July 1990). Another, by Anne Accola, et al., shows these effects parametrically. Appendix B.8 excerpts this paper, Sensitivity Study of SSF Operations Costs and Selected User Resources (presented at the International Academy of Astronautics Symposium on Space Systems Costs Methodologies and Applications, May 1990). • • • There are many potential tradeoffs that can be performed during the design stage of SSF. Many of them have major implications for crew safety, operations cost, and achievement of mission goals. Operations costs and important non-cost operations parameters are examined. One example of a specific area of concern in design is the reliability of the ORUs that comprise SSF. The implications of ORU reliability on logistics upmass and downmass to and from SSF are great, thus affecting the resources available for utilization and for other operations activities. In addition, the implications of reliability on crew time available for mission accomplishment (i.e., experiments) vs. station maintenance are important. The MTBF effect on operations cost is shown in Figure B-11. Repair and spares costs are influenced greatly by varying MTBF. Repair costs are inversely proportional to MTBF, as are replacement spares. The initial spares costs are also influenced by variables other than MTBF. The combined spares cost, consisting of initial and replacement spares are not as greatly affected as are repair costs. The five- year operations cost is increased by only ten percent if all ORU MTBF are halved. The total operations cost is reduced by three percent if all ORU MTBF are doubled. It would almost appear that MTBF is not as important as one would think. However, MTBF also affects available crew time and available upmass much more than operations cost as shown in Figures B-12 and B-13.
Available crew time is a valuable commodity because it is a limited resource. Doubling the number of ORU replacements (by decreasing the MTBF) increases the maintenance crew time by 50 percent, thus reducing the amount of time available to perform the SSF. Extra ORUs taken to orbit reduces available useful experiments or scientific work by 22 percent. upmass that could be used to take up experimental By halving the ORU replacements, the maintenance payloads. Essentially, by doubling the number of ORU crew time decreases by 20 percent and the available replacements, the available upmass is driven to zero. crew time increases by eight percent. Conversely, halving the number of ORU replacements Available upmass is another valuable increases the available upmass by 30 percent. resource because a fixed number of Space Shuttle flights can transport only a fixed amount of payload to
NASA Systems Engineering Handbook
Page 120 Doubling the number of ORU replacements would mean eight extra Space Shuttle flights would be needed over five years. Halving the ORU replacements would require two fewer Space Shuttle flights over five years. No attempt was made to assess the Space Shuttle capability to provide the extra flights or the design cost impacts to create the ORUs with the different reliabilities. Figure B-16 shows the effect of assessing the cost impact of the previous two figures and combining them with the five-year operations cost. The influence of MTBF is effectively doubled when Although the effects of MTBF on resources is interesting, it is a good idea to quantify the effectiveness of the scenarios based on total cost to maintain the nominal re- sources. Figure B-14 shows the number of crew members needed each year to maintain the available crew time. The figure shows that to maintain the nominal available crew time after doubling the number of ORU replacements, the Station would need two extra crew members. It should be noted that no attempt was made to assess the design capability or design cost impacts to accommodate these extra crew members. The savings of crew due to halving the number of ORU replacements is small, effectively one less crew member for half the year. Figure B-15 shows the number of Space Shuttle flights over five years needed to maintain the nominal available upmass. The Space Shuttle flights were rounded upward to obtain whole flights.
NASA Systems Engineering Handbook
Page 121 the resources of available upmass and crew time are maintained at their nominal values.
NASA Systems Engineering Handbook
Page 122 Appendix B.9 -- An Example of a Verification Requirements Matrix
Appendix B.9 is a small portion of the Verification Requirements Matrix (VRM) from the National Launch System Level III System Requirements Document, originally published at the Marshall Space Flight Center. Its purpose here is to illustrate the content and one possible format for a VRM. The VRM coding key for this example is shown on the next page.
NASA Systems Engineering Handbook
NASA Systems Engineering Handbook
Page 124 Appendix B.10—A Sample Master Verification Plan Outline 1.0 Introduction 1.1 Scope 1.2 Applicable Documents 1.3 Document Maintenance and Control 2.0 Program/Project Description 2.1 Program/Project Overview and Verification Master Schedule 2.2 Systems Descriptions 2.3 Subsystems Descriptions 3.0 Integration and Verification (I&V) Organization and Staffing 3.1 Program/Project Management Offices 3.2 NASA Field Center I&V Organizations 3.3 International Partner I&V Organizations 3.4 Prime Contractor I&V Organization 3.5 Subcontractor I&V Organizations 4.0 Verification Team Operational Relationships 4.1 Verification Team Scheduling and Review Meetings 4.2 Verification and Design Reviews 4.3 Data Discrepancy Reporting and Resolution Procedures 5.0 Systems Qualification Verification 5.1 Tests* 5.2 Analyses 5.3 Inspections 5.4 Demonstrations 6.0 Systems Acceptance Verification 6.1 Tests* 6.2 Analyses 6.3 Inspections 6.4 Demonstrations 7.0 Launch Site Verification 8.0 On-Orbit Verification 9.0 Post-Mission/Disposal Verification 10.0 Verification Documentation 11.0 Verification Methodology 12.0 Support Equipment 12.1 Ground Support Equipment 12.2 Flight Support Equipment 12.3 Transportation, Handling, and Other Logistics Support 12.4 TDRSS/NASCOM Support 13.0 Facilities * This section contains subsections for each type of test, e.g., EMI/EMC, mechanisms, thermal/vacuum. This fur -ther division by type applies also to analyses, inspections, and demonstrations.
NASA Systems Engineering Handbook
Page 125 Appendix C—Use of the Metric System Prefixes for SI Units
Factor Prefix Sym. Pronunciation C.1 NASA Policy 10 24 yotta Y YOTT-a (a as in about)
10 21 zetta Z ZETT-a (a as in about) It is NASA policy (see NM1 8010.2A and 10 18 exa E EX-a (a as in about) NHB 7120.5) to: 10 15 peta P PET-a (as in petal)
10 12 tera T TERR-a (as in terrace) Adopt the International System of Units, known by the 10 9 giga G GIGa (g as in giggle, a as in international abbreviation SI and defined by about ANSI/IEEE Std 268-1992, as the preferred system of 10 6 mega M MEG-a (as in megaphone) weights and measurements for all major system 10 3 kilo k KILL-oh** development programs. 10 2 hecto* h HECK-toe Use the metric system in procurements, grants and 10 deka* da DECK-a (as in decahedron) business-related activities to the extent economically 1 feasible. 10 -1 deci* d DESS-ih (as in decimal) Permit continued use of the inch-pound system of 10 -2 centi* c SENT-ih (as in centipede) measurement for existing systems. 10 -3 milli m MILL-ih (as in military) Permit hybrid metric and inch-pound systems when 10 -6 micro µ MIKE-roe (as in microphone) full use of metric units is impractical or will 10 -9 nano n NAN-oh (a as in ant) compromise safety or performance. 10 -12 pico p PEEK-oh
10 -15 femto f FEM-toe C.2 Definitions of Units 10 -18 atto a AT-toe (a as in hat)
10 -21 zepto z ZEP-toe (e as in step) Parts of Appendix C are reprinted from IEEE 10 -24 yocto y YOCK-toe Std 268-1992, American National Standard for Metric * The prefixes that do not represent 1000 raised to Practice, Copyright 1992 by the Institute of a power (that is hecto, deka, deci, and centi) Electrical and Electronics Engineers, Inc. The IEEE should be avoided where practical. disclaims any responsibility or liability resulting from ** The first syllable of every prefix is accented to the placement and use in this publication. Information assure that the prefix will retain its identity. is reprinted with the permission of the EKE. Kilometer is not an exception. • • • Outside the United States, the comma is candela (cd) The candela is the luminous intensity, in widely used as a decimal marker. In some a given direction, of a source that emits applications, therefore, the common practice in the monochromatic radiation of frequency 540 x 10 12 Hz United States of using the comma to separate digits and that has a radiant intensity in that direction of into groups of three (as in 23,478) may cause 1/683 watt per steradian. ambiguity. To avoid this potential source of confusion,
recommended international practice calls for kelvin (K) The kelvin, unit of thermodynamic separating the digits into groups of three, counting tempera-ture, is the fraction 1/273.16 of the from the decimal point toward the left and the right, thermodynamic tem-perature of the triple point of and using a thin space to separate the groups. In water. numbers of four digits on either side of the decimal
point the space is usually not necessary, except for kilogram (kg) The kilogram is the unit of mass; it is uniformity in tables. equal to the mass of the international prototype of the
kilogram. (The international prototype of the kilogram C.2.1 SI Prefixes is a particular cylinder of platinum-iridium alloy which The names of multiples and submultiples of is preserved in a vault at Sevres, France, by the SI units may be formed by application of the prefixes International Bureau of Weights and Measures.) and symbols shown in the sidebar. (The unit of mass,
the kilogram, is the only exception; for historical meter (m) The meter is the length of the path reasons, the gram is used as the base for traveled by light in a vacuum during a time interval of construction of names.) 1 299 792 458 of a second.
C.2.2 Base SI Units mole (mol) The mole is the amount of substance of a
system which contains as many elementary entities ampere (A) The ampere is that constant current as there are atoms in 0.012 kilogram of carbon-12. which, if maintained in two straight parallel conductors Note: When the mole is used, the elementary entities of infinite length, of negligible circular cross section, must be specified and may be atoms, molecules, and placed one meter apart in vacuum, would ions, electrons, other particles, or specified groups of produce between these conductors a force equal to 2 such particles. x 10 -7 newton per meter of length.
NASA Systems Engineering Handbook
Page 126 second (s) The second is the duration of 9 192 631 joule (J = N . m) The joule is the work done when the 770 periods of the radiation corresponding to the point of application of a force of one newton is transition between the two hyperfine levels of the displaced a distance of one meter in the direction of ground state of the cesium-133 atom. the force.
C.2.3 Supplementary SI Units lumen (lm = cd . sr) The lumen is the luminous flux
emit-ted in a solid angle of one steradian by a point radian (rad) The radian is the plane angle between source having a uniform intensity of one candela. two radii of a circle that cut off on the circumference
an arc equal in length to the radius. lux (lx = lm/m2) The lux is the illuminance produced
by a luminous flux of one lumen uniformly distributed steradian (sr) The steradian is the solid angle that, over a surface of one square meter. hav-ing its vertex in the center of a sphere, cuts off an
area of the surface of the sphere equal to that of a newton (N = kg . m/s 2 ) The newton is that force square with sides of length equal to the radius of the which, when applied to a body having a mass of one sphere. kilogram, gives it an acceleration of one meter per
second squared. C.2.4 Derived SI Units with Special Names
ohm (Ω Ω = V/A) The ohm is the electric resistance In addition to the units defined in this be-tween two points of a conductor when a constant subsection, many quantities are measured in terms of differ-ence of potential of one volt, applied between derived units which do not have special names—such these two points, produces in this conductor a current as velocity in m/s, electric field strength in V/m, of one ampere, this conductor not being the source of entropy in J/K. any electromotive force.
becquerel (Bq = 1/s) The becquerel is the activity of ascal (Pa = N/m 2 ) The pascal [which, in the a radionuclide decaying at the rate of one preferred pronunciation, rhymes with rascal] is the spontaneous nuclear transition per second. pressure or stress of one newton per square meter.
degree Celsius (°C = K) The degree Celsius is equal siemens (S = A/V) The siemens is the electric to the kelvin and is used in place of the kelvin for conduc-tance of a conductor in which a current of one expressing Celsius temperature defined by the ampere is produced by an electric potential difference equation t= T- T0, where t is the Celsius temperature, of one volt. T is the thermodynamic temperature, and To = 273.15
K (by definition). sievert (Sv = J/kg) The sievert is the dose equivalent
when the absorbed dose of ionizing radiation coulomb (C = A . s) Electric charge is the time multiplied by the dimensionless factors Q (quality integral of electric current; its unit, the coulomb, is factor) and N (product of any other multiplying factors) equal to one ampere second. stipulated by the International Commission on
Radiological Protection is one joule per kilogram. farad (F = C/V) The farad is the capacitance of a
capacitor between the plates of which there appears a tesla (T = Wb/m 2 ) The tesla is the magnetic flux difference of potential of one volt when it is charged density of one weber per square meter. In an by a quantity of electricity equal to one coulomb. alternative approach to defining the magnetic field
quantities the tesla may also be defined as the gray (Gy = J/kg) The gray is the absorbed dose when magnetic flux density that produces on a one-meter the energy per unit mass imparted to matter by length of wire carrying a current of one ampere, ionizing radiation is one joule per kilogram. (The gray oriented normal to the flux density, a force of one is also used for the ionizing radiation quantities: newton, magnetic flux density being defined as an specific energy imparted, kerma, and absorbed dose axial vector quantity such that the force exerted on an index.) element of current is equal to the vector product of
this element and the magnetic flux density. henry (H = Wb/A) The henry is the inductance of a
closed circuit in which an electromotive force of one volt (V = W/A) The volt (unit of electric potential differ- volt is produced when the electric current in the circuit ence and electromotive force) is the difference of varies uniformly at a rate of one ampere per second. electric potential between two points of a conductor
carrying a constant current of one ampere, when the hertz (Hz = 1/s) The hertz is the frequency of a power dissipated between these points is equal to periodic phenomenon of which the period is one one watt. second.
watt (W = J/s) The watt is the power that represents a rate of energy transfer of one joule per second.
NASA Systems Engineering Handbook
applications. The kilowatthour is widely used, weber (Wb = V . s) The weber is the magnetic flux however, as a measure of electric energy. This unit that, linking a circuit of one turn, produces in it an should not be introduced into any new areas, and electromo-tive force of one volt as it is reduced to eventually, it should be replaced by the megajoule. In zero at a uniform rate in one second. that same "temporary" category, the standard also
defines the barn (1 lb = 10 -28 m 2 ) for cross section, C.2.5 Units in Use with SI the bar (1 bar = 10 5 Pa) for pressure, the curie (1 Ci =
3.7 x 10 10 Bq) for radionuclide activity, the roentgen Time The SI unit of time is the second. This unit is (1 R = 2.58 x 10 -4 C/kg) for X- and gamma-ray pre-ferred and should be used if practical, particularly exposure, the rem for dose equivalent (1 rem = 0.01 when technical calculations are involved. In cases Sv), and the rad (1 rd = 0.01 Gy) for absorbed dose. where time relates to life customs or calendar cycles,
the minute, hour, day and other calendar units may be necessary. For example, vehicle speed will normally be expressed in kilometers per hour. minute (min) 1 min = 60 s hour (h) 1 h = 60 min = 3600 sec day (d) 1 d = 24 h = 86 400 sec week, month, etc.
Plane angle The SI unit for plane angle is the radian. Use of the degree and its decimal submultiples is permissible when the radian is not a convenient unit. Use of the minute and second is discouraged except for special fields such as astronomy and cartography. degree (°) 1°= (π/ 180) red minute (') 1' = (1/60)° = (π/10 800) rad second (") 1 " = (1/60)' = (π/648 000) rad
Area The SI unit of area is the square meter (m 2 ). The hectare (ha) is a special name for the square hectometer (hm 2 ). Large land or water areas are generally expressed in hectares or in square kilometers (km 2 ).
Volume The SI unit of volume is the cubic meter. This unit, or one of the regularly formed multiples such as the cubic centimeter, is preferred. The special name liter has been approved for the cubic decimeter, but use of this unit is restricted to volumetric capacity, dry measure, and measure of fluids (both liquids and gases). No prefix other than milli- or micro- should be used with liter. Mass The SI unit of mass is the kilogram. This unit, or one of the multiples formed by attaching an SI prefix to gram (g), is preferred for all applications. The megagram (Mg) is the appropriate unit for measuring large masses such as have been expressed in tons. However, the name ton has been given to several large mass units that are widely used in commerce and technology: the long ton of 2240 lb, the short ton of 2000 lb, and the metric ton of 1000 kg (also called tonne outside the USA) which is almost 2205 lb. None of these terms are SI. The term metric ton should be restricted to commercial usage, and no prefixes should be used with it. Use of the term tonne is deprecated.
Others The ANSI/IEEE standard lists the kilowatthour (1 kWh = 3.6 MJ) in the category of "Units in Use with SI Temporarily". The SI unit of energy, the joule, together with its multiples, is preferred for all
NASA Systems Engineering Handbook
Page 128 C.3 Conversion Factors One of the many places a complete set of conversion factors can be found is in ANSI/IEEE Std 268-1992. The abridged set given here is taken from that reference. Symbols of SI units are given in bold face type and in parentheses. Factors with an asterisk (*) between the number and its power of ten are exact by definition. To conform with the international practice, this section uses spaces -- rather than commas -- in number groups.
NASA Systems Engineering Handbook
NASA Systems Engineering Handbook
NASA Systems Engineering Handbook
Page 131 Bibliography Sons, Inc., New York, 1991. Referred to on Air Force, Department of the, page(s) 1. Producibility Measurement Boden, Daryl, and Wiley J. Larson (eds), for DoD Contracts, Office of the Cost-Effective Assistant for Reli-ability, Space Mission Operations, publication Maintainability, Manufacturing, and scheduled Quality, SAF/AQXE, Washington, DC, for August 1995. Referred to on page(s) 1. 1992. Boehm, Barry W., ''A Spiral Model of Referred to on page(s) 111. Software Develop-ment , Unmanned Space Vehicle Cost Model, and Enhancement", Computer, pp 61 -72, Seventh May 1988. Referred to on page(s) 13. Edition, SD TR-88-97, Air Force Systems Carter, Donald E., and Barbara Stilwell Com-mand/ Baker, Space Division (P. Nguyen, et al.), Concurrent Engineering: The Product August Development Environment for the 1990s, 1994. Referred to on page(s) 81. Addison-Wesley Publishing Co., Inc., Agrawal, Brij N., Design of Reading, Geosynchronous Spacecraft, MA, 1992. Referred to on page(s) 25. Prentice-Hall, Inc., Englewood Cliffs, NJ Chamberlain, Robert G., George Fox and 07632, William H. 1986. Referred to on page(s) 1. Duguette, A Design Optimization Armstrong, J.E., and Andrew P. Sage, An Process for Introduction to Space Station Freedom, JPL Publication Systems Engineering, John Wiley & 90-23, Sons, New June 15, 1990. Referred to on page(s) 18. York, 1995. Referred to on page(s) 1. Chestnut, Harold, Systems Engineering Army, Department of the, Logistics Tools, John Support Analysis Wiley & Sons, Inc., New York, 1965. Techniques Guide, Pamphlet No. 700-4, Referred to Army on page(s) 1. Materiel Command, 1985. Referred to on , Systems Engineering Methods, John page(s) 102. Wiley & Asher, Harold, Cost-Quantity Sons, Inc., New York, 1965. Referred to on Relationships in the page(s) 1. Airframe Industry, R-291, The Rand Churchman, C. West, Russell L. Ackoff Corporation, and E. Leonard 1956. Referred to on page(s) 82. Arnoff, Introduction to Operations Barclay, Scott, et al., Handbook for Research, John Decision Analysis, Wiley & Sons, Inc., New York, 1957. Decisions and Designs, Inc., McLean, Clark, Philip, et al., How CALS Can VA, Improve the DoD September 1977. Referred to on page(s) Weapon System Acquisition Process, 43. PL207RI, Blanchard, B.S., and W.J. Fabrycky, Logistics Management Institute, Systems November 1992. Engineering and Analysis (2nd Edition), Referred to on page(s) 103. Prentice-Hall, Inc. Englewood Cliffs, NJ, Committee on Space Research, COSPAR 1990. Information Referred to on page(s) 1. Bulletin No. 38 (pp. 3-10), June 1967. , System Engineering Management, John Referred Wiley & to on page(s) 115.
NASA Systems Engineering Handbook
Page 132 Defense, Department of, Transition from Engineering, EIA/IS-632, 2001 Development Pennsylvania Ave. to Production, DoD 4245.7-M, 1985. NW, Washington, DC, 20006, December Referred to 1994. on page(s) 40, 111. Referred to on page(s) 1, 4. , Metric System, Application in New Executive Order 11514, Protection and Design, DOD-STD- Enhancement of 1476A, 19 November 1986. Environmental Quality, March 5, 1970, as , Logistics Support Analysis and amended by Executive Order 11991, May Logistics Support 24, Analysis Record, MIL-STD-1388-1A/2B, 1977. Referred to on page(s) 112. revised Executive Order 12114, Environmental 21 January 1993, Department of Defense. Effects Abroad Referred to on page(s) 86, 100, 102. of Major Federal Actions, January 4, , Failure Modes, Effects, and Criticality 1979. Analysis, Referred to on page(s) 112. MIL-STD-1629A, Department of Defense. Fisher, Gene H., Cost Considerations in Re-ferred Systems to on page(s) 41. Analysis, American Elsevier Publishing Defense Systems Management College, Co. Inc., Scheduling New York, 1971. (Also published as R- Guide for Program Managers, GPO 490-ASD, #008-020-01196-1, January 1990. The Rand Corporation, December 1970.) , Test and Evaluation Management Referred to on page(s) 1, 67. Guide, GPO Forsberg, Kevin, and Harold Mooz, "The #008-020-01303-0, 1993. Relationship of
System Engineering to the Project
,Integrated Logistics Support (ILS) Cycle", Center Guide, for Systems Management, 5333 Betsy DTIC/NTIS #ADA 171 -087, 1993. Ross Dr., ,Systems Engineering Management Santa Clara, CA 95054; also available in Guide, A DTIC/NTIS #ADA223- 168, 1994. Referred Commitment to Success, Proceedings of to on the first page(s) 1, 40. annual meeting of the National Council ,Risk Management: Concepts and for Guidance, Systems Engineering and the 12th GPO #008-020-01164-9, 1994. annual DeJulio, E., SIMSYLS User's Guide, meeting of the American Society for Boeing Aerospace Engineering Operations, February 1990. Referred to on Management, Chattanooga, TN, 20-23 page(s) 87. October de Neufville, R., and J.H. Stafford, 1991. Referred to on page(s) 20. Systems Analysis for Fortescue, Peter W., and John P.W. Starl; Engineers and Managers, McGraw-Hill, (eds), New York, Spacecraft Systems Engineering, John 1971. Referred to on page(s) 1, 43. Wiley and Electronic Industries Association (EIA), Sons Ltd., Chichester, England, 1991. Systems Referred to on page(s) 1.
NASA Systems Engineering Handbook
Page 133 Gavin, Joseph G., Jr. (interview with), Hood, Maj. William C., A Handbook of Technology Supply Inventory Review, Vol. 97, No. 5, July 1994. Models, Air Force Institute of Referred to on Technology, School page(s) 92. of Systems and Logistics, September GAO/AIMD-94-197R, September 30, 1994. 1987. Re-ferred Hughes, Wayne P., Jr. (ed.), Military to on page(s) 103. Modeling, Military Goddard Space Flight Center, Goddard Operations Research Society, Inc., 1984. Multi-variable In-strument IEEE, American National Standard for Cost Model (MICM), Research Note #90- Metric Practice, 1, ANSI/IEEE Std 268- 1992 (supersedes IEEE Std Resource Analysis Office (Bernard Dixon
268-1979 and ANSI Z210.1-1976), Paul Villone, authors), NASA/GSFC, May American Na-tional 1990. Standards Institute (ANSI), 1430 Referred to on page(s) 81. Broadway, Green, A.E., and A.J. Bourne, Reliability New York, NY 10018. Referred to on Technology, page(s) Wiley Interscience, 1972. 139. Griffin, Michael D., and James R. French, , IEEE Trial-Use Standard for Application Space and Vehicle Design, AIAA 90-8, American Management of the Systems Engineering Institute of Process, IEEE Std 1220-1994., 345 E 47th Aeronautics and Astronautics, c/o St., TASCO, P.O. New York, NY 10017, February 28, 1995. Box 753, Waldorf, MD 20604 9961, 1990. Referred to on page(s) 1. Referred to on page(s) 1. Jet Propulsion Laboratory, The JPL Hickman, J.W., et al., PRA Procedures System Guide, The Development Management Guide, American Nuclear Society and The Version 1, JPL Institute of D-5000, NASA/JPL, November 15, 1989. Electrical and Electronics Engineers, Johnson Space Center, NASA Systems NUREG/CR-2300, Washington, DC, Engineering January Process for Programs and Projects, 1983. Referred to on page(s) 42. Version 1.O, Hillier, F.S. and G.J. Lieberman, JSC-49040, NASA/JSC, October 1994. Introduction to Opera-tions Referred Research, 2nd Edition, Holden-Day, Inc., to on page(s) xi, 22. 1978. Kececioglu, Dimitri, and Pantelis Hodge, John, "The Importance of Cost Vassiliou, 1992-1994 Considerations in Reliability, Availability, and the Systems Engineering Process", in Maintainability the NAL Software Handbook Reliability Monograph Series: Systems Engineering Engineering Papers, Program, University of Arizona, 1992. NASA Alumni League, 922 Pennsylvania Referred to Ave. on page(s) 95. S.E., Washington, DC 20003, 1990. Keeney, R.L., and H. Raiffa, Decisions Referred to with Multiple Ob-jechres: on page(s) 14.
NASA Systems Engineering Handbook
Page 134 Preferences and Value Tradeoffs, John Human Development (Code ND), NASA Wiley and Sons, Inc., New York, 1976. Headquarters, by DEF Enterprises, P.O. Referred Box to on page(s) 76. 590481, Houston, TX 77259, March 1990. Kline, Robert, et al., The M-SPARE Referred to on page(s) 117. Model, Logistics Marshall Space Flight Center, Program Management Institute, NS901R1, March Risk Analysis 1990. Handbook, NASA TM-100311, Program Referred to on page(s) 87. Planning Langley Research Center, Systems Office (R.G. Batson, author), Engineering NASA/MSFC, Handbook for In-House Space Flight August 1987. Projects, , Systems Engineering Handbook, LHB 7122.1, Systems Engineering Volume 1 — Division, Overview and Processes; Volume 2— NASA/LaRC, August 1994. Tools, Lano, R., The N 2 Chart, TRW Inc., 1977. Techniques, and Lessons Learned, Referred to MSFC-HDBK-1912, Systems Analysis on page(s) 68, 127. and Larson, Wiley J., and James R. Wertz Integration Laboratory, Systems (eds), Space Analysis Division, Mission Analysis and Design (2nd NASA/MSFC, February 1991. Referred to edition), on published jointly by Microcosm, Inc., page(s) 75, 89. 2601 Airport , Verification Handbook, Volume 1— Dr., Suite 23O, Torrance, CA 90505 and Verification Kluwer Process; Volume 2— Verification Academic Publishers, P.O. Box 17, 3300 Documentation AA Examples, MSFC-HDBK-2221, Systems Dordrecht, The Netherlands, 1992. Analysis Referred to and Integration Laboratory, Systems on page(s) 1. Integration , Reducing Space Mission Cost, Division, Systems Verification Branch, publication February scheduled for November 1995. Referred 1994. to on , General Environmental Test Guidelines page(s) 1. (GETG) Leising, Charles J., "System for Protoflight Instruments and Architecture", in System Experiments, Engineering at JPL, course notes MSFC-HDBK-67O, Systems Analysis and (contact: Judy Integra-tion Cobb, Jet Propulsion Laboratory), 1991. Laboratory, Systems Integration Referred Division, to on page(s) 11. Systems Verification Branch, February Lexicon—Glossary of Technical Terms 1994. Re-ferred and to on page(s) 109. Abbreviations Including Acronyms and McCormick, Norman, Reliability and Risk Symbols, Analysis, Aca-demic Draft/Version 1.0, produced for the NASA Press, Orlando, FL, 1981. Program/Project Management Initiative, Miles, Ralph F., Jr. (ed.), Systems Office of Concepts—Lectures
NASA Systems Engineering Handbook
Page 135 on Contemporary Approaches to , NHB 5300.4 (1A-1), Reliability Program Systems, John Require-ments Wiley & Sons, New York, 1973. Referred for Aeronautical and Space System to on Contractors, Office of Safety and Mission page(s) 1. Assurance (Code Q), NASA/HQ, January
Moore, N., D. Ebbeler and M. Creager, "A Referred to on page(s) 91, 92. Methodology , NHB 5300.4 (1B), Quality Program for Probabilistic Prediction of Structural Provisions Failures of for Aeronautical and Space System Launch Vehicle Propulsion Systems", Contractors, American Office of Safety and Mission Assurance Institute of Aeronautics and (Code Q), Astronautics, 1990. NASA/HQ, April 1969. Referred to on Referred to on page(s) 89. page(s) Morgan, M. Granger, and Max Henrion, 95. Uncertainty: A , NHB 5300.4 (1E), Maintainability Guide to Dealing with Uncertainty in Program Re-quirements Quantitative for Space Systems, Office of Safety Risk and Policy Analysis, Cambridge and Mission Assurance (Code Q), University NASA/HQ, Press, Cambridge, England, 1990. March 1987. Referred to on page(s) 96, Morris, Owen, ''Systems Engineering and 100. Integration , NHB 7120.5 (with NMI 7120.4), and Management for NASA Manned Management of Space Flight Major System Programs and Projects, Programs", in NAL Monograph Series: Office of Systems the Administrator (Code A), NASA/HQ, Engineering Papers, NASA Alumni November League, 922 1993 (revision forthcoming). Referred to Pennsylvania Ave. S.E., Washington, DC on 20003, page(s) ix, xi, 13, 14, 17, 99, 102, 139. 1990. Referred to on page(s) 9, 11. , NHB 8800.11, Implementing the NASA Headquarters, Initial OSSA Requirements Metrication Transition of the National Environmental Policy Act, Plan, Office of Space Science and Office of Applications Management Systems and Facilities (Code S), NASA/HQ, September 1990. (Code J), , NHB 1700.1B, NASA Safety Policy and NASA/HQ, June 1988. Referred to on Require-ments page(s) Document, Office of Safety and Mission 113. Assurance (Code Q), NASA/HQ, June , NHB 8020.12B, Planetary Protection 1993. Re-ferred Provisions to on page(s) 55. for Robotic Extraterrestrial Missions, , NHB 5103.6B, Source Evaluation Board Office of Space Hand-book, Science (Code S), NASA/HQ, Office of Procurement (Code H), forthcoming. Referred to NASA/HQ, on page(s) 115. October 1988. Referred to on page(s) 75. , NHB 9501.2B, Procedures for Contractor
NASA Systems Engineering Handbook
Page 136 Reporting of Correlated Cost and
Performance Data, Fi-nancial Navy, Department of the, Producibility Management Division (Code BF), Measurement NASA/HQ, Guidelines: Methodologies for Product February 1985. Referred to on page(s) 59. Integrity, , NMI 5350.1A, Maintainability and NAVSO P-3679, Office of the Assistant Maintenance Secretary Planning Policy, Office of Safety and for Research, Development, and Mission Assurance Acquisition, (Code Q), NASA/HQ, September 26, 1991. Washington DC, August 1993. Referred to Referred to on on page(s) 99, 100. page(s) 112. , NMI 7120.4 (with NHB 7120.5), Nuclear Regulatory Commission, U.S., Management of (by N.H. Major System Programs and Projects, Roberts, et al.), Fault Tree Handbook, Office of the NUREG-0492, Office of Nuclear Administrator (Code A), NASA/HQ, Regulatory November 1993 Research, January 1980. Referred to on (revision forthcoming). Referred to on page(s) page(s) ix, xi, 3, 94. 13. Office of Management and Budget, , NMI 8010.1A, Classification of NASA Guidelines and Dis-count Payloads, Rates for Benefit-Cost Analysis of Safety Division (Code QS), NASA/HQ, Federal 1990. Referred Programs, Circular A-94, OMB, October to on page(s) 38, 123. 1992. , NMI 8010.2A, Use of the Metric System Referred to on page(s) 79. of Pace, Scott, U.S. Access to Space: Measurement in NASA Programs, Office Launch Vehicle of Safety and Choices for 1990-2010, The Rand Mission Quality (Code Q), NASA/HQ, Corporation, June 11, 1991. R-3820-AF, March 1990. Referred to on page(s) 139. Presidential Directive/National Security , NMI 8020.7D, Biological Contamination Council Control Memorandum-25 (PD/NSC-25), Scientific for Outbound and Inbound Planetary of Spacecraft, Office Technological Experiments with of Space Science (Code S), NASA/HQ, Possible December 3, Large-Scale Adverse Environmental 1993. Referred to on page(s) 115. Effects and , NMI 8070.4A, Risk Management Policy, Launch of Nuclear Systems into Space, Safety December Division (Code QS), NASA/HQ, undated. 14, 1977. Referred to on page(s) 114. Referred to Procedures for Implementing the on page(s) 37, 44. National National Environmental Policy Act Environmental Policy Act (14 CFR (NEPA) of 1969, as 1216.3), amended (40 CFR 1500- 1508), Referred Referred to on page(s) 112. to on Reilly, Norman B., Successful Systems page(s) 112. Engineering for
NASA Systems Engineering Handbook
Page 137 Engineers and Managers, Van Nostrand Propulsion Laboratory, May 1, 1990. Reinhold, Referred to New York, 1993. Referred to on page(s) 1. on page(s) 76. Ruskin, Arnold M., "Project Management SP-7012 (NASA Special Publication), The and System International Engineering: A Marriage of System of Units; Physical Constants and Convenience", Conver-sion Claremont Consulting Group, La Canada, Factors, E.A. Mechtly, published by the California; presented at a meeting of the NASA Scientific and Technical Southern Information Office, California Chapter of the Project 1973; U.S. Government Printing Office, Management (stock Institute, January 9, 1991. Referred to on number 3300-00482). page(s) 11. Steuer, R., Multiple Criteria Optimization, Saaty, Thomas L., The Analytic Hierarchy John Wiley Process, and Sons, Inc., New York, 1986. McGraw-Hill, New York, 1980. Referred to Stewart, Rodney D., Cost Estimating, 2nd on ea., John page(s) 75. Wiley and Sons, Inc., New York, 1991. Sage, Andrew P., Systems Engineering, Taguchi, Genichi, Quality Engineering in John Wiley & Production Sons, New York, 1992. Referred to on Systems, New York: McGraw-Hill, 1989. page(s) 1. Referred Shishko, R., Catalog of JPL Systems to on page(s) 111. Engineering Tools Wagner, G. M., Principles of Operations and Models (1990), JPL D-8060, Jet Research— Propulsion With Applications to Managerial Laboratory, 1990. Decisions, , MESSOC Version 2.2 User Manual, JPL Prentice Hall, 1969. D-5749/
Rev. B, Jet Propulsion Laboratory,
1990. Referred to on page(s) 81. Sivarama Prased, A.V., and N. Somasehara, ''The AHP for Choice of Technologies: An Application", Technology Forecasting and Social Change, Vol. 38, pp. 151-158, September 1990. Smith, Jeffrey H., Richard R. Levin and Elisabeth J. Carpenter, An Application of Multiattribute Decision Analysis to the Space Station Freedom Program— Case Study: Automation and Robotics Technology Evaluation, JPL Publication 90-12, Jet
NASA Systems Engineering Handbook
Page 138 Index Cost-effectiveness 4, 5, 9, 10, 29, 91, 96, Advanced projects 13 99, 111 Alpha—see Space Station Alpha in trade studies 67-69, 72, 74, 77 Analytic Hierarchy Process (AMP) 75 Cost Estimating Relationship (CER) 80, Apollo 9, 10, 85 81, 88 Architecture—see system architecture Cost/Schedule Control System (C/SCS) Audits 30, 48, 49, 58, 96 59 Availability Critical Design Review (CDR) 18, 19, 45, measures of 62 52, 53, 56, 58, as a facet of effectiveness 83-85, 96 59, 96, 106 models of 85-87, 95, 98 Critical Item/Issue List (CIL) 39, 44, 91, as part of the LSA 101 125 Baselines 4, 8, 14, 17, 18, 27, 36, 45 Critical path 13, 33, 34 control and management of 10, 21, 28-30, Decision analysis 39, 41, 76 44-48, Decision sciences 7, 77 126 Decision support package 10, 71 in C/SCS 60 Decision trees 41, 70 in reviews 50-54 Design engineer(ing) 6, 8, 28, 49, 77 Bathtub curve 92, 93 Design-for-X 91 Bayesian probability 40 Design reference mission—see reference Budget cycle, NASA 18, 25, 26 mission Budgeting, for projects 31, 35, 37, 59 Design-to-cost—see design-to-life-cycle Change Control Board (CCB) cost composition of 46 Design-to-life-cycle cost 80 conduct of 46, 47 Design trades—see trade studies Change Request (CR) 46-49, 64, 65, 71, Digraphs 39, 41 80, 91, 95, 103 Discounting—see present discounted Concurrent engineering 21, 22, 25, 28, value 29, 39, 103 Earned value 7, 31, 60, 62 Configuration control 4, 8, 11, 20, 45-48, Effectiveness 4, 5, 9, 10 126 facets of 83-85, 91 Configuration management 10, 17, 29, in TPM 61 44-48, 126 in trade studies 67-70, 100, 102 Congress 3, 18, 25, 26, 114 Engineering Change Request (ECR)— Constraints 3, 5, 8-11, 18, 28, 35, 58, 67- see change 70, 72, 77 request Contingency planning 39, 44 Engineering specialty disciplines 6, 91- Continuous Acquisition and Life-Cycle 115 Support (CALS) in concurrent engineering 22-25 103 in SEMP 28, 29, 119 Control gates 13-22, 27, 45, 48, 49, 50-58, in trade studies 69, 77, 80, 85 79 Environmental Assessment (EA) 112-114 Cost (see also life-cycle cost) 4, 5, 9, 10 Environmental Impact Statement (EIS) account structure 27, 30, 31, 33 112 -114 caps 35, 37 Estimate at Completion (EAC) 31, 60, 88 estimating 80-83 Event trees 42 fixed vs variable 37 Failure Modes and Effects Analysis in trade studies 67-69, 74, 77 (FMEA)—see Fail-ure operations 81, 132-134 Modes, Effects, and Criticality Analysis overruns 17, 77 Failure Modes, Effects, and Criticality spreaders 82 Analysis
NASA Systems Engineering Handbook
Page 139 (FMECA) 39,41,91,95,98, 102, 105 Lexicon, NASA 117 Fault avoidance 94 Life-cycle cost (see also cost) 8, 10, 77- Fault tolerance 95 83 Fault tree 39, 42, 94 components of 77, 78 Feasibility 14, 18, 21, 50, 62 controlling 79, 80, 83, 111 Feasible design 5, 18 Linear programming 7, 72 Figure of merit 74, 75 Logistics Support Analysis (LSA) 53, 91, Flight Readiness Review (FRR) 19, 53, 96-103, 119 54, 96, 108, 115 Logistics Support Analysis Record Freedom—see Space Station Freedom (LSAR) 98-103 Functional Configuration Audit (FCA) 58, Lunar Excursion Module (LEM) 92 96 Maintainability 80, 92, 96-99 Functional Flow Block Diagram (FFBD) concept 97 68, 98, 111, definition of 96 127-129 models 98 Functional redundancy 94 Maintenance Galileo 94 plan 97 Game theory 7 time lines 98, 129, 131 Gantt chart 35, 36 Make-or-buy 29 Goddard Space Flight Center (GSFC) 81 Margin 43, 44, 62-64
Marshall Space Flight Center (MSFC) x
Hazard rate 92, 83 handbooks 75, 89, 109 Heisenberg—see uncertainty principle historical cost models 81 Hubble Space Telescope 98 Material Review Board (MRB) 96 IEEE 1, 42, 139, 141, 142 Mean Time Between Failures (M1BF) 80, Improvement 98, 132-134 continuous 64 Mean Time To Failure (MTTF) 86, 92, 98 product or process 11, 20, 25, 47, 78, 86 Mean Time to Repair (or Restore) (MTTR) Independent Verification and Validation 86, 98 (IV&V) 110 Metric system Inflation, treatment of 78, 79, 82 conversion factors for 142-144 Institutional Operating Plan (IOP) 25 definition of units in 139 -141 Integrated Logistics Support (ILS) 17, 29, Military standards 41, 86, 100, 102 30, 53, 78, 85-87, Mission analysis 7 92, 99-103, 105, 119 Mission assurance 91 concept 96, 97 Mission Needs Statement (MNS) 14, 17, plan 19, 87, 97-99 45, 56 Integration Models, mathematical conceptual 11 characteristics of good 72, 73 system 4, 11, 19, 22, 29, 30, 32, 33, 53 of cost 42, 80-83 Interface of effectiveness 42, 83-85 requirements 9-11, 17, 19, 45, 52, 53, 68, Markov 98 119, pitfalls in using 72, 88 127 programming 7, 72 control of 28, 64, 119, 126 relationship to SEMP 29, 83 Jet Propulsion Laboratory (JPL) ix, x, 81 types of 71, 72 Johnson Space Center (JSC) x, 43, 82 use in systems engineering 6, 7, 21, 67- Kennedy Space Center (KSC) 110 71, 87-89, Learning curve 82 106, 129 Lessons learned 11, 19, 30, 39, 41, 94, 98 Monte Carlo simulation 39, 42, 69, 88, 89, 95
NASA Systems Engineering Handbook
Page 140 Multi-attribute utility theory 75, 76 models 111 Network schedules 33-35, 42 Producing system (as distinct from the Non-Advocate Review (NAR) 14, 17, 18, product system) 56 1, 27, 59, 64 NASA Handbook (NHB) ix, xi, 13, 14, 17, Product Breakdown Structure (PBS) 27, 55, 59, 75, 77, 30-33, 59, 61, 79,91,95,96,99, 100, 102, 112, 113, 115, 120 139 Product development process 8, 13, 20- NASA Management Instruction (NMI) ix, 22 xi, 1, 3, 13, 37, Product development teams (PDT) 22, 38, 44, 99, 10(), 115, 123, 139 25, 27, 56, 91 Nuclear safety 114, 115 Product improvement 11, 78, 103 Objective function 4, 10, 74 Product system 1, 27, 99 Objectives, of a mission, project, or Program, level of hierarchy 3 system 3, 4, 8, 11, Program/project control xi, 44, 59-61 17, 28, 37, 38, 50, 51, 56, 67-71, 74-77, 83, Program Operating Plan (POP) 25 91 Project Office of Management and Budget (OMB) level of hierarchy 3 25, 79 management (see also system Operations concept 9, 14, 17. 22, 68-70, management) xi, 73, 77, 86, 101 27, 37, 46, 55, 59, 79 Operations research 7 plan 17-19, 28-30, 48, 49 Optimization 3, 6, 7, 9, 13, 67, 72, 80, 83, termination 49 119 Project life cycle Optimum repair level analysis (ORLA) 98 NASA 13-20 Orbital Replacement Unit (ORU) 80, 87, technical aspect of 20-26 97, 98, 132-134 Protoflight 106 Outer Space Treaty 115 Prototype 13 Parametric cost estimation 80-83 Quality Partitioning—see interfaces of systems engineering process 64, 65, Payload 17, 20, 132 75 classification 38, 93, 95, 105, 123 as a facet of effectiveness 84, 85 nuclear—see nuclear safety Quality Assurance (QA) 6, 29, 49, 52, 91, PERT 33, 39, 42 94-96, 119 Physical Configuration Audit (PCA) 48, Quality Function Deployment (QFD) 7 58, 96 Queueing theory 7 Precedence diagram 34 Red team 49 Preliminary Design Review (PDR) 17, 18, Reference mission 9, 69 21, 22, 25, Reliability 6, 22, 80, 91-95, 98, 100 45, 52,56,58,59,96, 106, 115 block diagrams 94