6.1    Technical Planning
6.2    Requirements Management
6.3    Interface Management
6.4    Technical Risk Management
6.5    Configuration Management
6.6    Technical Data Management
6.7    Technical Assessment
6.8    Decision Analysis

The purpose of this section is to provide an overview of the Decision Analysis Process, highlighting selected tools and methodologies. Decision Analysis is a framework within which analyses of diverse types are applied to the formulation and characterization of decision alternatives that best implement the decision-maker’s priorities given the decision-maker’s state of knowledge.

The Decision Analysis Process is used in support of decision making bodies to help evaluate technical, cost, and schedule issues, alternatives, and their uncertainties. Decision models have the capacity for accepting and quantifying human subjective inputs: judgments of experts and preferences of decision makers.

The outputs from this process support the decision authority’s difficult task of deciding among competing alternatives without complete knowledge; therefore, it is critical to understand and document the assumptions and limitation of any tool or methodology and integrate them with other factors when deciding among viable options.

6.8.1 Process Description

A typical process flow diagram is provided in Figure 6.8-1, including inputs, activities, and outputs. The first step in the process is understanding the decision to be made in the context of the system/mission. Understanding the decision needed requires knowledge of the intended outcome in terms of technical performance, cost, and schedule. For an issue that follows the decision analysis process, the definition of the decision criteria or the measures that are important to characterize the options for making a decision should be the next step in the process. With this defined, a set of alternative solutions can be defined for evaluation. These solutions should cover the full decision space as defined by the understanding of the decision and definition of the decision criteria. The need for specific decision analysis tools (defined in Section 6.8.3 in the NASA Expanded Guidance for Systems Engineering at https://nen.nasa.gov/web/se/doc-repository) can then be determined and employed to support the formulation of a solution. Following completion of the analysis, a description of how each alternative compares with the decision criteria can be captured for submission to the decision-making body or authority. A recommendation is typically provided from the decision analysis, but is not always required depending on the discretion of the decision-making body. A decision analysis report should be generated including: decision to be made, decision criteria, alternatives, evaluation methods, evaluation process and results, recommendation, and final decision.

Decision analysis covers a wide range of timeframes. Complex, strategic decisions may require weeks or months to fully assess all alternatives and potential outcomes. Decisions can also be made in hours or in a few days, especially for smaller projects or activities. Decisions are also made in emergency situations. Under such conditions, process steps, procedures, and meetings may be combined. In these cases, the focus of the systems engineer is on obtaining accurate decisions quickly. Once the decision is made, the report can be generated. The report is usually generated in an ongoing fashion during the decision analysis process. However, for quick or emergency decisions, the report information may be captured after the decision has been made.

Not all decisions require the same amount of analysis effort. The level and rigor required in a specific situation depend essentially on how clear-cut the decision is. If there is enough uncertainty in the alternatives’ performance that the decision might change if that uncertainty were to be reduced, then consideration needs to be given to reducing that uncertainty. A robust decision is one that is based on sufficient technical evidence and characterization of uncertainties to determine that the selected alternative best reflects decision-maker preferences and values given the state of knowledge at the time of the decision. This is suggested in Figure 6.8-2 below.

Note that in Figure 6.8-2, the phrase “net beneficial” in the decision node “Net beneficial to reduce uncertainty?” is meant to imply consideration of all factors, including whether the project can afford any schedule slip that might be caused by additional information collection and additional analysis.

6.8.1.1 Inputs

The technical, cost, and schedule inputs need to be comprehensively understood as part of the general decision definition. Based on this understanding, decision making can be addressed from a simple meeting to a formal analytical analysis. As illustrated in Figure 6.8-2, many decisions do not require extensive analysis and can be readily made with clear input from the responsible engineering and programmatic disciplines. Complex decisions may require more formal decision analysis when contributing factors have complicated or not well defined relationships. Due to this complexity, formal decision analysis has the potential to consume significant resources and time. Typically, its application to a specific decision is warranted only when some of the following conditions are met:

• Complexity: The actual ramifications of alternatives are difficult to understand without detailed analysis;
• Uncertainty: Uncertainty in key inputs creates substantial uncertainty in the ranking of alternatives and points to risks that may need to be managed;
• Multiple Attributes: Greater numbers of attributes cause a greater need for formal analysis; and
• Diversity of Stakeholders: Extra attention is warranted to clarify objectives and formulate TPMs when the set of stakeholders reflects a diversity of values, preferences, and perspectives.

Satisfaction of all of these conditions is not a requirement for initiating decision analysis. The point is, rather, that the need for decision analysis increases as a function of the above conditions. In addition, often these decisions have the potential to result in high stakes impacts to cost, safety, or mission success criteria, which should be identified and addressed in the process. When the Decision Analysis Process is triggered, the following are inputs:

• Decision need, identified alternatives, issues, or problems and supporting data: This information would come from all technical, cost, and schedule management processes. It may also include high-level objectives and constraints (from the program/project).
• Analysis support requests: Requests will arise from the technical, cost, and schedule assessment processes.

6.8.1.2 Process Activities

For the Decision Analysis Process, the following activities are typically performed.

It is important to understand the decision needed in the context of the mission and system, which requires knowledge of the intended outcome in terms of technical performance, cost, and schedule. A part of this understanding is the definition of the decision criteria, or the measures that are important to characterize the options for making a decision. The specific decision-making body, whether the program/project manager, chief engineer, line management, or control board should also be well defined. Based on this understanding, then the specific approach to decision-making can be defined.

Decisions are based on facts, qualitative and quantitative data, engineering judgment, and open communications to facilitate the flow of information throughout the hierarchy of forums where technical analyses and evaluations are presented and assessed and where decisions are made. The extent of technical analysis and evaluation required should be commensurate with the consequences of the issue requiring a decision. The work required to conduct a formal evaluation is significant and applicability should be based on the nature of the problem to be resolved. Guidelines for use can be determined by the magnitude of the possible consequences of the decision to be made.

6.8.1.2.1 Define the Criteria for Evaluating Alternative Solutions

This step includes identifying the following:

• The types of criteria to consider, such as customer expectations and requirements, technology limitations, environmental impact, safety, risks, total ownership and life cycle costs, and schedule impact;
• The acceptable range and scale of the criteria; and
• The rank of each criterion by its importance.

Decision criteria are requirements for individually assessing the options and alternatives being considered. Typical decision criteria include cost, schedule, risk, safety, mission success, and supportability. However, considerations should also include technical criteria specific to the decision being made. Criteria should be objective and measurable. Criteria should also permit differentiating among options or alternatives. Some criteria may not be meaningful to a decision; however, they should be documented as having been considered. Criteria may be mandatory (i.e., “shall have”) or enhancing. An option that does not meet mandatory criteria should be disregarded. For complex decisions, criteria can be grouped into categories or objectives.

6.8.1.2.2 Identify Alternative Solutions to Address Decision Issues

With the decision need well understood, alternatives can be identified that fit the mission and system context. There may be several alternatives that could potentially satisfy the decision criteria. Alternatives can be found from design options, operational options, cost options, and/or schedule options.

Almost every decision will have options to choose from. These options are often fairly clear within the mission and system context once the decision need is understood. In cases where the approach has uncertainty, there are several methods to help generate various options. Brainstorming decision options with those knowledgeable of the context and decision can provide a good list of candidate alternatives. A literature search of related systems and approaches to identify options may also provide some possible options. All possible options should be considered. This can get unwieldy if a large number of variations is possible. A “trade tree” (discussed later) is an excellent way to prune the set of variations before extensive analysis is undertaken, and to convey to other stakeholders the basis for that pruning.

A good understanding of decision need and criteria will include the definition of primary and secondary factors. Options should be focused on primary factors in the decision as defined by the decision criteria. Non-primary factors (i.e., secondary, tertiary) can be included in evaluations but should not, in general, define separate alternatives. This will require some engineering judgment that is based on the mission and system context as well as the identified decision criteria. Some options may quickly drop out of consideration as the analysis is conducted. It is important to document the fact that these options were considered. A few decisions might only have one option. It is a best practice to document a decision matrix for a major decision even if only one alternative is determined to be viable. (Sometimes doing nothing or not making a decision is an option.)

6.8.1.2.3 Select Evaluation Methods and Tools

Based on the decision to be made, various approaches can be taken to evaluate identified alternatives. These can range from simple discussion meetings with contributing and affected stakeholders to more formal evaluation methods. In selecting the approach, the mission and system context should be kept in mind and the complexity of the decision analysis should fit the complexity of the mission, system, and corresponding decision.

Evaluation methods and tools/techniques to be used should be selected based on the purpose for analyzing a decision and on the availability of the information used to support the method and/or tool. Typical evaluation methods include: simulations; weighted trade-off matrices; engineering, manufacturing, cost, and technical opportunity trade studies; surveys; human-in-the-loop testing; extrapolations based on field experience and prototypes; user review and comment; and testing. Section 6.8.2 provides several options.

6.8.1.2.4 Evaluate Alternative Solutions with the Established Criteria and Selected Methods

The performance of each alternative with respect to each chosen performance measure is evaluated. In all but the simplest cases, some consideration of uncertainty is warranted. Uncertainty matters in a particular analysis only if there is a non-zero probability that uncertainty reduction could alter the ranking of alternatives. If this condition is obtained, then it is necessary to consider the value of reducing that uncertainty, and act accordingly.

Regardless of the methods or tools used, results should include the following:

• Evaluation of assumptions related to evaluation criteria and of the evidence that supports the assumptions; and
• Evaluation of whether uncertainty in the values for alternative solutions affects the evaluation.

When decision criteria have different measurement bases (e.g., numbers, money, weight, dates), normalization can be used to establish a common base for mathematical operations. The process of “normalization” is making a scale so that all different kinds of criteria can be compared or added together. This can be done informally (e.g., low, medium, high), on a scale (e.g., 1-3-9), or more formally with a tool. No matter how normalization is done, the most important thing to remember is to have operational definitions of the scale. An operational definition is a repeatable, measurable number. For example, “high” could mean “a probability of 67 percent and above.” “Low” could mean “a probability of 33 percent and below.” For complex decisions, decision tools usually provide an automated way to normalize. It is important to question and understand the operational definitions for the weights and scales of the tool.

Note: Completing the decision matrix can be thought of as a default evaluation method. Completing the decision matrix is iterative. Each cell for each criterion and each option needs to be completed by the team. Use evaluation methods as needed to complete the entire decision matrix.

6.8.1.2.5 Select Recommended Solutions from the Alternatives Based on the Evaluation Criteria and Report to the Decision-Maker

Once the decision alternative evaluation is completed, recommendations should be brought back to the decision maker including an assessment of the robustness of the ranking (i.e., whether the uncertainties are such that reducing them could credibly change the ranking of the alternatives). Generally, a single alternative should be recommended. However, if the alternatives do not significantly differ, or if uncertainty reduction could credibly alter the ranking, the recommendation should include all closely ranked alternatives for a final selection by the decision-maker. In any case, the decision-maker is always free to select any alternative or ask for additional alternatives to be assessed (often with updated guidance on selection criteria). This step includes documenting the information, including assumptions and limitations of the evaluation methods used, and analysis of the uncertainty in the analysis of the alternatives’ performance that justifies the recommendations made and gives the impacts of taking the recommended course of action, including whether further uncertainty reduction would be justifiable.

The highest score (e.g., percentage, total score) is typically the option that is recommended to management. If a different option is recommended, an explanation should be provided as to why the lower score is preferred. Usually, if an alternative having a lower score is recommended, the “risks” or “disadvantages” were too great for the highest ranking alternative indicating the scoring methods did not properly rank the alternatives. Sometimes the benefits and advantages of a lower or close score outweigh the highest score. If this occurs, the decision criteria should be reevaluated, not only the weights, but the basic definitions of what is being measured for each alternative. The criteria should be updated, with concurrence from the decision-maker, to more correctly reflect the suitability of each alternative.

6.8.1.2.6 Report Analysis Results

These results are reported to the appropriate stakeholders with recommendations, impacts, and corrective actions

6.8.1.2.7 Capture Work Products

These work products may include the decision analysis guidelines, strategy, and procedures that were used; analysis/evaluation approach; criteria, methods, and tools used; analysis/evaluation assumptions made in arriving at recommendations; uncertainties; sensitivities of the recommended actions or corrective actions; and lessons learned.

6.8.1.3 Outputs

6.8.1.3.1 Alternative Selection and Decision Support Recommendations and Impacts

Once the technical team recommends an alternative to a NASA decision-maker (e.g., a NASA board, forum, or panel), all decision analysis information should be documented. The team should produce a report to document all major recommendations to serve as a backup to any presentation materials used. A report in conjunction with a decision matrix provides clearly documented rationale for the presentation materials (especially for complex decisions). Decisions are typically captured in meeting minutes and should be captured in the report as well. Based on the mission and system context and the decision made, the report may be a simple white paper or a more formally formatted document. The important characteristic of the report is the content, which fully documents the decision needed, assessments done, recommendations, and decision finally made.

This report includes the following:

• mission and system context for the decision
• decision needed and intended outcomes
• decision criteria
• identified alternative solutions
• decision evaluation methods and tools employed
• assumptions, uncertainties, and sensitivities in the evaluations and recommendations
• results of all alternative evaluations
• alternative recommendations
• final decision made with rationale
• lessons learned

Typical information captured in a decision report is shown in Table 6.8-1.

6.8.2 Decision Analysis Guidance

Refer to Section 6.8.2 in the NASA Expanded Guidance for Systems Engineering at https://nen.nasa.gov/web/se/doc-repository for additional guidance on decision analysis methods supporting all SE processes and phases including:

• trade studies,
• cost-benefit analysis,
• influence diagrams,
• decision trees,
• analytic hierarchy process,
• Borda counting, and
• utility analysis,

Additional information on tools for decision making can be found in NASA Reference Publication 1358, System Engineering “Toolbox” for Design-Oriented Engineers located at https://nen.nasa.gov/web/se/doc-repository.