5.0 Human Performance and Error
5. HUMAN PERFORMANCE AND ERROR
In addition to the physical capabilities and limitations discussed in Chapter 4, system design must accommodate human visual, auditory, sensorimotor, and cognitive capabilities and limitations to support effective and efficient human performance. These capabilities and limitations vary with age, sex, and fatigue, and are further impacted by the environmental conditions of spaceflight. Determination of anticipated levels of crew capability and anticipated levels of task demands can be made through a detailed task analysis.
- Visual capabilities include visual acuity, spatial contrast sensitivity, visual accommodation, field of regard, color discrimination, stereoscopic depth perception, and temporal contrast sensitivity. These capabilities apply to displays and controls, as well as other visual observations, such as out-the-window viewing.
- Auditory capabilities include, at a minimum, absolute threshold of hearing, auditory localization, and speech intelligibility. Audio-communications can play an essential role in completing mission operations.
- Sensorimotor capabilities include balance, locomotion, eye-hand coordination, visual control, tactile perception, and orientation perception. Controls and displays can provide information to the operator through sensorimotor perception channels.
- Cognitive capabilities include attention, memory, decision making, problem solving, logical reasoning, and spatial cognition. Accommodating cognitive capabilities is important to ensure optimal task performance, teamwork, and crew safety.
When human capabilities and limitations are considered in design, tasks can be accomplished within time and performance criteria; crew interfaces are legible and usable; cognitive workload levels are appropriate, avoiding underload and overload; and in case of inevitable overload or underload, avoiding scheduling critical tasks for that periods; crew situation awareness is sufficient to detect and respond to hazards; design-induced errors are minimized; and vehicles are controlled with ease and precision. The following human performance requirements support these goals and will provide important evidence that the human-system design supports crew safety and productivity.
For detailed discussions regarding human performance capabilities, e.g., visual perception, auditory perception, cognition, and workload, see chapter 5, Human Performance Capabilities, of the Human Integration Design Handbook (HIDH). For detailed discussions regarding the design of user interfaces, e.g., visual acquisition of displays, visual displays, layout of displays and controls, see chapter 10, Crew Interfaces, of the HIDH.
5.1 Human Performance
5.1.1 Operability
[V2 10003] The system shall provide crew interfaces that enable tasks to be performed successfully within the appropriate time limit and degree of accuracy.
[Rationale: Operability is the ability of the intended user to achieve the required or desired outcome, within the planned or required time to effect, using the system and procedures as designed. Successful task performance within the appropriate completion time is an objective measure of design effectiveness. Ineffective design may directly or indirectly impact operational timelines, crew stress and behavioral health, or safety. Design for operability is guided by function and task analysis, iterative design, and human-in-the-loop (HITL) evaluation. The task analysis will define tasks where timing is critical, as well as the performance outcome parameters.]
5.1.2 Usability
[V2 10001] The system shall provide crew interfaces that result in a minimum average satisfaction score of 85 or higher of the NASA Modified System Usability Scale (NMSUS).
[Rationale: Systems that are usable are acceptable and operable by the intended user for performing expected tasks. If a design does not meet the users’ needs, expectations, intuitions, or capabilities, and as a result causes frustration or confusion, the design is not effective. Ineffective design may directly or indirectly impact operational timelines, crew stress and behavioral health, or safety. Errors may occur, tasks may take longer to complete, or users may abandon, work around, or choose not to perform the tasks. Design for perceived crew acceptability is guided by task analysis and iterative prototyping and evaluation. Human-in-theloop acceptability evaluation is conducted early and throughout system design to gather user feedback on design effectiveness, efficiency, and potential design-induced errors to influence design improvements and measure design progress. While many tools exist for assessing user acceptability, NASA recommends use of the NMSUS with participants that have been trained on the tasks and system design to a pre-set performance criterion throughout design development. NMSUS is a reliable instrument that is short and easy to administer and is valuable for eliciting feedback on specific design elements for iterative improvement. The NMSUS scale and information on scoring can be found in Figure 5.1-1—NASA Modified System Usability Scale (NMSUS).]
Figure 5.1-1—NASA Modified System Usability Scale (NMSUS)
5.1.3 Design-Induced Error
[V2 10002] The system shall provide crew interfaces that do not exceed the maximum observed error rates listed in Table 5.1-1—Maximum Observed Design-Induced Error Rates.
[Rationale: Errors are detrimental to crew effectiveness, efficiency, acceptability, and safety. Even when recoverable or resulting in minimal impact, errors can still negatively impact crew performance in terms of productivity and satisfaction. Errors are defined as an action that does not result in the intended outcome or a failure by the crew to perform an action within the required limits of accuracy, sequence, or time which results in unwanted consequences. Designinduced errors include but are not limited to: missed or incorrect inputs or selections, display navigation errors, errors due to inadequate hardware component design, errors due to lack of system feedback to user inputs, errors due to inadequate information, errors due to design inconsistency or unfamiliar terminology, and the inability to complete a step or task. Unintentional errors that are related to human reliability (e.g., bumping a control due to fatigue) are not considered design-induced errors. It is crucial for design to be guided by an iterative, human-centered design process including task analysis, human error analysis, and human-in-the-loop (HITL) evaluations. Task analyses identify user tasks and task sequences. To ensure crew have situation awareness, detailed analysis of information needs are performed to identify the needed information is presented in the context necessary for crew to perform the correct actions at the correct time. Human error analysis identifies potential user errors at each step and the outcome or system consequence if the error is committed. Task errors that could result in a catastrophic hazard should be prevented through careful interface design and thorough evaluation. Tasks that are identified as complex, leading to critical or catastrophic hazards/events, or frequent need more rigorous developmental testing and are to be included in HITL verification testing.
For purposes of HITL testing, a scenario requiring evaluation will be defined as an activity driven by one or more related and sequential procedures. The procedure consists of a series of task steps, where a task step will be defined as a single instruction to the test subject, as is typical of current spaceflight procedures. Participants will maintain task completion times commensurate with the performance requirements.
- If any errors classified as having the potential of leading to a catastrophic outcome occur, the root cause of the error must be identified, mitigated satisfactorily (approved by NASA), and a re-test of the task performed to prove that the error has been eliminated.
- The percentage of errors (erroneous task steps) for each user is calculated by dividing the number of erroneous task steps and incomplete task steps by the total number of task steps and multiplying the result by 100.
- The percentage of users committing each error (erroneous task step) is calculated by dividing the number of users committing each erroneous task step by the total number of users and multiplying the result by 100.]
Table 5.1-1—Maximum Observed Design-Induced Error Rates
5.1.4 Cognitive Workload
[V2 5007] The system shall provide crew interfaces that result in Bedford Workload Scale ratings of 3 or less for nominal tasks and 6 or less for tasks performed under degraded system conditions.
[Rationale: Cognitive workload is the users’ perceived level of mental effort that is influenced by many factors, particularly task load and task design. Acceptability of cognitive workload level for critical or frequent operations/task sequences should be measured using a validated workload scale such as the Bedford Workload Scale. On the Bedford scale, acceptable level of workload is a rating of 3 or less for critical, novel, or frequent tasks. For tasks that are performed during cases of degraded system performance where there is a failure state that directly impacts the tasks being performed, ratings of 6 or less are allowed. The workload measurement enables standardized assessment of whether temporal, spatial, cognitive, perceptual, and physical aspects of tasks and the crew interfaces for these tasks are designed and implemented to support each other. Application of workload measurement for crew interface and task designs in conjunction with other performance measures, such as usability and designinduced error rates, helps assure safe, successful, and efficient system operations by the crew. Workload levels may be modulated (raised or lowered) through the combination of userinterface design and task design (e.g., task simplification, subtask combination and sequencing, and the distribution of tasks among multiple crewmembers and between crew and automation).]
5.1.5 Physical Workload
[V2 10200] The system shall provide crew interfaces that result in a Borg-CR10 rating of perceived exertion (RPE) of 4 (somewhat strong) or less.
[Rationale: The design of interfaces for physical tasks is important because of the risks of musculoskeletal injuries and disorders that arise out of mismatch between a crew capability and the physical demands of their task. Minimizing these risks is especially important for spaceflight where schedule and specialized crew training for unique tasks and environments cannot easily be adjusted. Attention should be paid to design of tasks that are high effort, extended duration, or involve repetitive motions that can result in over-exertion or fatigue, such as suit donning or doffing and EVAs. To ensure task and interface designs result in acceptable levels of physical workload, human-in-the-loop evaluation should be conducted early and throughout system design to gather user feedback on perceived exertion and task performance. Verification testing using the Borg RPE scale includes tasks that are suspected to be physically demanding and not those that are primarily cognitive tasks (unless they also include a significant physical aspect). The Borg RPE is a useful tool for measuring an individual’s effort and exertion, one with decades of use and validation that was designed to scale with workload intensity and heart rate during the execution of physical work (Borg, Gunnar. “Psychophysical scaling with applications in physical work and the perception of exertion.” Scandinavian journal of work, environment & health (1990): 55-58.).]
5.1.6 Situation Awareness
[V2 5006] Systems shall provide the Situation Awareness (SA) necessary for efficient and effective task performance and provide the means to recover SA, if lost, for anticipated levels of crewmember capability and anticipated levels of task demands.
[Rationale: SA refers to the process and outcome of understanding the current context and environment, evaluating that situation with respect to current goals, and projecting how that situation will evolve in the future. Lack of SA has been associated with numerous accidents and incorrect decisions by flight crews in commercial aviation and in ground-based simulation of spacecraft operations. To maximize SA and optimize operational accuracy and efficiency, designers are to perform a detailed information requirements analysis of all onboard operations and ensure that the crew-vehicle interfaces provide all required information to perform the operation. A useful and effective system design supports the crewmember’s ability to rapidly and accurately assess the current situation. Occasional loss of SA is expected in an operational setting where crew may have to unexpectedly move from task to task as events demand. It is important that the system design provides the necessary information, cues, or indicators to help the crewmember easily recover SA. Determination of anticipated levels of crew capability and anticipated levels of task demands is based on a detailed task analysis. SA can be directly assessed with a variety of validated industry tools (e.g., Situation Awareness Global Assessment Technique (SAGAT), Situation Awareness Rating Technique (SART)) and indirectly assessed through the measurement of usability, cognitive workload and design-induced error.]
5.1.7 Legibility
[V2 5051] The system shall provide crew interfaces that are legible under expected operating conditions.
[Rationale: Information presented to crew must be legible in all planned operating locations and conditions. Legibility includes both text elements and meaningful graphic elements such as symbols, icons, and maps and is important for the timely and accurate processing of information. Legibility depends upon display properties such as resolution and contrast, text properties such as font color, size, and contrast with background color and texture, visual capabilities of the operator, worksite illumination, and glare. In addition, the possible viewing angles, distance of the operator, the presence of a helmet or visor in a suit, and expected environmental operating conditions during use (e.g., high acceleration and vibration) need to be considered.]
5.1.8 Controllability and Maneuverability During Manual Control (Handling Qualities – Level 1)
[V2 10004] The spacecraft shall exhibit Level 1 handling qualities (Handling Qualities Rating (HQR) 1, 2 and 3), as defined by the Cooper-Harper Rating Scale, during manual control of the spacecraft’s flight path and attitude when manual control is the primary control mode or automated control is non-operational.
[Rationale: Handling qualities are defined as “those qualities or characteristics of [dynamic vehicle control] that govern the ease and precision with which a [user] is able to perform the tasks required” (Cooper and Harper, 1969). The Cooper-Harper rating scale is a standard method for measuring handling qualities. See Figure 5.1-2—Cooper-Harper Handling Qualities Rating Scale. Level 1 handling qualities are the accepted standard for manual control of flight path and attitude in military aircraft. Level 1 handling qualities will allow the crew to effectively control the spacecraft when necessary for mission completion or to prevent a catastrophic event. “Nonoperational” is defined as automated control system failed or manually disabled. Note that there are numerous vehicle performance factors that can significantly affect handling qualities, with two well documented factors including control and display latency. If the inception of a controller and the pilot’s observation of its system response occur with a significant latency, it is difficult to identify whether the operation of the control had the intended effect within sufficient time constraints, leading to adverse handling qualities and possibilities of adverse outcomes such as pilot-induced oscillation. Control latencies of less than 100ms for high gain tasks, and less than 200ms for low gain tasks, are associated with Level 1 Handling Qualities. High gain tasks are those for which temporal demands and task urgency are key drivers of task success or failure, while low gain tasks are those which place lower temporal demands on the crew (Reference MIL-STD-1797A Notice 3 – Flying Qualities of Piloted Aircraft, 2004). Display related information used in vehicle piloting tasks, including translation and rotation, also affect handling qualities. Display system latencies in the range of 50-100ms are associated with Level 1 handling qualities. Display system latency is defined as the time delay between the change in vehicle dynamics and the representation of associated new information on the display (total time from sensors to data presentation on the display). This is required to ensure piloting display elements that translate or rotate will do so smoothly without distracting or objectionable jitter, jerkiness, or ratcheting effects (Reference Funk, et al., 1993, Primary display latency criteria based on flying qualities and performance data). Additionally, if latencies (either control or display) exist that are not consistent and are highly variable during the execution of a task, it can be extremely difficult for crew to mitigate or adapt to the effects of such delays as the inconsistency makes adaptation difficult, resulting in poor handling qualities. A handling quality-related task is defined as the manual control capability that is being rated with the Cooper-Harper Rating Scale. Each task within a scenario is rated separately and has to meet the appropriate Level 1 handling qualities (handling quality ratings of 1, 2, or 3). Reference NASATN-D-5153, The use of pilot rating in the evaluation of aircraft handling qualities, for the Cooper-Harper Rating Scale.]
Figure 5.1-2—Cooper-Harper Handling Qualities Rating Scale
5.1.9 Controllability and Maneuverability During Manual Control with Deficiencies (Handling Qualities – Level 2)
[V2 5052] The system shall exhibit Level 2 (HQR 4-6) or better handling qualities during manual control in all other scenarios not specified in [V2 10004] Handling Qualities – Level 1.
[Rationale: Level 2 handling qualities are acceptable when failures of the flight control systems and/or displays and controls, prevent them from functioning as designed for nominal operations. If a failure exists that only affects automation, then Level 1 handling qualities are still expected. Select manual control scenarios that have to meet Level 1 handling qualities will be defined and scoped with applicable program/project agreement. A scenario includes one or more handling quality-related vehicle control tasks performed during a flight phase under specified conditions. Reference NASA-TN-D-5153, The use of pilot rating in the evaluation of aircraft handling qualities, for the Cooper-Harper Rating Scale as captured in Figure 5.1-2—Cooper-Harper Handling Qualities Rating Scale.]
5.2 Human Error
5.2.1 Controls for Human Error
[V2 5053] The system shall control for human error according to the following precedence:
- Design the system to prevent human error in the operation and control of the system.
- Design the system to reduce the likelihood of human error and provide the capability for the human to detect and correct or recover from the error.
- Design the system to limit the negative effects of the error.
[Rationale: Human error is an action that is not intended or desired by the operator or a failure on the part of the operator to perform a prescribed action within specified limits of accuracy, sequence, or time that fails to produce the expected result and has led or has the potential to lead to an unwanted consequence. Potential human errors include inadvertent/unintended operator actions, failure to perform an action, performing a wrong action, performing an action incorrectly, and performing an action with incorrect timing. The intent of this requirement is to identify potential human errors that can cause a catastrophic event by predictive human error analysis, determine the appropriate level of tolerance via integrated human error and hazard analyses, and determine the specific controls to mitigate error that could lead to a catastrophic event according to the described precedence.]
5.2.2 Protect Against Inadvertent Activation
[V2 10027] The system shall protect against inadvertent activation of controls.
[Rationale: Inadvertent crew actions and loose or uncontrolled movement of hardware could result in unintended activation of controls and cause undesired effects on the system and potentially hazardous consequences. For this requirement, a control is a device or interface that is used to send a command to the system or operate a component. Guards, covers, physical separation from other controls and requiring two operator actions are examples of methods to protect against inadvertent activation. Protections are to be provided for nominal and offnominal controls, including those used in response to failures and aborts.]
5.2.3 Error Detection and Recovery
[V2 10028] The system shall provide the capability to detect and recover from human error and inadvertent changes in system status.
[Rationale: Not all human error or inadvertent input can be prevented. When the system is unable to prevent the error, it is to provide a method that allows the system or operator to detect and recover without the error resulting in a catastrophic event. Incorrect control inputs are from unintended actions that were able to defeat the design protections and from scenarios where the operator chooses the wrong action forward due to lack of incorrect information and/or poor situation awareness. Detection methods include alerts provided within time to effect and recovery mechanisms include the ability for operators to reverse a previous command or send a new command to correctly address the hazard. Detections and recovery capabilities are to be provided for nominal and off-nominal scenarios, including responses to systems and aborts.]
