IV&V of Neural Networks
Computer scientists and engineers strive to make computers behave in ways that are more useful with less user intervention. Over the years computers have evolved from punch card interfaces to command lines to graphical user interfaces to make user accessibility easier. In the same way, computer behaviors have been improved so that misplacement of a comma, for example, does not cause hours or days of downtime on a shared computer system. All such improvements have hinged on the ability of the designer to anticipate user needs and problems that might occur and to create computer responses that prevent or minimize undesirable behavior. As computer systems and the problems solved by computer systems become more and more complex, designers are challenged to anticipate every set of events that must be accounted for in a given computer application.
Dealing with complexity under changing circumstances is a constant pastime of human beings. Humans have an ability to reason that is unique and far from completely understood. Artificial intelligence is a field that attempts to make computers problem solve more like humans. One area of study within artificial intelligence is that of artificial neural networks (ANNs). ANNs are computer software or hardware that attempt to model the inner workings of the human brain and consist of neurons and synapses, which are connections between neurons. Different ANN designs provide different interpretations of how to create and remove neurons and synapses as well as the relative impact that these entities have on the output solution.
Depiction of IV&V of Neural Network Methodology (Image Credit: ISR)
An ANN requires some amount of initial training, where for each input provided a human lets the ANN know whether the output emitted is correct or incorrect. The ANN adjusts its internal structure based on the input and feedback to yield better answers in the future. The idea is that, given sufficient exposure to training data and feedback, the ANN will be able to provide a correct output for any new input that was not used for training. After training, an ANN can be forced to stop adjusting its internal structure. It then becomes somewhat more like traditional software. Alternatively, it can be allowed to continue to adjust itself without human feedback to perform a kind of self improvement based on the data it receives.
ANNs are attractive for systems where detailed requirements are not well known or solutions are very complex. Spacecraft autonomy falls into this problem domain. As spacecraft work on more complex scientific tasks or fly farther from Earth where round trip communications time is prohibitive, autonomous systems like ANNs are very desirable. Developers can select an ANN and provide it a set of training data and human feedback rather than codify the results of every conceivable scenario. Safety is enhanced when the spacecraft can react successfully to unexpected situations without waiting for ground commands. Science opportunities are enhanced when the spacecraft can determine optimal pointing and filtering as data is collected rather than after the fact.
Traditionally, software assurance personnel rely on analysis of the source code of software coupled with test execution of the software to determine nonconformance to requirements. Source code is the mechanism used to implement requirements and thus its examination provides an objective statement of quality. A failed test can point a developer to a defect in a particular portion of source code. It is not possible to look at the source code or circuit design of an ANN and determine what output will be produced for a given input. The structure and arrangement of the neurons and synapses determine how an input is transformed to an output. Testing is still a valid technique but does not point the developer to any particular defect source.
The Development of Methodologies for IV&V Neural Networks initiative (called IVVNN) was initiated at the IV&V Facility in 2002 to address the need to provide assurance that ANNs for a given mission will not jeopardize the mission but will instead meet mission requirements. This initiative started at the ground level, with the contractor, the Institute for Scientific Research (ISR) with help from West Virginia University (WVU), researching fundamental ideas of ANNs to understand their inner workings and to see how the current state of practice of IV&V applied. This work was followed up by selecting the best known techniques for assuring ANNs and developing some new techniques where holes existed. These techniques were refined and meshed into the form of a guidance document that maps to the IEEE Std. 1012-1998, from which the IV&V Facility has tailored its tasking definitions. The result is a set of background knowledge related to ANNs (and to some degree, autonomous systems in general) coupled with techniques for software assurance practitioners to use to determine if a given ANN implementation meets its requirements.
The methodology developed by IVVNN was tested by ISR on the F-15 Intelligent Flight Control System (IFCS) project from Dryden Flight Research Center (DFRC). IFCS was an experiment using ANNs as part of the F-15 control system to improve aircraft safety by compensating for failures in the aircraft’s control surfaces. IFCS used two different ANN designs in two generations of development, permitting ISR to validate the IVVNN methodology on both designs. After the initial IVVNN guidance was developed and tested with the IFCS, feedback was sought from the IV&V Facility, WVU, DFRC, Ames Research Center, and the Air Force Research Lab. Feedback was incorporated and further refinements made to the guidance.
The detailed methodology was published in book form. The IEEE Std. 1012-1998 mapped guidance is in the process of publication. Online training materials were developed based on the IVVNN research to provide an introduction to the concepts needed to provide assurance of ANNs. This material is available now and is being transitioned to NASA’s online learning system. Development of tools in support of this methodology is a desire for the future to streamline the assurance of ANNs.
IVVNN paved the way for more confident use of ANNs in safety critical systems in NASA. The methodology and guidance are not limited to use by the IV&V Facility but are applicable to assurance personnel throughout NASA and anywhere else ANNs are used.
Written by Markland Benson