NASA Independent Verification and Validation Program Director Greg Blaney discusses the impact of IV&V on high-profile NASA missions.

Greg Blaney: It’s just instilled in our fiber to innovate and do excellent work because those astronauts, when they get on that vehicle, they are trusting us that the software is going to work.

We get to do independent verification and validation on NASA’s most critical missions on their most critical software.

We bring a lot of value to the projects. By finding those errors early we reduce their cost and their schedule impacts.

Deana Nunley (Host): Welcome back to Small Steps, Giant Leaps, a NASA APPEL Knowledge Services podcast that taps into project experiences to share best practices, lessons learned and novel ideas.

I’m Deana Nunley.

Our focus today is on independent verification and validation. NASA’s IV&V Program at the recently renamed Katherine Johnson Independent Verification and Validation Facility contributes significantly to the safety and success of NASA’s highest-profile missions by assuring software performs correctly. Greg Blaney is the Director of the IV& V Program, and is our guest. Greg, thank you for joining us on the podcast.

Blaney: My pleasure.

Host: Could you give us a brief overview of NASA’s IV&V program?

Blaney: Sure. Back after the Challenger tragedy, there was a Rogers Commission report, and out of that report came a recommendation that an independent entity be created to make sure that NASA’s safety and mission-critical software would work correctly. Even though software was not a problem with the Challenger accident, software was a growing entity at that time, and it was activities that could cause issues. So that’s how the IV&V Program was originated. So we’ve been in business for over 25 years. Our job is not to develop the software but to assess the developer’s artifacts throughout the entire life cycle of their development activity, just assuring that it’s done correctly. We’ve been doing that for 25 years. There’s a multitude of values that come out of that. If we find an error or a problem in the software early in the life cycle, it is much cheaper for the developer to fix it then than at the end. Right now, we’re finding critical errors in people’s software in phase with their development activities about 96 percent of the time, and they accept those findings about 95 or 96 percent of the time. My team is actually doing a fantastic job. We’re running at about a 99 percent customer satisfaction. So that’s what we do, and we’ve been doing it for about 25 years.

Host: So in July 2019, NASA renamed the IV&V facility in honor of the storied agency icon, Katherine Johnson, whose accomplishments were celebrated in the blockbuster movie, Hidden Figures. Have you had an opportunity to spend time with her?

Blaney: Oh, my. Well, my wife and I, when the movie came out, we watched the movie four or five times. We showed it to our parents and friends. We love the movie, and then when Congress renamed our facility, not the program, but the facility that we’re located here in West Virginia, they renamed that facility after Katherine Johnson. We were so excited because she is so inspirational. She was given such a gift with her brain and her mind and her thinking. The way that she used that to benefit, as NASA says, all of humankind, in helping people get to the Moon, it’s such an inspiration to all of America. And to have our facility named after her, it was such a privilege. It motivated the whole program, and we’re still excited that her name is on our building. So when we had the event, two of her daughters came, Katherine Moore and Joylette Hylick. Now when you think of daughters, you think of young people. Well, they’re not young, because Mrs. Johnson is 101.

But they came, and such gracious, humble, pleasant people. We just had a great time. My wife and I had the opportunity to take them out to dinner, and we spent three hours just talking about their mother, their life, and just families and things in general. Based on that relationship that we had actually established with the daughters, my wife and I were in D.C., and they invited us down to Virginia to meet her mother. So we went down, and we were in her apartment. I took her a lot of things from NASA, a picture book that we created from the renaming event. My wife and I spent a couple hours with her. I held her hand, and I told her how proud we were to have our facility named after her. She looked at me, and said, “I’m just doing my job.” I said, “Oh, Mrs. Johnson” – I almost wanted to call her Katherine, because I felt like she was my mother or grandmother. I said, “Mrs. Johnson, but you used your gifts so well, and in such a positive way, that you’ve inspired the entire world.” All she did was smile and look at me, and said, “Thank you.” Just being around such a gracious, humble, intelligent person, I’ll never forget that moment. It was awesome.

Host: That does sound awesome. Just for her to be able to spend some time with you, as the director of the program, and now that this facility has been renamed for her is so special. Could you tell us a little bit more about the facility? For anyone who hasn’t actually been to West Virginia and seen the facility, could you describe it for us?

Blaney: Congress allocated money, funds, to NASA to build the building in 1991. It opened its doors in ’93. It’s just a long, two-story building. The upstairs floor basically has people in it. We don’t have a lot of hardware because we’re software-focused. The downstairs was a raised-floor, pristine computer area, which we utilize. We also actually lease space in a building next to us where people sit, because we’ve grown so much. But it was the first building in the technology park located here in Fairmont. Because of our presence here, a lot of other buildings have been built around us – the West Virginia High Technology Park, the research facility up on the hill occupied by NOAA. We actually have NOAA tenants in our basement that we rent space out to. So it’s just one building that we’ve occupied for these last 25 years.

Host: When we talk about the IV&V Program, what do you think makes it unique?

Blaney: What makes it unique is that we are a rather small organization. We are NASA’s only IV&V agent. We get to do independent verification and validation on NASA’s most critical missions on their most critical software, created by many different developers. We’re talking Lockheed Martin, the Boeings, the APL, JPL, so we get to see the good, and the better, of all developers. We learn from them. They learn from us. So we get to see and work with all of NASA’s greatest missions. That’s what makes us unique, every launch, James Webb, Hubble, the shuttle, we worked on the shuttle for years – 20 years, the space station, the Mars Rovers. We’ve touched and had a contribution to each one of those. So when they are successful, we are just thrilled. Having this expertise, we have people here that have had years and years of experience developing software for spacecraft, testing software for spacecraft. Doing analysis, we have people that are developing tools and techniques. We are just a very specialized unit sitting out here in West Virginia. That’s what makes us unique.

Host: Could you explain the criteria for determining your IV&V team’s involvement in NASA missions, programs and projects?

Blaney: Sure. That has changed over the years. When we first started, there was no criteria. It was kind of subjective, but we realized that we couldn’t perform independent verification and validation on all of NASA’s software, so we had to limit it. So we actually have written in our NASA procedure requirements documents that we do IV&V on all category 1, which are human-rated missions, where humans ride on the shuttle and the station and some of the new missions, and then category 2 with a software class of A or B, which means they are expensive or very important to the agency. So within those, the most important missions within NASA, we only look at the most critical capabilities of that software. So we’re very specialized, and we’re very limited into what we look at when we use our budget to actually do that. So, it’s very precise on which missions we do IV&V for.

Host: What is the importance of the independent part of IV&V?

Blaney: Oh, it’s a great question. So, I’m going to talk about two aspects. First of all, being independent which means technically, managerially and financially independent, that allows my team to go in to a developer and if a developer is following all their processes, and they’re doing really great work in a certain area, we don’t really pay a lot of attention to that, but if we see weaknesses in an area like writing requirements or writing test scripts, we can delve into those areas, because we are not driven by their budget. We’re not driven by their schedules or their milestones. Our only goal is their success. So we want to make sure that they’re doing everything they can do to make that software reliable, safe and secure. Now the other piece of it, with us being independent, because we are paid for by the agency, so we don’t charge the projects, typically, for that, but we go in and we look at – there’s three questions.

We look at will the software do what it is supposed to do, but then we also bring these other aspects in. Will it not do what it’s not supposed to do, and will it respond appropriately under adverse conditions? So when we see something that is really critical, we can throw independent testing at it. We can put it on our software-only simulator, and we can throw all kind of errors at it and make sure that it’s really going to work. We will spend extra time making sure those critical aspects work.

So, that independence piece is so critical to have that independent agent come in. We’re not just a process checker or check the box. “You’re doing this or doing that right.” We actually get down and do software engineering and analysis on the actual code and the operation of that code. So independence is very important.

Host: How would you rate the value added when IV&V program expertise is applied to NASA missions?

Blaney: Well, the value comes in a multitude of flavors. Industry tries to assign a return on investment value. They have done that for IV&V in the industry, the medical field, the automotive industry, various people. Typically, they say that the return on value is 10 dollars for every dollar spent on IV&V. For NASA, I abandoned the effort to actually assign a return on investment figure to it, because we are so unique. We’re doing IV&V on one-of-a-kind missions. You can’t really measure the difference between having IV&V on a mission and not having one. But I would say we bring a lot of value to the projects. By finding those errors early we reduce their cost and their schedule impacts because they can fix them cheaper and faster in the early phases of the life cycle. We bring knowledge to the table when we find these errors in their software. They see how they might be able to change how they do work, so that they don’t actually create those errors. There’s a multitude of benefits, and then we learn from it as well so that we can pass on those lessons learned to other developers as well.

Host: Are there any specific examples that you might want to share with us, some way that you’ve seen that process work, and it’s really helped a program or a project?

Blaney: Sure, there’s a multitude of examples. One, years ago, we were doing an activity called static code analysis on a customer’s software. We were finding these coding errors that they were making over and over and over again. They came to us, and they said, “What are you – why are we making these errors?” We told them, “Well, we’re doing static code analysis. We have these automated tools. You could do this yourself.” Now, we have a large array of static code analysis tools, which they probably couldn’t afford to buy all of them, but we said, “The primary ones, you could buy one or two of those and do this yourself, so that you don’t produce these errors.” So they did that, and they’ve been doing it for the last 15 years.

We had some NASA missions recently where there were three separate projects, and they all had to interface with each other eventually, but they were building their systems separately at the time, and they had what they called unique identifiers for data that the software was using. Each project was assigning a different ID code to that same piece of data. We said, “You really shouldn’t do that, because it’s really going to cause confusion when you get down the road and you integrate your systems.” They said, “Oh, we’ll create some kind of spreadsheet or something to where they’ll do the translation.” I actually spent probably a year convincing them not to do that. And it has eliminated and reduced confusion. Now, how do you measure that? I don’t know, but I think they finally realized that it was going to be much better for them to have a standard language between these unique identifiers.

Host: Really interesting. How does innovation factor into the IV&V Program?

Blaney: In software, innovation is critical. Software is changing so rapidly, and various entities out there in the industry that are developing software, they have different styles, techniques. Some people are going to the agile development approach. Some are using different languages. Some are starting to use model-based systems engineering. Cybersecurity is becoming a very critical element in software. So here at the IV&V Program, if you think about software, it’s easier to find young people that are interested in the software area, so innovation is natural. Seems like it’s natural for them. I will say I probably struggle more in holding them back, trying to not let them go off and automatically write their own scripts and do their own tools. So we have to control that a little bit and make sure that it’s going to work before we abandon an old proven approach, but we are constantly innovating.

We are leading the agency in model-based software assurance. We have expertise in cybersecurity. We have software-only simulation, which a lot of projects, they have to wait until they actually build a system and put the hardware in place to actually do testing. We can actually simulate that hardware in software, create that much cheaper, create multiple versions of that that people can use for testing or training or whatever. So we’re innovating constantly. We had a team that had to adjust our approaches, our IV&V approaches to the agile development approach. We did that very successfully. We’ve written some papers. It’s out there in the world and a lot of people, a lot of other agencies are coming to us, and asking us, “How did you do that? Can you help us adapt our processes to the same way you did?”

Innovation is one of our core values here at the program. It’s expected. It’s encouraged. We welcome it. And our mission actually drives us to want to do that. If you think about what NASA does, even in the NASA mission statement, it says to benefit all humankind. We all know astronauts here. My boss is an astronaut. He sends me pictures of his grandchildren. I sent him pictures of my grandchildren. We know astronauts. We’re friends with astronauts. I was on console when the Challenger accident happened. I heard all the traffic across NASA’s networks when that happened. I was on console when Columbia happened. I guess we just don’t want that to happen again.

NASA’s IV&V Program — our mission is to make sure NASA’s missions are reliable, safe and secure. And the last thing we want is for software to cause another tragedy, not just the astronauts, but their families, and everyone around them to experience what happened earlier. So it’s just instilled in our fiber to innovate and do excellent work because those astronauts, when they get on that vehicle, they are trusting us that the software is going to work. That’s what makes – I think that’s what drives us to do excellent work and innovate, because we just want to do as much as we can, and if we automate, and innovate, then we can do more for more software with the money that we get from Congress.

Host: Greg, as you’ve described this thirst for innovation, you’ve touched on the creativity and commitment of the IV&V team. Could you talk to us about the culture at IV&V and share your perspective of what makes this a high performing team?

Blaney: A lot of things drive the culture. I will say a lot of our people are from West Virginia, and they are so thrilled and appreciative that they can either stay here in West Virginia and work for NASA on these great missions that NASA is creating, or they can come back to West Virginia. Now we have folks that come in from outside, but they fit right in to that culture. We are not a large organization — 51 civil servants, 275 contractors — but we are a family, a rather large family, but we think of ourselves as a family. We are not in the Beltway. We are not mixed in with a lot of other companies. We are just out here in West Virginia, kind of by ourselves. So the people here have a tremendous work ethic. They love working here. They love working for NASA. The people, the local people in the area, when we tell them we work for NASA, they are just so excited and “Wow, you work for NASA?” So in our mission, the NASA mission in itself is so exciting. And then the mission that we have is so important. I think all that just combined just drives us to have this culture of doing the best we can and supporting each other as family members.

Host: Do you have customers and partners outside of NASA?

Blaney: Yes. And it’s growing rapidly. I have a risk in my risk database that we have to manage that growth. Other government agencies like the Department of Energy, the Department of Defense, Air Force, Missile Defense Agency, the FAA, are coming to us and asking us for our expertise. We’re actually doing work for DOE, the Department of Energy. The Missile Defense Agency came to me about a year ago, and we’ve entered into a couple of interagency agreements, and basically, they want us to help them, their agency, develop an IV&V program just like ours. They don’t want it to take 25 years like it did for us. So, we’re actually going to help them create their own IV&V program.

A couple years ago, we had New York City came to us and hired us to do systems engineering for their upgrade of their 911 system. So, yes, we have outside entities coming to us all the time wanting to learn from us and partner with us. We’re glad to do that because we’re always – we want to be a learning organization. We want to learn from them as well. So besides just other government agencies, even within NASA itself, we have a multitude of organizations within NASA coming to us and providing funding. We call it reimbursable work or project-funded work. About 25 percent of our budget is from either reimbursable or project-funded work. We could probably do a lot more if we weren’t managing our growth.

Host: I want to follow up on what you said about the 911 system in New York. So what does your team do in a situation like that? What is the kind of work that you do to assist with that?

Blaney: Well, in New York City, the deputy mayor, Cas Holloway at the time, he was the deputy mayor, and Bloomberg was the mayor. The city was procuring an upgrade to their 911 system in New York City. They didn’t have the expertise within the city employees or the city group to understand the systems engineering aspects of what the developers or their companies were doing for them. So they hired us to do systems engineering. Basically, that is to help make sure that the requirements are well written that needed to be done, and help the integration activities and make sure the testing was being done. We were kind of a translator between the city and the contractors that were doing the work for them. They appreciated our work a lot.

Host: That’s really fascinating. What do people usually find otherwise to be fascinating about the work the IV&V team is doing?

Blaney: It depends on what group you’re talking about. Our customers, I believe, are surprised with our technical expertise, and our ability to find errors in their software. They say, “Well, you don’t” – their anticipation is that we won’t understand their system. It’s surprising – I think they’re pleasantly surprised, always pleasantly surprised of the technical expertise that we can bring to the table, and our tools that we can bring to the table. Actually, a lot of times, they’re impressed of the cost, the low cost, that we can actually bring to the table. That expertise, the multitude of tools that we have here that are available to us, I think, impresses our customers, our projects that we work on.

The people in the area that live around us, that we live with on a daily basis, they are impressed with “You guys actually do that for NASA? You get to work on that? Your boss is an astronaut?” It’s fascinating that out here in West Virginia, a lot of people don’t even know that NASA is out here. People find it fascinating that we can actually work on the pristine NASA missions right here in West Virginia. My bosses, I believe, are also fascinated in that our technical skills and our dedication to the mission, and our willingness to just step in and do whatever needs to be done to make one of our customers successful.

Host: Greg, you’ve been with NASA for over 30 years. What motivates you?

Blaney: Well, there’s a lot of things. I guess NASA has been good to me. Their mission, which is to help humankind, motivates me. The people that I work for motivates me. Helping West Virginia’s economy, and creating jobs here in West Virginia motivates me. One of the primary things that motivates me is my dedication to not letting software cause another tragedy in NASA. I’ve experienced that twice, and I didn’t know astronauts back then, but I do now. I’m very close to many of them. A lot of them are not astronauts anymore. When you say astronauts, people are like, “Those are awesome people.” People don’t understand how awesome these people are. They are smart. They are capable. They’re physical, their attitudes, these are just the top, the cream of the crop, from my perspective, of the people in this country. They are willing to step on that rocket and put their life at risk trusting us. That’s what motivates me.

Host: Many thanks to Greg for joining us on the podcast. You’ll find his bio along with links to topics mentioned on the show and a transcript of today’s episode on our website at APPEL.NASA.gov/podcast.

We invite you to take a moment and subscribe to the podcast and tell your friends and colleagues about it.

As always, thanks for listening to Small Steps, Giant Leaps.