Lessons from the National Ignition Facility
By Matthew Kohut
The National Ignition Facility (NIF) is home of the world's largest laser. With 192 laser beams that can deliver more than sixty times the energy of any previous laser system, NIF represents a significant step in enabling the study of high-energy density science. The design and construction of this unique, highly complex facility posed management challenges that the project team overseeing its development could not foresee.
The final optics assemblies, shown here mounted on the lower hemisphere of the target chamber, contain special optics for beam conditioning, color conversion, and color separation. (Click image for close-up) Photo Credit: Lawrence Livermore National Laboratory
A One-of-a-Kind Facility
The Department of Energy (DOE) formally broke ground on NIF at Lawrence Livermore National Laboratories (LLNL) in May 1997. The federal government used a lean management structure for NIF, relying on a small core team of civil servants to oversee a contractor organization of hundreds. DOE chose to implement this project through one of its existing Management and Operating (M&O) contracts—broad agreements that provide the government with access to the capabilities of its national laboratories.
NIF, a ten-story building the length of three football fields, actually consists of three connected buildings: the Optics Assembly Building, the Laser and Target Area Building, and the Diagnostics Building. In the Optics Assembly Building, assembly of precision-engineered laser components takes place under Class 100 clean-room conditions. The Laser and Target Area Building houses the 192 laser beams in two identical bays. Large mirrors direct the laser beams into a target bay, where they are focused to the center of a 10-meter-diameter target chamber. The cleanliness inside the beam enclosures typically exceeds that of a semiconductor or pharmaceutical manufacturing plant.
LLNL had built several complex laser systems prior to NIF, but nothing approaching its size and scope. "When you do work planning and estimating for something where there isn't a direct comparison that provides you the basis for a parametric estimate, you go with what you understand," said Scott Samuelson, who began working on the project in the mid-1990s. "All the prior laser systems at the lab were put together one beamline at a time by trained technicians who worked for the lab. Our initial thinking assumed we'd use the in-house [workforce], and they'd put it together just like the previous systems."
The challenge proved greater than expected. "As we got closer to having to start the assembly of the beam-path infrastructure—as the building was getting finished on the inside and you could start to get a physical feel for the scale of the laser system and the components we would be assembling, and the challenge of doing that work in a clean environment—I thought, 'You know, I don't think we can do it the way the previous systems were built; I don't see how those techniques and people can successfully put this thing together.'" Samuelson, then acting as field director for the project, agreed with LLNL management that a new look at the original plan for building the laser system should be conducted.
Resetting the Baseline
In 1999, one of the largest cranes in the world lifted NIF's 10-meter-diameter target chamber into the target bay in a dramatic event attended by Secretary of Energy Bill Richardson. Shortly after the event, the project team came forward with an estimate to complete, which identified deficiencies in the existing project planning and projected schedule delays and several hundred million dollars of cost overruns. When this came to light, Secretary Richardson issued a forceful press release setting out a six-step approach for addressing NIF's cost and schedule problems. "These are project management issues, and we will get ahead of these problems and turn them around with aggressive and tighter management action from this department," Richardson wrote. This action plan led to management changes at LLNL, increased and more-focused oversight from DOE, and reviews by a Secretarial NIF Task Force as well as the General Accounting Office.
The reviews resulted in significant revisions to the project's baseline cost and schedule. "Basically what happened is the project just didn't understand how hard the job of building that ship in the bottle and keeping it as clean as it needed to be inside while we were doing it would be," Samuelson said. "The work scope was there, but the effort associated with that work scope was underestimated."
Samuelson attributed the cost-estimation errors to the lack of a model that could offer a meaningful basis for comparison: "The lab did their best estimate. We had external reviewers come in, we had independent cost estimators come in, but in the end, they could only look at what they understood; they had to take our word for the things they simply had no experience with. As a result, the estimates for the 'conventional' parts of the facility were pretty much right on, but our existing management systems never gave us a good independent estimate of the 'first-of-a-kind' portion of the project. That's where you end up when you're doing something that nobody's ever done before. The real lesson to me is that we proved one more time that the old RAND study on mega projects was right: on these highly complex, one-of-a-kind projects, you make your best estimate of what it's going to cost and then double it. Don't plan on spending the extra, just sit on it until you find out what you didn't know and understand what you really need it for."
After the Shakeup
The multiple reviews led to changes in the management approach. "It changed a lot of things," said Samuelson. "It redefined a lot of management relations. It certainly redefined the teams that were working on the project, and the way we were using the suite of project management tools and techniques that we had available to us."
Managers tightened the scope of the project. "We took the opportunity to review our project-completion criteria and scrubbed them very carefully," explained Samuelson. "We made some small changes based on input from the scientific community and incorporated them through the change-control process. The final product was included in the project execution plan so there wouldn't be any doubt what we were signed up for; it never changed after that."
The management chain on the federal side shortened. "Back when this started, you had a program manager in Washington, and you had a project manager in Washington who was part of another organization, and they had no real contractual authority to direct the people at the M&O contractor doing the work. All that came down a separate line through what was then the operations office manager to one of his assistant managers, and then eventually to the guy out in the field, which was the job I held back then. It made for a pretty difficult situation when you needed to reach the person where all those lines came together," said Samuelson.
The reviews led to the creation of the Office of the NIF Project, whose director reported to the assistant secretary for defense programs. "Anytime you do anything that's this big and this hard, I think that's what you ought to do," said Samuelson. "You need to focus on the project. You can't have other priorities. You can't have distractions from other places. The person you're reporting to must be able to make decisions and allocate resources consistent with the Department's commitment to the success of the project."
For the next two or three years, the project continued with an office director in Washington, D.C., and a deputy who served as the field director on-site at NIF. There was more formality than in the past in the use of external review groups as well as more rigor in reporting. The changes in the project management approach and tools made a difference.
"In the earlier time period, we were using an earned value management–like system, not a real earned value management system [EVMS], because a full EVMS wasn't required in those days. We probably had a little less insight into future performance based on what we had observed under that old system than we did later in the project, but we thought it was adequate at the time," Samuelson said.
He also noted the forecasting limitations of project management tools: "Earned value management is a great tool, but for better or for worse, it measures past performance against your plan. You can use that information to project what may happen in the future against that plan, but what we identified that led to the revised estimate to complete (prior to the 1999–2000 re-baseline) wouldn't have been identified until later through an EVMS. We were performing against the plan; the problem was that the plan was underscoped going forward, but what we'd done so far was not an indication of that. It was just a matter of looking at the work ahead and saying, 'It's not going to happen that way.'"
The project also made greater use of external reviews than it had in the past. "We really ran the reviews using the model the Office of Science uses. Dan Lehman has run reviews on their projects for years, and he has a pretty specific formula for how you conduct them. We would have a group of very experienced people who'd come in and turn us inside out, and tell us what they thought, ask us what we thought, and then they would leave us with their opinions," Samuelson said. "I'd say 99 percent of the time, we took care of what they told us. It was almost completely embraced, and it paid a lot of dividends over the years."
The Role of the Federal Project Director
After the director of the Office of the NIF Project retired, Samuelson and others rotated responsibilities for a time. In 2004, DOE established the federal project director (FPD) designation and began the process of certifying Samuelson to fill the new position for NIF.
One of the biggest differences having an FPD made was focusing contact between the government and its contractor. "The project organization got even tighter, and there was increased emphasis on the importance of providing direction via the contract. It made it a lot easier to keep straight what the government was officially saying to the contractor with regard to the project. The contractor knew that while program managers and other senior managers in HQ made decisions and communicated directly with them, project direction implementing those decisions came out of my [FPD] office through the contract," said Samuelson. "That really helped on both sides. It's hard to overplay how important that was."
Samuelson described the FPD's role as one that shapes the reaction the system will have to events that occur. "Project management is about solving the hundred problems that come up every day. If every time something happens, you just say, 'They're normal, and we're taking care of them, and I know it's happening—thanks for keeping me informed, and let me know if any problem is getting bigger,' then people tell you about them," he said. "If you make a big deal and stir up the whole system every time you hear that something didn't happen when it was supposed to, you're going to have a hard time finding out about it, and you're going to be too busy with things that the contractor should just be handling to pay attention to the stuff you need to be paying attention to. That's especially true with something the size of NIF."
NIF's federal employees focused on creating a strong team with the M&O office. "For all intents and purposes, everything that was going on there was transparent," said Samuelson. "We had access to all their information and management systems, and we defined very carefully what the thresholds and boundaries were on responsibilities." He emphasized the importance of developing a certain level of trust with the contractor workforce: "I don't know any way to do that other than to be here [in the field] in the middle of what's going on, being able to have the trust of the people so they feel comfortable talking to you about what's really happening and they understand that you're going to be focused on helping find a solution and moving things forward."
From Samuelson's perspective, the oversight role revolves around access to information and the ability to evaluate it. "I got lots of numbers and pieces of paper and statements of whether milestones were being met. That's all good. But it's developing the ability to determine whether or not what you're seeing matches what's really going on. Is the system giving you data you should believe?" he asked rhetorically. "Then when you look at that data and it matches your other observations, and you do see something going on, do you understand why that's happening?"
Most importantly, Samuelson and his team focused on eliminating roadblocks and keeping the project moving toward completion. "I think that one of the things that all of us as a group did well for the last nine years was just handle stuff: resolve the issues and make sure people were aware and informed," he said.
Crossing the Finish Line
On March 31, 2009, DOE announced that the National Nuclear Security Administration had certified the completion of the National Ignition Facility. Eighteen months later, the Project Management Institute named it the 2010 Project of the Year.
This story has been provided by the Department of Energy's Office of Engineering and Construction Management.
Write to the Author
More Articles by Matthw Kohut