The control system for the National Ignition Facility (NIF) monitors thousands of components, such as this calorimeter, which measures the energy of the laser.

WHEN completed, the National Ignition Facility (NIF) will be, by far, the world’s largest and most energetic laser and a major international scientific resource. Designed to study the physics of matter at extreme densities, pressures, and temperatures, NIF will use 192 laser beams to compress fusion targets to conditions required for thermonuclear ignition and burn. In the process, more energy will be liberated than is used to initiate the fusion reactions. (See the box below.)
Every NIF experimental shot requires the coordination of complex laser equipment. In the process, 60,000 control points of electronic, optical, and mechanical devices—such as motorized mirrors and lenses, energy and power sensors, video cameras, laser amplifiers, pulse power, and diagnostic instruments—must be monitored and controlled. The precise orchestration of these parts will result in the propagation of 192 separate nanosecond-long bursts of light over a 1-kilometer path length. These 192 beams must arrive within 30 picoseconds of each other at the center of a target chamber 10 meters in diameter, and they must strike within 50 micrometers of their assigned spot on a target measuring less than 1 centimeter long.
Indeed, fulfilling NIF’s promise requires a large-scale computer control system as sophisticated as any in government service or private industry. Conceived and built by a team of 100 software developers, engineers, and quality control experts, NIF’s integrated computer control system (ICCS) software, now nearly 80 percent complete, will soon have about 1.4-million lines of code running on more than 750 computers. ICCS, which is operated from a main control room, fires the laser and conducts these experiments automatically.
ICCS proved itself over the past two years during the NIF Early Light campaign, which used beams from the first four completed lasers, or quad, to conduct more than 400 shots. With the first quad in operation, at least one of every type of hardware device was successfully monitored and controlled by ICCS. Among its many accomplishments, the control system demonstrated that it can use deformable mirrors to maintain the optical quality in laser beams, synchronize the beams’ arrival at their targets, and align the laser’s optical elements to ensure that beams hit their targets precisely. “NIF Early Light served as the ultimate test bed for the control system software and was crucial to our development efforts,” says Paul VanArsdall, associate project manager for ICCS.

Ralph Patterson, NIF deputy project manager for controls and information systems, says, “NIF Early Light was a challenge we placed on ourselves to acquire as much information as possible about the performance of the hardware and software.” Perhaps more importantly, physicists from both Lawrence Livermore and Los Alamos national laboratories, with collaborators from Sandia National Laboratories and the University of Rochester’s Laboratory for Laser Energetics, were able to conduct experiments on NIF, including studies of laser beam propagation in plasmas, energy delivery into targets of importance to the National Ignition Program, and the hydrodynamics of materials subjected to laser-driven shocks. ICCS managers obtained valuable feedback from the physicists to better fulfill the physicists’ experimental needs and increase NIF’s operational efficiency.
Over the next few years, as laser “bundles” of eight beams—the basic modular unit of NIF—are completed, computers and software that were fielded for the first bundle will be replicated. NIF’s independent bundle architecture simplifies the task of controlling the laser because each bundle is prepared for the upcoming shot independently. The bundles are synchronized just before shot time so that even the most complex experiments can be carried out efficiently with a short turnaround time.
VanArsdall emphasizes the importance of this concept. “With the bundle approach, we have a highly manageable way to bring additional lasers on line. We designed each bundle to be controlled by its own software segment. As a result, performance remains constant regardless of the number of bundles installed.” A more traditional approach would have resulted in a control system of overwhelming complexity because software would have to be scaled up to control all 192 laser beams simultaneously.
“Instead, we just have to deploy 24 copies of the control system,” says VanArsdall. “We’ve demonstrated the architecture during extensive commissioning shots and user experiments. Once we control one bundle, it is a straightforward task to extend controls to all. In this way, we’ve simplified our design and dramatically improved the performance of ICCS.”
The modular control system concept dovetails well with plans for NIF experiments. For example, although achieving ignition will require all 192 beams, many experiments will require fewer laser beams.

Control System’s Stiff Requirements
ICCS was designed to fire and diagnose laser shots every four hours. This requirement includes software for setting up the shot and countdown sequence; performing automatic alignment, laser beam diagnosis, and control of power conditioning and electro-optic subsystems; monitoring the status of all subsystems and components; and providing operators with graphics interfaces to display those data. ICCS also must maintain records of system performance and archive the experimental data recorded by NIF’s advanced diagnostic instruments.
“We started building the first software prototypes for NIF in 1997 with a team of about eight people,” recalls Bob Carey, lead software architect and one of the original eight developers. From that modest group, the organization grew to include many additional software developers, information technologists, systems engineers, quality control managers, and an independent testing group. Computer scientists and engineers on the team average 20 years of experience in such fields as database design, real-time controls, test engineering, graphics user interfaces, and object-oriented programming.
“Our staff is very talented and experienced,” says Larry Lagin, associate project manager for software engineering and a division leader in the Computation Directorate. Lagin points out that many of the people on the ICCS team helped develop control systems for past Livermore projects including the Atomic Vapor Laser Isotope Separation Program and the Shiva and Nova lasers. “The experience gained from previous large laser systems has been invaluable in developing an integrated, scalable, and robust system with the flexibility and automation required,” says Jerry Krammen, a computer scientist who has worked on many laser projects at the Laboratory. “We were also successful at building up the team with specialists who had appropriate skills from working at other research laboratories and industry,” says Lagin, who worked previously for a major aerospace firm and on Princeton University’s fusion energy program.
NIF was designed to be operational over a 30-year lifetime. Therefore, control software must be flexible and easy to update. “We wanted an architecture that provided an integrated control system we could maintain for the foreseeable future,” says Lagin.
ICCS’s architecture is hierarchical in nature. The two main layers are a front-end or bottom layer consisting of about 700 front-end processors (FEPs) and a supervisory or top layer of 50 powerful computers—all managed in the main NIF control room from an ensemble of 14 operator consoles. The supervisory layer includes operator-controlled graphics displays and automated controls that work with the FEPs to coordinate components in all 192 beams. Databases and common services incorporated into the supervisory layer support control system operation.
A high-performance network, with a throughput of 1 gigabit per second, interconnects these computers for passing commands, assessing bundle status, and retrieving diagnostics data. The network also carries video images of the laser beams generated by over 300 high-resolution digital cameras that serve as the eyes of the control system for monitoring and adjusting the laser alignment automatically.

These x-ray images from a NIF experiment show beam transport through a target at (a) 1.5, (b) 2.5, and (c) 3.5 nanoseconds. The integrated computer control system (ICCS) records and archives all experimental data. Operators use the ICCS’s graphics interfaces to display the results.

Microprocessors and Supervisors
The FEPs, which attach to the laser hardware, operate as real-time applications running on industrial-grade microprocessors. They are organized to support hardware in NIF’s functional systems: injection laser, beam controls, laser diagnostics, pulse power, and target diagnostics.
Different types of FEPs optimize the control of similar devices, such as beam motion or the main power supplies. Installed in racks, the FEPs interface to devices such as stepping motors, transient digitizers, calorimeters, and photodiodes. For example, a single beam control FEP drives as many as 100 motors to precisely adjust the laser beam so that the laser is kept on course within 50 micrometers (about half the width of a human hair). In keeping with the independent bundle concept, these FEPs are wired to control devices associated with a single bundle.
Supervisor systems run on servers and workstations located near the main NIF control room and provide operators with system status and other data from the FEPs. Operators access supervisor-system data through a hierarchy of on-screen graphics interfaces. Operators can also view video images of the laser beams from any of the hundreds of sensor cameras located throughout the complex.
Daily operation of NIF is managed by the on-duty shot director, who oversees control room activities and operates the laser and target systems for conducting shot experiments. The ICCS team developed shot-supervisor software that assists the shot director and the control room staff to prepare and fire each shot by automatically sequencing the system’s many functions. “The software puts everything within each bundle in a specific time sequence and makes sure all the components play together to achieve the required laser performance,” says Dave Mathisen, lead designer of the shot-supervisor software. The shot director interacts with this software to ensure that experiments run successfully.
In designing the NIF central control room, VanArsdall studied the layout of the National Aeronautics and Space Administration’s (NASA’s) mission control room in Houston, Texas. NIF laser physicist and former NASA astronaut Jeff Wisoff notes that both control rooms have operator stations corresponding to different hardware systems. In NIF’s case, each console corresponds to a functional system on the laser. Similar to NASA operators in the Launch Center control room, operators located in the NIF control room continuously track data on their monitors.
Wisoff sees other similarities between executing a NIF shot and launching a Space Shuttle. “Launch of a Space Shuttle is controlled by software centered in the Launch Center control room until T minus 31 seconds—or 31 seconds before liftoff,” says Wisoff. “Then computers onboard the shuttle take over.” Similarly, countdown for a NIF shot includes computer checks of every subsystem, and the control system will automatically stop events from proceeding unless all conditions are satisfactory. At T minus 2 seconds, the ICCS software turns over control to a high-precision integrated timing system designed to trigger thousands of laser modules and diagnostics at exactly the right instant.

The architecture of NIF’s integrated computer control system is divided into layers. A front-end or bottom layer consists of about 700 front-end processors, and a supervisory or top layer has 50 powerful computers. Each bundle of eight beams has its own independent control system, which is replicated for the others.

Automation Does It All
Achieving the 4-hour shot turnaround time requires automating the numerous tasks involved in NIF shots. Efficient shot campaigns begin with careful advance planning. Information technology (IT) analysts evaluated the work processes needed for physicists and managers to plan, approve, and review experimental campaigns. IT developers then implemented campaign management tools that captured user requirements and prepared electronic shot plans for the control system.
A laser performance operations model (LPOM), which is an integral part of the supervisor software, translates user goals from the shot plan into the optimal operating parameters to be set by the control system at the start of each experiment. According to NIF physicist Mike Shaw, “Efficient operation of NIF experiments depends on obtaining precisely specified energy waveforms and producing energy balance among the beams.” The energy of every beam could differ because of slight differences in amplifier gains and optical transmission losses in each beamline. By running a computational model of the facility before each experiment, the team determines in advance how to configure the system so that the total requested beam energy and power output are delivered.
Shot automation software, delivered in April 2005, computerizes the preparation of each NIF bundle independently from the others. When the 24 bundles are ready, the software then synchronizes the countdown of all 192 beams to fire the shot. “We always knew we would have to develop this layer, but we needed NIF Early Light to provide the exact requirements,” says Carey. During the Early Light campaign, the ICCS and shot operations teams developed a 6,000-line checklist procedure showing how operators used the software during a busy shot schedule, which allowed the control system managers to improve the automation and reduce manual tasks.
“We want NIF operations to be as efficient and automated as possible,” says VanArsdall. He notes the tremendous success developers achieved in aligning NIF’s beams automatically. The alignment control system software determines the position of NIF’s laser beams on the optics by analyzing sensor video images with a variety of computer-vision algorithms. Motor control robotics software uses the sensor information to remotely position more than 9,000 stepping motors and other actuators. These devices point the beams through pinholes, center them on mirrors and lenses, and focus them onto the target—achieving greater precision and effectively eliminating the need for personnel to adjust the beamlines manually. “The precision we have achieved is comparable to hitting the strike zone with a baseball thrown from 350 miles away,” says Patterson.
At the same time, the system retains the facility’s flexibility so that the beams can be configured in many different modes depending on the experiment. “Our experience during NIF Early Light demonstrated that we can efficiently perform shots,” says Lagin. Thanks to the underlying control system architecture, adding the automation did not require rewriting other software layers.

Architecture Offers Flexibility
Control system software is written in two languages: Ada for the FEPs and the common services, such as timing; and Java for user interfaces and databases. Ada is often used in mission-critical applications such as real-time transportation and military systems to improve reliability and reduce maintenance. The mixed language environment offers support for engineering the core of the control system in Ada while providing tools to quickly develop graphics and other applications in Java.
The ICCS team developed a software framework of tools and patterns for building the large number of FEPs and supervisory systems. With this dynamic configuration framework, the team can replicate the software completed for one bundle simply by modifying parameters in the database and running another copy. ICCS software accesses the database during initialization to assign each process in the control system to prescribed bundles. “We wanted a plug-and-play capability for our system to make it simple to expand and service,” says VanArsdall. “Rather than writing new software each time, the FEPs and supervisors are simply configured in the database to bring each new bundle online.”
NIF managers adopted a strategy of incremental cycles of software development and formal testing to successfully deliver the large-scale system. Numerous software releases and updates have been delivered to date. “We plan and develop each release consistent with project goals,” says Lagin. “Then we test and, if necessary, modify it to meet the system’s requirements.”
Quality control and assurance processes are part of the development effort. Before software releases are approved for deployment, they are tested extensively to verify their performance. “We’ve made a significant investment in testing prior to deployment,” says Lagin. Quality control is performed independent from software development and constitutes 20 percent of the total software effort. “The ratio of software developers to testers is about four to one, which is consistent with the computer industry’s best practices,” says Lagin.

Offline Testing Followed by Online
Software testing begins with offline tests conducted in the ICCS Integration and Test Facility. “We have had tremendous success with our offline test program, finding the majority of software defects before deployment—when it is most cost effective,” says Drew Casavant, Controls Verification and Validation manager. “Offline testing substantially decreases the time we need to validate a software release.”
The Integration and Test Facility contains servers, workstations, network equipment, FEPs, embedded controllers, and example devices to be controlled during testing. Because it is impractical to reproduce NIF in the test bed, simulation software and other test aids are often used. “Simulations can offer high fidelity in testing the behavior of NIF systems,” says Casavant, “but we also run tests with real hardware as the final confirmation that the systems operate correctly and meet the performance requirements.”
Software testing can range from controlling an individual motor to executing a full shot sequence. In addition, fault conditions can be introduced during offline tests to confirm expected system behavior without risk of misoperating equipment or adversely affecting the facility. NIF shot directors and control room console operators participate toward the end of offline test cycles to learn about the new software and prepare for online deployment.
Once fully integrated and qualified in the test bed, the software release is approved for deployment to NIF. The configuration management team installs the software and verifies the release is complete and starts correctly. Testers ensure that software continues to work as expected and that new functionality operates as designed. Test personnel receive extensive training in safety, site work controls, and operation of the laser equipment.

NIF Early Light Proved the System
During the next 4 years, as more laser equipment is installed, the ICCS software and controls proven on the first quad will be installed to activate the remaining bundles. Additional automation is being developed for the target area control system. The team wants to reduce the number of manual activities required to control the target positioner, diagnostic manipulators, target diagnostic instruments, and other equipment in and around the target chamber.
“We can keep building onto the control system because the right architecture was laid down early in the project,” says Patterson. Looking to the future when all 192 beams are firing regularly, he envisions a continual process for improving ICCS to make operations even more efficient and easier for experimenters.
“We’ve established disciplined software engineering to deliver a majority of the software, and we’ve proven the control system architecture,” says VanArsdall. The team is increasingly confident that NIF will be a vital resource for keeping the U.S. nuclear stockpile safe and reliable, advancing scientific knowledge of the physics of matter under extreme conditions, and taking the next steps in fusion energy toward achieving ignition. “Many of us have devoted most of our careers to achieving ignition in the laboratory,” says Lagin. “It’s a grand challenge, and it’s also a great privilege to be part of the team working to achieve it.”
—Arnie Heller

Key Words: Front-end processor (FEP), integrated computer control system (ICCS), laser performance operations model (LPOM), National Ignition Facility (NIF), NIF Early Light, stockpile stewardship.

For further information contact Paul VanArsdall (925) 422-4489 (vanarsdall1@llnl.gov) or Larry Lagin (925) 424-3331 (lagin1@llnl.gov).

Download a printer-friendly version of this article.