GPU-accelerated system of at least 10 petaflops to provide exceptional scientific computing

The Department of Energy’s Oak Ridge National Laboratory has awarded a contract to Cray Inc. to increase the Jaguar supercomputer’s science impact and energy efficiency. The upgrade, which will provide advanced capabilities in modeling and simulation, will transform the DOE Office of Science-supported Cray XT5 system, currently capable of 2.3 million billion calculations per second (petaflops), into a Cray XK6 system with a peak speed between 10 and 20 petaflops.

The new system will employ the latest AMD Opteron central processing units (CPUs) as well as NVIDIA Tesla graphics processing units (GPUs) — energy-efficient processors that accelerate specific types of calculations in scientific application codes. The last phase of the upgrade is expected to be completed in late 2012. The system, which will be known as Titan, will be ready for users in early 2013.

With 299,008 cores and 600 terabytes of memory for solving critical problems, Titan will continue the legacy of its predecessor as one of the nation’s most powerful tools for science. The computationally intense numerical experiments that will run on the system will focus on DOE priorities in energy technology and science research.

Titan’s pioneering simulation projects will include the commercially viable production of biofuels and biomaterials from cellulosic materials, such as switchgrass and poplar trees. Another project investigates combustion, responsible for most energy use, to burn fuels cleanly in efficient engines.

Simulations will aid the development of new materials for photocells that will convert more sunlight into electricity and new battery technology to store that energy for use when the sun isn’t shining. Moreover, Titan’s users will study safe extension of the lifecycles of nuclear power plants. Titan will also be used to study the reactions and flow of contaminants in the ground as well as the impacts of energy use on climate.

“All of these areas of science will benefit from Titan’s enormous increase in computational power,” said ORNL Director Thom Mason. “Titan will allow for significantly greater realism in models and simulations and the resulting scientific breakthroughs and technological innovations will provide the return on this national investment. Discoveries that take weeks even on a system as powerful as Jaguar might take days on Titan.”

“Cray has been fortunate to share in a great partnership with the Oak Ridge National Laboratory and the Department of Energy’s Office of Science, and we are excited that our relationship will continue with Titan,” said Peter Ungaro, president and CEO of Cray. “Scientists at Oak Ridge and across the DOE have been researching some of the world’s most pressing energy and environmental challenges. As a company, it’s an honor to know that this vitally important work has been, and will now continue to be, done on supercomputers built by Cray. It’s a partnership we are very proud of.”

The initial system upgrade will replace all of the processors with the latest AMD Opteron processors codenamed “Interlagos,” taking the system from two 6-core processors per node to one 16-core processor per node, and facilitate the addition of GPUs.

As one of the first deployments of the latest generation of AMD Opteron processors, the upgrade delivers one-third more cores in the same physical space compared with Jaguar, while also doubling the memory and outfitting the system with Cray’s powerful “Gemini” network to improve performance on scientific applications.

“But the game-changing new technology will be the inclusion of advanced GPUs,” Mason said. “These Tesla GPUs are able to perform many more calculations for the same amount of power as conventional microprocessors and will work hand-in-hand with CPUs to deliver new levels of energy-efficient application acceleration.”

Buddy Bland, project director for the upgrade, described the deployment timeline. “In the first phase of the upgrade, to begin this year, we will replace all of the processor boards, memory, and system interconnect network in Jaguar with the new Cray XK6 processor board. This upgrade will also include approximately 1,000 current NVIDIA Tesla 20-series GPUs,” he said.

The GPUs will allow users to begin modifying their scientific applications to work well with the new technology. In the second phase, to be performed in the second half of 2012, between 7,000 and 18,000 of NVIDIA’s next-generation Tesla GPUs will be added to the system, increasing the peak performance to between 10 and 20 petaflops, depending on architectural options executed.

Additional information about Titan is available on the Oak Ridge Leadership Computing Facility website.