ACADEMIA

The latest upgrade to the Cray XT Jaguar supercomputer at the Department of Energy’s (DOE’s) Oak Ridge National Laboratory (ORNL) has increased the system's computing power to a peak 1.64 “petaflops,” or quadrillion mathematical calculations per second, making Jaguar the world’s first petaflop system dedicated to open research. Scientists have already used the newly upgraded Jaguar to complete an unprecedented superconductivity calculation that achieved a sustained performance of more than 1.3 petaflops. “Jaguar is one of science’s newest and most formidable tools for advancement in science and engineering,” said Dr. Raymond L. Orbach, DOE’s Under Secretary for Science. “It will enable researchers to simulate physical processes on a scale never seen before, and approach convergence for dynamical processes never thought possible. High end computation will become the critical third pillar for scientific discovery, along with experiment and theory.” The upgrade at DOE’s Oak Ridge National Leadership Computing Facility represents a major milestone in a four-year project, begun in 2004 when DOE’s Office of Science launched a sustained effort to upgrade supercomputing capabilities for unclassified research at DOE’s complex of national laboratories. The project to build a petaflops machine--completed on time, on budget and exceeding the original scope--included partnerships with industry to develop new hardware and computer architectures. “With the expansion of the leadership computing resources at Oak Ridge, the Department of Energy is continuing to deliver state-of-the-art computational platforms for open, high-impact scientific research," said Michael Strayer, Associate Director of the DOE Office of Science for Advanced Scientific Computing Research. "The new petaflops machine will make it possible to address some of the most challenging scientific problems in areas such as climate modeling, renewable energy, materials science, fusion and combustion." Within hours of access to the Oak Ridge supercomputer, an ORNL team became the first to achieve sustained petascale performance on a scientific application. In 1998, another ORNL team was the first to achieve sustained terascale performance for science. Thomas Zacharia, Associate Laboratory Director for Computing and Computational Sciences, said he expects that Jaguar "will drive new developments that in turn will lead to energy technology innovations.” Supercomputing holds significant promise for U.S. economic competitiveness, including the promise of enabling American industry to perform “virtual prototyping” of complex systems and products. Jaguar will enable companies to reduce development costs and shorten the time required to market new technologies. Jaguar is the result of a partnership among DOE, ORNL and Cray that has pushed computing capa­bility at a rapid pace. The current upgrade is the result of an addition of 200 cabinets of Cray XT5 to the existing 84 cabinets of the XT4 Jaguar system. During the third quarter of 2008 Cray achieved a major milestone by successfully deploying all of the cabinets for the petaflops system, ahead of schedule. Starting at 26 TF (26 trillion calculations per second) in 2006, the XT system grew 60-fold in capability through a series of upgrades to what is today the world’s most capable system dedicated to open scientific research. Jaguar uses over 45,000 of the latest quad-core Opteron processors from AMD and features 362 terabytes of memory and a 10-petabyte file sys­tem. The machine has 578 terabytes per second of memory bandwidth and unprec­edented input/output (I/O) bandwidth of 284 gigabytes per second to tackle the biggest bottleneck in leading-edge systems—moving data into and out of processors. The upgraded Jaguar will undergo rigorous acceptance testing in late December before transitioning to production in early 2009. Among the most powerful open scientific computing systems in the world, Jaguar is already in high demand by scientists who are honing their codes to take advantage of the machine’s blistering speed. The Jaguar petaflops system is unique in the balance it represents among speed, power, and other elements essential to scientific discovery. Sev­eral design choices make it an excellent machine for computational sciences—including more memory than any other machine by almost a factor of three, more powerful processors, more I/O bandwidth and the high-speed SeaStar network developed specifically for very-high-performance computing. Annually, 80 percent of the Leadership Computing Facility resources are allocated through DOE’s Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program, a competitively selected, peer reviewed process open to researchers from universities, industry, government and non-profit organizations. Scientists and engineers at DOE’s Oak Ridge National Laboratory are finding an increasing variety of uses for the Cray XT system. A recent report identified 10 breakthroughs in U.S. computational science during the past year. Six of the breakthroughs involved research conducted with the Jaguar supercomputer, including a first-of-its-kind simulation of combustion processes that will be used to design more efficient automobile engines. The report is available at its Web site. The landmark DOE Office of Science report, “Facilities for the Future of Science: A Twenty Year Outlook,” published in 2003 listed ultrascale scientific computing capability as the second highest priority. The report was the first long-range facilities plan prioritized across disciplines ever issued by a government science funding agency anywhere in the world. In preparing the report, Dr. Orbach set out to increase by a factor of 100 the computing capability available to support open (as opposed to classified) scientific research – reducing from years to days the time required to simulate complex systems to understand the combustion process, model thermal reactions, analyze climate change data, reveal chemical mechanisms of catalysts, and study the collapse of a supernova. The report also stipulated that the Office of Science supercomputing facilities would be available to all researchers, subject to proposal peer review. Additional information on the Leadership Computing Facility is available at its Web site.