John Melchi, NCSA Senior Associate Director for Administration
By John R. Melchi -- In 2011, a supercomputer of unprecedented power will be deployed by the National Center for Supercomputing Applications (NCSA) on the campus of the University of Illinois at Urbana-Champaign. Called Blue Waters, this IBM system will have greater computing capacity than all the current Top 500 supercomputers combined. This resource, which is funded by the National Science Foundation, will be capable of sustained performance of at least 1 petaflop when running the real codes and applications used by scientists and engineers. To put it another way, if you could multiply two 14 digit numbers every second, it would take about 31 million years to complete the 1 quadrillion calculations Blue Waters will complete every second. This computing power will enable scientists at universities and research centers across the United States to make extraordinary leaps in knowledge and scientific discovery: predicting the structure of complex biological systems, designing new materials atom by atom, predicting climate and ecosystem changes, and improving intricate engineering systems such as chemical plants and airplanes. Such breakthroughs will dramatically advance our understanding of the world around us and create enormous benefits for society—better healthcare and emergency response, less dependence on oil, increased sustainability of the environment—as well as renewed economic competitiveness of U.S. industries in a global economy that is fueled by innovation. Before these breakthroughs can be realized, however, NCSA and its partners—IBM as the hardware vendor, EYP Mission Critical Facilities (EYP MCF) and Gensler as the engineering and architectural firms, and Clayco/Nova as the construction manager—must plan and build a state-of-the-art greenfield, purpose-built, high-performance computing data center to house Blue Waters. This involves balancing the need for physical and cybersecurity with the requirements of operating an open research infrastructure and determining the most efficient strategies for powering and cooling the system with the least impact on the environment—all on a fixed budget. And to make the project super-challenging, we must do all of this for a computer and components that so far exist only as plans on paper. This requires a tremendous level of open communication and deep collaboration among IBM, NCSA and the University of Illinois, and EYP MCF.
An architectural rendering of the Illinois Petascale Computing Facility that will house Blue Waters.
Design key to staying on time, on budget
If you want to bring your data center project in on time and on budget, get the design right the first time. It's all about design and thoroughly understanding and communicating your requirements. It may sound simple, but the key is to design to a budget and not the other way around. The Illinois Petascale Computing Facility (PCF) project included an initial feasibility study followed by a schematic design study. Each step in the process brought to light issues that required well‑informed decision making. Because we have a fixed budget, over‑provisioning is not an option. The initial design team (a rather large one I must admit) consisted of engineers from IBM, the University, NCSA, and EYP MCF. The facility design process resulted in some changes to the Blue Waters system that otherwise would have not been made. The synergy between the engineers was fascinating to watch as ideas were shared and expanded upon in dynamic workshops. This face-to-face interaction forced us to address real issues impacting the functionality of the facility and the viability of the budget. We have taken great pains in ensuring that the facility is sited properly and can be expanded when additional raised floor space, power, and cooling capacity are required. In addition, we hired a construction manager with expertise in data center design and construction to ensure that our estimates are as accurate as possible and to keep the construction on time and on budget. We are currently developing the final construction documents for the design, with the final design to be completed in September and groundbreaking to follow this fall. It is important to note that the PCF actually consists of three projects: construction of the facility, extension of campus power to the site, and extension of campus chilled water to the site. All three will be completed no later than the summer of 2010. Strategies for energy efficiency
Energy efficiency is an integral part of the Blue Waters/PCF project. The University is committed to meeting environmental stewardship and infrastructure sustainability goals. As part of this commitment, the PCF will achieve LEED certification, with LEED Gold certification as the target. The Blue Waters system will be water-cooled, a departure from the air cooling typically used for supercomputing systems. As with the recently announced IBM POWER6 chip and POWER 575 server, chilled water will be delivered directly to the CPU package avoiding the inefficiency of using chilled water to cool air which is then used to cool processors. In the case of the POWER 575, IBM expects that water cooling will reduce energy consumption by approximately 40 percent, and we expect to see a similar gain in energy efficiency with the Blue Waters system. The facility will be equipped with three on-site cooling towers. About 60 percent of the year, these towers are expected to provide water that has been passively chilled by the cold outdoor air, thus substantially reducing the electricity required for cooling and reducing Blue Waters' annual operating costs. Because the facility will house additional infrastructure that will be air-cooled, the building is designed for both water- and air-cooled systems. We are using computational fluid dynamics models to precisely design the interior of the building to maximize the efficiency of the air conditioning system for those air-cooled computing systems. In addition, by using the University's highly reliable electricity supply, and distribution lines from two other power grids, we can avoid the installation of a costly uninterruptible power supply (UPS), which is traditionally used as a power backup. Eliminating the UPS minimizes floor space used and the costs of electrical infrastructure, while increasing energy efficiency. We'll also eliminate the substantial energy losses associated with power conversions for the UPS. Balancing security with accessibility
Blue Waters will be available to scientists in the national research community who will access the system from across the United States through the nation's research and education network infrastructure. While it must be an open, accessible resource, it is also essential that the integrity and security of the building, system, and data produced by users be assured. The building is designed to withstand an F3 tornado with winds up to 165 mph. In addition, the building's security plan includes physical measures such as doors, locks, lighting, and barriers; technical measures such as cameras, digital video recording, and biometric devices; and operational policies, procedures, and training. Cybersecurity, meanwhile, will be provided by NCSA's experts. The center's staff has more than 20 years of experience designing and deploying systems that meet stringent cybersecurity requirements; these comprehensive policies, procedures, and technologies will be extended to Blue Waters. System and facility evolving together
Through the Blue Waters/PCF project, an innovative computer architecture is driving the design of an innovative data center. Rather than dealing with legacy systems or the constraints of an existing facility, we are free to start from scratch, creating a perfectly paired system and facility that will work in harmony to meet the nation's scientists' needs for more computational horsepower to solve the world's most complex questions. In the end, it's always about the science. John R. Melchi is the Senior Associate Director for Administration at NCSA.
