Maintaining its place at the forefront of scientific research, Princeton has brought together three high-performance supercomputers to establish one of the nation’s leading university-based research computing facilities. The move comes as research that used to take place almost exclusively in Princeton’s laboratories is moving increasingly toward its computing facilities. Many chemists are setting up virtual “dry” labs; neuroscientists, astrophysicists and geneticists are running experiments dependent on the availability of high-speed computing; and engineers are moving aerospace tests out of wind tunnels and into computing centers. Cooperation between Princeton’s Office of Information Technology (OIT), the Princeton Institute for Computational Science and Engineering (PICSciE), the School of Engineering and Applied Science and the provost’s office — along with leading computational scientists on campus — has brought together the speed and versatility of high-speed supercomputers nicknamed “Della,” “Hecate” and “Orangena” to respond to these evolving needs. “With three large central supercomputers dedicated to research computing, Princeton is today a leader among higher education institutions in providing central support for researchers,” said Betty Leydon, the University’s vice president for information technology and chief information officer. “Each of the three high-performance machines — Della, Orangena and Hecate — has a different performance profile suitable for handling different kinds of computational tasks,” Leydon added. “Together, these three machines provide Princeton faculty a world-class computational research environment and solidify the emerging partnership between OIT and faculty in support of research computing at Princeton.” In a process that began last fall, the University worked with research faculty and affiliated departments to design the computing facility. It includes the Dell cluster of computers Della, which has 512 processors capable of very high speeds; the IBM Blue Gene brand supercomputer Orangena, which has 2,048 processors capable of “scaling” to perform large and smaller tasks; and the SGI Altix computer Hecate, which has 64 processors with large memory capabilities. “We have put together a leading, cutting-edge computational research center that catapults us into the ranks of the leading university-based research computing institutions in the country,” said Jeremiah Ostriker, the director of PICSciE, which led the effort to bring Hecate to campus. Campus computing experts spent six months working with faculty to acquire the resources to buy all three machines, another six months installing and configuring them for use, and now are making the computing center widely available to the entire Princeton research community. The goals are to sustain the groundbreaking research already taking place at Princeton and to encourage future generations of research among current faculty, and also recruited scholars and scientists who may be attracted by the opportunities offered by the new facility. Eager faculty members are already proposing problems for the most recently acquired computer Della. Emily Carter, the Arthur W. Marks ’19 Professor of Mechanical and Aerospace Engineering and Applied and Computational Mathematics, plans to use the super-fast Dell computer for her work on metal alloys. “By understanding the mechanical response of materials as a function of composition and structure, we hope to design stronger and harder materials for use in a variety of technological applications,” Carter said. Scientists at the Princeton Plasma Physics Laboratory (PPPL) use Orangena for its ability to tackle exceptionally large, as well as smaller, problems. “My particular area of interest is the simulation of turbulence and its influence on the magnetic confinement of a plasma, which is a very hot gas made up of charged particles,” said William Tang, the chief scientist of PPPL and a contributor to the computing center as associate director of PICSciE. “It’s important to understand the dynamical behavior of hot plasmas so that we can effectively control fusion reactions to create energy in an efficient manner.” The University computing experts who initiated the partnerships that brought the three supercomputers to campus commonly refer to the three systems as the three legs of a stool, saying Princeton now has a well-balanced set of resources that can support a wide variety of computations. Faculty members in genomics, for example, are expected to benefit from all three systems. “We need more and more computing resources to keep up with the rapid, and seemingly exponential, increase in demand for computing in biology,” said David Botstein, director of the Lewis-Sigler Institute for Integrative Genomics and professor of molecular biology. “Contributing to the Dell cluster reserved a proportional amount for our use, and having a shared cluster managed by OIT as a central resource obviates the need to dedicate our own personnel, and also may potentially allow more efficient appropriation of computing resources.” Different needs, different machines According to Princeton computing resource managers, among the greatest challenges faced by researchers is finding resources to dedicate to equipment to support their particular research needs. Space and maintenance for the equipment also become factors. “One of the things I noticed when I was provost was that the community of scientists and engineers at Princeton whose research and teaching depends on high-speed computing is very rapidly growing,” said Ostriker, who was the University’s provost from 1995 to 2001. He estimated that the total number of faculty in research computing is doubling every three years, including biologists and others who traditionally were not engaged in heavy computing in the past. “Faculty in mechanical and aerospace engineering who used to test wings in a wind tunnel now do it via computation, so there are huge numbers of different areas where people are using computers for things that they didn’t before, and they are spread throughout the University,” Ostriker said. “It seemed vital to try to combine resources from many sources so people could do their work effectively. We’re doing this by combining hardware, software, courseware and infrastructure support.” OIT announced last September that it had entered a unique partnership with Princeton scientists to acquire one part of the newly established computing facility. That part was Orangena, ranked one of the top supercomputers in the world and given a moniker to incorporate one of Princeton’s colors into the name IBM gave the system for its work in the field of genetics. Months earlier, PICSciE had set up a similar partnership to acquire the SGI company’s Hecate. OIT recognized that the two supercomputers offered Princeton researchers different opportunities, said Curt Hillegas, manager of computational science and engineering support in OIT’s academic affairs department. He has worked closely with PICSciE and the faculty to find a balance between all of the systems. “There are some scientific models that can’t be broken down in small enough pieces to scale to Orangena,” Hillegas said, “and there are some kinds of problems that can’t be broken down at all, and the entire problem needs to fit into one big piece of memory, which is what the SGI [Hecate] is good for.” Orangena is ideal for problems that can be broken up among many processors, such as the complex modeling of the universe, and the SGI computer Hecate — named after the Greek goddess — can run one problem for a long period of time. The super-fast Dell computer Della has four gigabytes of memory for each of its 512 processors, making it optimal for jobs requiring fast speeds. Opening the facility’s doors Managers of the systems emphasized that the willingness of individuals and departments to contribute, not only to their own research but also to the future of computational research at Princeton, will prove a benefit to the entire University community. Della’s contributors were genomics, PICSciE, OIT and astrophysics, while five individual faculty from a wide range of departments contributed with OIT, PICSciE and the engineering school to bring Orangena to campus. Hecate’s contributors were PICSciE, PPPL, astrophysics and faculty from other academic departments. “The selection of the systems wasn’t arbitrary,” said Serge Goldstein, director of academic services for OIT. “It is actually an extremely well-balanced broad range of systems that support a broad range of scientific and engineering models, so any researcher at the University can find a system that is well suited to his or her needs.” Resource managers may make researchers aware of another option called Condor, which is software that allows OIT to run jobs on computers across the University that are idle, Goldstein said. OIT can submit jobs that run on machines essentially at night, complementing the high-speed capabilities of the research computing facility as another means of providing support for research. An application process that includes a one- to two-page research proposal will be used to determine schedules and uses for the various systems that make up the research computing facility. A faculty committee will ensure that research projects require the computational capacity the facility offers. “This new model for research computing support at Princeton involved the pooling of funds and resources to create collaborative, shared central facilities that can be efficiently maintained and that avoid the costly duplication of effort that comes with traditional models where everyone has his or her own cluster,” Leydon said. The University’s collaborative model encourages progress in new directions, Leydon added. With support from the provost’s office, the goal is to promote further scientific discovery, collaboration both on campus and with other institutions, and better use of the resources already available on campus. “High-speed computing is a thread that ties together exciting research programs in an amazing range of departments,” Provost Christopher Eisgruber said. “The capacity to analyze vast amounts of data rapidly is producing new discoveries about phenomena that range from the microscopic to the galactic. I am pleased that collaborations among our scientists, engineers and OIT professionals have enhanced Princeton’s stature as a leader in this important field.” For more information about the application process to gain access to the research computing facility, members of the Princeton community should contact Hillegas at curt@princeton.edu.