The discovery of a vast new oil field in the Santos Basin off the coast of Brazil was exciting news. The next challenge would be extracting the oil. Before going into production, Brazil’s state-run oil firm, Petróleo Brasileiro (Petrobras), had to clear numerous technical and environmental hurdles. The deep-water find is submerged more than a mile beneath the surface and under approximately two miles of ocean floor with a thick layer of sedimentary salt. The area is also riddled with geological faults, which require multifractured, horizontal wells to safely extract oil. To address those challenges, Petrobras is sponsoring an HPC grid managed by an alliance of five federal universities. The HPC project, called the Petróleo Brasileiro (BR) Network, would be used for computer simulations to aid oil production in Brazil's pre-salt basin.

The Alberto Luiz Coimbra Institute-Graduate School and Research in Engineering from the Federal University of Rio de Janeiro (COPPE/UFRJ), which has been working with Petrobras on HPC projects since the late 1980s, realized that the universities needed a high-performance computing grid to handle the complex project. After building an innovative, ecologically efficient datacenter near Rio de Janeiro Bay, in September 2008 the alliance issued an RFP to several vendors, including Sun Microsystems, IBM, Dell, Bull, SGI, and HP. The universities were looking for a high-performance, affordable solution that they could expand easily over the next few years.

After an extensive review process, the alliance decided that Sun would best meet its goals for performance, scalability, and energy efficiency. In addition, the universities discovered that the Sun solution would cost about 30% less than an offering from the closest competing vendor. “Sun provided the best solution in terms of meeting the technical requirements together with the lowest price — it’s as simple as that,” says Alvaro Coutinho, professor at the Center for Parallel Computing and the Department of Civil Engineering at COPPE/UFRJ.

The new HPC solution includes 448 Sun Blade X6275 Server Modules with the new Intel Xeon processor series 5500, Sun Blade 6048 Modular Systems, and the Lustre file system. The solution features the first blade server with on-board QDR InfiniBand for high performance and low latency. The alliance runs the Red Hat Linux operating system on a Sun open-network solution, which supports multiple operating systems including the Solaris OS and Windows.

Installed in June 2009, the BR Network currently includes three universities: the University of São Paulo, the Federal University of Alagoas, and the Alberto Luiz Coimbra Institute-Graduate School and Research in Engineering from the Federal University of Rio de Janeiro. The grid provides approximately 100 teraflops of peak performance and 21.5 TB of memory. The alliance is currently running benchmark tests to submit to the Top500 list of the world’s fastest supercomputers. When the tests are complete, the universities will deploy custom software on the BR Network and prepare for production. The alliance expects that the BR Network will be fully operational by February 2010.

Approximately 300 researchers, including professors, postdoctoral researchers, and graduate students, will work on the new cluster. Sun Professional Services is also providing ongoing support for the project’s air-conditioning and hi-visualization environments.

The Sun solution more than doubles the universities’ processing speed with ample room for growth in the new datacenter. “We are very excited about the scalability of our Sun HPC system,” says Coutinho. “The idea when we submitted the RFP was for the solution to be expandable to hundreds of teraflops. We are currently at about 100 teraflops, and we are thinking of doubling or even tripling it in the next two years.”