ACADEMIA

New System Replaces Older Cluster and Speeds Analysis and Development of Offshore Oil and Gas Production Systems for Petrobras: To help Brazil maintain its recently achieved goal of oil self-sufficiency, the Numerical Offshore Tank-TPN laboratory at the State University of Sao Paulo (USP) selected shared memory, high-performance compute and scalable SGI InfiniteStorage systems from SGI to employ, and further develop, an exciting new computational fluid dynamics (CFD) code. The laboratory's main focus is to develop and analyze floating offshore production systems for deep-water oil and gas production managed by Petrobras, Brazil's national oil and gas company. In the six months the USP lab has been using the SGI Altix system, purchased with funding from the Brazilian government and Petrobras, researchers report the code scales extremely well and runs up to 100 times faster on the SGI Altix system than on their home-grown cluster environment. "We chose SGI Altix, first, because Altix is simply the best machine to run our codes," said Antonio Augusto Russo, System Engineer, Numerical Offshore Tank-TPN Lab, University of Sao Paulo. "On the SGI 16-processor machine, our codes run at least a hundred times faster than when we used the old cluster. We need very large amounts of memory to run our codes, and they scale very well on the Altix because of the shared memory and the NUMAflex architecture -- there's a lot of communication. The second reason we selected Altix is the ability to use OpenMP, which helped a lot with code development; it was much faster than it would have been if we used MPI. Also, SGI is the most reliable storage, by far, that we have. We even use it to hold our financial data -- invoices, contracts and documents related directly to our research -- as well as for analysis and development." The CFD code, introduced at Supercomputing 2006 in Tampa by University of Tokyo researchers, is called Moving Particle Semi-explicit (MPS), a Linux based numerical technique to solve hydrodynamics problems, in particular sloshing water, and fluid-solid interactions. The powerful processing and memory SGI NUMAflex architecture of SGI Altix systems was absolutely key to the university's purchase. The SGI Altix is also being used to develop the lab's main code, known in Portuguese as TPN, translated as "Numerical Offshore Tank." TPN is a numerical simulator of offshore structures, such as floating oil production rigs. MPS is a complementary code to TPN at present, but through development on the SGI Altix system, the Lab foresees MPS becoming their main research tool in the near future. "The Numerical Offshore Tank-TPN Lab's use of SGI technology is yet another example of SGI's ability to deliver solutions for the most demanding compute and data-intensive workflows," said Michael Brown, sciences segment manager, SGI. "The Altix system's extreme scalability, flexibility, and reliability, coupled with its open architecture, provide the best combination of compute, special purpose processing, memory and I/O elements, to match the diverse needs of the Lab's environment." Brazil reached oil self-sufficiency in 2006; 80% of that oil comes from deep waters off the coast of Brazil in the South Atlantic Ocean. The continuing and main focus of the USP Numerical Offshore Tank-TPN lab is to help Petrobras explore oil and gas reservoirs in deeper offshore waters, and the new SGI Altix system and SGI InfiniteStorage will play an important part because they not only supply the computational power to run the latest CFD codes but the storage capacity for the initial data as well as results from analysis. CFD Codes in Development at University of Sao Paulo MPS is a particle matter tracking system that simulates solids or fluids as very small particles on unstructured, moveable meshes, allowing scientists to more realistically demonstrate rolling and rotational motions. Simulating water flow, or flow around sand or rock formations or floating rigs, pinpoints drilling sites with a much greater degree of accuracy than conventional CFD applications. USP Professor Kazuo Nishimoto and Mr. Russo are working in collaboration with University of Tokyo Professor Seiichi Koshizuka, who originally developed the MPS code; Dr. Koshizuka is also working closely with SGI in Japan on optimizing the code on very large SGI Altix systems. "This code requires heavy computational power that was impossible a few years ago," said Prof. Nishimoto, Department of Naval Architecture & Ocean Engineering, Numeric Offshore Tank -- TPN Lab, University of Sao Paulo. "With the accelerated evolution of high-performance computers and processors, the MPS method can start to be used in practical calculations. While we continue to use conventional HPC clusters linked by conventional communication cards for other simulation calculations, the SGI Altix machine, with the NUMAflex architecture, shows very promising results. The Altix can run this kind of particle code very fast, even at high particle numbers. Compared with the conventional HPC cluster using high-speed communication systems, the Altix is faster and provides us with more accurate results using the MPS method." The MPS method simulates fluid or solid by particles using Lagrangian representation. By simulating floating units in the wave, current and wind conditions, scientists can consider almost all physical phenomena that cause dynamic effects on floating bodies. The advantages of MPS include incorporating non-linear effects on the free surface of fluid, like sloshing and wave impacts that facilitate the representation of the fluid-structure interaction. The lab, using their TPN code, has started to incorporate the MPS method in their simulator to represent more precise non-linear phenomena of the waves and fluid structure interaction. The University of Sao Paulo purchased an SGI Altix system with 16 Intel Itanium 2 processors with Novell SUSE Linux Enterprise 9, with SGI ProPack 4. USP also selected an SGI InfiniteStorage solution, with an initial capacity of 2.4TB.