ACADEMIA

By Uwe Harms, Harms Supercomputing Consulting -- DKRZ (German Climate Computer Centre) was officially opened by Staatssekretär Dr. Klaus Thomas. Today it delivers a performance of 1 TFlop/s using 16 NEC SX-6 nodes (128 processors). Next year, in a third step, another 8 SX-6 nodes will be added. This leads to a peak performance of 1.5 TFlop/s. September 10, 2002, DKRZ was officially opened at University Hamburg by Staatssekretär Dr. Uwe Thomas, Ministry of Research (BMBF). This German Ministry paid for the computer system, the data server and the new infrastructure of HLRE about 35 million EURO. DKRZ was founded in 1988 as a joint effort of Max-Planck-Institute for Meteorology in Hamburg, the Centre for Ocean and Climate Research of University Hamburg, the Alfred-Wegener-Institute for Polar Research in Bremerhaven and the GKSS Research Centre in Geesthacht. The partners pay the operational costs of DKRZ, about 8 million EURO per year. They can use 50% of the computing power, the is dedicated for German climate researchers. They can propose scientific projects, which are reviewed and undergo an allowance procedure. Now HLRE/DKRZ is one of the national supercomputer centre like Jülich, Stuttgart and Munich with Max-Planck and Leibnizrechenzentrum. In his opening address he mentioned the climate changes, a challenge to find solution. As an example he said that the USA need a factor of 10 more energy than people in fast developing countries like China or India. In the next years there will be massively changes in requests for resources. BMBF funded ecological projects, 35 million Euros to produce ecological products and production in textile industry. He sees Europe ahead the USA in environmental behavior. Germany has a powerful climate model to better understand the earth system. A specific research topic has to be the earth system in specific regions, the early forecast of the heavy rains and floods in Germany. DKRZ, by its cooperative approach with partners from academia and research in Germany, could extend its service to Europe. All can benefit from the supercomputer, the scientific knowledge and the climate models. The Computers DKRZ is well known of its supercomputers, starting with a Control Data Cyber 205, a Cray 2, followed by a 16 processor Cray C90, installed in 1994. Thus this was a time of frustration, neglecting the importance of climate research by German politicians. Last year the centre signed the contract with NEC, which installed as step zero NEC SX-4 systems, and the Cray was switched off end of October 2001. This spring NEC installed the first 8 nodes (64 processors) and in August/September another 8 nodes. Now the computer system has a peak performance of 1 TeraFlop/s and a memory of 1 TeraByte. Mid next year another 8 nodes will be added, leading to a peak of 1.5 TeraFlop/s and a memory of 1.5 TeraByte. Then the computer covers 100 square meters, about 50% of the computer room. Thus there is enough space on the 15th floor of the Geomatikum to expand the capacity in all directions. By the way, the Cray 2 needed a crane to be put into the 15th floor. The bandwidth of a memory access is 256 GB/s. With this speed the 8 processors of a node read from and write to memory per second. This is about 256 time faster than that of a PC. The capacity of the disks sum up to 50 TeraByte, the tapes to 1.4 PetaByte. The high-speed network has a performance of 450 MByte/s. DKRZ expects an application performance of about .5 TeraFlop/s in its codes, compared to about 10% of the peak with massively scalar processors. Wolfgang Sell, technical director of DKRZ, mentioned that the scientists have a performance improvement of a factor of 100 compared to the old Cray. This fact means that the grid size can be reduced from 300 km to a 50 to 100 km resolution. Hundred years modeling time on a 50 km grid needs 15 years computing time on the Cray. This is now reduced to 3 months, he expects on the NEC. The turbulence resolution in the ocean would require about 7 years on the old machine but only 2 months on the NEC. The third example is a 100 km grid including chemistry in the model, 3.5 years reduced to 1 months. This is a time frame a climate researcher can wait. Additionally Sell told that the computer was accessed very well, in the first months a usage of 75 to 80%. But the users approach real big problems very slowly, the preparation has not been done. He expects a high application performance on this vector machine, as the climate models can efficiently use this architecture. Prof. Dr. Guy Brasseur, Scientific Director of DKRZ, tries to put this national solution into a broader, European way. Probably Europe needs a computer solution like the Earth Simulator in Japan. There has to be build a European network of climate researchers in academia, research and computer centers, software and hardware companies. He discussed the researchers model approach. The computer solves the climate questions, the complexity within the model. The climate system is complex, coupling the atmosphere, the ocean, the land, the arctic ice, the chemistry and more influencing components. New is the regional aspect of the climate. Changes in the ocean need centuries to be observed. There are two approaches, a high-resolution, complex long integration or a sample of runs with a coarse grid. The model is computed with variations, deterministic, 10 to 20 times. The researcher now looks for the resulting changes in the solution. He additionally want to redefine, renew the scientific approach. He wants a connection of human and natural science. The science of the future is interdisciplinary, in the climate research economics and philosophy have to be involved. He closed that Watanabe's Earth Simulator, vector machines, are best suited to earth simulation. DKRZ will adapt the system to the needs of Germany. Prof. Dr. K. Mehldorn, Vice President of Max-Planck-Society, now wants to improve the operation of DKRZ in close coordination with the partners. They proposed to strengthen the science in DKRZ, a better incorporation of users, a better integration of mathematics and informatics - computer science, and an intensive cooperation with the model and data group, which is outside DKRZ. Professor Dr. Hans von Storch, scientific steering committee, underpinned the request for more theoretical science in DKRZ. Models and observations produce figures and data. If there is no understanding what they mean, and statistics only condense the figures, no research can be done. For an understanding, one needs theory to interpret the figures and to put them into a context. T. Watanabe, Vice President, NEC Solutions, the chief architect of the NEC SX-series, gave an outlook into the computer development and architecture of the next 10 years. In 2010 the chip size is 6.2 square centimeters, the size of a microprocessor core (MPC) is only 0.1 square centimeters. The free chip area can be used for many special engines like graphics, or vector engines, multiple MPCs for HPC, high end affordable HPC, or VLIW, superscalar, multithreading. He expects an evolutionary development in the next 10 years. W. Zwieflhofer, head of the ECMFW supercomputer centre, presented the users requirements for Earth Simulation at such a centre. There are three basic elements in Earth Simulation: * processed studies, modeling paradigm, field experience, scientific work on model computation (Theory) * observation, space platforms, Satellite, insitu measurements (Experiment) * models, modeling framework, multiple models, computing resources (Simulation) There must be interaction between these elements. Then he listed another field of model interaction: atmospheric model climate research ocean seasonal forecast chemistry weather forecats land marine forecast hydrology hydrology forecast He compared clusters with vector and scalar CPUs: Clusters with vector -- scalar CPUs 16 nodes -- 100 nodes 8 proc/node -- 8 proc/node 8 vector pipes -- 2 scalar FPUs ILP (M+A, etc) <2 -- ILP < 2 FP ops 2048 -- 3200 units of serial work 1024 -- 800 ILP (Instruction Level Parallelism) A parallel system is busy with 1000 units serial work. With a single model, T42, 300 km, you get 8000 point, 8 columns. T511, 40 km, 350 000 points, 350 columns. Small units with different work run good on a parallel machine. He looked deeper into some architectural issues. For good application performance there is sufficient inherent parallelism. The coded application exploits the parallelism without significant load imbalance and uses efficient numerical algorithms. Real world application requires significant amount of intercommunication between units of work. Thus a high bandwidth and low latency intercommunication is necessary. The bandwidth to memory is a factor for performance. He meant that memory size is less an issue but large memory allows on the fly post processing. Then he gave an important hint on the three basic elements of Earth Simulation: * processed studies, 1000s of scientists are involved * observations cost billions/year, e.g. Europe pays 1 billion for observation platform * models, only 10s of millions, the opening of HLRE is an important step for more model continuity. http://www.dkrz.de http://www.ess.nec.de (NEC supercomputing)