GOVERNMENT

Bristol, UK, At SC2002 the top 500 supercomputing systems (http://www.top500.org/) saw a dramatic increase in the number of large linux-based clusters. Most notably the MCR system housed at the Department of Energy National Laboratory at Lawrence Livermore (LLNL) entered at number five in the list with a Linpack performance of 5.69 Tflops. Since that time Quadrics, LLNL and the University of Texas at Austin Department of Computer Sciences have worked together to optimize the performance of Linpack on IA32 Quadrics based clusters. With this work MCR has now reached a figure of 7.63 delivered Tflops out of a peak of 11.06 Tflops for an efficiency of 69.03% of peak on Linpack (http://www.llnl.gov/linux/news_events.html). The QsNet cluster fabric allows Linpack performance to scale linearly with system size. Building on the success of MCR, LLNL have partnered with IBM® and Quadrics to install a second large linux cluster, called ALC. This system uses IBM® x-servers with a 960-way Quadrics QsNetI switch and a similar software environment to MCR. The ALC system delivers a Linpack result of 6.58 Tflops from a peak of 9.21, an efficiency of 71.6 %. Based on these results, MCR will be elevated to the number four slot and ALC will enter the list in the number five position of the current top500. Quadrics machines now comprise six of the top eight most powerful computers in the world. These machines are used both for basic scientific research and for the most critical of national security issues. Both systems at LLNL are built from standard Intel® Xeon® processors running at 2.4 GHz with 2 Gbytes/memory per processor. The Quadrics network consists of a customized fat-tree network that can accommodate up to 1,536 nodes. Individual cluster nodes run the LLNL CHAOS Linux distribution - see http://www.llnl.gov/linux/chaos. Quadrics performance reaches some 400 Mbytes/sec of bandwidth together with sub 5us latency for MPI and 1.8us for DMA. In order to support the operational requirements of these systems, QsNet is also used to carry IP traffic for the Lustre Parallel File System allowing bulk transfer of data direct to applications. IP over QsNet reaches 275 Mbytes/sec in standard TCP/IP test for bulk transfers. "One of the underestimated aspects of commodity cluster technology is that there are smart people working all around the world with the base node as their personal computers. These contributors, working around the clock, come up with stunning new performance optimizations and invent new Open Source software. They then make this software available to everyone via the web. By utilizing the highly optimized GOTO DGEMM routine we were able to achieve a 34% improvement on the Linpack benchmark with MCR. This aspect of commodity cluster computing is quite compelling," says Mark Seager, LLNL Integrated Computing and Communications Asst. Dept Head for Advanced Technology. "It is the performance of the sequential matrix-matrix multiplication (DGEMM) kernel that limits the overall performance of the parallel Linpack benchmark. The implementation of this kernel by Mr. Kazushige Goto, Visiting Scientist at UT-Austin, for the Intel® Xeon® 2.4GHz achieves an impressive 81.2% of the peak performance of a single processor (http://www.cs.utexas.edu/users/flame/goto/) Since the HPL benchmark, using this kernel, scales from achieving 71.4% for a single 2-way Xeon® node to 69% for the entire machine, the architecture shows remarkable scalability on this benchmark. Particularly noteworthy is the cost/performance ratio thus achieved." notes Robert van de Geijn, Professor of Computer Sciences at The University of Texas at Austin. “MCR and ALC represent major milestones in the delivery of cost-effective production supercomputer systems”, says Drazen Stilinovic, Quadrics General Manager. “For both of these systems, Quadrics has provided the right balance of products and services to ensure that delivery timescales and production goals were satisfied. The working relationship we have with LLNL has been of tremendous value to Quadrics and we look forward to providing the continued commitment necessary in this area”.