ACADEMIA

The Department of Energy’s (DOE’s) Oak Ridge National Laboratory (ORNL), Myricom, and Juniper Networks are demonstrating “wide-area” 100 Gigabit Ethernet (100 GbE) between the ORNL and Juniper Networks booths at SC11, the premier international conference on high performance computing. ORNL operates the DOE Office of Science-supported Jaguar supercomputer, America’s fastest open-research system. With the increasing size of scientific data sets and with the accelerating need to share these data sets between different organizations across the U.S. and around the world, this demonstration highlights data movement between distinct organizational domains that are bridged into a single Ethernet fabric using a high-performance, hardware-optimized communication software stack.

“The Oak Ridge Leadership Computing Facility provides world-class computational resources to teams of researchers from around the world,” said Galen Shipman, leader of ORNL’s Technology Integration Group. “Supporting access to and collaboration among these researchers requires the use of leading-edge technologies for data movement and high-performance remote access. Our work with Juniper and Myricom demonstrates advances in wide-area networking, end-system network architecture, and an optimized software stack for data movement.”

To create the unified Ethernet broadcast domain for the SC11 demonstration, each booth has a Juniper Networks MX 3D Universal Edge Router connected by a 100 GbE link, separate from SCinet’s backbone. These routers provide Virtual Private LAN Services to allow a machine in one domain to access a machine in another domain.

Connected to the routers in each booth is a single host with four Myricom network interface cards, each with dual 10-GbE ports. Each host is connected to the Juniper Networks router via eight 10-GbE links for a combined capacity of 80 gigabits per second (Gbps). Using a hardware-accelerated implementation of the Common Communication Interface (CCI), a single thread using CCI will be able to push 80 Gbps of data to the other host.

“In addition to this demonstration at SC11, both Juniper Networks and Myricom are providing ORNL the infrastructure required to connect to DOE’s Advanced Networking Initiative (ANI) 100G testbed, announced today,” Shipman said. The ANI project connects the National Energy Research Scientific Computing Center at Lawrence Berkeley National Laboratory, the Argonne Leadership Computing Facility at Argonne National Laboratory (ANL), and the Oak Ridge Leadership computing facility at ORNL via a 100 Gbps backbone.

Scott Atchley, an HPC system engineer at ORNL, added, “CCI is an open-source communication interface that aims to provide a simple and portable application programming interface, high performance, scalability for the largest deployments, and robustness in the presence of faults. CCI provides a common network abstraction layer for applications so that they can take advantage of high-performance networking hardware, such as the Myricom extreme performance Ethernet NICs [Network Interface Cards], when available, and standard hardware otherwise. Using CCI in combination with the single Ethernet fabric provided by the Juniper edge routers, we are able to illustrate how large data can be moved between DOE facilities and around the world.”

Researchers struggling to move the mountains of data that their petascale simulations generate welcome the enhanced capabilities. “100 Gbps is a critical technology in helping ORNL to meet its mission of driving new technological and scientific discoveries that have a long-term impact on critical issues like clean energy and national security,” said Luc Ceuppens, vice president of marketing, Platform Systems Division at Juniper Networks. “By demonstrating 100 GbE on Juniper Networks MX 3D Universal Edge Routers, ORNL is showcasing how it will maintain its world-class leadership in innovative research programs that advance the fundamental science behind economic progress today and tomorrow.”

Said Nan Boden, CEO of Myricom, “Extreme network performance is essential to Oak Ridge National Laboratory in its effort to push the boundaries of science through technological innovation. Myricom’s adaptable hardware platform enables development of new communication interfaces such as CCI. The ORNL/Juniper/Myricom 100GbE demonstration leverages Myri-10G 10-Gigabit Ethernet network adapters to show the benefits of CCI in a cutting-edge network.”