SCIENCE

Public-Private Collaborative Demo Leveraged New ESnet Advanced Networking Initiative (ANI) Testbed

Researchers at the Department of Energy’s Energy Sciences Network (ESnet), Orange Silicon Valley and Bay Microsystems have collaborated to showcase the world’s first 40 Gbps (gigabits per second) wide area network demonstration of RDMA over InfiniBand®. RDMA, or Remote Direct Memory Access, is an alternative model to TCP/IP for bulk data transfer designed to significantly improve the utilization of available bandwidth resources and the processing efficiency of the end hosts ‐ and does so without the need for any special network tuning.

While RDMA over InfiniBand is not a new concept, this experiment marks the first time that it has been demonstrated at 40 Gbps data rates over long-haul distances, in this case approximately 200 miles. The demonstration showed that using RDMA moves data at up to 96 percent of the peak capacity of the network as opposed to other network technologies where utilization can drop significantly over longer distances.

“The volume of science data that needs to be shared among global research sites is reaching levels beyond what current 10 Gbps networks and traditional network protocols can handle,” said Brian Tierney, ESnet Staff Scientist and lead for the Advanced Networking Initiative (ANI). “RDMA communication opens new possibilities for researchers by enabling quick and efficient data transfer for ‘big data’ applications. It does so by eliminating many of the typical network performance bottlenecks and helps users make much more efficient use of the WAN infrastructure ‐ this helps to ensure that scientific progress is unimpeded even as data needs grow.”

The long-haul RDMA demonstration is part of a research project being conducted at Brookhaven National Laboratory (BNL) in New York and leverages ESnet’s Advanced Networking Initiative (ANI) 100 Gbps testbed. The testbed is being made available to researchers both in the public and private sector as a first of its kind platform to test different and potentially disruptive approaches to new network protocols and architectures in a greater than 10 Gbps network environment. The testbed is currently available in the Long Island, New York region but will soon have a national footprint to enable experimentation at even greater distances.

“While RDMA over extended distances has been tested and shown to scale up to 10 Gbps data rates, this is the first time RDMA over distance has been proven to work at full bandwidth for 40 Gbps data rates” said Eric Dube, Senior Product Manager of Systems at Bay Microsystems, Inc. “Gaining access to a 40 Gbps wide area optical circuit is very costly and had prohibited this kind of research in the past. Using the ANI testbed, we are now able to prove these concepts in a live network environment setting the stage for deploying scalable RDMA-enabled applications over 100G networks. This is especially important as more geographically dispersed data centers and science sites will require this type of bandwidth and capability.”

RDMA works by enabling network adapters to transfer data to or from application memory, eliminating the overhead of copying data between application memory and the data buffers in the operating system. When an application performs an RDMA operation, the application data is delivered directly to the network eliminating the additional latency the transfer would incur if the data were to be copied into the operating system memory buffers. This dramatically increases the application-messaging rate and increases the utilization of available network bandwidth.

Bay Microsystems provided its IBEx G40 40 Gbps InfiniBand WAN acceleration platform as well as provided input on the network design, bring-up, and overall support during the testing effort. Bay specializes in providing secure, high-performance networking solutions to commercial enterprises, government agencies and service providers. In particular, Bay delivers a portfolio of wide area network acceleration solutions to data centers to maximize the utilization of their computation, storage, and interconnect assets, and increase their security and reliability.

Dube added, “Applications that combine RDMA with InfiniBand achieve additional benefits including reliable data transfer, sustained high bandwidth utilization, and reduced CPU overhead, all critical items to consider when ‘Big Data’ Cloud computing, high frequency trading, and high performance computing applications are deployed across a WAN infrastructure.”

Orange is an international operator for mobile and broadband Internet services and, under the brand Orange Business Services, is one of the world leaders in providing telecommunication services to multinational companies. Orange Silicon Valley, a US subsidiary of France Telecom-Orange, participated in the project to explore RDMA’s potential for meeting the growing challenge of the company and its enterprise customers.

Soumik Sinharoy, Project Manager at Orange Silicon Valley, said “Orange believes that the high bandwidth efficiency that comes with RDMA would be beneficial considering the significant increase in traffic anticipated across our enterprise and consumer markets. Orange is also using this opportunity to evaluate how increased efficiency could help us increase the utilization of our existing assets.”

By using the IBEx G40 40 Gbps InfiniBand WAN acceleration platform and Infinera’s DTN switched DWDM system along with high-performance InfiniBand-connected hosts provided by BNL, researchers were able to extend 4X InfiniBand QDR outside the data center and transport it on a 40 Gbps wavelength over the 370.36 km fiber loop of ESnet’s wide area network circuit from BNL in Upton, NY to Manhattan, NY. The IBEx G40, provides enhanced buffering and credit management allowing all native InfiniBand protocols to be extended over SONET OC-768/SDH STM-256 or ITU-T G.709 OTU3 at 40 Gbps data dates over thousands of miles.

The proof is in the results where application testing produced near theoretical line rate performance across the WAN circuit with RDMA-enabled application communication providing 96 percent link efficiency. These improvements can be attributed to the lossless nature of InfiniBand allowing for reliable transport when used as a wide area network technology and the reduced protocol overhead capabilities of RDMA.

Further demonstrations will be presented at the SC11 conference being held Nov. 12-18 in Seattle. Researchers will build a 40 Gbps transcontinental network circuit enabling RDMA application data transfers over InfiniBand at near line-rate performance.