ACADEMIA

In a large, loud computer equipment room at NASA's Goddard Space Flight Center, amidst the humming of fans and trilling of transistors, is a gadget about the size of a small paperback. Network engineers call them "pluggables." These devices can pump data into a fiber optic line at rates up to 100 gigabits per second (100 Gbps).

That's "gigabit" as in "a billion bits." It is 10,000 times faster than a typical broadband cable modem connection, which operates at a mere 10 million bits per second, or 10 Mbps. 100 Gbps is fast enough to transfer a 25 Gb Blu-ray (HD) movie over the Internet in 2 seconds flat.{modal url=http://www.supercomputingonline.com/images/sc11_demo-team_680.jpg}{/modal}

A data superhighway as speedy as this one doesn't come cheap. The pluggable across the hall costs nearly as much as a luxury sports car. It converts electronic signals into pulses of laser light that travel down fiber optic wires and zip out onto the Internet at near-light speed.

A team of Goddard network engineers borrowed two of the super-fast 100 Gbps pluggables in preparation for a major technology demonstration in Seattle at the Supercomputing 2011 (SC11) conference, November 12-18. The demo gave the high-performance computing world a glimpse of how the Internet will be used in the future to conduct research involving extraordinarily large transfers of data.

Today's leading edge is known as petascale computing. That's "peta" as in "petabyte," or 1,000 terabytes. One terabyte is 1,000 gigabytes, and a gigabyte is 1,000 megabytes, so a petabyte is a LOT of data. A petabyte of storage could hold 40,000 Blu-ray movies.

Now computer scientists are even talking about the coming era of exascale computing: "exa" as in one quintillion bytes, or one billion gigabytes.

Connected to the MAX

Quite a bit of network "plumbing" is required to move data at this scale. Various pieces of equipment and fiber-optic connections had to be put in place to create a 100 Gbps network connection from Goddard to Seattle.

The everyday Internet can't transfer data anywhere near 100 Gbps. The Goddard team had to make special arrangements with various entities on the Internet to create a big enough data pipeline. This required connecting multiple high-speed, fiber-optic research and education networks.

In Goddard's local neighborhood, a key partner in the demo was the Mid-Atlantic Crossroads — affectionately known in the network world as "the MAX," which is based in College Park, Md. The MAX provides Goddard access to Internet2 and other high-speed networks, such as ESnet, SCinet, and Starlight.{modal url=http://www.supercomputingonline.com/images/sc11_nasa-booth_800.jpg}{/modal}

Specialized equipment, some of it not yet released on the open market, is also required to switch and route data at such blisteringly high speeds. NASA's vendor partners loaned leading-edge network and server technologies worth around $2 million for the SC11 demonstrations. These companies include Acadia Optronics, Alcatel-Lucent, Brocade, Ciena, cPacket, Force10, Fujitsu, Intel, Juniper, and Supermicro.

After much pre-meeting preparation, network engineers Bill Fink, Paul Lang, and Jeff Martz, members of Goddard's High End Computer Networking (HECN) team, flew out to Seattle to demonstrate technology that could be used to transfer exceedingly large sets of data across a 100 Gbps network. Crates of computer workstations and networking equipment were shipped to the site. The team used the equipment to build a local network at the convention center, connecting the booths of NASA and its various demo partners.

SC11 demo

The demos consisted of transferring data between a computer workstation at Goddard and the NASA booth at SC11, and also among the booths of NASA and its partner organizations at SC11: Ciena, the SCinet Network Operations Center, and the University of Illinois at Chicago Laboratory for Advanced Computing/Northwestern University International Center for Advanced Internet Research.

One of the toughest tricks in high-performance networking is a so-called "disk to disk" (DTD) transfer of large data sets. In a DTD transfer, the speed limit is determined by how fast the data moves between two hard drives.

Hard drives are a notorious bottleneck in high-performance networks. In a typical personal computer, the fastest hard drives can transfer data at about 130 megabytes per second, or roughly 1,000 Mbps.

But the Goddard team has developed innovative ways to set up the hardware, software, and networking paths to allow the fastest possible data flow. One important dimension of this is using solid-state memory drives, which are scaled up versions of the small thumb drives people use to transport files between computing devices.

The computer workstation at Goddard used for disk-to-disk networking experiments contains some 40 Vertex3 120-Gb solid-state hard drives. Each drive can spew data at a maximum rate of 550 Mb per second — fast enough to download hundreds of iTunes songs per second. You would need more than 100 conventional spinning hard drives to match the performance of 40 solid-state drives.{modal url=http://www.supercomputingonline.com/images/sc11_gear-display_800.jpg}{/modal}

Last year, at Supercomputing 2010, the team demonstrated disk-to-disk transfers up to 30 Gbps. This year they doubled it to 60 Gbps, transferring data between the NASA booth in Seattle and the workstation at Goddard. The top-speed disk-to-disk demo this year transferred data between two booths in Seattle at 72 Gbps (9 GB per second). That is equivalent to a Blu-ray movie download every 3 seconds.

The NASA networking research has its immediate applications in high-performance computing for understanding and predicting climate change, as well as other computer modeling tasks, carried out at the NASA Center for Climate Simulation and other organizations. Ultimately, networking technology enhances our ability to use computers to solve pressing social problems.