SCIENCE

Zoom down to one artery in your body, and the commotion is constant: blood cells hurtle down the passage with hundreds of their kin, bumping against other cells and the walls as they go. The many variables—and the sheer immensity of the human circulatory system—have kept scientists from closely documenting the rough-and-tumble life inside blood vessels.

This is an area of science called "biophysics", for the forces that govern red blood cells' movements at this level are best described by the laws of physics and can be mapped with mathematics. That's exactly what a team of scientists from Brown University led by G. E. Karniadakis and the U.S. Department of Energy's (DOE) Argonne National Laboratory are doing on the lab's supercomputer, hoping that a better map will lead to better diagnoses and treatments for patients with blood flow complications.

Though we've come a long way from the ancient Greeks, who believed blood came from the liver, there's a surprising amount that we don't know about blood. Newer, faster supercomputers have allowed scientists to create detailed models of blood flow that help doctors understand what happens at the molecular level and, consequently, how heart and blood diseases can be treated.

Argonne's Blue Gene/P supercomputer, housed at the Argonne Leadership Computing Facility (ALCF), allows scientists to tackle these immense problems with the power of 500 trillion calculations per second.

One part of the study is mapping exactly how red blood cells move through the brain. For example, last year the team used similar modeling to discover that the malaria parasite makes its victims' red blood cells 50 times stiffer than normal.

Healthy red blood cells are smooth and elastic; they need to squeeze and bend through tiny capillaries to deliver blood to all areas of the brain. But malaria-infected cells stiffen and stick to the walls, creating blockages in arteries and vessels. Malaria victims die because their brain tissues are deprived of oxygen. A more complete picture of how blood moves through the brain would allow doctors to understand the progression of diseases that affect blood flow, like malaria, diabetes and HIV.

"Previous computer models haven't been able to accurately account for, say, the motion of the blood cells bending or buckling as they ricochet off the walls," said Joe Insley, a principal software developer at Argonne who is working with the team. "This simulation is powerful enough to incorporate that extra level of detail."

Another part of the study seeks to understand the relationship between cerebrospinal fluid and blood flow in the brain. "Blood vessels expand if blood pressure is high; and since they are located between brain tissues, this can put dangerous pressure on the brain," said Leopold Grinberg, a Brown University scientist on the team. In healthy people, spinal fluid can drain to relieve pressure on brain tissues, but occasionally the system breaks down—leaving the brain vulnerable to damage.

"Understanding how the system interacts will allow us to more accurately treat the problem," Grinberg said.

But before the simulations are even run, there's a hurdle that researchers must face.

It is a peculiarity of large computers that code for one computer doesn't always work well on another. A code written for a computer with two cores—what's probably in your home computer—doesn't translate well into a computer that has 160,000 cores, as Argonne's Blue Gene/P does.

"I liken it to driving the family car on the Daytona 500 racetrack," explained Michael Papka, deputy associate laboratory director for Computing, Environment and Life Sciences at Argonne. "What may be suitable for driving around town isn't designed for high-speed racing. The ALCF staff helps the researchers rework their code for optimal performance on the big machines."

For example, each core, performing its own small slice of the work, has to transmit its data to other cores once it finishes a particular task. If the work isn't equally distributed, some cores might finish earlier than others and sit idle as they wait for the others to catch up. Or if the network connecting them isn't well managed, the transmission of data might slow down the whole process.

Because each supercomputer is individually designed, the Blue Gene/P's architecture is different from other supercomputers.

"For example, one of the Blue Gene/P's strengths is good interconnects," said Vitali Morozov, a computational scientist at the ALCF. "The cores are beautifully arranged, and if you know how to use them it's very efficient—but it's tricky." Thus, to get the best performance out of the machine, the code has to be tuned to fit the computer.

The team was allotted 50 million processor-hours on the Blue Gene/P through DOE's Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program. INCITE is a DOE program supported by the Office of Science's Office of Advanced Scientific Computing Research that provides access to computing power and resources to support computationally intensive, large-scale research projects to researchers from industry, academia and government research facilities.