Ranger helps scientists model 3-D blood flow through the arterial tree: George Karniadakis’ conversion to computational science as a PhD student at the Massachusetts Institute of Technology was practically a matter of necessity. “After burning my eyes several times doing an arc welding experiment with very high radiation, I said, ‘Forget it, I’ll simulate the whole thing.’ From there I started simulating all kinds of things.”
Bats in flight, crashing waves, the jolt of churning pistons — as a professor of applied mathematics at Brown University, Karniadakis has simulated all of these phenomena. But for the last decade, he has focused his attention on computational biology, developing and optimizing algorithms to model blood flow through the human body. His research is in the spirit of the
Physiome Project, a corollary to the Human Genome Project that precisely maps the workings of the body, from the molecular level to the entire system, to establish essential information about how the body works. With research teams around the world working on computer models of the heart, the kidneys, the lungs, and other physiological systems, Karniadakis focuses on the arterial tree — the link that connects all of these systems — to reconstruct and simulate the arteries of a virtual human with greater accuracy than ever before. Using
Ranger, currently the world’s most powerful supercomputer for open science research, Karniadakis and his team will soon create the first fully three-dimensional model of the arterial tree. Integrating multiple scales of vessels throughout hundreds of connected branches, Karniadakis’ study may one day help clinicians obtain quick and accurate information about how a blocked artery will affect blood pressure or how efficiently drugs are dispersed in the bloodstream. His decision to study the arterial tree derived in part from past failures with single-system simulations. “We have seen, when we do simulations, that what happens in the carotid artery affects the entire brain, and in fact the entire body,” Karniadakis said. “These are global interactions, and unless you include them in a more integrated systems approach, you will miss important interactions, especially when you have pathologies.”
3D model of major vessels and bifurcations of the human arterial tree. Geometry of arterial walls is reconstructed from a set of CT and MRA images with gOREK, a computer software developed at Brown University. Colors represent different parts of the model. Left: Aorta and adjacent arteries. Right top: Cranial arterial network. Right bottom: Part of computational domain of the cranial arteries; the spectral-element method allows accurate representation of complex flows.
This insight led Karniadakis to model the vast highway of vessels that connects our vital organ and feeds our cells oxygen, energy, and other necessities. The arterial tree is composed of three systems: the macrovascular system, made up of arteries only a few millimeter in diameter but clearly visible under the skin; the mesovascular system with arterioles (100 microns in diameter) that split from the arteries like branches of a tree; and the microvascular system, made up of capillaries so narrow (5 microns in diameter) that red blood cells have to hunch to travel their length. To simulate this three-level arterial tree, computational scientists translate clinical data about blood pressures and flow rates from a real patient into a computational grid. This grid is mapped onto the complex geometric architecture of the blood vessels using a mesh of triangles, pyramids and tetrahedrons. As the elements that represent the distinct points of blood flow travel over hundreds of time-steps — the equivalent of dozens of cardiac cycles — they speed or slow, or head down a capillary, based on their location and flow rate within the arterial tree. Even with Ranger’s unprecedented computational power, it is impossible to fully model these three scales of vessels. “Our body has several thousands of arteries, millions of arterioles, and billions of capillaries,” Karniadakis explained. But, with thousands of processors and careful computational algorithms, Karniadakis and his team can get closer than they ever have before. “At the macrovascular level, we know the equations and the geometry and everything is very accurate,” Karniadakis said. “But at the mesoscopic and microscopic level, we cannot image the vessels, therefore we don’t know their exact diameter. So we have to improvise.” This improvisation includes adapting the algorithms from soil geology to describe the smallest scales of vessels, where capillaries act like a porous medium, delivering nutrients to the cells. By describing the three scales algorithmically and coupling the different models in the computer code, Karniadakis is able to take into account the billions of vessels in the human body, providing greater accuracy and predictive value. The evolution of Karniadakis’ research reflects the incredible strides that high performance computing resources and computational science have taken together over the last decade. “In the past, myself and others were doing just a single bifurcation,” or splitting of an artery, Karniadakis noted. “Now, three years later, we’re simulating a hundred arteries.”
Flow pattern in the Circle of Willis. Arrows depict the instantaneous flow direction. The simulation indicates that during part of a cardiac cycle blood supplied through the left carotid arteries feeds both hemispheres of the brain.
Not only has Karniadakis’ model grown from one artery to hundreds in three years, his simulations have evolved from one dimensional flow charts to fully three-dimensional interactive movies. His arterial simulations now contain half a million elements, and because of the multiple degrees of freedom for each element, the number of parts rises to 200 to 400 million. “The complexity of the problem is tremendous,” Karniadakis exclaimed. Leopold Grinberg, a PhD student at Brown, and Karniadakis’s programming guru on the arterial tree project, has spent three years optimizing the spectral element analysis code called NekTar, which their team uses to model arterial blood flow. Scaling their code from hundreds of processors to Ranger’s tens of thousands of parallel processors is a non-trivial problem, and has required a reevaluation of the basic structure and Message Passing Interface of their code. “In scaling our code, we try to take advantage of the hierarchical architecture of Ranger. We want to maximize the fast communication and minimize the slower communication,” Grinberg explained. “These are challenges for everyone working on Ranger. You can’t model the code on a hundred processors and claim that it will run the same of 30,000 processors. It’s a very painstaking process.” Their computational problem requires not only thousands of processors and good communication, but also heavy-duty memory capacity and fast I/O speeds to efficiently collate and checkpoint the streams of data emerging from Ranger’s parallel processors. Using the early user period, Karniadakis and Grinberg scaled up their research up to 11,152 processors, working in an iterative fashion to improve their computational codes for this system. Once the codes are fully optimized, they’ll be able to see the circulatory system in action as no one ever has before. “Computers like Ranger allow us to implement our vision of fully simulating the macrovascular network for the first time ever,” Karniadakis said. “It lets us incorporate the multi-scalar nature of the problem and opens up new possibilities that didn’t exist before.” Connecting the body with his intensely modeled arterial tree, Karniadakis acts as a synthesizer of physiological knowledge, whose computational simulations and creative parallel computing solutions are helping the virtual human evolve towards reality. ********************************************************** To follow Dr. Karniadakis' modeling of the arterial tree, visit the
Crunch group at Brown University. Aaron Dubrow
Texas Advanced Computing Center
Science and Technology Writer
March 26, 2008
