IBM and Uppsala University and the Swedish Institute of Space Physics announced today a major new Stream Computing project to analyze massive volumes of information in real time to better understand “space weather”. By using IBM InfoSphere Streams to analyze data from sensors that track high frequency radio waves, endless amounts of data can be captured and analyzed on the fly. Over the next year, this project is expected to perform analytics on at least 6 gigabytes per second or 21,600 gigabytes per hour – the equivalent of all the Web pages on the Internet.

Scientists sample high frequency radio emissions from space to study and forecast “space weather” or the effect of plasma eruptions on the sun that reach the earth and adversely affect energy transmission over power lines, communications via radio and TV signals, airline and space travel, and satellites. However, the recent advent of new sensor technology and antennae arrays means that the amount of information collected by scientists surpassed the ability to intelligently analyze it. IBM InfoSphere Streams, new software derived from IBM Research project System S, enables large volumes of data to be analyzed in real time making an entirely new level of analytics possible.

“IBM InfoSphere Streams is opening up a whole new way of doing science, not only in this area, but any area of e-Science where you have lots of data coming in from external sources and sensors, streaming at such high data rates you can’t handle it with conventional technology,” said Dr. Bo Thidé, professor and head of research, Swedish Institute of Space Physics and director of LOIS Space Center. “It has helped create a paradigm shift in the area of online observation of the earth, space, sun and atmosphere.” 

Sunspot activity, electromagnetic storms, and other types of solar activity can impact communications signals. As critical infrastructure such as power grids and telecommunications networks become more digitally aware, instrumented and interconnected, it is increasingly important to understand how these can be affected by influences such as electromagnetic interference or other changes in the atmosphere. 

Researchers at Uppsala University and the Swedish Institute of Space Physics worked with the LOIS Space Center facility in Sweden to develop a new type of tri-axial antenna that streams three-dimensional radio data from space, extracting a magnitude more physical information than any other type of antennae array before. Since researchers need to measure signals from space over large time spans, the raw data generated by even one antenna quickly becomes too large to handle or store.

“We’ve embarked upon an entirely new way of observing radio signals using digital sensors that produce enormous amounts of data,” Thidé said. “With this type of research, you have to be able to analyze as much data as possible on the fly. There is no way to even consider storing it. InfoSphere Streams is playing a pivotal role in this project. Without it, we could not possibly receive this volume of signals and handle them at such a high data rate because until now, there was not a structured, stable way of analyzing it.”

The technology addresses this problem by analyzing and filtering the data the moment it streams in, helping researchers identify the critical fraction of a percent that is meaningful, while the rest is filtered out as noise. Using a visualization package, scientists can perform queries on the data stream to look closely at interesting events, allowing them not only to forecast, but to nowcast events just a few hours away. This will help predict, for example, if a magnetic storm on the sun will reach the earth in 18-24 hours.