Anomalo has announced availability on Databricks Partner Connect. Partner Connect helps Databricks customers discover validated data, analytics, and AI tools and integrate these tools directly within their Databricks Lakehouse. One of Databricks’ first partners in data quality, Anomalo is providing an exclusive free trial to Databricks customers so that they can start monitoring their tables immediately to detect and root cause data quality issues if they are not already Anomalo customers.

“Data quality has become a foundational part of the modern data stack,” said Roger Murff VP, ISV Partners at Databricks. “Businesses shouldn’t have to wait until data quality issues pile up when they become harder and more costly to fix. With Databricks Partner Connect, we’re excited about the potential to help enterprises be proactive about data quality, allowing them to try out Anomalo with a few clicks and instantly see the value of high-coverage, accessible, automated monitoring.”

Databricks combines the best of data warehouses and data lakes to offer an open and unified platform for data and AI. This helps organizations streamline their data ingestion and management and make that data available for everything from business decision-making to predictive analytics and machine learning.

However, dashboards and data-powered products are only as good as the quality of the data that powers them. When scaling their data efforts, many companies quickly encounter one unfortunate fact: much of their data is missing, stale, corrupt, or prone to unexpected and unwelcome changes. As a result, companies spend time dealing with issues in their data rather than unlocking that data’s value and are at risk of silent failures in the data that might go undetected for months or more.

Anomalo addresses the data quality problem by monitoring enterprise data and automatically detecting and root-causing data issues, allowing teams to resolve any hiccups with their data before making decisions, running operations, or powering models. Anomalo leverages machine learning to uncover a wide range of data failures with minimal human input. This is in contrast to legacy approaches to monitoring data quality that require extensive work writing data validation rules or setting limits and thresholds.

With today’s announcement, Databricks customers can now begin monitoring the quality of the data in their Databricks Lakehouse immediately within their Databricks workspace.

“Whether you’re using your Databricks Lakehouse for analytics or machine learning and AI, your results are only as good as the quality of the underlying data. So, we’re excited to partner with Databricks to give their customers a great tool for automatically detecting and understanding the root causes of data issues – thus preventing such issues from leading to incorrect BI dashboards or broken machine learning models,” said Elliot Shmukler, co-founder and CEO of Anomalo.

First direct visual evidence – ring-like structure like M87* - Theoretical Physicists of Goethe University Frankfurt instrumental in interpreting the data

The image is a long-anticipated look at the massive object that sits at the very center of our galaxy. Scientists had previously seen stars orbiting around something invisible, compact, and very massive at the center of the Milky Way. This strongly suggested that this object — known as Sagittarius A* (Sgr A*, pronounced "sadge-ay-star") — is a black hole, and today’s image provides the first direct visual evidence of it. 

Although we cannot see the black hole itself, because it is completely dark, glowing gas around it reveals a tell-tale signature: a dark central region (called a “shadow”) surrounded by a bright ring-like structure. The new view captures light bent by the powerful gravity of the black hole, which is four million times more massive than our Sun.

“We were stunned by how well the size of the ring agreed with predictions from Einstein’s theory of general relativity,” says EHT Project Scientist Geoffrey Bower from the Institute of Astronomy and Astrophysics, Academia Sinica, Taipei. “These unprecedented observations have greatly improved our understanding of what happens at the very center of our galaxy and offer new insights on how these giant black holes interact with their surroundings.”

Because the black hole is about 27,000 light-years away from Earth, it appears to us to have about the same size in the sky as a donut on the Moon. To image it, the team created the powerful EHT, which linked together eight existing radio observatories across the planet to form a single “Earth-sized” virtual telescope. The EHT observed Sgr A* on multiple nights, collecting data for many hours in a row, similar to using a long exposure time on a camera.

The enormous amount of observational data collected had to be interpreted theoretically. For this, a research team led by theoretical astrophysicist Luciano Rezzolla from Goethe University Frankfurt used supercomputers to simulate how a black hole could look like when observed by the EHT – based on what had already been known about Sgr A*. In this way, the scientists created a library of millions of images. Then, they compared this image library with the thousands of different images of the EHT to deduce the properties of Sgr A*.

The breakthrough follows the EHT Collaboration’s 2019 release of the first image of a black hole, called M87*, at the center of the more distant Messier 87 galaxy.

The two black holes look remarkably similar, even though our galaxy’s black hole is more than a thousand times smaller and less massive than M87*. “We have two completely different types of galaxies and two very different black hole masses, but close to the edge of these black holes they look amazingly similar,” says Sera Markoff, Vice-Chair of the EHT Science Council and a professor of theoretical astrophysics at the University of Amsterdam, the Netherlands. “This tells us that general relativity governs these objects up close, and any differences we see further away must be due to differences in the material that surrounds the black holes.”

This achievement was considerably more difficult than for M87*, even though Sgr A* is much closer to us. EHT scientist Chi-kwan (‘CK’) Chan, from Steward Observatory, the Department of Astronomy and the Data Science Institute at the University of Arizona, US, explains: “The gas in the vicinity of the black holes moves at the same speed — nearly as fast as light — around both Sgr A* and M87*. But where gas takes days to weeks to orbit the larger M87*, in the much smaller Sgr A* it completes an orbit in mere minutes. This means the brightness and pattern of the gas around Sgr A* was changing rapidly as the EHT Collaboration was observing it — a bit like trying to take a clear picture of a puppy quickly chasing its tail.”

The researchers had to develop sophisticated new tools that accounted for the gas movement around Sgr A*. While M87* was an easier, steadier target, with nearly all images looking the same, that was not the case for Sgr A*. The image of the Sgr A* black hole is an average of the different images the team extracted, finally revealing the giant lurking at the center of our galaxy for the first time.

The effort was made possible through the ingenuity of more than 300 researchers from 80 institutes around the world that together make up the EHT Collaboration. In addition to developing complex tools to overcome the challenges of imaging Sgr A*, the team worked rigorously for five years, using supercomputers to combine and analyze their data, all while compiling an unprecedented library of simulated black holes to compare with the observations.

Luciano Rezzolla, professor of Theoretical Astrophysics at Goethe University Frankfurt, explains: “The mass and distance of the object were known very precisely before our observations. We thus used these tight constraints on the size of the shadow to rule out other compact objects – such as boson stars or wormholes – and conclude that: ‘What we're seeing definitely looks like a black hole!’”

Using advanced numerical codes, theorists in Frankfurt have performed extensive calculations on the properties of the plasma accreting onto the black hole. Rezzolla: “We managed to calculate three million synthetic images varying the accretion and radiation emission models, and considering the variations seen by observers at different inclinations with respect to the black hole.”

This last operation was necessary because the image of a black hole can be radically different when seen by observers of different inclinations. “Indeed, a reason why our images of Sgr A* and M87* are rather similar is because we’re seeing the two black holes from an almost identical angle,” Rezzolla explains.

“To understand how the EHT has produced an image of Sgr A* one can think of producing a picture of a mountain peak based on a time-lapse video. While most of the time the peak will be visible in the time-lapse video, there are times when it is not because it is obscured by clouds. On average, however, the peak is clearly there. Something similar is true also for Sgr A*, whose observations lead to thousands of images that have been collected in four classes and then averaged according to their properties. The result is a clear first image of the black hole at the center of the Milky Way.” Rezzolla concludes.

Scientists are particularly excited to finally have images of two black holes of very different sizes, which offers the opportunity to understand how they compare and contrast. They have also begun to use the new data to test theories and models of how gas behaves around supermassive black holes. This process is not yet fully understood but is thought to play a key role in shaping the formation and evolution of galaxies.

“Now we can study the differences between these two supermassive black holes to gain valuable new clues about how this important process works,” says EHT scientist Keiichi Asada from the Institute of Astronomy and Astrophysics, Academia Sinica, Taipei. “We have images for two black holes — one at the large end and one at the small end of supermassive black holes in the Universe — so we can go a lot further in testing how gravity behaves in these extreme environments than ever before.”

Progress on the EHT continues: a major observation campaign in March 2022 included more telescopes than ever before. The ongoing expansion of the EHT network and significant technological upgrades will allow scientists to share even more impressive images as well as videos of black holes shortly.