- The Hobby-Eberly Telescope Experiment (or HETDEX) seeks to measure the expansion of the Universe in order to better understand dark energy.
- The HETDEX team uses RAIC to label artifacts in their massive dataset of telescope imagery, increasing their usable data by about 50%.
- RAIC massively accelerates the analysis of visual data and opens new avenues for discovery.
The Biggest Mystery in the Universe
There’s a lot we don’t know about the Universe. For Karl Gebhardt, an astronomer and professor at The University of Texas at Austin, the unanswered questions are a fascination, and his life’s work. “I’ve always been excited by trying to find the things you can’t see,” Gebhardt said. “Most of my research has been in measuring black hole masses. You can’t ‘see’ them, but by using supercomputers, writing my own code, and modeling galaxies, I’ve measured them by how fast stars are moving around them.”
In the late ‘90s, it was discovered that the Universe was expanding more quickly than previously known. The expansion couldn’t be explained by existing frameworks. Enter the concept of dark energy: the unknown component of the Universe driving its expansion.
Dark energy is arguably the biggest mystery in the Universe. It’s not an exaggeration to say that we know nothing about it, and that it could explain everything.
Gebhardt was immediately drawn to the concept. He said, “In 2000, I came to UT Austin as a professor. We have one of the largest telescopes in the world, and one of the most powerful supercomputers — I thought, ‘Surely there’s a way to get some answers.’”
His thinking went something like this: if unmeasurable phenomena like black holes could be measured by their effects on the visible objects around them, the same could be done with dark energy, by measuring the clustering of galaxies.
That idea would become HETDEX — the Hobby-Eberly Telescope Dark Energy Experiment — an ongoing project to map the early Universe, allowing for calculations of expansion at different points in the Universe’s history. These measurements will help narrow down the numerous competing explanations of dark energy. The HETDEX team of researchers expects to complete their work in June 2024.
By the time they’re done, they’ll have taken and analyzed one trillion resolution elements, or discrete independent measurements of an area in the Universe, across one billion spectra. Enabling the analysis is Gebhardt’s own software, which he coded for the experiment.
By the time they’re done, they’ll have taken and analyzed one trillion resolution elements, or discrete independent measurements of an area in the Universe, across one billion spectra. Enabling the analysis is Gebhardt’s own software, which he coded for the experiment.
The world-class telescope instrument used by HETDEX records spectral data — which goes beyond the spectrum of visible light to capture a "cosmic footprint" of each of the one billion points measured by the project.
Credit: G. Zeimann/HETDEX Collaboration More about this image
With the sheer quantity of data, there were bound to be some artifacts, or data imperfections, to be dealt with. Accuracy matters, especially when you’re calculating a new clue to the biggest mystery in the Universe.
RAIC: A Partner in Exploration and Discovery
Gebhardt explained that the artifacts in the data would require a new solution: “There’s one type of defect, as an example, that creates a weak vertical streak in the image. Because it’s so weakly correlated, it’s difficult to write code that can detect it. But the human eye picks it up instantly. I realized that if the eye could do it, so could machine learning.”
RAIC, or Rapid Automatic Image Categorization, made it possible for HETDEX to label and categorize anomalies in their trillion-resolution-element unlabeled data— no small feat for a team of five.
“Without this tool, the data artifacts would have been an intractable problem,” said Erin Mentuch Cooper, a research scientist at UT Austin and the HETDEX Data Manager. “In RAIC, instead of having to manually label hundreds of thousands of examples to build a model, we were able to prototype from a single example, and iterate at a quick pace.”
Given just one example image, RAIC searches your entire dataset in seconds, returning a context map of objects sorted by similarity to the seed image and to each other. Users can then select the best matches for their object of interest to improve the search.
HETDEX uses RAIC to categorize artifacts in their spectral data, which they've ingested as image files. RAIC's human-machine collaboration brings together the speed-at-scale of an AI model with the user's expertise.
Exploring vast quantities of data in RAIC, Mentuch Cooper started detecting patterns in the anomalies that otherwise may have gone unrecognized. “There were relatively rare artifacts we’d noticed from the start of the survey that suddenly became more meaningful once we could quantify them in RAIC,” she said. Namely, the number of satellite detections within the HETDEX observations correlated to the rate of new satellites launched into Earth’s orbit.
This is exactly how the loop of human-AI collaboration within RAIC is meant to work: as a user asks questions of their data in RAIC, RAIC surfaces additional insights from the data, which then inspire new questions, and so on. All the while, RAIC’s classifier is rapidly trained, one “nudge” from the user at a time.
RAIC’s speed and sophisticated algorithm make it a powerful tool for analysis. Its interactivity and flexibility make it a partner in exploration and discovery.
Mentuch Cooper felt that partnership as soon she started using RAIC: “This is what scientists love to do. When something works, we want to explore it. RAIC allows you to see what works and what doesn’t.”
She foresees additional uses for RAIC within her field of astronomy research, including analyzing 3D representations of HETDEX data. “We capture data in the spectral direction and the spatial direction, so you can imagine creating cubes of our data and not only doing object detection and classification, but simultaneously measuring galaxy distances, as well.”
For the HETDEX team, RAIC was a force multiplier of their work. Mentuch Cooper said:
“RAIC allowed us to do the work of a much, much larger team. The scale of our data is massive, and RAIC made it possible to keep it all clean.”
Gebhardt quantified RAIC’s impact on the project: “RAIC gave us the confidence to push the data harder and use more sources, knowing RAIC would be able to keep out the ‘junk.’ That increased the size of our useable data sample by about 50%.”
(A 50% increase in Universe? Not a bad ROI.)
Final Thoughts
Humans have been asking questions about the Universe for as long as we’ve been around. We build tools to help us find the answers — telescopes, supercomputers, artificial intelligence — but it’s ultimately up to us to explore the mysteries.
As an intuitive, collaborative AI tool, RAIC accelerates data analysis, and opens new avenues for discovery.
As HETDEX shows, RAIC makes discovery easier, faster, and more scalable so that you can make discoveries.
The Hobby-Eberly Telescope, one of the largest optical telescopes in the world, captures the data that HETDEX will analyze to further our understanding of the Universe. Photo credit: Bill Nowlin Photography