Key Takeaways:
- Researchers have proposed a unifying mathematical framework to explain why many successful multimodal AI systems work
- The framework, called the Variational Multivariate Information Bottleneck Framework, helps organize and guide the decision process for designing AI systems
- The framework links the design of loss functions directly to decisions about which information should be preserved and which can be ignored
- The approach has the potential to reduce the amount of computational power needed to run an AI system and make it less environmentally harmful
- The framework can be used to propose new algorithms, predict which ones might work, estimate the needed data for a particular multimodal algorithm, and anticipate when it might fail
Introduction to Multimodal AI
Artificial intelligence is increasingly relied on to combine and interpret different kinds of data, including text, images, audio, and video. However, one obstacle that continues to slow progress in multimodal AI is deciding which algorithmic approach best fits the specific task an AI system is meant to solve. As Ilya Nemenman, Emory professor of physics and senior author of the paper, says, "We found that many of today’s most successful AI methods boil down to a single, simple idea — compress multiple kinds of data just enough to keep the pieces that truly predict what you need." This idea has led to the development of a new framework that helps explain why many successful multimodal AI systems work.
A Unifying Framework
The researchers at Emory University developed a mathematical framework that brings structure to how algorithms for multimodal AI are derived. The framework, called the Variational Multivariate Information Bottleneck Framework, links the design of loss functions directly to decisions about which information should be preserved and which can be ignored. As co-author Michael Martini explains, "Our framework is essentially like a control knob. You can ‘dial the knob’ to determine the information to retain to solve a particular problem." This approach has the potential to reduce the amount of computational power needed to run an AI system and make it less environmentally harmful.
The Development Process
The researchers brought a unique perspective to the problem of optimizing the design process for multimodal AI systems. As physicists, they wanted to understand how and why something works, rather than just achieving accuracy in a system. As Eslam Abdelaleem, first author of the paper, explains, "The machine-learning community is focused on achieving accuracy in a system without necessarily understanding why a system is working. As physicists, however, we want to understand how and why something works." The team spent years working on mathematical foundations, discussing them with Nemenman, trying out equations on a computer, and repeating these steps after running down false trails.
The Eureka Moment
The researchers vividly recall the day of their eureka moment. They had come up with a unifying principal that described a tradeoff between compression of data and reconstruction of data. As Martini says, "We tried our model on two test datasets and showed that it was automatically discovering shared, important features between them. That felt good." Abdelaleem’s Samsung Galaxy smart watch, which uses an AI system to track and interpret health data, even misinterpreted his racing heart as three hours of cycling, highlighting the excitement and impact of their discovery.
Applying the Framework
The researchers applied their framework to dozens of AI methods to test its efficacy. As Nemenman says, "We performed computer demonstrations that show that our general framework works well with test problems on benchmark datasets. We can more easily derive loss functions, which may solve the problems one cares about with smaller amounts of training data." The framework also holds the potential to reduce the amount of computational power needed to run an AI system, making it less environmentally harmful. The researchers hope others will use the generalized framework to tailor new algorithms specific to scientific questions they want to explore.
Future Directions
The researchers are building on their work to explore the potential of the new framework. They are particularly interested in how the tool may help to detect patterns of biology, leading to insights into processes such as cognitive function. As Abdelaleem says, "I want to understand how your brain simultaneously compresses and processes multiple sources of information. Can we develop a method that allows us to see the similarities between a machine-learning model and the human brain? That may help us to better understand both systems." The framework has the potential to open up new avenues of research and discovery, and the researchers are excited to see where it will lead.
Scientists Create a “Periodic Table” for Artificial Intelligence