Key Takeaways
- Auditing AI is essential to move the technology from “testing the water” to a trusted, routine service—much like drinking tap water after years of safety inspections.
- Effective audits require a skeptical mindset, continuous monitoring, and a blend of technical, statistical, and communication skills.
- The book Auditing AI (MIT Press, 2026) uses historical analogies (indoor plumbing, early airline automation) to show how routine evaluation can become ordinary practice.
- Auditors should understand human and organizational needs, be comfortable with technology and statistics, and be able to produce trustworthy evidence that both experts and laypeople can believe.
- Regular AI audits act as maintenance tools, catching drift and emerging harms before they erode public trust, while still allowing innovation to flourish when safety is demonstrated.
- A growing industry of AI auditors is emerging across nonprofits, startups, and consultancies, offering career paths for students interested in responsible technology.
The Water‑Analogy Framework for Trustworthy AI
J. Nathan Matias opens the conversation by likening today’s AI uncertainty to the era after indoor plumbing was introduced, when people wondered whether the water flowing from their taps was safe. “Over time, because we built systems of inspection and of safety testing, it’s now possible for your local water authority to say, yes, we are very confident: You can drink the water, and that means we don’t have to think about it,” he explains. He argues that AI must reach a similar point where users can rely on it without constant vigilance, and that systematic auditing is the pathway to get there.
Auditing as a Skeptical Practice
When asked whether the mantra “The deft auditor begins with the mindset of a skeptic” should also guide everyday AI users, Matias affirms the idea, noting that a good life principle is “trust but verify.” He emphasizes that individuals often miss subtle AI failures, especially when harms appear as patterns across organizations rather than isolated incidents. “And on top of that, sometimes the problems of AI don’t show up for just one person. They’re a pattern of decisions or actions that play out over an entire organization,” he says, underscoring why auditing—especially analysis of cumulative outputs—is uniquely valuable.
AI Audits as Ongoing Maintenance, Like Car Inspections
Matias compares AI audits to yearly automobile inspections, stressing that AI behavior is not static. “One of the things we’ve seen is that the behavior of AI systems does change over time. Some things you can test once and say, ‘Great, I know it’s going to work fine forever.’ But take the mechanical condition of your car: It changes over time, and you need to get it inspected regularly.” He points out that models
https://news.cornell.edu/stories/2026/05/regular-audits-would-build-trust-confidence-ai