Key Takeaways
- Weak AI‑safety regulations that target only downstream (application‑specific) firms can actually lower overall safety, producing a net‑negative effect compared with no regulation at all.
- The problem stems from a free‑rider incentive: when the downstream specialist is forced to meet a safety floor, the upstream model provider can cut its own safety investments, relying on the specialist to “do the work.”
- Regulations that impose safety standards on both the model maker and the adapter, set at an appropriate level, can increase safety and leave both firms better off by removing mutual distrust.
- Trust between the two parties is a crucial mechanism; a binding rule acts like a contract with teeth, enabling a jointly preferred safety‑profit outcome that would otherwise be unattainable.
- The findings echo in related work showing that rules encouraging model openness can backfire in the same way if they fall on the wrong link in the AI supply chain.
- Real‑world policy debates—such as the EU AI Act and various U.S. state bills—are already grappling with where to place safety responsibilities, making this research timely for designers of future AI law.
Introduction and Core Finding
New research warns that weak AI safety regulations can actually make artificial intelligence less safe. Instead of encouraging companies to improve their products, poorly designed rules may shift responsibility in ways that reduce overall safety. Benjamin Laufer, a doctoral researcher at Cornell University, built the analysis with his adviser Jon Kleinberg, in collaboration with Professor Hoda Heidari of Carnegie Mellon University (CMU). Together they modeled how safety rules ripple through the chain of companies that build and sell AI. Their central result is blunt: “A weak rule placed only on the second company can lower the safety of the finished product. It sinks below the level those same companies would have reached with no rule at all.” This counter‑intuitive outcome held up across a wide range of settings in their model, not just a single lucky example.
Model of the AI Supply Chain
The researchers treated the two companies as players in a game. Each one chooses how much to invest in AI safety and raw performance. Safety costs money; so does performance. Whatever they build, they split the revenue it earns. The model maker moves first and sets the starting point. Then the domain specialists who adapt the model decide how much further to push it. Before any of that, the two sides strike a deal on how to divide the eventual payout. A regulator sits above all this. It can set a minimum safety level for the first company, the second, both, or neither. The researchers then solved for how rational, profit‑seeking firms would respond. This builds on earlier work from the same group on how general and specialist firms bargain over fine‑tuning. The new twist is the safety floor and the question of who should have to clear it.
The Free‑Rider Problem Explained
The backfiring comes from a simple piece of self‑interest. Consider a rule that forces the downstream company to meet a set safety bar. “The model maker now knows the final product will clear that bar no matter what because the law requires it.” So the maker can quietly spend less on safety. The downstream safety floor does the work instead. In the unregulated version, the maker had reason to invest more because no one else was guaranteed to. “There’s a free‑riding behavior that occurs,” said Laufer. “The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist.” The net effect runs backward: the rule was meant to raise safety, but it hands the upstream company an excuse to cut corners. Total safety ends up lower than before. They found this holds whenever both companies share revenue and both put in real effort—a broad slice of how AI is actually built today.
When Regulations Backfire: Weak Rules on Downstream Firms
Because the upstream model provider can rely on the downstream specialist to satisfy the safety requirement, it reduces its own safety investment. The overall safety level of the finished AI system therefore drops below what would have been achieved without any regulation. This outcome is not a fluke; the researchers observed it across numerous parameter settings in their game‑theoretic model. The regulation unintentionally creates a moral hazard: the specialist bears the cost of compliance while the provider enjoys a free ride on safety. The study thus highlights a trap for policymakers who aim to improve safety by targeting only the application‑level firms.
Effective Regulation: Targeting Both Links
The second finding runs the other way. Rules aimed at both companies, set at the right level, can make products safer and leave both firms better off. The reason is trust. Two companies building a product may both want higher safety, yet neither can trust the other to follow through. Each has a private incentive to skimp at the last minute. So the safer, more profitable path never gets taken. A rule that binds both sides removes the guesswork. “Neither has to take the other at its word. That lets them reach a combination of safety and profit they both prefer but could not lock in alone,” Laufer explained. This is why some companies openly ask to be regulated: a well‑placed rule can act like a contract with teeth, and both parties may be willing to pay for it.
Trust and Mutual Benefit: How Rules Enable Cooperation
By imposing a shared safety floor, the regulator eliminates the opportunity for either firm to exploit the other’s caution. When both know the other must meet the same standard, the incentive to cut corners disappears. The joint investment in safety can then be coordinated to a point where the marginal benefit of extra safety equals its marginal cost for each firm, yielding a Pareto‑improving outcome. The model shows that, under these conditions, both the model maker and the adapter can achieve higher safety and higher expected profits than in the unregulated baseline. This mutual‑gain scenario underscores the importance of designing regulations that align incentives rather than creating asymmetric burdens.
Policy Implications: Lessons from the EU AI Act and State Bills
The team’s findings may also extend beyond AI safety. A related analysis from the same researchers found that rules encouraging AI models to become more open can backfire in much the same way if they fall on the wrong link in the supply chain. The study is still a simplified picture, and the authors want to test their predictions against real‑world regulations as they emerge. Even so, the work highlights a trap policymakers may want to avoid: weak rules aimed at the wrong companies could do less than nothing. The study is published in Proceedings of the National Academy of Sciences.
Broader Implications: Beyond AI Safety to Model Openness
Although the paper focuses on safety, its logic applies to any regulatory tool that seeks to shift responsibility along a production chain. For instance, mandates that require downstream adapters to disclose model details or to enforce openness can unintentionally relieve upstream providers of the incentive to invest in transparent, robust foundations. If the upstream firm knows the downstream partner will bear the compliance cost, it may reduce its own efforts, potentially degrading overall model quality. Policymakers should therefore examine the entire value chain when crafting rules, ensuring that obligations are distributed in a way that preserves incentives for all parties to contribute to the desired outcome.
Conclusion and Future Research Directions
The research offers a clear lesson: aiming safety rules only at the companies adapting AI for niche uses can quietly make things worse. Spreading the requirements across both the model maker and the adapter tends to work better. As the European Union’s AI Act and various state bills continue to evolve, regulators have an empirical basis to debate where to place responsibility. Future work will test these theoretical predictions against actual regulatory environments, refining the model with empirical data on firm behavior, revenue sharing, and safety outcomes. Until then, the study serves as a cautionary reminder that well‑intentioned regulation must be carefully designed to avoid unintended free‑riding that undermines the very safety it seeks to enhance.
https://www.earth.com/news/weak-ai-regulations-may-leave-artificial-intelligence-less-safe/