Key Takeaways
- Large Reasoning Models (LRMs) often outperform plain LLMs on reasoning benchmarks, but the “chain‑of‑thought” text they emit may not faithfully reflect internal computation.
- Research shows that many thinking tokens have little causal impact on answers; replacing them with irrelevant or even meaningless symbols frequently leaves performance unchanged.
- Some experts argue LRMs succeed through approximate retrieval of patterns from vast training data rather than genuine step‑by‑step reasoning.
- Despite doubts about interpretability, LRMs can produce verifiable, high‑impact results (e.g., solving open math problems, improving theorem‑proving) when paired with external validation tools.
- The debate hinges on whether we should demand transparent reasoning traces or treat LRMs as useful “black‑box” engines whose outputs we verify independently.
The Rise of Large Reasoning Models
When “large reasoning models,” or LRMs, first appeared around 2024 they were marketed as the next evolution of language models, capable of explicit, step‑by‑step thinking. As the author notes, “What the hell is going on with AI ‘reasoning’?” reflects the whiplash felt by observers who saw early critiques of LRMs as an “Illusion of Thinking” quickly followed by triumphs such as OpenAI’s model solving a famous open mathematical problem in one shot in May 2026. The pendulum swung again when LRMs began winning gold medals at the International Mathematical Olympiad, a feat described by Gary Marcus and Ernest Davis as something “even very successful mathematicians and scientists may well highlight [it] on their CVs all their lives.” These contrasting outcomes set the stage for a deeper investigation into what LRMs are actually doing.
Chains of Thought: Promise and Peril
LRMs are trained to emit a “chain of thought” before answering—a synthetic text trace meant to mimic human reasoning. Melanie Mitchell, a veteran AI researcher, summarized the current consensus in three points: “Number one: It works. It improves things… Number two: The actual text that’s generated… isn’t necessarily faithful to what’s going on [inside the model]. And number three: A lot of that text isn’t even useful. You can actually take it out.” This candid assessment highlights that while the traces often accompany correct answers, they may be more decorative than diagnostic.
Evidence That Thinking Tokens Are Not Causally Central
Several studies have probed whether the intermediate tokens truly drive model behavior. Subbarao Kambhampati’s lab at Arizona State demonstrated that “fully replacing a model’s correct ‘traces’ with incorrect or irrelevant ones didn’t degrade its performance on a formal reasoning task.” Similarly, a 2024 NYU paper showed that “meaningless filler tokens — literally, strings of dots — could function effectively in place of a human‑readable ‘chain of thought.’” William Merrill, one of the NYU authors, put it bluntly: “There’s no guarantee the chain of thought has to be meaningful in any sense.” These findings suggest that the linguistic content of reasoning traces can be largely incidental to the model’s output.
Limited Causal Impact of Reasoning Traces
Further work from Northeastern University and UC Berkeley quantified this insignificance, finding that “between 30% and 60% of their ‘thinking steps’ had ‘minimal causal impact’ on the answers the models produced to benchmark math questions.” Removing roughly half of the thinking steps barely affected performance, prompting Weiyan Shi to caution, “We want to be careful when we review these chain‑of‑thought prompts because they may not be linked to the final output.” Such results challenge the intuition that a visible reasoning process must be responsible for a correct answer.
The Approximate‑Retrieval Hypothesis
Kambhampati proposes a mechanistic alternative: LRMs are essentially LLMs with more specific training, performing “approximate retrieval” across their vast corpora rather than executing a true logical algorithm. He likens the role of thinking tokens to “mumbling words to yourself to jog your memory”: the exact words matter little; they merely shift the model’s internal state toward predicting reasoning‑shaped text. In his view, the model does not need to learn a general reasoning process; it only needs to absorb enough examples of what correct steps look like to “stitch together” a plausible solution that can later be verified.
Verification Through External Tools
Despite doubts about internal fidelity, many state‑of‑the‑art LRMs are deployed alongside external verification systems. Agentic AI frameworks and tools like Google DeepMind’s AlphaProof Nexus, which relies on the theorem prover Lean, check the model’s outputs after the fact. When asked whether OpenAI’s unit‑distance proof used Lean, Sébastien Bubeck dismissed the question, stating, “We have released the chain of thought. You can just go and look at it. The whole point is that the model is reasoning like a human would. And when humans reason, we don’t use Lean.” Yet the released trace was a “rewritten summary” produced by human experts using Codex, and raw chains of thought remain unpublished, underscoring a tension between claimed transparency and actual accessibility.
Why the Debate Matters: Trust and Scientific Progress
The core concern is whether LRMs can be trusted to give the “right answer for the right reason.” Melanie Mitchell warns, “You want the right answer for the right reason, so you can trust these things,” noting that reliance on potentially spurious reasoning traces could mislead future research. Conversely, proponents like Bubeck argue that demanding perfect interpretability may be counterproductive: “It’s more interesting and more productive to talk about what they can do, rather than, ‘Oh, but they can only do that because of X [reasons].’” The field thus faces a trade‑off between insisting on transparent mechanisms and embracing useful, verifiable results even when the inner workings remain opaque.
Wishful Mnemonics and the Human Tendency to Anthropomorphize
The author draws on the concept of “wishful mnemonics,” coined by Drew McDermott in 1976, to explain why researchers and observers alike are prone to treat LRMs’ thinking tokens as genuine reasoning. As Mitchell observes, “We react to language in a way that is very anthropomorphizing. That’s just the way that we humans work.” Calling a process “chain of thought” can be a useful shorthand, but it risks becoming a “fake theory” that obscures our genuine ignorance. Kambhampati captures this sentiment: “A fake theory is worse than admitting that we don’t have a theory.” Until a clearer scientific account emerges, we may need to view LRMs’ outputs much like we view horsepower—an impressive performance whose underlying mechanics we accept on trust while continuing to probe what truly lies under the hood.
Looking Forward
The ongoing back‑and‑forth between critics and advocates shows that LRMs occupy a strange middle ground: they deliver striking, often verifiable successes while their internal “reasoning” remains partially inscrutable and potentially misleading. As the field advances, the challenge will be to develop evaluation methods that separate genuine algorithmic progress from superficial pattern‑matching, and to decide whether we demand faithful explanations or settle for reliable, externally validated outcomes. Either way, the conversation sparked by those early air quotes around AI “reasoning” is far from over.