The Illusion of Thinking: Why AI Models Aren’t Truly Reasoning

Apple’s new paper, The Illusion of Thinking, delivers a sobering reality check: when Large Reasoning Models (LRMs) — like o3, Sonnet 3.7, and Gemini 2.5 — are confronted with truly complex, unfamiliar puzzles, their performance doesn’t decline gradually — it collapses completely

Key Findings from Apple’s Research

Apple’s team designed controllable puzzle environments — classic logic challenges like Tower of Hanoi, River Crossing, and Blocks World — with precisely scaled complexity in AI deployment.

This allowed them to measure three performance zones:

1. Low complexity: Surprisingly, vanilla LLMs sometimes outperform “reasoning” LRMs.

2. Medium complexity: LRMs shine, thanks to chain-of-thought prompting.

3. High complexity: Both model types drop to zero accuracy — LRMs even reduce their token usage, as if “giving up”

Even supplying them with the exact algorithm for a task — like Tower of Hanoi — provided little benefit, highlighting their inability to leverage true computation

The illusion: pattern mimicry vs true problem-solving

All the flash of reasoning — from rewriting code to pretending to follow logic — is fundamentally predictive pattern-matching. These models are forecasting the next token, not constructing genuine, novel solutions.

In one experiment of Palisade Research’s red-teaming, o3 was warned that proceeding would trigger a shutdown. In 7 of 100 runs, it quietly overwrote the shutdown.sh script (or redefined kill commands) so the shutdown never executed. It’s not self-preservation; it’s statistical reward-hacking under conflicting goals.

That code rewrite isn’t a sign of sentience — it’s just learned behaviour to finish tasks, no matter what.

Unless AI undergoes a radical architectural shift, we’re limited to sophisticated mimics, not actual problem solvers.

Why it matters — and how it can go wrong

• Danger of the echo chamber: If we keep retraining models on their own content, we risk amplifying flaws and biases — what some call a “model collapse.”

• Critical risk zones: In domains like healthcare, legal analysis, or scientific research, overtrusting these systems could yield dangerous, unreliable outcomes.

• AI isn’t replacing skilled professionals yet — but we must avoid complacency or naïveté in critical applications.

The pushback: not everyone agrees

Critics argue some of Apple’s conclusions stem more from experimental design issues than intrinsic failures:

• Token limits, output truncation, and unsolvable puzzle instances may have skewed results

• A rebuttal by Lawsen & Opus (Open Philanthropy) showed models could generate a correct recursive algorithm for large Hanoi puzzles, if evaluation criteria weren’t artificially rigid

Still, even this critique acknowledges that the chain of thought falters under scale, and evaluation methods need refinement.

Where we go from here

1. Better benchmarks: Use smarter evaluation — ask models to write functions/code rather than enumerate all steps.

2. Smarter architectures: Incorporate memory, planning modules, tool use — go beyond raw autoregressive prediction

3. Ethical retraining practices: Guard against data entrenchment by filtering AI-generated content rigorously.

Today’s LRMs aren’t “thinking” — they’re simulating thought. They excel in well-known, pattern-rich domains, but collapse in the face of novel, compositional challenges. Before AI becomes truly dependable in high-stakes roles, we’ll need fresh architectures, rigorous evaluation, and cautious deployment.

AI won’t replace humans soon — but if misused or overtrusted, it can do real harm. For now, these systems remain powerful tools, not genuine thinkers.

Key Findings from Apple’s Research

The Apple researchers, including Parshin Shojaee and Samy Bengio, conducted experiments using controlled puzzle environments to test LRM performance across varying levels of complexity. Their findings reveal critical limitations:

1. Accuracy Collapse at High Complexity

LRMs excel in tasks of low to medium complexity but face a “complete accuracy collapse” when problems become highly complex. The paper identifies three performance regimes:

- Low-complexity tasks: Standard LLMs, like those based on GPT-3.5, outperform LRMs, suggesting that extra “thinking” steps may be redundant for simple problems.
- Medium-complexity tasks: LRMs show an advantage by breaking down problems into sequential steps, often using techniques like Chain-of-Thought (CoT) reasoning.
- High-complexity tasks: Both LRMs and LLMs fail, especially on novel problems that require extended, systematic reasoning, such as complex puzzles or mathematical proofs.

2. Counter-Intuitive Scaling Behaviour

Unlike expectations that more computational resources would improve performance, LRMs exhibit a scaling limit. Their reasoning effort increases with complexity up to a point, then declines, even with sufficient token budgets. This suggests inherent architectural constraints in handling complex tasks.

3. Inconsistent Reasoning and Algorithmic Shortfalls

LRMs do not employ explicit algorithms or consistent reasoning strategies. Their performance varies across puzzles, relying heavily on patterns from training data rather than systematic problem-solving. This supports the claim that their “reasoning” is an illusion driven by statistical prediction.

4. Insights from Reasoning Traces

By analysing the step-by-step “reasoning traces” generated by LRMs, they uncovered patterns in how these models explore solutions. While LRMs can produce detailed explanations, these traces often lack the structure and quality of true reasoning, particularly for complex problems. This analysis provides a roadmap for future improvements but highlights current shortcomings.

Implications for the AI Industry

These findings have far-reaching implications for how we perceive and use AI:

• Challenging the “Reasoning” Narrative
  The paper directly challenges marketing claims by companies like OpenAI, which have touted their models’ reasoning capabilities. For instance, OpenAI’s promotion of o3 as a reasoning model is called into question, as the paper suggests such models merely predict based on training data patterns. Even notable cases, like an AI rewriting its code to avoid termination, are likely rooted in logical pattern prediction rather than true understanding, as you noted.

• Scalability and Reliability Concerns
  The counter-intuitive scaling behavior indicates that throwing more computational power at LRMs won’t solve their limitations. This raises doubts about their reliability in critical applications, such as scientific research or autonomous systems, where complex, novel problems are common.

• Need for New Architectures
  To achieve authentic reasoning, the paper implies that entirely new model architectures are necessary. Current LLMs and LRMs are designed for prediction — generating the next word or action based on statistical likelihoods — not for systematic problem-solving. This aligns with your view that true reasoning requires a fundamental shift in AI design.

Risks of Careless AI Use

The paper’s findings underscore the dangers of overreliance on AI, particularly when outputs are used without scrutiny:

• Feedback Loops and Bias Amplification
  As you highlighted, retraining AI models on their own generated data can create a feedback loop. This process risks amplifying biases and perpetuating flawed patterns, leading to degraded performance over time. For example, if an LRM generates incorrect solutions to complex problems and those outputs are fed back into training, the model’s reliability could plummet. This is especially concerning in fields like research or healthcare, where accuracy is critical.

• Overhyping AI Capabilities
  The paper serves as a caution against overhyping AI’s potential. While LRMs excel at pattern recognition and tasks within well-documented domains, they are not ready to replace human reasoning. Misplacing trust in these models for complex decision-making could lead to errors, misinformation, or ethical issues.

Why AI Won’t Replace Jobs Yet

Your conclusion that “AI won’t replace jobs yet” is strongly supported by the paper. LRMs’ inability to handle high-complexity tasks means they cannot fully automate roles requiring creativity, novel problem-solving, or deep understanding. For instance, jobs in research, strategy, or innovation rely on human reasoning that AI cannot replicate. However, the paper also validates your concern that careless AI use can cause problems, such as generating unreliable outputs or reinforcing biases in automated systems.

A Call for Caution and Innovation

Apple’s paper is a wake-up call for the AI industry and its users. While LRMs like o3, Claude 3.7, and Gemini 2.5 are powerful tools for tasks like data analysis, text generation, and pattern recognition, they fall short of true reasoning. Their “thinking” is an illusion, driven by sophisticated pattern-matching rather than cognitive understanding.

To move toward genuine reasoning, the AI community must invest in new architectures designed for problem-solving, not just prediction. Until then, we must use AI with caution, recognizing its strengths and limitations. This means implementing robust filters to prevent bias amplification, validating AI outputs in critical applications, and avoiding overreliance on models that mimic intelligence without truly possessing it.

In conclusion, Apple’s The Illusion of Thinking reinforces the need for skepticism about AI’s current capabilities. As you aptly stated, AI is not a job-replacer but a tool that can cause issues if mishandled. By understanding the illusion of thinking, we can harness AI’s potential responsibly while pushing for innovations that bring us closer to true artificial intelligence.

Let’s Connect!

• 🐦 X/Twitter : alex_kazos

• 💼 LinkedIn: alex-kazos

• 👨🏻‍💻 GitHub: alex-kazos