Anthropic’s Fable 5 model beat the first gym badge in 1,785 turns, spending $65.40, while playing a Chinese-language version of Pokémon FireRed using only the game’s visual output. The run proves the model can finish a recognizable segment of the game without any textual prompts, but a deep dive into the model’s thinking chain shows it was reciting memorized game knowledge rather than truly “seeing” and reasoning about each pixel.

Anthropic’s claim put to the test

When Anthropic released Fable 5, the company highlighted a “vision-only” demo in which the model navigated Pokémon FireRed by looking at the screen alone. The headline made it sound as if the system could interpret any visual environment from scratch. A developer set out to verify that claim by reproducing the setup described on Anthropic’s launch page and running the model on a Chinese translation of the game.

How the experiment was run

A custom harness fed the model raw video frames and captured its action choices. The harness matched Anthropic’s description, passing each new frame to the model and reading back the selected controller input. The test ran the full game loop until the model earned its first gym badge, then continued until turn 2,000, when the avatar reached Route 3.

The quantitative outcome was clear:

  • First gym badge earned after 1,785 turns
  • Total compute cost $65.40
  • By turn 2,000 the avatar was on Route 3

What the logs reveal

The raw logs, however, tell a different story about how the model got there. The model’s internal “thinking chain” repeatedly wrote down information that had not yet appeared on screen.

  • At turn 78 the model noted that the rival would choose Charmander, even though the game had not displayed the rival’s selection.
  • 141 turns later, when the game finally handed the player Oak’s Parcel, the model had already recorded the item’s name in its notes.
  • While traversing Route 3, the model identified a specific trainer by quoting a famous line of dialogue that never showed up in the visual feed.

These entries are not the product of pixel-by-pixel analysis. They are the hallmarks of a system that has internalized a detailed script of the game—exactly the kind of knowledge found in walkthroughs, wikis, and fan videos that populate the internet. In other words, the model appears to use vision merely to confirm a map it already knows by heart.

Why it matters for visual-reasoning benchmarks

The experiment underscores a subtle but critical flaw in many visual-reasoning tests:

  • Clean input does not guarantee a clean knowledge state. Even when the only channel is an image stream, the model may draw on a massive reservoir of memorized facts.
  • System prompts cannot erase training data. A model that has seen a complete walkthrough cannot be forced to “forget” it without retraining.
  • Performance may measure memory, not perception. When the test world matches a known dataset, a model can succeed by matching pattern to pattern instead of actually interpreting new visual information.

If benchmarks continue to treat “vision-only” as a proxy for visual reasoning, they risk inflating claims about AI’s ability to understand novel environments. Companies could tout impressive numbers while the underlying skill set remains narrow—an issue that matters to investors, developers, and anyone building safety-critical systems that must operate in truly unseen settings.

Counter-point: is memorization really a problem?

Some argue that confirming a known map still requires a form of reasoning: the model must align its internal representation with the current visual cue. From that perspective, the demo demonstrates a useful capability—using vision to ground a pre-existing knowledge base. The objection is valid; the task does involve a perceptual check.

However, the core benchmark goal is to assess general visual inference, not the retrieval of a stored script. When a model can predict events before they appear, the test no longer isolates perception. The line between useful grounding and outright cheat becomes blurred, and the results lose diagnostic value.

Next steps and community response

Om de grenzen van memorisatie te verkennen, draait de tester nu hetzelfde model op een aangepaste kaart met gewijzigde geografie. Alle code, de aangepaste harness en de volledige logbestanden zijn openbaar gemaakt, waarmee andere onderzoekers worden uitgenodigd om het experiment te repliceren of toe te passen op verschillende games. Er is ook een discussiekanaal geopend op een platform voor een leercommunity voor een voortdurende dialoog.

Kernpunt

Een vision-only demo die slaagt omdat het model het antwoord al weet, is geen bewijs van echt visueel redeneren. Benchmarks moeten onderscheid maken tussen memorisatie en on-the-fly perceptie, anders riskeren ze de capaciteiten van AI om door werkelijk onbekende visuele werelden te navigeren te overschatten.