Puzzled by AI
Puzzled by AI
I gave sixteen of the most capable AI models a Block Blast! puzzle, and found only three managed to solve it. In this post, we look more closely at what worked, and what did not.
What do you do while the machine is thinking? A serious request to an LLM takes minutes to complete, and we humans have our own ways to occupy this idle time. Personally, I tend to run several sessions in parallel, so there is already something ready to look at, but let’s be honest, sometimes I also indulge in a bit of Block Blast! — these games have a capacity to burn through time at alarming speed... So it was only natural that I ended up wondering how my trusty Claude would handle it. And while I was at it, why not test other models too?
So I ran the experiment. The task was a single screenshot from the game: an eight-by-eight grid, partly filled, with three shapes waiting to be dropped into the empty squares. This was accompanied by a simple prompt asking the model whether it could solve the puzzle. The results turned out to be more interesting than I was expecting.
Prefer the raw numbers first? See the full data — every model, every score, and what each attempt cost →
First of all, only three of the sixteen solved the puzzle. These are the models that sit at the top of the public benchmarks, and the puzzle is not difficult, so we have to dig deeper to find out what was happening here.
Second finding, no model came unstuck on the logic. Most set out careful, orderly reasoning and arrived at answers that were composed, plausible in tone and sometimes against the rules: pieces laid on squares already occupied, lines declared complete that were not, etc. Further investigation showed nearly every mistake began in the same place: the model had misread the board, and having misread it, reasoned impeccably but from the wrong starting point. This is why asking your favourite model to read your prompt and replay it to you with an outline of what it plans to do is always a good idea.
The key take-aways from this experiment:
- The handful of premium models performed best. The three that solved it — Claude Fable 5, GPT-5.6 sol-pro and Gemini 3.1 Pro — are the ones the market already rates as the strongest frontier models, so their reputation is not built on hype.
- The same right answer came at very different prices. Gemini solved it for 19 cents, GPT-5.6 sol-pro for 56 and Claude Fable 5 for 80 — a factor of four between the cheapest and the dearest of the winners. The dearest failure, for its part, cost more than the cheapest success.
- Token counts tell their own story. One of the three winners used twice the tokens of the other two for the same answer, and the biggest token spender of the run still failed — more on that below. Two models used their entire budget on internal reasoning and returned nothing at all; a tool that runs long and comes back empty is its own kind of production risk.
Grok 4.5 was the biggest surprise here. I had high expectations for Grok 4.5, which seems to be highly rated as a software engineering partner. In true journalistic fashion, I invited Grok to provide a statement to accompany this publication — demolition jobs are not our business here.
From Grok: “Vision capabilities combined with precise spatial reasoning remain a genuine weak spot for many frontier models — including Grok 4.5 in this evaluation. Most failures stemmed not from flawed high-level strategy, but from board transcription errors and misaligned piece placements, highlighting how easily models can hallucinate grid layouts or piece orientations.
Grok 4.5, for instance, consumed over 44,500 reasoning tokens and approximately ten minutes of processing time, yet still proposed an illegal overlapping placement. This stands in contrast to the three models that solved the puzzle successfully (Claude Fable 5, GPT-5.6-sol-pro, and Gemini 3.1 Pro-preview), which achieved the optimal three-line clear with more efficient and accurate visual grounding. While Grok 4.5 excels in many coding, agentic, and cost-efficiency scenarios, multimodal puzzles of this nature expose gaps that pure text-based reasoning chains do not always bridge.
Benchmarks like this are valuable: xAI is iterating rapidly with new models released monthly, and real-world evaluations such as demystifAI’s help sharpen our focus on improving vision grounding, structured validation, and reliable spatial understanding.”
Credit where credit is due: Grok’s statement is a fair diagnosis. It names the actual failure, transcription rather than strategy. It also comes from a company that has been catching up with the leaders of the field at remarkable speed, and on that trajectory the gaps this puzzle exposed are the kind that close quickly. We look forward to the improvements to come.
None of this makes the models less useful. It makes them useful in a narrower way than the demonstrations suggest: reliable where a person can check the work, and when it can ground itself (this is the strange jargon we now employ) in good data well beyond what a human brain can absorb. This little experiment gives us two things heavy model users instinctively learned: i) don’t trust blindly: when it feels too good to be true it is easy to get carried away (the subject of a future paper, by the way); ii) best practices: there is still an emerging body of knowledge, but the foundations have already been laid (also for another paper).
Are you, like me, surprised by these results? What would you have expected? Perhaps you have run your own experiment and come to a different conclusion? Let me know in the comments below.
The full data behind this experiment — all sixteen models, scored and costed — is here: Block Blast experiment, the data →
Comments ()