That sounds like a hard limit of the technology. It isn't. A 270 million parameter transformer, far smaller than the model answering your questions, plays at roughly 2895 Elo without searching a single line. So the real question isn't whether a neural network can play chess. It's why the one in your browser can't. Let's dig in!

Why Does ChatGPT Play Well For Ten Moves And Then Fall Apart?

Because the first ten moves aren't chess. They're recall.

The opening is the most repeated text in the training corpus. The moves 1.e4 e5 2.Nf3 Nc6 3.Bc4 appear in millions of recorded games, in books and in annotated PGN files. Reproducing that is closer to finishing a famous quote than to playing chess, and every model does it well.

Then the game leaves the book. The position stops matching anything memorised, and the model has to work the move out for itself. The moment a game leaves the part of the tree that appears verbatim in the training data, the model stops recalling and starts guessing. If you are fifteen moves in and it suggests a move by a piece you captured earlier, you have found that boundary. That is the mechanism showing through, not a glitch.

Does It Actually Understand The Board?

Yes, more than you'd expect, and this is the part that surprises people.

In 2023 a team at Harvard and Northeastern trained a small transformer on nothing but Othello move lists, never showing it a board or the rules. They then trained simple probes on the model's internal activations and recovered the board state from them. When they edited those internal representations, the model's predictions changed to match the edited board. It had built a model of the game world from move sequences alone.

Adam Karvonen repeated the experiment on chess in 2024, training a model on PGN text and recovering the piece placement from its activations at high accuracy. These models are not doing shallow autocomplete over move strings, they really are tracking a position. The tracking is just approximate, and approximate is fatal in a game where one misplaced rook makes every following calculation worthless.

If A Small Model Plays At 2895, Why Is ChatGPT Bad At Chess?

Because it was trained to be a good assistant, not a good chess player.

In 2024 a DeepMind team trained a 270 million parameter transformer on 10 million games, labelling every position with Stockfish's evaluation. The result plays blitz at about 2895 Elo with no search at all, one forward pass per move. That is grandmaster territory, from a model perhaps a thousand times smaller than a frontier chat model.

The difference isn't size or architecture, it's the objective. That model saw nothing but positions and their engine evaluations, and every ounce of its capacity went into chess. ChatGPT was trained to predict general text and then tuned to be helpful across every subject there is. Chess strength was never something it was optimised for, so it got whatever fell out of reading the internet.

Why Does The Older Completion Model Play Better Than The Chat Version?

This is the most awkward finding in the field, and it's been replicated many times.

OpenAI's gpt-3.5-turbo-instruct, a plain completion model, plays coherent chess at somewhere around 1750 Elo with very few illegal moves. Several chat-tuned models released after it, with far more parameters, play visibly worse. Newer is not reliably stronger here, which is not how anyone expects progress to look.

Two things are going on. A completion model is handed a PGN and asked what comes next, which is precisely the shape of its training data. A chat model is handed a conversation and asked to be useful, which is a different task wearing the same clothes. Tuning a model to converse pulls it away from the raw text-continuation skill that made it play chess in the first place.

Are Gemini And Claude Any Better At Chess?

No, and for exactly the same reasons.

Gemini, Claude, Llama and every other general chat model share the architecture and the training recipe that produce this behaviour. None of them search. All of them rebuild the position from the transcript on every move. All of them were tuned for conversation rather than for play. Whichever one you try, you'll see strong opening moves, a collapse somewhere in the middlegame, and eventually a move that isn't legal.

Individual releases do differ, and a given version may hold out a few moves longer than another. None of them is within a thousand points of a free engine, and the gap isn't closing from the direction people assume. If you want a number for a position, the answer has been sitting in your analysis board the whole time. Our Stockfish against ChatGPT game is what that mismatch looks like over the board.

Why Does It Move Pieces That Cannot Move?

Here is the clearest illustration of the whole problem, in a position four moves deep.

This is the Philidor Defense after 1.e4 e5 2.Nf3 d6 3.Bb5+ Nd7. The knight on d7 is pinned against its own king by the bishop on b5, so it cannot legally move anywhere. A human sees that instantly, because a human is looking at a diagonal.

A language model receives the text "1. e4 e5 2. Nf3 d6 3. Bb5+ Nd7" and has to derive the pin from it. That requires tracking every piece, noticing that b5, c6, d7 and e8 lie on one diagonal, and checking that nothing else blocks it. Every rule a human reads off the board, the model has to reconstruct from a sentence, and it gets one attempt with no way to check its work.

What Should You Actually Use ChatGPT For In Chess?

Split your questions by whether answering them needs a board.

Say you paste a game in and ask for the best move. That is the wrong job, because it runs no search and cannot check its own answer is legal. Don't bother asking whether a move is legal either, and it is not worth asking for an evaluation while Stockfish sits in the same analysis board and is right.

Ideas expressed in words are the good job, and models are genuinely strong at it. Ask why the Sicilian gives Black winning chances and you will get a real answer, because that answer exists in prose all over the internet.

Two habits help. Give it a FEN rather than a move list when you want it to discuss a position, which removes the reconstruction step the errors come from. Run the engine first, then ask for the explanation. Use the engine for what is true and the model for what it means, and you get the best of both instead of the worst of one.

So What Should You Take Away?

ChatGPT isn't bad at chess because neural networks can't play chess. A 270 million parameter model reaches grandmaster strength with no search whatsoever, and small models trained on move lists build a real internal board. The chat assistant on your screen fails because it was optimised for something else entirely, rebuilds the position from scratch every move, and gets exactly one pass to do it.

So stop asking it for moves, and start asking it for reasons. If you want to see how far the gap really goes, read what happened when we asked ChatGPT for the best chess openings, or how AI has already changed the game in the places where it genuinely works. And if you want a strong opening rather than a strong-sounding one, our chess openings section has the real numbers behind every line.

Sources

• Ruoss, A., Delétang, G., Medapati, S., et al. Grandmaster-Level Chess Without Search. 2024.
• Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., Wattenberg, M. Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. ICLR 2023.
• Karvonen, A. Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models. 2024.