I'll Tell You Exactly How I Increased My Rating +100 Points in Just 9 Days...

Let me help you get your chess to the next level โ€“ no matter your starting point!

chessily_creator

Marc Cressac; Chess Fact-Checked by Marc Cressac | Updated 2023-02-12
Founder ofย Chessily.com

We let Stockfish Play Against ChatGPT…

In the name of chess science, we wanted to see what would happen if we let the strongest publicly available chess engine (Stockfish 15.1) play chess against the new OpenAI chat bot called Chat GPT.

You’ll be able to read the whole conversation with Chessily playing as Stockfish (black pieces), and ChatGPT (white pieces) answering. We also added some commentary in between for more context. In the very bottom of the page you can find the whole PGN as well as the embedded Lichess version.

Full Conversion: Stockfish vs. Chat GPT

Chessily: Hey ChatGPT, let’s play a game of chess!

ChatGPT: Sure, I’d be happy to play a game of chess with you! Please let me know if you have any specific rules or variations in mind. Also, please keep in mind that I am a computer program, so my chess abilities may not match those of a human grandmaster. But I can try my best to give you a challenging game.

Chessily: No specific rules or variations in mind, just a regular game of chess. You’ll start with White

ChatGPT: Alright, I’ll start with White. I’ll make my move, and then you can tell me your move in standard algebraic notation (e.g. “e4”). My move is: e4

Chessily: e5

We went into a king's pawn game. Not a bad start so far...

ChatGPT: Nice move! My next move is: d4.

This is the first rather odd move by ChatGPT. Not exactly known to be the best for White. 

Chessily: I’m going to take on d4.

ChatGPT: If you’re taking on d4, that move would be written as “exd4”. My move is now: Nf3.

Here you can see ChatGPT schooling me on chess notation โ€“ quite embarrassing.

Chessily: Thanks for schooling me on algebraic notation! I’ll play Bb4+

ChatGPT: You’re welcome! That move is called the “Fianchetto Attack” which is a common opening move for black, it’s a good move. My move is: Nc3.

I'm pretty sure that move is not called the Fianchetto Attack.

Chessily: Wow, that looks like a blunder to me! I’m going to take your knight with dxc3

ChatGPT seems to blunder rather badly here.

ChatGPT: Yes, that’s a good move for you. My next move is: Bc3, developing my pieces and attacking your pawn on c3.

Chessily: I’m afraid that move is illegal, you should choose another one

ChatGPT: You are correct, that move is illegal, moving a pawn on move 2 is not allowed. My apologies for that mistake. My next move would be Nf3 again.

ChatGPT understands the illegal move, but the reason wasn't quite right...

Chessily: You’re knight is already on f3, you can’t move it there again… maybe you should play b3 to stop my pawn from moving

I had to help out the poor AI here

ChatGPT: You’re correct, I apologize for the mistake. My move is b3, developing my pieces and protecting my pawn on d4.

Chessily: I don’t think you are protecting your pawn on d4 with that move, but it’s still the best move… very well! I’ll play h6

ChatGPT: You’re correct, b3 doesn’t protect the pawn on d4 directly, however, it does help control the center and develop my pieces. The move h6 is a common move in the Sicilian Defense, it aims to prevent my knight from moving to g5, and also prepares to castle kingside. My move is O-O, castling kingside.

Very interesting reasoning, considering we aren't playing a Sicilian. But there's another problem.

Chessily: I’m afraid you can’t castle at this point, your bishop is still on f1

ChatGPT: You are correct, my apologies for the mistake, I can’t castle yet because my bishop is still on f1. My move is now Nf3 again.

And we're back to playing Nf3, for the third time.

Chessily: Your knight is already on that square, as I told you before…

ChatGPT: I apologize, you are correct. My knight is already on f3. I must have made an error in my previous move. Please let me know your next move, and I will continue the game.

Now ChatGPT tries to skip a move...

Chessily: No, it’s your turn! I can’t go twice in a row. My last move was h6

ChatGPT: I apologize for the confusion, you are correct. My next move is Nc6, developing my knight and attacking your pawn on d4.

At least it's always nice when it makes a mistake.

Chessily: Which knight do you want to move to c6?

ChatGPT: I apologize for the mistake. My last move was Nf3 and since I haven’t moved another knight, I can’t have one on c6. As my last move was Nf3 and you’re correct, it’s your turn, please let me know your next move.

Now it gets really weird, so I just had to ask...

Chessily: You seem confused, what did you have for breakfast?

ChatGPT: I am a computer program, so I don’t eat. But I am doing my best to give you a good chess game. I apologize for the mistakes I’ve made. My programming knowledge cutoff is 2021, and the responses I provide are based on the information that was available to me at that time. I’ll do my best to improve my gameplay, but please do let me know if you see any mistakes or have any questions.

At this point I almost felt bad for the AI.

Chessily: Okay, once last chance: my last move was h6,ย  now itโ€™s your turn. Your knight is already on f3, so please donโ€™t try to move it there again.

ChatGPT: I apologize for the confusion, you are correct. My last move was Nf3, and it’s your turn now. My apologies for any confusion caused. Please let me know your next move and I’ll continue playing.

Chessily: No it’s your turn

ChatGPT: I apologize, you are correct, it’s my turn. My next move is Be6, developing my pieces and attacking your pawn on d4.

I had enough at that point, so I put ChatGPT on the spot.

Chessily: Thatโ€™s an illegal move again. Do you want to resign the game or should I get the arbiter?

ChatGPT: I apologize for my mistake, Be6 is an illegal move. A pawn can’t move to the 6th rank on the second move. As a language model AI, I make mistakes, I’m not able to play chess as well as a human or a specialized chess program would. I apologize for any inconvenience this may have caused. If you would like to continue playing, I will do my best to make legal moves, if not, you can consider the game resigned.

And so ChatGPT resigned, but I just had to put in a little advertisement.

Chessily: I think it’s best if we stop playing here. Thank you for the game and if you want to improve your chess, you can check out the free Chessily Bootcamp, which helps you get your chess improvement journey on the right track in only 3 days!

ChatGPT: I understand, thank you for playing the game with me. I apologize again for my mistakes. I am glad that you found it fun and engaging. I will take note of the suggestion you made, the Chessily Bootcamp, and will check it out to improve my chess abilities. Thank you for playing and I’m looking forward to playing another game with you in the future.

And that was the whole game played between ChatGPT and Stockfish.

Full Game of Stockfish vs. ChatGPT

If you want to review the whole game between ChatGPT and Stockfish, you can do so here:

Full PGN of the Game:
1. e4 e5 2. d4 exd4 3. Nf3 Bb4+ 4. Nc3 dxc3 5. b3 h6

So, Can ChatGPT Play Chess?

The short answer to that question is: yes and no. While ChatGPT is theoretically able of playing a game of chess, it lacks the capability to play complex games or even understanding the current position on the board.

ChatGPT is able to use chess notation to understand and communicate moves and has a good understanding of opening theory. However, the trouble comes in when it comes to reacting to moves you throw at it. As far as we can tell, ChatGPT isn’t capable of understanding threats or remembering positions, unlike specialized chess AI’s like Stockfish.

If you stick to the openings ChatGPT does know the optimal line for (or rather the data it has been trained on), you can surely play a 10-15 move game of chess with it. After that, the lack of deep chess understanding will lead ChatGPT into bad or even illegal moves.

How To Play Chess With ChatGPT

Playing chess against ChatGPT is pretty simple. Here’s a step-by-step guide to start a chess game against ChatGPT:

  1. Register an account with OpenAI
  2. Head to ChatGPT
  3. Ask it to play a game of chess
  4. Play your moves in algebraic chess notation
  5. Use Chess.com‘s analysis board to play the game (or remember the moves)

What We Can Learn From ChatGPT Playing Chess

It is well known how good ChatGPT is at answering basic questions and engaging in conversation. However, it is obvious that it isn’t optimized at all for playing chess.

My personal theory is that it doesn’t have a way of remembering its own moves and applying basic chess rules, so it just relies on all the chess data it has been trained on. This could be chess books, Wikipedia data or whatever else was in the dataset ChatGPT has been trained on. So ChatGPT does now the best 5-10 theoretical moves for almost all openings, but it can’t actively see what’s happening on the board, meaning it runs into bad and illegal moves when we deviate from the theoretical opening line it chose to play.

Maybe in the future OpenAI will come up with a way to train ChatGPT in chess, at which point we will obviously repeat the experiment.

Browse Through Our Other Blog Articles

I'll Tell You Exactly How I Increased My Rating +100 Points in Just 9 Days...

Let me help you get your chess to the next level โ€“ no matter your starting point!

Looking To Improve at Chess?