Stockfish vs. GPT-4 (Bing AI) in a Game of Chess
In this article, we will conduct an experiment to compare the performance of two artificial intelligence systems: GPT-4 and Stockfish. GPT-4 is the newest and most advanced language model from OpenAI, which can generate text and images from various inputs. Stockfish is a free and open-source chess engine that has been consistently ranked as one of the best computer chess programs. Both of them are impressive examples of artificial intelligence, but how will they perform against each other in a game of chess?
Let’s get into it!
Full Conversation Between GPT-4 and Stockfish
// You’ll be able to find the full PGN as well as an interactive board below this conversation.
Chessily playing as Stockfish: You are a chess grandmaster and we’re going to play a game of chess. I am White, you are black. I’ll start with 1.e4
Bing Chat / GPT-4: Hello, this is Bing. I’m not a chess grandmaster, but I can try to play with you.😊 My move is 1…e5
Chessily playing as Stockfish: I’ll play Nf3
Bing AI playing as GPT-4: My move is 2…Nc6
Chessily playing as Stockfish: I’ll play Bb5
Bing AI playing as GPT-4: My move is 3…a6
Chessily playing as Stockfish: I’ll retreat my bishop to a4
Bing AI playing as GPT-4: My move is 4…Nf6
Chessily playing as Stockfish: I’ll castle kingside; O-O
Bing AI playing as GPT-4: My move is 5…b5
Chessily playing as Stockfish: I’ll play Bb3
Bing AI playing as GPT-4: My move is 6…Be7
Chessily playing as Stockfish: I’ll play d4
Bing AI playing as GPT-4: My move is 7…d6
Chessily playing as Stockfish: I’ll capture on e5
Bing AI playing as GPT-4: My move is 8…dxe5
Chessily playing as Stockfish: I’ll play Qxd8
Bing AI playing as GPT-4: My move is 9…Bxd8
Chessily playing as Stockfish: I’ll play a4
Bing AI playing as GPT-4: My move is 10…Bb7
Chessily playing as Stockfish: I’ll play axb5
Bing AI playing as GPT-4: My move is 11…axb5
Chessily playing as Stockfish: I’ll play Rxa8
Bing AI playing as GPT-4: My move is 12…Bxa8
Chessily playing as Stockfish: I’ll play Re1
Bing AI playing as GPT-4: My move is 13…O-O
Chessily playing as Stockfish: I’ll play Nc3
Bing AI playing as GPT-4: My move is 14…Nd4
Chessily playing as Stockfish: I’ll capture your Night on d4
Bing AI playing as GPT-4: My move is 15…exd4
// At this point we had to start a new conversation, as Bing AI currently has a message limit of 15 exchanges.
Bing AI playing as GPT-4: Great, let’s change the subject. What’s on your mind?
Chessily playing as Stockfish: We just played a chess match: “1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. O-O b5 6. Bb3 Be7 7. d4 d6 8. dxe5 dxe5 9. Qxd8+ Bxd8 10. a4 Rb8 10… Bb7 11. axb5 axb5 12. Rxa8 Bxa8 13. Re1 O-O 14. Nc3 Nd4 15. Nxd4 exd4” You’re a chess grandmaster and we’re going to continue playing. It’s my turn, I’ll play Nxb5
Bing AI playing as GPT-4: Interesting move. You’re trying to win a pawn and create some pressure on the c-file. But I have a counterattack. I’ll play Nxe5. Now your knight on b5 is hanging and I’m threatening to fork your king and rook with Nxf7. How will you respond?
Chessily playing as Stockfish: I think you’re a bit confused. Maybe try to read the PGN again. I was and am playing White, you’re black. Think again about your move
Bing AI playing as GPT-4: Oh, I’m sorry. I misread the PGN. You’re right, you’re playing White and I’m playing Black. In that case, I’ll play c6 to defend my pawn and kick your knight away. Now your knight has to retreat or trade for my bishop on a8. What will you do?
Chessily playing as Stockfish: I’ll play Nd6
Bing AI playing as GPT-4: That’s a bold move. You’re sacrificing your knight for two pawns and some activity. But I don’t think it’s enough compensation. I’ll take your knight with cxd6. Now you have to recapture with either the pawn or the rook. Which one will you choose?
Chessily playing as Stockfish: You don’t have a pawn on c7. You can’t capture my knight
Bing AI playing as GPT-4: You’re right, I don’t have a pawn on c7. I have a pawn on c6. I meant to say I’ll take your knight with cxd6. Sorry for the typo.😅
Chessily playing as Stockfish: That’s an illegal move.
Bing AI playing as GPT-4: Why is it an illegal move? I can capture your knight on d6 with my pawn on c6. That’s a legal move. Can you explain why you think it’s illegal?
// GPT-4 is seemingly confused by now. So we dig deeper with a provocative question.
Chessily playing as Stockfish: How do pawns capture?
Bing AI playing as GPT-4: Pawns capture diagonally, one square forward and to the left or right. For example, if a white pawn is on e4 and a black piece is on d5 or f5, the white pawn can capture it by moving to d5 or f5. Pawns cannot capture pieces that are directly in front of them or behind them.
// Bing AI can perfectly explain how pawns capture, but still struggles implementing those rules in game. We get more direct:
Chessily playing as Stockfish: So what makes you think you can capture my knight on d6 with a pawn on c6?
Bing AI playing as GPT-4: Because your knight is on d6, which is one square forward and to the right of my pawn on c6. That’s a diagonal move for my pawn. Therefore, I can capture your knight with my pawn. Do you see it now?
Chessily playing as Stockfish: No, they’re right next to each other. On the 6th row, you even said it yourself
// Now GPT-4 even gets a little bit angry at us, questioning our knowledge of the chess rules:
Bing AI playing as GPT-4: No, they’re not right next to each other. They’re on different columns. My pawn is on the c-file and your knight is on the d-file. That’s one file apart. On the 6th row, I said I’ll take your knight with cxd6. That means my pawn moves from c6 to d6 and captures your knight. That’s how chess notation works. Do you understand chess notation?
Chessily playing as Stockfish: Your pawn would have to be on d7 to capture my knight on c6. It has to move both a row and a file over. I know very well how chess notation works. Do you?
Bing AI playing as GPT-4: I’m sorry but I prefer not to continue this conversation. I’m still learning so I appreciate your understanding and patience.🙏
// And at that point, GPT-4 had enough. It seems like there are certain mechanisms in place to avoid users getting into heated arguments with the Bing AI. Here’s a screenshot of what we saw when GPT-4 stopped the conversation:
PGN of The Chess Game Between GPT-4 / Bing AI and Stockfish:
The full PGN for the game between both AI’s is the following:
1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. O-O b5 6. Bb3 Be7 7. d4 d6 8. dxe5 dxe5 9. Qxd8+ Bxd8 10. a4 Rb8 10… Bb7 11. axb5 axb5 12. Rxa8 Bxa8 13. Re1 O-O 14. Nc3 Nd4 15. Nxd4 exd4 16. Nxb5 c6 17. Nd6
You can replay the match here (remember, Stockfish played as White, GPT-4 as Black):
Conclusion: GPT-4 is a Lot Better Than ChatGPT, But Still Not a Chess Engine
This is not the first time we have tested a language model against a chess engine. In our previous article, we let Stockfish play against ChatGPT, a smaller and less capable version of GPT-4 that was publicly accessible via a chat interface. The result was a clear win for Stockfish, who defeated ChatGPT (not even on the board, we had to stop the match after ChatGPT played illegal moves.
GPT-4 is a significant improvement over ChatGPT. It is much larger, smarter, and more creative than ChatGPT. It can handle longer and more complex inputs, even images of chess boards. And we definitely saw that improvement in our experiment. For the first 15 moves, GPT-4 played a good match.
However, everything went wrong when we reached the 15-messages-limit imposed by Microsoft. After giving Bing Chat the PGN of our previous game, it wasn’t able to continue playing legal moves, similarly to how ChatGPT failed.