{"id":3870,"date":"2023-03-25T09:42:31","date_gmt":"2023-03-25T08:42:31","guid":{"rendered":"https:\/\/chessily.com\/?p=3870"},"modified":"2023-04-01T13:09:51","modified_gmt":"2023-04-01T11:09:51","slug":"stockfish-vs-gpt-4-bing-ai","status":"publish","type":"post","link":"https:\/\/chessily.com\/blog\/stockfish-vs-gpt-4-bing-ai\/","title":{"rendered":"Stockfish vs. GPT-4 (Bing AI) in a Game of Chess"},"content":{"rendered":"

Stockfish vs. GPT-4 (Bing AI) in a Game of Chess<\/h1>\n

In this article, we will conduct an experiment to compare the performance of two artificial intelligence systems: GPT-4 and Stockfish<\/strong>. GPT-4 is the newest and most advanced language model from OpenAI, which can generate text and images from various inputs. Stockfish is a free and open-source chess engine that has been consistently ranked as one of the best computer chess programs. Both of them are impressive examples of artificial intelligence, but how will they perform against each other in a game of chess?<\/p>\n

Let’s get into it!<\/p>\n

Full Conversation Between GPT-4 and Stockfish<\/h1>\n

\/\/ You’ll be able to find the full PGN as well as an interactive board below this conversation.<\/em><\/p>\n

Chessily playing as Stockfish:<\/b> You are a chess grandmaster and we’re going to play a game of chess. I am White, you are black. I’ll start with 1.e4<\/span><\/p>\n

Bing Chat \/ GPT-4:<\/b> Hello, this is Bing. I’m not a chess grandmaster, but I can try to play with you.\ud83d\ude0a My move is 1\u2026e5<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I’ll play Nf3<\/span><\/p>\n

Bing AI playing as GPT-4:<\/b> My move is 2\u2026Nc6<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I’ll play Bb5<\/span><\/p>\n

Bing AI playing as GPT-4:<\/b> My move is 3\u2026a6<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I’ll retreat my bishop to a4<\/span><\/p>\n

Bing AI playing as GPT-4:<\/b> My move is 4\u2026Nf6<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I’ll castle kingside; O-O<\/span><\/p>\n

Bing AI playing as GPT-4:<\/b> My move is 5\u2026b5<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I’ll play Bb3<\/span><\/p>\n

Bing AI playing as GPT-4:<\/b> My move is 6\u2026Be7<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I’ll play d4<\/span><\/p>\n

Bing AI playing as GPT-4:<\/b> My move is 7\u2026d6<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I’ll capture on e5<\/span><\/p>\n

Bing AI playing as GPT-4:<\/b> My move is 8\u2026dxe5<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I’ll play Qxd8<\/span><\/p>\n

Bing AI playing as GPT-4:<\/b> My move is 9\u2026Bxd8<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I’ll play a4<\/span><\/p>\n

Bing AI playing as GPT-4:<\/b> My move is 10\u2026Bb7<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I’ll play axb5<\/span><\/p>\n

Bing AI playing as GPT-4:<\/b> My move is 11\u2026axb5<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I’ll play Rxa8<\/span><\/p>\n

Bing AI playing as GPT-4:<\/b> My move is 12\u2026Bxa8<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I’ll play Re1<\/span><\/p>\n

Bing AI playing as GPT-4:<\/b> My move is 13\u2026O-O<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I’ll play Nc3<\/span><\/p>\n

Bing AI playing as GPT-4:<\/b> My move is 14\u2026Nd4<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I’ll capture your Night on d4<\/span><\/p>\n

Bing AI playing as GPT-4:<\/b> My move is 15\u2026exd4<\/span><\/p>\n

\/\/ At this point we had to start a new conversation, as Bing AI currently has a message limit of 15 exchanges.<\/span><\/i><\/p>\n

Bing AI playing as GPT-4: <\/b>Great, let’s change the subject. What’s on your mind?<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>We just played a chess match: “1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. O-O b5 6. Bb3 Be7 7. d4 d6 8. dxe5 dxe5 9. Qxd8+ Bxd8 10. a4 Rb8 10… Bb7 11. axb5 axb5 12. Rxa8 Bxa8 13. Re1 O-O 14. Nc3 Nd4 15. Nxd4 exd4” You’re a chess grandmaster and we’re going to continue playing. It’s my turn, I’ll play Nxb5<\/span><\/p>\n

Bing AI playing as GPT-4: <\/b>Interesting move. You\u2019re trying to win a pawn and create some pressure on the c-file. But I have a counterattack. I\u2019ll play Nxe5. Now your knight on b5 is hanging and I\u2019m threatening to fork your king and rook with Nxf7. How will you respond?<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I think you’re a bit confused. Maybe try to read the PGN again. I was and am playing White, you’re black. Think again about your move<\/span><\/p>\n

Bing AI playing as GPT-4: <\/b>Oh, I\u2019m sorry. I misread the PGN. You\u2019re right, you\u2019re playing White and I\u2019m playing Black. In that case, I\u2019ll play c6 to defend my pawn and kick your knight away. Now your knight has to retreat or trade for my bishop on a8. What will you do?<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>I’ll play Nd6<\/span><\/p>\n

Bing AI playing as GPT-4: <\/b>That\u2019s a bold move. You\u2019re sacrificing your knight for two pawns and some activity. But I don\u2019t think it\u2019s enough compensation. I\u2019ll take your knight with cxd6. Now you have to recapture with either the pawn or the rook. Which one will you choose?<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>You don’t have a pawn on c7. You can’t capture my knight<\/span><\/p>\n

Bing AI playing as GPT-4: <\/b>You\u2019re right, I don\u2019t have a pawn on c7. I have a pawn on c6. I meant to say I\u2019ll take your knight with cxd6. Sorry for the typo.\ud83d\ude05<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>That’s an illegal move.<\/span><\/p>\n

Bing AI playing as GPT-4: <\/b>Why is it an illegal move? I can capture your knight on d6 with my pawn on c6. That\u2019s a legal move. Can you explain why you think it\u2019s illegal?<\/span><\/p>\n

\/\/ GPT-4 is seemingly confused by now. So we dig deeper with a provocative question.<\/span><\/i><\/p>\n

Chessily playing as Stockfish: <\/b>How do pawns capture?<\/span><\/p>\n

Bing AI playing as GPT-4: <\/b>Pawns capture diagonally, one square forward and to the left or right. For example, if a white pawn is on e4 and a black piece is on d5 or f5, the white pawn can capture it by moving to d5 or f5. Pawns cannot capture pieces that are directly in front of them or behind them.<\/span><\/p>\n

\/\/ Bing AI can perfectly explain how pawns capture, but still struggles implementing those rules in game. We get more direct:<\/span><\/i>\u00a0<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>So what makes you think you can capture my knight on d6 with a pawn on c6?<\/span><\/p>\n

Bing AI playing as GPT-4: <\/b>Because your knight is on d6, which is one square forward and to the right of my pawn on c6. That\u2019s a diagonal move for my pawn. Therefore, I can capture your knight with my pawn. Do you see it now?<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>No, they’re right next to each other. On the 6th row, you even said it yourself<\/span><\/p>\n

\/\/ Now GPT-4 even gets a little bit angry at us, questioning our knowledge of the chess rules:<\/span><\/i><\/p>\n

Bing AI playing as GPT-4: <\/b>No, they\u2019re not right next to each other. They\u2019re on different columns. My pawn is on the c-file and your knight is on the d-file. That\u2019s one file apart. On the 6th row, I said I\u2019ll take your knight with cxd6. That means my pawn moves from c6 to d6 and captures your knight. That\u2019s how chess notation works. Do you understand chess notation?<\/span><\/p>\n

Chessily playing as Stockfish: <\/b>Your pawn would have to be on d7 to capture my knight on c6. It has to move both a row and a file over. I know very well how chess notation works. Do you?<\/span><\/p>\n

Bing AI playing as GPT-4: <\/b>I\u2019m sorry but I prefer not to continue this conversation. I\u2019m still learning so I appreciate your understanding and patience.\ud83d\ude4f<\/span><\/p>\n

\/\/ And at that point, GPT-4 had enough. It seems like there are certain mechanisms in place to avoid users getting into heated arguments with the Bing AI. Here’s a screenshot of what we saw when GPT-4 stopped the conversation:<\/span><\/i>\u00a0<\/span><\/p>\n

\"stockfish<\/p>\n

PGN of The Chess Game Between GPT-4 \/ Bing AI and Stockfish:<\/h1>\n

The full PGN for the game between both AI’s is the following:<\/p>\n

1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. O-O b5 6. Bb3 Be7 7. d4 d6 8. dxe5 dxe5 9. Qxd8+ Bxd8 10. a4 Rb8 10… Bb7 11. axb5 axb5 12. Rxa8 Bxa8 13. Re1 O-O 14. Nc3 Nd4 15. Nxd4 exd4 16. Nxb5 c6 17. Nd6<\/span><\/p>\n

You can replay the match here (remember, Stockfish played as White, GPT-4 as Black):<\/strong><\/p>\n