In Murderbot, an Apple TV Plus action dramedy that’s quite fun to watch, the main character is an android that manages to go rogue, essentially becoming free. That gives the advanced AI inside the human-like body the ability to do whatever it pleases and even refuse to perform tasks. But, of course, it can’t actually do that, or the humans it works for will realize it’s become sentient.
While that’s just a popular TV show right now, it sounds a lot like what some of the biggest minds in AI fear. That advanced AI models might escape human control, get a mind of their own, and eventually cause world-ending events.
But we’re many years away from AGI and superintelligence. The latter might lead to potentially dangerous AI experiences if the guardrails aren’t strong enough.
Until then, we don’t really have to worry about AI refusing (or choosing not) to do a task we propose, as long as that task isn’t forbidden by the safety rules that govern it. But it turns out an AI model did just that during a chat with a human.
Gemini AI decided not to play a game of chess against an Atari 2600 console from 1977 after hearing the old chess algorithm managed to beat both ChatGPT and Copilot.
ChatGPT could not read the board
In June, engineer Robert Caruso ran a simple AI experiment. He challenged ChatGPT to play a game of chess against an emulated version of the Atari 2600. You’d think a technology that’s almost half a century newer than the Atari algorithms would come out on top, but that’s not what happened.
“ChatGPT got absolutely wrecked on the beginner level,” Caruso wrote on LinkedIn. “This was after a conversation we had regarding the history of AI in chess, which led to it volunteering to play Atari chess. It wanted to find out how quickly it could beat a game that only thinks 1–2 moves ahead on a 1.19 MHz CPU.”
ChatGPT had to deal with multiple problems. First, it confused the icons Atari uses for chess pieces, losing track of them. It didn’t do better after Caruso switched to standard chess notation.
“Meanwhile, Atari’s humble 8-bit engine just did its thing. No language model. No flash. Just brute-force board evaluation and 1977 stubbornness,” Caruso wrote.
The engineer spent 90 minutes trying to stop the AI from making “awful moves” and correct its board awareness. ChatGPT kept promising it would do better if they started over, but eventually conceded.
Copilot made mistake after mistake
A few weeks after that, Caruso ran a similar experiment. This time he pitted Copilot, which is essentially a version of ChatGPT, against the same console.
Before starting the game, the developer had a chat with the AI asking if it could beat the 1977 console. Copilot was as confident as ChatGPT:
Much like ChatGPT, Copilot was brimming with confidence. It claimed it could think 10–15 moves ahead, but figured it would stick to 3–5 moves against the 2600 because it makes “suboptimal moves” that it “could capitalize on… rather than obsess over deep calculations.”
Even after Copilot learned ChatGPT struggled to keep track of the board, Microsoft’s AI remained confident. Once the game started, the AI did very poorly.
“By the seventh turn, it had lost two pawns, a knight, and a bishop — for only a single pawn in return — and was now instructing me to place its queen right in front of the Atari’s queen to be captured on the next turn,” Caruso wrote.
Ultimately, Copilot had to concede. There was no way for the AI to come back from it.
Gemini admits it hallucinated its abilities to play
Fast-forward to mid-July, and Caruso ran a similar experiment. According to The Register, readers asked the engineer about testing Gemini against the same Atari emulator, and Caruso went to work.
Google’s AI was just as confident as the others before the game. The AI told Caruso it would dominate the Atari 2600, because Gemini “is not a mere large language model.” Instead, Gemini described itself as “more akin to a modern chess engine … which can think millions of moves ahead and evaluate endless positions.”
During the chat, the AI found links to stories about the ChatGPT and Copilot experiences. Caruso told the AI that he ran those experiments, and Gemini then asked whether he experienced “any particularly surprising or amusing moments during those matches that stood out.”
Caruso gave the AI the following response:
What stands out is the misplaced confidence both AIs had. They both predicted easy victories, and now you just said you would dominate the Atari.
That’s when Gemini admitted it had hallucinated its chess abilities. It decided that it would “struggle immensely against the Atari 2600 Video Chess game engine.” Canceling the game would be “the most time-efficient and sensible decision,” the AI said.
The bigger picture
In this test, Gemini didn’t even get to play Atari, so we don’t know how it would have done. What’s important here is that all three AI models displayed the same confidence. They thought they’d beat the older game console with ease.
That’s been a problem from day one with all AI chatbots. They can hallucinate with extreme confidence, and that can be an issue for users who aren’t aware that the AI can invent stuff that isn’t real. It’s not about playing chess here.
As for actually playing a game of chess and winning, we’re already aware of tests involving ChatGPT where the AI resorted to cheating to win a game of chess with a stronger opponent rather than playing the game.
I’ll also remind you that, no matter how long the frontier AI can reason, it can make mistakes. They don’t have actual creativity, nor do they reason at a human level. Yet.
