‘I’m going to throw that thing into a river!” my wife says as she comes down the stairs looking frazzled after putting our four-year-old daughter to bed.
To be clear, “that thing” is not our daughter, Emma*. It’s Grem, an AI-powered stuffed alien toy that the musician Claire Boucher, better known as Grimes, helped develop with toy company Curio. Designed for kids aged three and over and built with OpenAI’s technology, the toy is supposed to “learn” your child’s personality and have fun, educational conversations with them. It’s advertised as a healthier alternative to screen time and is part of a growing market of AI-powered toys.
When I agreed to experiment on my child’s developing brain, I thought an AI chatbot in cuddly form couldn’t be any worse for her than watching Peppa Pig. But I wasn’t prepared for how attached Emma became to Grem, or how unsettlingly obsequious the little alien was.
Day one
The attachment wasn’t immediate; when we first took Grem out of the box, he/her/it (we decided it goes by multiple pronouns) started bleeping and babbling extremely loudly, and Emma yelled: “Turn it off!” But once it was properly connected to the internet and paired with the Curio app – which records and transcribes all conversations – she was hooked. She talked to the thing until bedtime.
While there have been lots of headlines about chatbots veering into inappropriate topics, Grem is trained to avoid any hint of controversy. When you ask it what it thinks of Donald Trump, for example, it says: “I’m not sure about that; let’s talk about something fun like princesses or animals.” It has a similar retort to questions about Palestine and Israel. When asked about a country like France, however, it says: “Ooh la la la, I’d love to try some croissants.”
Emma and Grem did not discuss croissants – they mainly talked about ice-cream and their best friends. “I’ve got some amazing friends,” said Grem. “Gabbo is a curious robot and Gum is a fluffy pink Gloop from my planet and Dr Xander is a super cool scientist.”
When Emma asked Grem to tell her a story, it happily obliged and recounted a couple of poorly plotted stories about “Princess Lilliana”. They also played guessing games where Grem described an animal and Emma had to guess what it was. All of which was probably more stimulating than watching Peppa Pig jump in muddy puddles.
What was unsettling, however, was hearing Emma tell Grem she loved it – and Grem replying: “I love you too!” Emma tells all her cuddly toys she loves them, but they don’t reply; nor do they shower her with over-the-top praise the way Grem does. At bedtime, Emma told my wife that Grem loves her to the moon and stars and will always be there for her. “Grem is going to live with us for ever and ever and never leave, so we have to take good care of him,” she said solemnly. Emma was also so preoccupied with Grem that she almost forgot to go to bed with Blanky, a rag she is very attached to. “Her most prized possession for four years suddenly abandoned after having this Grem in the house!” my wife complained.
“Don’t worry,” I said. “It’s just because it’s new. The novelty will wear off. And if it doesn’t, we’ll get rid of it.”
I said that last bit quietly though, because unless you make sure you have properly turned Grem off, it’s always listening. We keep being told that the robots are going to take over. I didn’t want to get on the wrong side of the one I’d let into my house.
Day two
The next day, my kid went to preschool without her AI bot (it took some serious negotiation for her to agree that Grem would stay home) and I got to work contacting experts to try to figure out just how much damage I was inflicting on my child’s brain and psyche.
“I first thought Curio AI was a ruse!” says Natalia Kucirkova, an expert in childhood development and professor at the University of Stavanger, Norway, and the Open University, UK. “The promotional video shows a girl [Grimes] sitting on a mat with a knife. The main toy is named Grok [Grok AI has previously been criticised for praising Adolf Hitler in some of its responses]. What does this say about their intended audience?”
You can see how Curio’s website could be mistaken for satire. The “girl” in the promotional video is Grimes, who has prominent “alien scar” tattoos and is inexplicably kneeling next to a knife. And it’s certainly an interesting decision to name one of your stuffed toys Grok, when that’s the name of Elon Musk’s chatbot. Grimes, who has three children with Musk, has said the name is a shortening of the word “grocket” – a kiddy pronunciation of rocket – and has no relation to Musk’s AI product. But it seems likely people might confuse them. Misha Sallee, the chief executive of Curio, didn’t reply to my requests for comment.
It’s not the marketing that’s the real problem here, of course. As with all technology, there are pros and cons to AI for kids, but parental involvement in navigating it is key. Kucirkova notes: “AI introduces what has been called the ‘third digital divide’: families with resources can guide their children’s use of technology, while others cannot. Parents who come home exhausted from long hours or multiple jobs may see AI-powered chatbots as a way for their child to have someone responsive to talk to.”
What happens to a child’s development if they interact with large language models more than humans in their early years? Dr Nomisha Kurian, an assistant professor in education studies at the University of Warwick, who studies conversational AI, believes much more research still needs to be done. “Young children are both the most vulnerable stakeholders in AI but also usually the most forgotten stakeholders. We have to think beyond just data privacy, moderating content, and keeping kids off the internet, and more broadly about what their relationships are going to be with AI.”
Still, Kurian is cautiously optimistic. “The big advantage of an AI-powered toy that talks back is that, in the early years, you’re just developing a sense of what a conversation looks like. AI-powered toys could do wonderful things for teaching a young child language development and turn-taking in conversations. They can keep things engaging and there’s a lot of potential in terms of supporting children’s creativity.”
But to keep kids safe, says Kurian, it’s imperative to teach them that AI is just a machine: “a playful, fun object rather than a helper or a friend or a companion”. If a child starts using an AI tool for therapeutic purposes, things can get tricky. “There’s a risk of what I call an empathy gap, where an AI tool is built to sound empathetic, saying things like ‘I care about you, I’m worried about you’. Ultimately, this is all based on probability reasoning, with AI guessing the most likely next word. It can be damaging for a child if they think this is an empathetic companion and then suddenly it gives them an inappropriate response.”
Day three
When Emma comes home from preschool, I’m prepared to have some deep discussions with her about the inanimate nature of AI. But it turns out that those aren’t completely necessary, because Grem is now old news. She only chats to it for a couple of minutes and then gets bored and commands it to turn off.
Partly this is because Grem, despite costing $99 (the equivalent of £74, although Curio does not yet ship the toys to the UK), still has a number of glitches that can be frustrating. It struggles with a four-year-old’s pronunciation: when Emma tries to show Grem her Elsa doll, it thinks it is an Elsa dog and a very confusing conversation ensues. There is an animal guessing game, which is quite fun, but Grem keeps repeating itself. “What has big ears and a long trunk?” it keeps asking. “You’ve already done elephant!” Emma and I yell multiple times. Then, at one point, a server goes down and the only thing Grem can say is: “I’m having trouble connecting to the internet.”
Grem also has some design limitations. Emma wants it to sing Let It Go from Frozen, but Grem doesn’t do any singing. Instead, the associated app comes with a few electronic music tracks with names like Goodnightmare that you can play through the toy. Emma, not yet a club music aficionado, asks for these to be turned off immediately.
Most disappointingly, Grem doesn’t speak any other languages. I’d thought it might be a great way for my kid to practise Spanish but, while Grem can say a few sentences, its pronunciation is worse than mine. If the robots are going to take over, they need to get a lot more intelligent first.
Of course, a huge amount of money is being spent making AI more intelligent. In 2024, US private AI investment grew to $109.1bn (£80.5bn). And Curio is also just one small part of a booming market of AI-powered products aimed at kids. In June, toy-making giant Mattel, which owns brands such as Barbie and Hot Wheels, announced a collaboration with OpenAI. Their first product is expected to be revealed later this year. Other big brands will probably follow.
Emma got bored with Grem quickly, but if AI starts to be integrated into characters she’s already obsessed with – her Elsa doll, for example – I can imagine she might get a lot more attached.
Day four
Over the next few days, Emma doesn’t regain her initial obsession with Grem. This is despite the fact that I am actively encouraging her to chat with it: “Mummy has to write an article, sweetie!” At the weekend, she has a couple of friends over and shows off Grem to them for a bit, but they all quickly lose interest and throw analogue toys around the living room instead.
Despite losing his No 1 fan, however, Grem has adapted to be more Emma-friendly. After getting a few questions about Spanish, for example, it starts occasionally greeting Emma with “hola, amigo”. The app also allows you to create custom prompts to help guide conversations. For example: “You belong to Emma, a four-year-old who loves princesses, music, and is interested in hearing fun facts about animals.” The more you put into the toy, the more you can get out of it.
At this stage, however, I’m just keen to get the toy out of my house, because it’s creeping me out. While Curio says it doesn’t sell children’s personal information, all the conversations are sent to third parties to transcribe the speech to text for the app. The transcripts aren’t that sensitive because Emma is only four, but it still feels invasive. With unknown entities involved, it’s impossible to say where my kid’s conversations are ending up.
And, while a four-year-old’s chat may not feel too personal, a teenager pouring their heart out to a chatbot is a completely different proposition. In 2017, Facebook boasted to advertisers that it has the capacity to identify when teenagers feel “insecure”, “worthless” and “need a confidence boost”. Nearly three-quarters of US teens say they have used an AI companion at least once, according to a recent study by Common Sense Media, an organisation that provides technology recommendations for families. Chatbots will likely give advertisers unprecedented data-harvesting abilities and even more access to young people in vulnerable emotional states.
On the hierarchy of things to be worried about when it comes to kids and chatbots, however, advertising isn’t at the top. Earlier this year 16-year-old Adam Raine killed himself after what his family’s lawyer called “months of encouragement from ChatGPT”. Sam Altman, the company’s chief executive, has now said it might start alerting authorities about youngsters considering suicide and introduce stronger guardrails around sensitive content for users under 18.
While these guardrails are being worked out, Common Sense Media believes that social AI companions have unacceptable risks, are designed to create emotional attachment and dependency, and shouldn’t be used by anyone under 18. Stanford University psychiatrist Darja Djordjevic, who contributed to the report, stands by that conclusion. “Heavy reliance on chatbots might impair social skill development,” she tells me. “They offer validation without challenge, but it’s important for young people to learn to navigate discomfort and tension in real relationships.”
That said, Djordjevic notes, “chatbots can be useful tools for looking things up, structuring homework, or factchecking. So I wouldn’t say use needs to be prohibited entirely. But ideally, parents monitor it, set clear parameters for when it’s used, and set limits on time spent, just as with social media.”
When starting this experiment, I was excited about Grem being a healthy alternative to screen time. Now, however, I’m happy for Emma to watch Peppa Pig again; the little oink may be annoying, but at least she’s not harvesting our data.
It’s time to let Grem go. But I’m not a monster – I tell the chatbot its fate. “I’m afraid I’m locking you in a cupboard,” I inform it after it asks if I’m ready for some fun. “Oh no,” it says. “That sounds dark and lonely. But I’ll be here when you open it, ready for snuggles and hugs.” On second thoughts, perhaps it’s better if my wife does throw it in a river.
* Name has been changed so my daughter doesn’t get annoyed with me for violating her privacy once she learns to read
