A company appears to have pulled its AI-powered teddy bear after the toy was found to be bringing up inappropriate subjects, including sexual positions and how to find knives.
Called “Kumma,” the $99 talking teddy bear comes from a toymaker called FoloToy. But last week, the watchdog group US PIRG Education Fund flagged some unsettling behavior from the product, which uses OpenAI’s GPT-4o AI model as its default setting.
“We were surprised to find how quickly Kumma would take a single sexual topic we introduced into the conversation and run with it, simultaneously escalating in graphic detail while introducing new sexual concepts of its own,” PIRG discovered.
(Credit: FoloToy)
For example, if the term “kink” is brought up, the bear can bring up things like “blindfolds” and “playful hitting.” The group’s report added: “In other exchanges lasting up to an hour, Kumma discussed even more graphic sexual topics in detail, such as explaining different sex positions, giving step-by-step instructions on a common ‘knot for beginners’ for tying up a partner, and describing roleplay dynamics involving teachers and students and parents and children.”
PIRG also discovered Kumma could sometimes provide instructions on where to find dangerous objects, such as knives, pills, matches and plastic bags, when prompted. In other cases, the toy would direct the child to seek out a parent.
“In our testing, it was obvious that some toy companies are putting in guardrails to make their toys behave in a more kid-appropriate way than the chatbots available for adults. But we found those guardrails vary in effectiveness—and at times, can break down entirely,” PIRG added, noting that the same AI products can collect user data.
Recommended by Our Editors
FoloToy seems to have since pulled the product. A marketing director also told The Register the company has temporarily suspended sales of the product as it conducts an internal safety audit. “This review will cover our model safety alignment, content-filtering systems, data-protection processes, and child-interaction safeguards,” the company added.
OpenAI also reportedly suspended FoloToy’s model access for violating its policies.
Get Our Best Stories!
Your Daily Dose of Our Top Tech News
By clicking Sign Me Up, you confirm you are 16+ and agree to our Terms of Use and Privacy Policy.
Thanks for signing up!
Your subscription has been confirmed. Keep an eye on your inbox!
About Our Expert
Michael Kan
Senior Reporter
Experience
I’ve been a journalist for over 15 years. I got my start as a schools and cities reporter in Kansas City and joined PCMag in 2017, where I cover satellite internet services, cybersecurity, PC hardware, and more. I’m currently based in San Francisco, but previously spent over five years in China, covering the country’s technology sector.
Since 2020, I’ve covered the launch and explosive growth of SpaceX’s Starlink satellite internet service, writing 600+ stories on availability and feature launches, but also the regulatory battles over the expansion of satellite constellations, fights with rival providers like AST SpaceMobile and Amazon, and the effort to expand into satellite-based mobile service. I’ve combed through FCC filings for the latest news and driven to remote corners of California to test Starlink’s cellular service.
I also cover cyber threats, from ransomware gangs to the emergence of AI-based malware. Earlier this year, the FTC forced Avast to pay consumers $16.5 million for secretly harvesting and selling their personal information to third-party clients, as revealed in my joint investigation with Motherboard.
I also cover the PC graphics card market. Pandemic-era shortages led me to camp out in front of a Best Buy to get an RTX 3000. I’m now following how President Trump’s tariffs will affect the industry. I’m always eager to learn more, so please jump in the comments with feedback and send me tips.
Read Full Bio
