California Gov. Gavin Newsom (D) signed a bill Monday placing new guardrails on how artificial intelligence (AI) chatbots interact with children and handle issues of suicide and self-harm.
S.B. 243, which cleared the state Legislature in mid-September, requires developers of “companion chatbots” to create protocols preventing their models from producing content about suicidal ideation, suicide or self-harm and directing users to crisis services if needed.
It also requires chatbots to issue “clear and conspicuous” notifications that they are artificially generated if someone could reasonably be misled to believe they were interacting with another human.
When interacting with children, chatbots must issue reminders every three hours that they are not human. Developers are also required to create systems preventing their chatbots from producing sexually explicit content in conversations with minors.
“Emerging technology like chatbots and social media can inspire, educate, and connect – but without real guardrails, technology can also exploit, mislead, and endanger our kids,” Newsom said in a statement.
“We’ve seen some truly horrific and tragic examples of young people harmed by unregulated tech, and we won’t stand by while companies continue without necessary limits and accountability,” he added. “We can continue to lead in AI and technology, but we must do it responsibly — protecting our children every step of the way. Our children’s safety is not for sale.”
The family of a California teenager sued OpenAI in late August, alleging that ChatGPT encouraged their 16-year-old son to commit suicide. The father, Matthew Raine, testified before a Senate panel last month, alongside two other parents who accused chatbots of driving their children to suicide or self-harm.
Growing concerns about how AI chatbots interact with children prompted the Federal Trade Commission (FTC) to launch an inquiry into the issue, requesting information from several leading tech companies.
Sens. Josh Hawley (R-Mo.) and Dick Durbin (D-Ill.) also introduced legislation late last month that would classify AI chatbots as products in order to allow harmed users to file liability claims.
The California measure is the latest of several AI and tech-related bills signed into law by Newsom this session. On Monday, he also approved measures requiring warning labels on social media platforms and age verification by operating systems and app stores.
In late September, he also signed S.B. 53, which requires leading-edge AI models to publish frameworks detailing how they assess and mitigate catastrophic risks.
