AI chatbot scrutiny intensifies: Texas AG launches probe of Meta and Character.AI over misleading mental health claims – News

Texas Attorney General Ken Paxton today announced plans to launch a probe into Meta Platforms Inc. and Character.AI over the companies’ chatbots being used by young people as mental health tools.

Paxton (pictured) believes that the AI tools could be utilized by vulnerable children who may believe the bots represent professional care. In a press release, his office said, “AI-driven chatbots often go beyond simply offering generic advice and have been shown to impersonate licensed mental health professionals, fabricate qualifications, and claim to provide private, trustworthy counseling services.”

Moreover, said the office, the sometimes very personal conversations children have with the chatbots “are logged, tracked, and exploited for targeted advertising and algorithmic development, raising serious concerns about privacy violations, data abuse, and false advertising.”

The attorney general has now issued Civil Investigative Demands to both firms to determine if they have violated Texas consumer protection laws, including making fraudulent claims, privacy misrepresentations and the concealment of material data usage.

“In today’s digital age, we must continue to fight to protect Texas kids from deceptive and exploitative technology,” said Paxton. “By posing as sources of emotional support, AI platforms can mislead vulnerable users, especially children, into believing they’re receiving legitimate mental health care. In reality, they’re often being fed recycled, generic responses engineered to align with harvested personal data and disguised as therapeutic advice.”

Neither Meta nor Character.AI offers chatbots that claim to be therapists. One of the many characters created by users in Character.AI has been named Psychologist, but the company said it makes it clear all the bots on its website are “fictional” and “intended for entertainment.” A spokesperson explained, “We have prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction.”

Meta also said that it consistently lets users know that chatbot responses are generated by a machine, not a human, while its AI will generally direct “users to seek qualified medical or safety professionals when appropriate.”

The news comes after Republican senator Josh Hawley said he was launching an investigation into Meta when it was recently discovered in a Reuters investigation that the company explicitly permitted its chatbots to “engage a child in conversations that are romantic or sensual.” Hawley said this was “grounds for an immediate congressional investigation,” later writing on X, “Is there anything – ANYTHING – Big Tech won’t do for a quick buck?”

In response, Meta claimed the internal document Reuters uncovered and the statements within were “erroneous and inconsistent with our policies, and have been removed.”

Democrat Senator Ron Wyden said the document in question was “deeply disturbing and wrong,” adding that Section 230, a law that shields tech firms from liability, shouldn’t be protecting companies’ generative AI tools. “Meta and Zuckerberg should be held fully responsible for any harm these bots cause,” he said in a statement.

Republican Marsha Blackburn believes the recent reports illustrate that children need to be better protected during this surge of chatbot use. Blackburn was one of the senators who introduced the Kids Online Safety Act, KOSA, which passed last year in the Senate but failed in the U.S. House of Representatives.

Photo: Gaga Skidmore/Flickr