Video games these days involve chatting, and when people are given the chance to chat, they’re not always going to keep it G-rated. The popular online game-creation platform Roblox, among many other games, has used simple substitution systems to censor chats for years. The game would simply remove the offending word and replace it with a bunch of pound signs. Now, Roblox is tapping AI to rephrase the offending phrase rather than just block out its letters.
“Using AI filters to block problematic text (visible to users as #####) has always been central to our approach to safety and increasing civility,” said Rajiv Bhatia, Roblox VP of User and Discovery Product, in a blog post. “Today, we’re adding an additional feature to help keep in-experience text chat civil: We’re leveraging AI to automatically rephrase messages, starting with profanity.”
When a player uses a problematic word in chat, the AI will rewrite the message to remove it before posting it to the chat, thereby eliminating the pound signs. So, “oh shit, are you OK?” turns into, “are you OK?” And “hurry tf up” becomes “hurry up.” (Yes, even though that last one doesn’t spell it out, Roblox knows what the F stands for.)
Everyone in the chat will see that AI has rephrased the message, but only the sender will receive a warning about using improper language in chat.
The change comes just two months after Roblox introduced facial recognition, which sorts users into age groups to limit contact between children and adults. It hasn’t stopped legal action. Nebraska filed a lawsuit on Wednesday targeting Roblox’s alleged child-safety failures, joining Texas, Florida, Louisiana, Kentucky and several municipalities.
Users who send bad words will be notified that their sentence was rewritten by AI and warned about following the rules.
Language as it evolves
People make up new slang all the time, and much of it can be considered harsh or harmful. The company says the AI is being trained on “a set of specialized (AI) models” based on a larger model and in-game chat samples to help block current lingo. New slang is handled when the AI system can’t decide whether a new word or phrase is actually bad, sending it to a larger, more highly trained AI model, which can then deduce its meaning from context.
Players who repeatedly attempt to circumvent the new system or continue using banned words will still be disciplined as they normally would under the game’s Community Standards rules.
The new system may still produce false positives and take time to learn new lingo, but Roblox is hopeful that it will get it right most of the time.
“Our experiments show that this combined approach has significantly improved our filters,” Bhatia said. “The filters can now better detect leet-speak, or letters replaced with numbers or symbols, and more sophisticated attempts to bypass our filters.”
