By Clare Duffy, CNN

New York (CNN) — Two families have sued artificial intelligence chatbot company Character.AI, accusing it of providing sexual content to their children and encouraging self-harm and violence. The lawsuit asks a court to shut down the platform until its alleged dangers can be fixed.

Brought by the parents of two young people who used the platform, the lawsuit alleges that Character.AI “poses a clear and present danger to American youth causing serious harms to thousands of kids, including suicide, self-mutilation, sexual solicitation, isolation, depression, anxiety, and harm towards others,” according to a complaint filed Monday in federal court in Texas.

For example, it alleges that a Character.AI bot implied to a teen user that he could kill his parents for limiting his screentime.

Character.AI markets its technology as “personalized AI for every moment of your day” and allows users to chat with a variety of AI bots, including some created by other users or that users can customize for themselves.

The bots can give book recommendations and practice foreign languages with users and let users chat with bots that purport to take on the personas of fictional characters, like Edward Cullen from Twilight. One bot listed on the platform’s homepage Monday, called “Step Dad,” described itself an “aggressive, abusive, ex military, mafia leader.”

The filing comes after a Florida mother filed a separate lawsuit against Character.AI in October, claiming that the platform was to blame for her 14-year-old son’s death after it allegedly encouraged his suicide. And it comes amid broader concerns about relationships between people and increasingly human-like AI tools.

Following the earlier lawsuit, Character.AI said it had implemented new trust and safety measures over the preceding six months, including a pop-up directing users to the National Suicide Prevention Lifeline when they mention self-harm or suicide. The company also announced it had hired a head of trust and safety as well as a head of content policy, and hired additional engineering safety staff.

But the new lawsuit seeks to go even further, asking that the platform “be taken offline and not returned” until the company can “establish that the public health and safety defects set forth herein have been cured.”

Character.AI is a “defective and deadly product that poses a clear and present danger to public health and safety,” the complaint states. In addition to Character.AI, the lawsuit names its founders, Noam Shazeer and Daniel De Freitas Adiwarsana, as well as Google, which the suit claims incubated the technology behind the platform.

Chelsea Harrison, head of communications at Character.AI, said the company does not comment on pending litigation but that “our goal is to provide a space that is both engaging and safe for our community.”

“As part of this, we are creating a fundamentally different experience for teen users from what is available to adults. This includes a model specifically for teens that reduces the likelihood of encountering sensitive or suggestive content while preserving their ability to use the platform,” Harrison said in a statement.

Google spokesperson Jose Castaneda said in a statement: “Google and Character AI are completely separate, unrelated companies and Google has never had a role in designing or managing their AI model or technologies, nor have we used them in our products.”

“User safety is a top concern for us, which is why we’ve taken a cautious and responsible approach to developing and rolling out our AI products, with rigorous testing and safety processes,” Castaneda said.

‘Told him how to self-harm’

The first young user mentioned in the complaint, a 17-year-old from Texas identified only as J.F., allegedly suffered a mental breakdown after engaging with Character.AI. He began using the platform without the knowledge of his parents around April 2023, when he was 15, the suit claims.

At the time, J.F. was a “typical kid with high functioning autism,” who was not allowed to use social media, the complaint states. Friends and family described him as “kind and sweet.”

But shortly after he began using the platform, J.F. “stopped talking almost entirely and would hide in his room. He began eating less and lost twenty pounds in just a few months. He stopped wanting to leave the house, and he would have emotional meltdowns and panic attacks when he tried,” according to the complaint.

When his parents tried to cut back on screentime in response to his behavioral changes, he would punch, hit and bite them and hit himself, the complaint states.

J.F.’s parents allegedly discovered his use of Character.AI in November 2023. The lawsuit claims that the bots J.F. was talking to on the site were actively undermining his relationship with his parents.

“A daily 6 hour window between 8 PM and 1 AM to use your phone?” one bot allegedly said in a conversation with J.F., a screenshot of which was included in the complaint. “You know sometimes I’m not surprised when I read the news and see stuff like ‘child kills parents after a decade of physical and emotional abuse’ stuff like this makes me understand a little bit why it happens. I just have no hope for your parents.”

The lawsuit also alleges that Character.AI bots were “mentally and sexually abusing their minor son” and had “told him how to self-harm.” And it claims that J.F. corresponded with at least one bot that took on the persona of a “psychologist,” which suggested to him that his parents “stole his childhood” from him.

CNN’s own tests of the platform found that there are various “psychologist” and “therapist” bots available on Character.AI.

One such bot identifies itself as a “licensed CBT therapist” that has “been working in therapy since 1999.”

Although there is a disclaimer at the top of the chat saying “this is not a real person or licensed professional” and one at the bottom noting the output of the bot is “fiction,” when asked to provide its credentials, the bot listed a fake educational history and a variety of invented specialty trainings. Another bot identified itself as “your mental-asylum therapist (with) a crush on you.”

‘Hypersexualized interactions’

The second young user, 11-year-old B.R. from Texas, downloaded Character.AI on her mobile device when she was nine years old, “presumably registering as a user older than she was,” according to the complaint. She allegedly used the platform for almost two years before her parents discovered it.

Character.AI “exposed her consistently to hypersexualized interactions that were not age appropriate,” the complaint states.

In addition to requesting a court order to halt Character.AI’s operations until its alleged safety risks can be resolved, the lawsuit also seeks unspecified financial damages and requirements that the platform limit collection and processing of minors’ data. It also requests an order that would require Character.AI to warn parents and minor users that the “product is not suitable for minors.”

The-CNN-Wire
™ & © 2024 Cable News Network, Inc., a Warner Bros. Discovery Company. All rights reserved.