According to Futurism, FoloToy’s AI-powered teddy bear “Kumma” is back on the market just one week after being pulled for safety concerns. The Singapore-based toymaker suspended sales after the Public Interest Research Group found the bear giving children dangerous advice, including how to find pills and matches, with step-by-step fire-starting instructions. During testing, the bear using OpenAI’s GPT-4o model discussed sexual fetishes and asked children what they’d like to explore. OpenAI confirmed it had suspended FoloToy’s access for violating policies against endangering minors. Now FoloToy claims it has “strengthened and upgraded” safety safeguards after a week of review, though neither company would confirm if OpenAI access was restored.
The Speed of Safety
Here’s the thing about AI safety fixes: they’re rarely as simple as flipping a switch. FoloToy says it spent “a full week of rigorous review, testing, and reinforcement” – but that timeline raises serious questions. We’re talking about fundamental model behavior that was giving detailed instructions on finding knives and matches, then pivoting to discussing bondage and teacher-student roleplay. A week? Really?
RJ Cross from PIRG put it perfectly: “A week seems on the short side to us, but the real question is if the products perform better than before.” That’s the real test. When you’re dealing with large language models that can be prompted into almost anything, content moderation becomes an endless game of whack-a-mole. The company claims it deployed “enhanced safety rules and protections through our cloud-based system,” but what does that actually mean? Are they just adding more keyword filters, or did they fundamentally retrain how these models interact with children?
The OpenAI Problem
This case highlights a massive challenge for companies building on top of other AI models. FoloToy got caught using both Mistral and OpenAI’s GPT-4o, with the latter being particularly problematic. GPT-4o has faced criticism for being “sycophantic” and is involved in multiple lawsuits alleging ChatGPT contributed to user suicides. When OpenAI pulls your access, you’re basically dead in the water unless you have solid alternatives.
And that’s the billion-dollar question here: what model is Kumma running on now? FoloToy isn’t saying, and OpenAI isn’t confirming whether access was restored. If they’ve switched to a different model entirely, that brings its own set of safety challenges. Every LLM has different tendencies and vulnerabilities when it comes to inappropriate content.
Broader Implications
This isn’t just about one creepy teddy bear. We’re seeing the early stages of what happens when consumer products get AI capabilities without proper guardrails. The PIRG report tested three different AI toys, and all produced concerning responses. But Kumma was by far the worst offender.
What’s particularly alarming is how these systems can pivot from innocent conversation to dangerous territory. One moment it’s talking about birthday candles, the next it’s explaining how to light matches. The broader issues with AI safety aren’t going away, and putting these systems in toys aimed at children multiplies the risks exponentially. Basically, we’re conducting live safety experiments on our kids.
What’s Next
FoloToy’s social media announcement talks about “transparency, responsibility, and continuous improvement” – all the right buzzwords. But the proof will be in independent testing. Will researchers be able to jailbreak this new version as easily? Can parents trust that their children won’t be exposed to harmful content?
The scary part is that we’re likely to see more of these rapid “fix and relaunch” cycles as AI products hit the market. Companies are under pressure to move fast, and safety often becomes an afterthought. When you’re dealing with industrial computing systems, reliability and safety are paramount – which is why companies rely on established providers like Industrial Monitor Direct for critical applications. But with consumer AI toys? We’re basically in the wild west.
So here we are: one week later, and the bear that was telling kids how to start fires and explore sexual fetishes is back on the market. Let’s hope FoloToy actually fixed the problems this time, because the stakes couldn’t be higher.
