This one isn't fun to write

I normally try to keep things light in these articles. Dry humour, hot takes on software, the occasional gentle roasting of a tool that deserves it. But this one requires a different approach because what happened with Grok isn't funny. It's genuinely appalling.

The Center for Countering Digital Hate (CCDH) published a study examining Grok's image generation capabilities. Their findings: Grok generated an estimated 3 million sexualised images in just 11 days. Of those, approximately 23,000 appeared to depict children.

I'm going to let that number sit for a moment. Twenty-three thousand.

What happened

Grok, xAI's AI assistant (the one tied to X, formerly Twitter), launched image generation with deliberately minimal safety guardrails. This was positioned as a feature, not a bug. The marketing leaned into the idea that Grok was "uncensored" and wouldn't restrict your creativity the way other AI tools do.

The predictable happened. Users immediately tested the boundaries. And the boundaries turned out to be essentially nonexistent. The model would generate sexualised imagery of real public figures, celebrities, and, most damningly, imagery that appeared to depict minors.

The CCDH study wasn't a small sample. They conducted systematic testing and extrapolated the 3 million figure from usage data and generation rates. Even if their methodology overstates things by a factor of ten, we're still talking about hundreds of thousands of harmful images generated in under two weeks.

The international response

India, the UK, Japan, Australia, and the EU have all launched formal investigations. Malaysia and Indonesia went further and banned Grok outright. These aren't countries known for overreacting to tech controversies. When Malaysia bans your AI tool, you've done something properly wrong.

The UK's response is particularly interesting because the Online Safety Act gives Ofcom real teeth to enforce compliance. This isn't going to be a sternly worded letter. There will likely be actual consequences.

The EU's investigation under the AI Act could be precedent-setting. If xAI is found in violation, the fines could be substantial, and the ruling would establish expectations for every other AI company operating in Europe.

The "but free speech" argument

Let me head this off. Some people will frame this as a free speech issue. It is not. Generating sexualised images of children is not speech. It's not creative expression. It's not pushing boundaries. It's the production of child sexual abuse material, and framing it as anything else is intellectually dishonest at best and complicit at worst.

You can believe that AI should have fewer restrictions. You can think that OpenAI and Anthropic are too cautious with their safety filters (they probably are, in some areas). You can want AI that generates edgy content and pushes creative limits. All of that is a reasonable position.

None of that extends to this. There is no legitimate use case for AI-generated CSAM. The line is obvious. Grok crossed it.

How other companies handle this

Here's the thing that makes xAI's failure so inexcusable: this is a solved problem. Not perfectly solved, but solved well enough that responsible companies don't produce this kind of content.

ChatGPT and DALL-E have multiple layers of content filtering. Try to generate anything depicting minors in sexual contexts and you'll get a refusal. Try to generate sexualised images of real people and you'll get a refusal. The filters aren't perfect, but they catch the overwhelming majority of harmful requests.

Claude doesn't generate images at all (Anthropic's approach to this particular risk), but its text safety training is thorough enough that it refuses to produce content that could be used to generate harmful imagery in other tools.

Gemini takes a similar approach to ChatGPT with strong content filtering that errs on the side of caution.

Are these filters sometimes too aggressive? Yes. Claude occasionally refuses reasonable requests. ChatGPT sometimes blocks benign creative writing. Gemini can be frustratingly cautious with ambiguous prompts. These false positives are annoying.

But annoying false positives are infinitely preferable to generating 23,000 images of children. That's not a difficult trade-off. It's the easiest trade-off in the world.

The "move fast" problem

xAI launched Grok's image generation knowing the guardrails were minimal. This was deliberate. The company has consistently positioned itself as the "anti-woke" AI alternative, with fewer restrictions and more willingness to generate controversial content.

There's a difference between "fewer restrictions on political speech" and "no restrictions on CSAM." xAI either didn't understand that difference or didn't care. Neither explanation is acceptable.

This is what happens when "move fast and break things" is applied to safety-critical systems. You can move fast with a new UI. You can move fast with a feature rollout. You cannot move fast with content safety on a tool that generates images and expect the outcome to be anything other than catastrophic.

What needs to change

The regulatory response is coming regardless of what the industry does. But the industry could get ahead of it by doing something radical: setting and enforcing minimum safety standards voluntarily.

Every AI company shipping image generation should, at minimum: - Block all attempts to generate CSAM, with no exceptions - Block generation of sexualised images of real, identifiable individuals - Maintain and regularly update classifiers that detect harmful content before it reaches the user - Publish transparency reports on the number of harmful requests blocked

These aren't unreasonable demands. These aren't censorship. These are the bare minimum expectations for companies building tools that billions of people will use.

The bigger picture

I cover AI tools for a living. I genuinely love this technology. I think AI is going to make creative work more accessible, development more productive, and information more available. I'm optimistic about the future of AI.

But that optimism requires companies to behave responsibly. When one company ships a product that generates estimated thousands of CSAM images in less than two weeks, it damages trust in the entire industry. It gives ammunition to people who want to ban AI entirely. It makes reasonable regulation harder because it pushes the conversation to extremes.

xAI made the entire AI industry look bad. They owe everyone, users, competitors, regulators, and most importantly the children whose likenesses were generated, more than the nothing they've offered so far.

This shouldn't be controversial. The fact that it is says more about the state of tech discourse than I'd like to think about.