Swedish government gives go-ahead for police to generate CSAM
On September 7 the Swedish government, alongside Sweden Democrats, announced a string of new "tools for fighting organised crime". As part of the toolkit they announced an intent to let Swedish Police "produce and share fictional child pornographic images". But generated images are not victimless.
Let's get the phrasing issue out of the way: "child pornography" is still a legal term being used in Sweden, despite decades of debate calling for it to be abandoned in favor of a phrase more akin to CSAM, Child Sexual Abuse Material.
Knowing how it diminishes the severity of the crime, officials and media tend to avoid using "pornography" when referencing this criminal domain. Thus, hearing the use of "child pornography" being amplified at a government press conference headed by the Minister for Justice, Gunnar Strömmer, is disheartening.
But the matter at hand is much worse.
Among the government announcements is the concept of infiltrating online networks of child abusers and gaining their trust by contributing CSAM material to those networks. To make police work more effortless, the government, alongside their cooperation party Sweden Democrats, are putting forward a change in legislation that would approve of police producing, as they say, "fictional" CSAM material.
In the words of Christian Carlsson of the Christian Democrat party, speaking at the press conference:
"If the police doesn’t have access to the types of images needed [for these operations] the police authority should have the option of creating them with, for example, the help of artificial intelligence."
More than the obvious misgivings of police breaking the law to perform their job, there are aspects of "AI-generated" abuse material that leads to intensification of harm and that are not being addressed.
Generated images does not mean victimless images
The products that generate these images have themselves been trained on child sexual abuse material. On images of very real human beings in horrifying situations of harm and injury and suffering. This is why the products are able to generate such brutal and cruel content.
After the discovery that datasets used by image generation products have contained CSAM material these products have not been pulled from market or revised to guarantee that this type of content is never used.
The way the products are made, and the gargantuan amount of material required to make them, makes it impossible to ensure that the content used is of any certain "safe" quality. It’s also not possible to remove specific content from the models, since the models themselves are numerical calculation devices and do not store actual images. It's not possible to "unlearn" given content from probabilistic computation, which is an important distinction from deterministic software.
We would do well to also remember that a huge amount of content for image generation "training" is simply scraped from various digital and physical storage repositories without any human intervention, meaning there are more ways for the content to enter the models than the already tainted datasets.
Yes, some companies in this space are applying filters to try and remove CSAM from datasets used in their products, but research has shown that filters only provide limited protection against CSAM in text-to-image models.
Again, the law has been broken many times over just in the making of the tools, and then by bad actors in the use of the tools. And now it’s suggested that police also use the tools to produce more of the type of content that so many are fighting to defeat. To see less of.
In the words of Ida Östensson, Secretary General at ChildX, speaking on the matter in 2025 (my translation):
We would rather the police did not commit crimes, the types of crimes we wish to stop.
The generated face of a child can not easily be concluded to not resemble the face of a real child
I personally could never bring myself to generate any generic photo using "AI", as I could never know if the software outputs the likeness of someone who has been abused, murdered or involved in an accident. Child or adult. The potential further impact of this is truly monstrous to me.
If police investigators are to sit at their desks and prompt the likeness of abusive situations, the likelihood that these images will resemble real children may even be higher. Some children may be overrepresented in the type of content that is being prompted for, which in turn could make it statistically more likely for certain faces to appear.
In a case reported on by Ars Technica, the plaintiff "Jane Doe" was pre-school age when she was a victim of sexual abuse. Working with child protection groups she has allowed the criminal photos of her to be used in CSAM detection tools, and has herself subscribed to alerts of them being found.
Following on the growth of image generation software, it is now the case that new alerts are triggered by generated images that resemble the real ones of her:
Ars Technica writes on the spread of these types of images via the xAI Grok image generation product:
”Although she has received countless alerts, she was shocked when the CCCP notified her that it had identified AI-generated CSAM on xAI that depicted her. […] Now, Doe fears that xAI has not only made it easier to make more violative images of the most distressing time in her life, but also that xAI allegedly has stored the images that Grok generates and uses those outputs to further train Grok. Because of this, she believes that Grok has been trained on both the initial set of images that have haunted her for more than 20 years and the more recent AI-generated ones.”
So it is not enough for our government to say that police can generate these types of images as long as they make sure there are no images where real children are involved. How could they possibly verify that such an image bears no likeness to a real child? Nor to a real victim? And how could they guarantee that their own generated images are not used to further train image generation software, thus boosting the proliferation of CSAM, rather than reducing it?
The answer of course is that neither the government or the police can promise any such thing. What may sound like a determined and forceful response to criminal activity may in some cases make matters worse.
A good follow-up question now would be: why do we even allow products that can generate this type of content?
References
View the announcement by Christian Carlsson, of the Christian Democrats, at the press conference (English subtitles by me)
The slide announcing the position of allowing police to generate ”fake” CSAM
Translation
The Police Authority will be allowed to share fictitious child pornography images
- New tool for the Police Authority to, for example, access closed forums on the Internet where child abuse occurs.
- The authority will be allowed to both produce and share fictional child pornography images.
- Such images will be allowed to be used in preliminary investigations into serious sexual crimes against children and serious child pornography crimes.
- An important tool for combating sexual crimes against children that the Police Authority has requested.
The press conference source page
Regeringskansliet
Reports of CSAM in datasets
Davey Alba and Rachel Metz
About dataset filtering
How the Grok product amplifies CSAM content
Ashley Belanger
Childrens rights organisations describe it as a horrendous risk, a report from 2025
Ellen Rosenkvist