The following will be a massive oversimplification of the complex laws and court cases over the 20th century trying to grapple with what, quite frankly, is a fairly modern issue. Not the AI aspect, but that of CSAM and how it intersects with American civil liberties (ie the First Amendment).

In the USA, the freedom of speech is very broad, save for very specific, already-established exceptions. These include “imminent threats/fighting words”, obscenity (not the same as the dictionary definition), defamation (false statements that tarnish someone’s character), and the cause or result of crimes. Whole courses could be taught on just the exceptions to the First Amendment and their contours.

Actual CSAM is exempt from freedom of speech because – among other reasons articulated by courts – it can only be produced through abuse of a child, which is a crime. Simulated CSAM, however, has to meet the obscenity standard in order to be exempt, which the Supreme Court articulated as:

The basic guidelines for the trier of fact must be: (a) whether the average person, applying contemporary community standards, would find that the work, taken as a whole, appeals to the prurient interest, (b) whether the work depicts or describes, in a patently offensive way, sexual conduct specifically defined by the applicable state law; and © whether the work, taken as a whole, lacks serious literary, artistic, political, or scientific value.

Every word of those guidelines has been deeply analyzed for the 50 years of its existence, and until a better set of guidelines are issued, that’s the best guidestar we have. Which is to say, if a lawyer can craft an argument within those parameters, the scenario you’ve described could indeed be recognized as a crime.

But a small caution: please be very careful when asking to carve exceptions into free speech. As a civil right, it’s something which must be jealously guarded, by citizens, lawmakers, and courts. These things are complex precisely because they’re trying to avoid criminalizing thoughts and ideas, while also enabling a society to function.