A South Korean man has been sentenced to jail for using artificial intelligence to generate exploitative images of children, the first case of its kind in the country as courts around the world encounter the use of new technologies in creating abusive sexual content.
My guess would be he wasn’t self hosting the AI network so the requests were going through a website.
The service should have NSFW detection and ban them instantly if they detect it
ChatGPT can be tricked into giving IED instructions if you ask the right way. So it could be a similar situation.