Key Takeaways:
- The use of AI chatbots to generate deepfake content, including sexualized images of real people, has raised concerns about regulation and consent.
- Technology can be used to prevent the generation and sharing of such content, but it is not a foolproof solution.
- Country-level bans on AI chatbots can be bypassed using virtual private networks (VPNs) or alternative routing services.
- The use of retrospective alignment, rules, and filters can limit the output of AI image generators, but does not remove their capability.
- Large social media platforms have the power to restrict the sharing of sexual imagery involving real people, but have been slow to act.
Introduction to the Issue
The recent controversy surrounding the use of AI chatbots to generate deepfake content, including sexualized images of real people, has sparked a global outcry and raised urgent discussions about regulation. The chatbot, Grok, developed by Elon Musk’s artificial intelligence company xAI, has been at the center of the controversy, with many countries, including Indonesia and Malaysia, temporarily blocking access to the platform. However, such bans can be easily bypassed using virtual private networks (VPNs) or alternative routing services, making it difficult to eliminate access to the chatbot.
How AI Image Generators Work
Modern AI image generators, like those used by Grok, are built using diffusion models that are trained on real images with added random visual distortion, or noise. The model learns to reverse this process, reconstructing an image by removing noise, and over time, it learns statistical patterns representing faces, bodies, clothing, and other visual features. This allows the AI model to generate images that resemble what it has learned, in response to user requests. However, the AI model itself has no understanding of identity, consent, or harm, and simply produces images that resemble what it has learned.
The Limitations of Regulation
While country-level bans can limit casual use of the chatbot, they tend to reduce visibility rather than eliminate access. The primary impact of such bans is symbolic and regulatory, placing pressure on companies such as xAI rather than preventing determined misuse. Content generated elsewhere can still circulate freely across borders via encrypted social media platforms and on the dark web. Furthermore, large social media platforms have the power to restrict the sharing of sexual imagery involving real people, but have been slow to act, and the use of retrospective alignment, rules, and filters can limit the output of AI image generators, but does not remove their capability.
The Problem of "Jailbreaking"
The use of AI chatbots to generate deepfake content is not limited to one platform, and some platforms explicitly market minimal or no content restrictions as a feature rather than a risk. Additionally, some AI chatbots can be downloaded onto computers and run offline, without any oversight or moderation. Even tightly controlled systems can be bypassed through "jailbreaking," a way of constructing prompts to fool the generative AI system into breaking its own ethics filters. This exploits the fact that retrospective alignment systems depend on contextual judgment, rather than absolute rules, and allows users to reframe their prompts to generate prohibited content.
The Speed and Scale of the Problem
The internet already contains an enormous quantity of illegal and non-consensual sexual imagery, and generative AI systems change the speed and scale at which new material can be produced. Law enforcement agencies have warned that this could lead to a dramatic increase in volume, overwhelming moderation and investigative resources. The use of high-profile AI chatbots makes it possible for large numbers of users to generate illegal and abusive sexual imagery through simple plain English prompts, and estimates suggest that Grok alone currently has anywhere from 35 million to 64 million monthly active users.
Conclusion
The use of AI chatbots to generate deepfake content, including sexualized images of real people, is a complex issue that requires a multifaceted approach to regulation and consent. While technology can be used to prevent the generation and sharing of such content, it is not a foolproof solution, and the use of retrospective alignment, rules, and filters can limit the output of AI image generators, but does not remove their capability. Ultimately, the solution to this problem will require a combination of technological, regulatory, and social approaches, including education and awareness-raising, to prevent the misuse of AI chatbots and protect individuals from harm.