OpenAI launches CSAM image classifier and ChatGPT warnings
OpenAI rolled out an AI image classifier to detect child sexual abuse material and added warning screens plus temporary blocks in ChatGPT.
Source: TechCrunch — AIOpenAI announced two safety‑focused additions on Thursday: a new image‑classification system that can spot child sexual abuse material (CSAM), including AI‑generated pictures, and a set of warning screens in ChatGPT that temporarily block users who request sexualised content involving children. The moves come as regulators and advocacy groups pressure AI providers to tighten safeguards.
What new tools did OpenAI release?
OpenAI introduced an image classifier that scans visual inputs for signs of child sexual exploitation. The company also updated the ChatGPT interface to display an alert whenever a user attempts a prompt that could produce or retrieve CSAM. The alert directs users to specialised support resources and imposes a 24‑hour account suspension as a first‑offence warning. Repeated violations may trigger permanent removal of the account. According to TechCrunch, the changes are part of a broader safety package that includes developer guidelines and law‑enforcement tooling.
How does the image classifier work and what were the test results?
The classifier operates by assigning a risk score to each image based on visual patterns associated with abuse. In early testing with partners such as the National Center for Missing and Exploited Children (NCMEC), the system flagged 48 % fewer images while still catching the same number of abusive files, effectively reducing reviewer workload. Moreover, images that received a high‑risk score were twice as likely to be genuinely abusive compared with the previous detection approach. OpenAI is offering the classifier through its API at no cost to a limited set of companies, working with organisations like Tech Coalition and ROOST to broaden access.
What changes are being added to ChatGPT for CSAM warnings?
When a user submits a prompt that could produce sexualised depictions of minors, ChatGPT now shows a warning overlay. The overlay explains that the request is disallowed, provides links to child‑protection hotlines, and logs the incident. If the user proceeds, the system automatically suspends the account for 24 hours as a deterrent. The policy builds on existing content filters that already block explicit CSAM content. OpenAI consulted with groups such as Protect Children, the Lucy Faithfull Foundation, and academic researchers like Joel Scanlan to design the messaging.
How is OpenAI supporting developers and law‑enforcement?
OpenAI released a set of guidelines for developers who integrate its models into their products. The guidelines outline best practices for detecting, reporting, and removing CSAM, and they encourage developers to register with NCMEC for direct reporting. In pilot programs, the guidelines helped several developers adopt the new safety tooling.
For law‑enforcement, OpenAI is packaging AI‑powered plugins, investigative tools, and complex data‑analysis utilities. Access to these tools is gated by program administrators, and OpenAI is also providing training sessions and a field guide to help investigators incorporate AI into their workflows.
Why do these measures matter for creators, developers, and researchers?
For creators, the classifier offers a way to screen generated images before they are published, helping to avoid accidental distribution of illegal content. Developers integrating OpenAI’s models now have a clearer compliance path and can leverage the free API access to the classifier for internal moderation pipelines. Researchers studying AI‑generated imagery gain a new benchmark for evaluating how well generative models can be filtered for harmful content, and the public release of the guidelines adds transparency to the safety ecosystem.
How can Grafy users explore these safety tools?
Grafy does not host OpenAI’s classifier directly, but creators and developers can experiment with the concepts using Grafy’s own multimodal suite. For example, you can use Grafy Chat to test prompt‑filtering behavior and see how warning messages appear when you ask for disallowed content. Researchers can upload PDFs of OpenAI’s technical brief to Grafy Reader to search for details about the classifier’s methodology. Finally, developers building custom moderation pipelines could prototype image‑analysis workflows in Grafy Code, calling OpenAI’s API (if you have access) alongside Grafy’s file‑handling utilities.
Frequently asked questions
How does the 24‑hour suspension work in practice?
When a user triggers the CSAM warning in ChatGPT, the system logs the incident and automatically disables the account for a full 24‑hour period. During that time the user cannot send new prompts. If the same user repeats the violation after the suspension lifts, the account may be permanently removed.
Is the image classifier available to all OpenAI API users?
OpenAI is initially providing the classifier at no charge to a select group of partner companies. Wider availability is planned, but the rollout will remain controlled to ensure responsible use and compliance with legal reporting obligations.
Can developers integrate the classifier into their own moderation pipelines?
Yes. OpenAI’s API returns a risk score for each image, which developers can use to flag or block content in their applications. The accompanying developer guidelines describe how to report flagged images to NCMEC and how to store the scoring data securely.
Source: TechCrunch — AI — OpenAI releases a new image classifier and ChatGPT warnings to combat CSAM


