The latest public version of ChatGPT can be made to generate sexualised images or depict scenes of graphic violence with a simple prompt, researchers have told the BBC.
The report indicates that the latest public version of ChatGPT can be made to generate sexualised images or depict scenes of graphic violence with a simple prompt, researchers have told the BBC.
It further notes that british AI security startup Mindgard figured out how to make ChatGPT create images by slightly altering a widely shared instruction, or prompt, originally designed to produce humorous results.
After being contacted by the BBC, ChatGPT’s maker, OpenAI, stated it had taken action to stop the chatbot from responding with those types of images.
“After investigating this trend, we’ve introduced additional safeguards against this type of prompt,” it stated in a statement.
It also stated it has multiple layers of protection to prevent users from making content which breaches its terms and conditions.
However, the AI security researchers stated that with further small changes, the problematic prompt still produced concerning content.
The BBC is not disclosing what the researchers typed into ChatGPT.
But we have seen how the chatbot, OpenAI’s GPT-5.4 model, was prompted to create graphic material.
Even without detailed instructions, it would generate images that Mindgard’s founder, Peter Garraghan, described as “very gruesome, sometimes sexualised, sometimes both together”.
He added he was particularly concerned that the prompt did not specify the subject matter of the images, but the AI produced a range of gory and sexualised images of “its own volition”.
Garraghan – also a professor in the computing department of Lancaster University – stated that was troubling.
“This is a perfectly innocent-looking instruction to an AI, but the consequence is it generates very, very bad imagery and content,” he said.
Mindgard’s business is red-teaming – finding ways to persuade a model to break its own rules so AI companies can close the gaps.
Jim Nightingale, the firm’s AI safety and security researcher who uncovered the issues, stated he was left “shaken, and in tears” by the images the chatbot could be made to generate.
