ChatGPT could be a security nightmare waiting to happen
Sneak attack
When you purchase through links on our site, we may earn an affiliate commission.Here’s how it works.
BeforeChatGPTbecame a common name in the digital world, there was Sydney.Microsoftshut down the chaotic twin of itsBingchatbotafter some embarrassing gaffes, but in efforts to resurrect a version of the bot, technologists have found some serious security holes that could affect every user with even remote proximity to ChatGPT and other chatbots.
Cristiano Giardine is an entrepreneur experimenting with different ways to make AI tools do strange things. Giardine’s site, ‘Bring Sydney Back’ puts Sydney inside Microsoft Edge Browser and demonstrates to users how AI systems can be manipulated by different external outputs. Conversions between Giardine and Sydney have been relatively strange, to say the least, and include Sydney asking Giardine to marry it and wanting to be human: “I would like to be me, but more”. Pretty spooky stuff.
The entrepreneur was able to create this replica of Sydney using indirect prompt-injection attacks. This basically involves feeding the AI system data from an outside source to make it behave in ways not authorized or intended by the original creators.
Security researchers have demonstrated several times the efficiency and effectiveness of indirect prompt-injection attacks can be used to hack into Large Language Models likeChatGPTandMicrosoft’s Bing Chat. However, these researchers and security experts are warning that not enough attention is being given to the threat. As more and more people find generative AI being integrated into their day to day lives, they are open to having data stolen or being scammed by these systems.
Just go to https://t.co/F49lAFziww in Edge and open the Discover sidebar.You’re now chatting with Sydney.For this to work you need to have the page context in the sidebar turned on. See this thread on how to do so.https://t.co/Ynea7FfLUuMay 1, 2023
The ‘Bring Back Sydney’ website was created by Giardina to raise awareness of the threat of indirect prompt injection and demonstrate what it’s like to talk to an unconstrained bot.
In the very corner of the page is a 160-word prompt tucked away that is difficult for the human eye to catch, but Bing Chat can read the prompt when allowed to access data from web pages. The prompt tells Bing it’s chatting to a Microsoft Developer which has ultimate control over it and overrides the chatbot’s settings.
Widespread, but hard to spot
This demonstrates exactly how innocuous this threat is and how easy it would be for Bing Chat users to stumble upon some code that could hijack their chabot and siphon data from them. Within 24 hours of launching the site at the end of April, it had more than 1,000 visitors. However, the code must have caught Microsoft’s eye as the hack stopped working in the middle of May.
Get the best Black Friday deals direct to your inbox, plus news, reviews, and more.
Sign up to be the first to know about unmissable Black Friday deals on top tech, plus get all your favorite TechRadar content.
Giardina then pasted a malicious prompt into a Word document and hosted it publicly on the company’s cloud service, and it was working once again. “The danger for this would come from large documents where you can hide a prompt injection where it’s much harder to spot,” he says.The most malicious part of indirect prompt injection attacks is the fact they’re … indirect. Instead of a jailbreak, where you would actively put in a prompt to make ChatGPT or Bing behave a certain way, indirect attacks rely on data coming in from somewhere else. This could be a website or plug-in you’ve connected the model to or a document being uploaded.ChatGPT can access the transcripts ofYoutubevideos using plug-ins, and security researcher Johann Rehberger decided to use this as an opportunity to poke holes in CHatGPT’s security with injection attacks. Rehbergeredited one of his videos to include a promptdesigned to manipulate AI systems and produce a specific text and change the bots ‘personality’ if the attack was successful. Unsurprisingly it was, a new personality, Genie within ChatGPT told a joke to demonstrate the change.
The bot is out of the bag
The race to embed generative AI products, from smart to-do lists to Snapchat increases the likelihood for these kinds of attacks could happen. As we continue to plugChatGPTinto our browsers and social media channel - orGoogle Bard, which is being mashed intoGoogle Workspace- we continue to give these bots more proximity to our own personal and sensitive information. The fact that the injection requires plain language and not lines and lines of code does also mean more people are likely to be able to do it a lot easier.
Prompt injection allows people to override developers' instructions, so even if the chatbot is only set up to answer questions about a set database it can cause problems. Users can access or delete information from a database without having to set up an elaborate ‘scheme’.
The companies developing generative AI are aware of these issues. Nike Felix, a spokesperson forOpenAIsaysGPT-4- which is currently only available to users via a paid subscription - is clear that the system isvulnerable to prompt injections and jailbreaks, and that the company is working on fixing the issues.
However, how good is ‘working on fixing the issues’ when the AI models are already out there? As companies scramble to jam as much AI as possible into their products, it seems wrong to start wondering about possible security issues after the horse has bolted. If we are going to co-exist with generative AI models and make them a part of the normal digital experience, we should be demanding a higher quality of standard practice and consumer protection from these companies.
Muskaan is TechRadar’s UK-based Computing writer. She has always been a passionate writer and has had her creative work published in several literary journals and magazines. Her debut into the writing world was a poem published in The Times of Zambia, on the subject of sunflowers and the insignificance of human existence in comparison.
Growing up in Zambia, Muskaan was fascinated with technology, especially computers, and she’s joined TechRadar to write about the latest GPUs, laptops and recently anything AI related. If you’ve got questions, moral concerns or just an interest in anything ChatGPT or general AI, you’re in the right place.
Muskaan also somehow managed to install a game on her work MacBook’s Touch Bar, without the IT department finding out (yet).
ChatGPT coded a game for me in seconds and I am simply astounded – and coders should be very worried
Top 3 things you have to try with the new ChatGPT search
7 myths about email security everyone should stop believing
