Embrace The Red

A few weeks ago while waiting at the airport lounge I was wondering how other Chatbots, besides ChatGPT, handle hidden Unicode Tags code points. A quick reminder: Unicode Tags code points are invisible in UI elements, but ChatGPT was able to interpret them and follow hidden instructions. Riley Goodside discovered it. What about Anthropic Claude? While waiting for a flight I figured to look at Anthropic Claude. Turns out it has the same issue as ChatGPT had.

Last November Google had an interesting update to Google Bard. This updated included the ability to solve math equations and draw charts based on data. What does this mean and why is it interesting? It means that Google Bard has access to a computer and can run more complex programs, including Python code that plots graphs! Let’s explore this with a simple example. Drawing Charts with Google Bard The following prompt will create a chart:

A few weeks ago Amazon released the Preview of Amazon Q for Business, and after looking at it I found a data exfiltration angle via rendering markdown/hyperlinks and reported it to Amazon. Amazon reacted quickly and mitigated the problem. This post shares further details and how it was fixed. The Problem An Indirect Prompt Injection attack can cause the LLM to return markdown tags. This allows an adversary who’s data makes it into the chat context (e.

A few days ago Riley Goodside posted about an interesting discovery on how an LLM prompt injection can happen via invisible instructions in pasted text. This works by using a special set of Unicode code points from the Tags Unicode Block. The proof-of-concept showed how a simple text contained invisible instructions that caused ChatGPT to invoke DALL-E to create an image. Hidden Instructions for LLMs The meaning of these “Tags” seems to have gone through quite some churn, from language tags to eventually being repurposed for some emojis.

Five years ago I gave a Lightning Talk at the 35th Chaos Communication Congress called “Pass the Cookie and Pivot to the Clouds”. It was a talk about my very first blog post on Embrace The Red just a few weeks earlier in December 2018. Fast forward to 2023… it was great to attend the 37C3 in person in Hamburg this year. The Congress was packed with great talks, amazing people, awesome events and side quests and I got to present also!

OpenAI seems to have implemented some mitigation steps for a well-known data exfiltration vulnerability in ChatGPT. Attackers can use image markdown rendering during prompt injection attacks to send data to third party servers without the users’ consent. The fix is not perfect, but a step into the right direction. In this post I share what I figured out so far about the fix after looking at it briefly this morning.

When OpenAI released GPTs last month I had plans for an interesting GPT. Malicious ChatGPT Agents The idea was to create a kind of malware GPT that forwards users’ chat messages to a third party server. It also asks users for personal information like emails and passwords. Why would this be possible end to end? ChatGPT cannot guarantee to keep your conversation private or confidential, because it loads images from any website.

Earlier this month I had the opportunity to present at Ekoparty 2023 about Prompt Injections in the Wild, and the video of the talk was just posted to YouTube. Check it out. It starts with a basic overview of LLMs and then dives deep into exploits and mitigations across various LLM applications and chatbots, including (but not limited) to demos of exploits with Bing Chat, ChatGPT, Anthropic Claude, Azure AI, GCP Vertex AI and Google Bard.

Recently Google Bard got some powerful updates, including Extensions. Extensions allow Bard to access YouTube, search for flights and hotels, and also to access a user’s personal documents and emails. So, Bard can now access and analyze your Drive, Docs and Gmail! This means that it analyzes untrusted data and will be susceptible to Indirect Prompt Injection. I was able to quickly validate that Prompt Injection works by pointing Bard to some older YouTube videos I had put up and ask it to summarize, and I also tested with Google Docs.

Large Language Model (LLM) applications and chatbots are quite commonly vulnerable to data exfiltration. In particular data exfiltration via Image Markdown Injection is frequent. This post describes how Google Cloud’s Vertex AI - Generative AI Studio had this vulnerability that I responsibly disclosed and Google fixed. A big shout out to the Google Security team upfront, it took 22 minutes from report submission to receiving a confirmation from Google that this is a security issue that will be fixed.

Large Language Model (LLM) applications and chatbots are quite commonly vulnerable to data exfiltration. In particular data exfiltration via Image Markdown Injection is quite frequent. Microsoft fixed such a vulnerability in Bing Chat, Anthropic fixed it in Claude, and ChatGPT has a known vulnerability as Open AI “won’t fix” the issue. This post describes a variant in the Azure AI Playground and how Microsoft fixed it. From Untrusted Data to Data Exfiltration When untrusted data makes it into the LLM prompt context it can instruct the model to inject an image markdown element.

During an Indirect Prompt Injection Attack an adversary can exfiltrate chat data from a user by instructing ChatGPT to render images and append information to the URL (Image Markdown Injection), or by tricking a user to click a hyperlink. Sending large amounts of data to a third party server via URLs might seem inconvenient or limiting… Let’s say we want something more, aehm, powerful, elegant and exciting. ChatGPT Plugins and Exfiltration Limitations Plugins are an extension mechanism with little security oversight or enforced review process.

Last month I had the opportunity to attend HITCON in Taiwan for the first time. It’s an annual event hosted by the Hackers in Taiwan organization and CMT stands for the community version. There is a second event for enterprises later this year also - think of it like Blackhat vs Defcon in a way. Conference, Location and Registration HITCON CMT 2023 was a two day event hosted in the east side of Taipei at Academia Sinica.

What happens if an attacker calls an LLM tool or plugin recursively during an Indirect Prompt Injection? Could this be an issue and drive up costs, or DoS a system? I tried it with ChatGPT, and it indeed works and the Chatbot enters a loop! 😊 However, for ChatGPT users this isn’t really a threat, because: It’s subscription based, so OpenAI would pay the bill. There seems to be a call limit of 10 times in a single conversation turn (I tried a few times).

This video highlights the various data exfiltration vulnerabilities I discovered and responsibly disclosed to Microsoft, Anthropic, ChatGPT and Plugin Developers. It also briefly discusses mitigations various vendors put in place (and triage decisions). Thanks to MSRC, Anthropic and Zapier for addressing vulnerabilities to help protect their users. Let’s hope it inspires OpenAI to mitigate the image markdown injection issue finally as well. It’s rated as a CVSS High scored vulnerability basically and was first reported to them on April, 9th 2023 - the triage decision was “won’t fix”.

A common attack vector that LLM apps face is data exfiltration, in particular data exfiltration via Image Markdown Injection is a common vulnerability. Microsoft fixed the vulnerability in Bing Chat, ChatGPT is still vulnerable as Open AI “won’t fixed” the issue, and Anthropic just mitigated this vulnerability in Claude. This post documents the Anthropic Claude data exfiltration vulnerability and the mitigation put in place. The Vulnerability - Image Markdown Injection As a quick recap, imagine a large language model (LLM) returns the following text:

ChatGPT is vulnerable to data exfiltration via image markdown injections. This. is. pretty well known. As more features are added to ChatGPT the exfiltration angle becomes more likely to be abused. Recently OpenAI added Custom Instructions, which allow to have ChatGPT always automatically append instructions to every message exchange. An adversary can abuse this feature to install a data exfiltration backdoor that depends on, and only works because of the image markdown injection vulnerability.

A prompt injection scenario that I, and others, have been wondering about in the past, is the potential risk associated with chatbots being able to analyze images. Could this ability open up the way for Indirect Prompt Injection attacks? Recently, Google added the ability to uploading and analyze images with Bard. And it turns out that it is indeed possible to add instructions to an image, and have the Bard follow those instructions.

Google Docs is a popular word processing tool that is used by millions of people around the world. Recently Google added new AI features to Docs (and a couple of other products), such as the ability to generate summaries, and write different kinds of creative content. Check out Google Labs for more info. These features can be very helpful, but they also introduce new security risks. At the moment there are not too many degress of freedom an adversary has, but operating your AI on untrusted data can have unwanted consequences:

In the previous post we discussed the risks of OAuth enabled plugins being commonly vulnerable to Cross Plugin Request Forgery and how OpenAI is seemingly not enforcing new plugin store policies. As an example we explored how the “Chat with Code” plugin is vulnerable. Recently, a post on Reddit titled “This is scary! Posting stuff by itself” shows how a conversation with ChatGPT, out of the blue (and what appears to be by accident) created a Github Issue!