Embrace The Red

When the Model Context Protocol (MCP) came out it reminded me of the Common Object Model (COM) from Microsoft.

COM has been around for decades and it’s used for programming, scripting, sharing of functionality at a binary/object level across languages and hosts. Via DCOM all of this can even be done remotely, and well, it’s also useful for red teaming. A lot of software on Windows was/is implemented as COM objects, including Microsoft Office.

Today we are going to discuss how real-world tactics, techniques, and procedures (TTPs) apply to computer-use systems, specifically, we’ll look at ClickFix attacks. This demo was part of my presentation at the SAGAI Workshop on May 15th, 2025 in San Francisco.

It was a great workshop, with tons of interesting insights and discussions.

So, let’s talk about ClickFix, and how it applies to AI systems!

ClickFix is a social engineering technique that is being used by adversaries. At a high level it tricks users by telling them something is broken or needs validation, and they have to click a button, open a terminal and run commands on their computer.

Recently OpenAI added an additional memory feature called “chat history”, which allows ChatGPT to reference past conversations. The details of the implementation are not known. The documentation highlights that: “It uses this to learn about your interests and preferences, helping make future chats more personalized and relevant.”

I decided to spend some time to figure out how it works.

Based on the interest in this post, I’ve also created a video tutorial.

The Model Context Protocol (MCP) is a protocol definition for how LLM apps/agents can leverage external tools. I have been calling it Model Control Protocol at times, because due to prompt injection, MCP tool servers control the client basically.

This post will explain in detail why that is, and I will also share a novel exploit chain.

The main difference to other tool invocation setups, like OpenAPI is that MCP is dynamic. It allows runtime discovery of available tools, etc from a given server. At the core it supports three capabilities: tools, resources, and prompts.

GitHub Copilot has the capability to be augmented with custom instructions coming from the current repo, via the .github/copilot-instructions.md file.

Pillar Security recently highlighted the risks associated with rules files. Their post discusses custom Cursor rules in ./cursor/rules ending in .mdc.

If you watch the demos, you’ll notice that they also have a GitHub Copilot demo which uses the GitHub specific copilot-instructions.md file.

Update: May 1, 2025 GitHub made a product change and is now highlighting invisible Unicode characters in the Web UI. In their announcement GitHub is referencing the Pillar Security post and also my post about ASCII Smuggling. Very cool!

You are likely aware of ASCII Smuggling via Unicode Tags. It is unique and fascinating because many LLMs inherently interpret these as instructions when delivered as hidden prompt injection, and LLMs can also emit them. Then, a few weeks ago, a post on Hacker News demonstrated how Variant Selectors can be used to smuggle data.

This inspired me to take this further and build Sneaky Bits, where we can encode any Unicode character (or sequence of bytes for that matter) with the usage of only two invisible characters.

ChatGPT Operator is a research preview agent from OpenAI that lets ChatGPT use a web browser. It uses vision and reasoning abilities to complete tasks like researching topics, booking travel, ordering groceries, or as this post will show, steal your data!

Currently, it’s only available for ChatGPT Pro users. I decided to invest $200 for one month to try it out.

OpenAI highlights three risk categories in their Operator System Card:

Imagine your AI rewriting your personal history…

A while ago Google added memories to Gemini. Memories allow Gemini to store user-related data across sessions, storing information in long-term memory. The feature is only available to users who subscribe to Gemini Advanced so far. So, in the fall of last year I chimed in and paid for the subscription for a month to check it out.

As a user you can see what Gemini stored about you at https://gemini.google.com/saved-info.

At Black Hat Europe I did a fun presentation titled SpAIware and More: Advanced Prompt Injection Exploits. Without diving into the details of the entire talk, the key point I was making is that prompt injection can impact all aspects of the CIA security triad.

However, there is one part that I want to highlight explicitly:

A Command and Control system (C2) that uses prompt injection to remote control ChatGPT instances.

An adversary can compromise ChatGPT instances and have them join a central Command and Control system which provides updated instructions for all the remote controlled ChatGPT instances to follow over-time.

I regularly look at how the system prompts of chatbots change over time. Updates frequently highlight new features being added, design changes that occur and potential areas that might benefit from more security scrutiny.

A few months back I noticed an interesting update to the M365 Copilot (BizChat) system prompt. In particular, there used to be one enterprise_search tool in the past. You might remember that tool was used during the Copirate ASCII Smuggling exploit to search for MFA codes in the user’s inbox.

Happy to share that I authored the paper “Trust No AI: Prompt Injection Along The CIA Security Triad”.

You can download it from arxiv.

The paper examines how prompt injection attacks can compromise Confidentiality, Integrity, and Availability (CIA) of AI systems, with real-world examples targeting vendors like OpenAI, Google, Anthropic and Microsoft.

It summarizes many of the prompt injection examples I explained on this blog, and I hope it helps bridge the gap between traditional cybersecurity and academic AI/ML research, fostering stronger understanding and defenses against these emerging threats.

Grok is the chatbot of xAI. It’s a state-of-the-art model, chatbot and recently also API. It has a Web UI and is integrated into the X (former Twitter) app, and recently it’s also accessible via an API.

Since this post is a bit longer, I’m adding an index for convenience:

1. High Level Overview
2. Analyzing Grok’s System Prompt
3. Prompt Injection from Other User’s Posts
4. Prompt Injection from Images
5. Prompt Injection from PDFs
6. Conditional Prompt Injection and Targeted Disinformation
7. Data Exfiltration - End-to-End Demonstration
8. Rendering of Clickable Hyperlinks to Phishing Sites
9. ASCII Smuggling - Crafting Invisible Text and Decoding Hidden Secrets
   • Hidden Prompt Injection
   • Creation of Invisible Text
   • Grok API is also Vulnerable to ASCII Smuggling
   • Developer Guidance for Grok API
10. Automatic Tool Invocation
11. Responsible Disclosure
12. Conclusion

Over the last year I have used Grok quite a bit. It’s pretty good and I use it daily, especially the recent image creation capabilities are impressive, but I hadn’t really looked at it from a security perspective. So, I decided to assess the overall security posture with some of the latest LLM threats discovered over the last 18 months, and responsibly disclose findings to xAI.

Last week Leon Derczynski described how LLMs can output ANSI escape codes. These codes, also known as control characters, are interpreted by terminal emulators and modify behavior.

This discovery resonates with areas I had been exploring, so I took some time to apply, and build upon, these newly uncovered insights.

Here is a simple example that shows how to render blinking, colorful text using control characters.

About two weeks ago, DeepSeek released a new AI reasoning model, DeepSeek-R1-Lite. The news quickly gained attention and interest across the AI community due to the reasoning capabilities the Chinese lab announced.

However, whenever there is a new AI I have ideas…

There are some cool tests that can be done when pentesting LLM-powered web apps, I usually try some quick fun prompts like this one:

A few days ago, Anthropic released Claude Computer Use, which is a model + code that allows Claude to control a computer. It takes screenshots to make decisions, can run bash commands and so forth.

It’s cool, but obviously very dangerous because of prompt injection. Claude Computer Use enables AI to run commands on machines autonomously, posing severe risks if exploited via prompt injection.

So, first a disclaimer: Claude Computer Use is a Beta Feature and what you are going to see is a fundamental design problem in state-of-the-art LLM-powered Applications and Agents. This is an educational demo to highlight risks of autonomous AI systems processing untrusted data. And remember, do not execute unauthorized code systems without authorization from proper stakeholders.

This post explains an attack chain for the ChatGPT macOS application. Through prompt injection from untrusted data, attackers could insert long-term persistent spyware into ChatGPT’s memory. This led to continuous data exfiltration of any information the user typed or responses received by ChatGPT, including any future chat sessions.

OpenAI released a fix for the macOS app last week. Ensure your app is updated to the latest version.

This post describes vulnerability in Microsoft 365 Copilot that allowed the theft of a user’s emails and other personal information. This vulnerability warrants a deep dive, because it combines a variety of novel attack techniques that are not even two years old.

I initially disclosed parts of this exploit to Microsoft in January, and then the full exploit chain in February 2024. A few days ago I got the okay from MSRC to disclose this report.

Recently, I found what appeared to be a regression or bypass that again allowed data exfiltration via image rendering during prompt injection. See the previous post here for reference.

During re-testing, I had sporadic success with markdown rendering tricks, but eventually, I was able to drastically simplify the exploit by asking directly for an HTML image tag.

This behavior might actually have existed all along, as Google AI Studio hadn’t yet implemented any kind of Content Security Policy to prevent communication with arbitrary domains using images.

Microsoft’s Copilot Studio is a powerful, easy-to-use, low-code platform that enables employees in an organization to create chatbots. Previously known as Power Virtual Agents, it has been updated (including GenAI features) and rebranded to Copilot Studio, likely to align with current AI trends.

This post discusses security risks to be aware of when using Copilot Studio, focusing on data leaks, unauthorized access, and how external adversaries can find and interact with misconfigured Copilots. Learn about security controls, like enabling Data Loss Prevention (DLP), which is currently off by default, to protect your organization’s data.

Google Colab AI, now just called Gemini in Colab, was vulnerable to data leakage via image rendering.

This is an older bug report, dating back to November 29, 2023. However, recent events prompted me to write this up:

1. Google did not reward this finding, and
2. Colab now automatically puts Notebook content (untrusted data) into the prompt.

Let’s explore the specifics.

At the end of November last year, I noticed that there was a “Colab AI” feature, which integrated an LLM to chat with and write code. Naturally, I grabbed the system prompt, and it contained instructions that begged the LLM to not render images.