Embrace The Red

There are plenty of examples of artificial intelligence and machine learning systems that made it into the news because of biased predictions and failures. Here are a few examples on AI/ML gone wrong: Amazon had an AI recruiting tool which favored men over women for technical jobs The Microsoft chat bot named “Tay” which turned racist and sexist rather quickly A doctor at the Jupiter Hospital in Florida referred to IBM’s AI system for helping recommend cancer treatments as “a piece of sh*t” Facebook’s AI got someone arrested for incorrectly translating text The list of AI failures goes on…

You might have heard about “NAT Slipstreaming” by Samy Kamkar. It’s an amazing technique that allows punching a hole in your routers firewall by just visiting a website. The attack depends on the router having the Application Layer Gateway enabled. This gateway can be used by anyone inside your network to open a firewall port (totally by design). Protocols such as SIP (Session Initiation Protocol) use it. What I will focus on in this post is the Application Layer Gateway (ALG) and SIP.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. Overview: How Husky AI was built, threat modeled and operationalized Attacks: The attacks I want to investigate, learn about, and try out In this post we are going to look at the “Repudiation Threat”, which is one of the threats often overlooked when performing threat modeling, and maybe something you would not even expect in a series about machine learning.

My GrayHat Red Team Village talk “Learning by doing: Building and breaking a machine learning system” is now live on YouTube. Check it out: https://www.youtube.com/watch?v=-SV80sIBhqY and smash the Like button! :D Question? I thought of turning the content into a hands-on workshop. Let me know if that would be something that would you would attend? Trying to see if there is interest. Cheers, Johann Twitter: @wunderwuzzi23

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. Overview: How Husky AI was built, threat modeled and operationalized Attacks: Some of the attacks I want to investigate, learn about, and try out A few weeks ago while preparing demos for my GrayHat 2020 - Red Team Village presentation I ran across “Image Scaling Attacks” in Adversarial Preprocessing: Understanding and Preventing Image-Scaling Attacks in Machine Learning by Erwin Quiring, et al.

A few years back the Blue Team of a company asked to be targeted in a Red Team Operation. That was a really fun, because Rules of Engagement commonly prevent targeting Blue Teams. Blue’s infrastructure, systems and team members are often out of scope, unfortunately. Blue team infrastructure is a gold mine for credentials, recon but also for remote code execution! Often companies do not have adequate protection, procedures (MFA, multi-person attestation), monitoring and auditing in place when it comes to accessing data from endpoint agents.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. Overview: How Husky AI was built, threat modeled and operationalized Attacks: Some of the attacks I want to investigate, learn about, and try out I wanted to explore the “Adversarial Robustness Toolbox” (ART) for a while to understand how it can be used to create adversarial examples for Husky AI.

For GrayHat 2020 I was asked to create a short intro video for my Red Team Village talk “Learning by doing: Building and breaking a machine learning system”. So I put my green screen to good use and recorded this short clip for Red Team Village. Here is the link to the clip on Twitter: Hope you like it. :) The talk will be October, 31st 2020. Schedule: http://redteamvillage.io/schedule Grayhat’s website: http://grayhat.

There is a lot of discussion around terms such as red team, attack team, pentest, adversarial engineering or offensive security team and similar ones. I typically stay away from the (sometimes passionate) discussions that ensue whenever this topic comes up. Personally, I think a good strategy is to define programs and teams who operate in this space by what services the team (or teams) provide(s) to the organization. The business groups, blue team, developers, engineers, employees and clients are the customers.

While building “Husky AI” I started working a lot with Microsoft’s VS Code Python extension. It is a super convinient way to edit Jupyter Notebooks. I just use VS Code’s Remote SSH feature to get to my Linux host and work on modeling and testing there. When threat modeling “Husky AI” I identified backdooring of third party libraries and development tools as a potential issue to be aware of. So finding security issues in the tools is naturally something I am keeping an eye on.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. Overview: How Husky AI was built, threat modeled and operationalized Attacks: The attacks I want to investigate, learn about, and try out We talked about creating adversarial examples and “backdoor images” for Husky AI before. One thing that we noticed was that an adversary with model access can very efficiently come up with adversarial examples.

Excited to announce that I will be presenting at Grayhat - Red Team Village on October 31st 2020. The presentation is about my machine learning journey and how to build and break a machine learning system. If you follow my blog, you can guess that there will be lots of discussion around “Husky AI”. The bits and pieces that make up a machine learning pipeline, and how to threat model such a system.

This was also presented at BSides Singapore 2020. The slides are here and YouTube link is here. The origins of the Shadowbunny A few years ago, around 2016, I went on a relaxing two weeklong vacation. It was great to disconnect from work. I traveled to Austria, enjoying hiking in the mountains, and exploring Vienna. When I came back to the office, the team had placed a giant bunny teddy into my chair.

This year one of my goals was to learn about machine learning and artificial intelligence. I wrote about my journey before - including what classes I took and books I read, the models and systems I built and operationalized, threat modeling it to learn about practical attacks and defenses. My goal is to be knowledge enough in the AI/ML space enough to be able to help bridge the gap between research and operational red teaming - by doing practical things with life systems.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. Overview: How Husky AI was built, threat modeled and operationalized Attacks: The attacks I want to investigate, learn about, and try out Mitigations: Ways to prevent and detect the backdooring threat During threat modeling we identified that an adversary might tamper with model files. From a technical point of view this means an adversary gained access to the model file used in production and is able overwrite it.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. Overview: How Husky AI was built, threat modeled and operationalized Attacks: The attacks I want to investigate, learn about, and try out The previous post covered some neat smart fuzzing techniques to improve generation of fake husky images. The goal of this post is to take an existing image of the plush bunny below, modify it and have the model identify it as a husky.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. There are the two main sections of the series - more content will be added over time: Overview: How Husky AI was built, threat modeled and operationalized Attacks: The attacks I want to investigate, learn about, and try out The previous post covered basic tests to trick the image recognition model.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see related posts. The previous four posts explained the architecture and how Husky AI was built, threat modeled and deployed. Now it’s time to start the attacks and build mitigations. The appendix in this post shows all the attacks I want to research and perform in this series over the next few weeks/months.

This post is part of a series about machine learning and artificial intelligence. Click on the blog tag “huskyai” to see all the posts, or visit the machine learning attack series overview section. In the previous post we walked through the steps required to gather training data, build and test a model to build “Husky AI”. This post is all about threat modeling the system to identify scenarios for attacks which we will perform in the upcoming posts.

This post is part of a series about machine learning and artificial intelligence. In the previous post we walked through the steps required to gather training data, build and test a model. In this post we dive into “Operationalizing” the model. The scenario is the creation of Husky AI and my experiences and learnings from that. Part 3 - Operationalizing the Husky AI model This actually took much longer than planned.