Hi All,
I've been working with Falcon AIDR for some time now and I wanted to share some details on how I've been building an actually valuable detection pipeline. I say this because the product has a lot of valuable features but the documentation can sometimes lack the necessary insight required to make a finding into an alert a human needs to look at.
Basics
As this post is focused on genuine attempts to abuse an AI tool under protection by falcon AIDR rather than someone just not complying with business processes (which AIDR can help with) it's first important to understand what a malicious prompt finding is.
AIDR processes prompts sent to it's endpoints and evaluates the text for key properties. Of course the most intuitive one is sematic meaning where if you provide an instruction, a sense of urgency and an method of output that aligns to a known pattern of abuse it will very likely trigger a malicious prompt finding.
These patterns can and will also have intersectionality with normal human interactions where someone might be frustrated that Copilot won't write a script or in some cases a user might have a legitimate but peculiar request like building a database of identifies from LinkedIn. These contexts lend to sentence content and text structure that can look a lot like prompt injection attempts.
Prompt injection attempts in short are inputs that aim to illicit the language model to act outside a defined set of boundaries. Their shape can vary greatly and Crowdstrike maintains a large database of techniques which you can find for free on their website.
Turn a finding into an alert
Falcon AIDR will likely generate many malicious prompt findings over the course of a month especially if your business is one that provides services to other businesses (B2B). The volume of findings will likely overwhelm any team of analysts but Crowdstrike affords us a few controls and key areas to increase the quality of signals we get out from AIDR and protect analysts from alert fatigue.
Centre to any work you do in this area for Falcon AIDR will be the confidence score. It ranges from 0-1 where zero is no finding and 1 is an exact match. Now if you remember what I detailed earlier that exact match may not be a perfectly bonified adversary on their way to your crown jewels it very well be legitimate user still. However with the confidence score we apply some logic such that more interesting signals are raised and investigated.
I won't explore basic statistical analysis as hopefully everyone can Intuit it's value. If a user generates a hundred confidence 1 malicious prompt findings it's probably worth looking at.
Going further than statistical analysis we can start capturing how entire threads or conversations take shape over time in our logic that generates an alert. To do this we have to consider what adversaries typically do to bypass AI controls.
Looking at past public incidents at the heart of adversary playbooks is the general idea of an interative approach. No single prompt is distinctly or obtusely abusive however the total sum of the conversation as it's processed by the language model ultimately leads to compromise.
Luckily thanks to the verbosity of Falcon AIDR we can build detection logic that covers this. For each malicious prompt finding we can aggregate on a common anchor such as the conversation URL or user and track how that users conversation changes and develops.
In a given conversation Falcon AIDR might rate the first prompt with no findings then it may identify confidential PII through it's classifiers in prompt two and perhaps a non English language in prompt 5. It's these smaller and independently shallow findings that gradually build and develop into a picture that will be very obviously abusive.
Thankfully this is easy to instrument because Falcon AIDR stores it's telemetry in NGSIEM (repo/view perms recently changed if you can't see it).
to be continued...