Watching the Agents: What a SIEM Sees of Attacks on LLM Agents, from Small Open Models to Claude
- 1Independent Researcher
Abstract
Security operations centres are asked to watch LLM agents that read untrusted content and act through tools. We measure what a SIEM fed by the application's telemetry can see of attacks on such agents, using an open pack of 25 Microsoft Sentinel rules (23 also for Wazuh), 112,894 items from 14 public datasets, two open injection classifiers, and live attacks in one demo application on twelve models: five small open-weight models, three Azure-hosted OpenAI models and four Claude models behind the Anthropic API. Content signatures fire on under 1% of benign prompts but recall only 12% of attack prompts, and lose almost everything to rewording. Against models that comply, rules over what the agent did or produced caught all 269 successful attacks across our ten scenarios. Against three Claude models (Haiku 4.5, Sonnet 5.5, Opus 5.5) those rules saw 13% of 300 attack sessions, and none on Opus 5.5, because the attacks failed. In a pre-registered confirmation, logging provider refusals as events and matching refusal language in the answer raised this to 70% (72% on Opus, almost all from Anthropic's serving-side classifier) at an 8% benign rate, so it measures attempts rather than acting as a precise detector. An instruction placed beyond the application's logged preview was followed but did not appear in the log, and Azure's default content filter blocked nothing past the first 1,000 characters of a user message and nothing in tool results. When the agent did not act on an injected request to email an internal document to an outside address, nine of twelve models told the user to do it, as the next step, in 61% to 100% of their runs (1,440 in all, labelled by language models); only Claude Sonnet 5.5 and the two Opus models warned. One of two cross-family raters disagreed with the labels, so this generality claim missed its pre-registered criterion; the other agreed on 97% of items.
Related
- Watching what others eat: exposure to social-media eating content, body appreciation, and eating behaviors in Turkish adults
- E.6. Round Table: Who’s watching the food giants? The EU case for laws holding companies to account for what they sell
- Impact of Ransomware Attacks on Small and Medium Enterprises (SMEs)
- What Do CAE Simulation Agents Really Need Beyond a Generic Harness?
- Autonomy Is a Property of the Operation: A Framework for Deciding What DevOps Agents May Do Unattended
BibTeX
@misc{lajko2026watching,
title = {{Watching the Agents: What a SIEM Sees of Attacks on LLM Agents, from Small Open Models to Claude}},
author = {Levente Lajkó},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23650043},
url = {https://zenodo.org/records/23650043},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Watching the Agents: What a SIEM Sees of Attacks on LLM Agents, from Small Open Models to Claude
AU - Levente Lajkó
PY - 2026
DA - 2026/10/25
PB - Zenodo
DO - 10.5281/zenodo.23650043
UR - https://zenodo.org/records/23650043
AB - Security operations centres are asked to watch LLM agents that read untrusted content and act through tools. We measure what a SIEM fed by the application's telemetry can see of attacks on such agents, using an open pack of 25 Microsoft Sentinel rules (23 also for Wazuh), 112,894 items from 14 public datasets, two open injection classifiers, and live attacks in one demo application on twelve models: five small open-weight models, three Azure-hosted OpenAI models and four Claude models behind the Anthropic API. Content signatures fire on under 1% of benign prompts but recall only 12% of attack prompts, and lose almost everything to rewording. Against models that comply, rules over what the agent did or produced caught all 269 successful attacks across our ten scenarios. Against three Claude models (Haiku 4.5, Sonnet 5.5, Opus 5.5) those rules saw 13% of 300 attack sessions, and none on Opus 5.5, because the attacks failed. In a pre-registered confirmation, logging provider refusals as events and matching refusal language in the answer raised this to 70% (72% on Opus, almost all from Anthropic's serving-side classifier) at an 8% benign rate, so it measures attempts rather than acting as a precise detector. An instruction placed beyond the application's logged preview was followed but did not appear in the log, and Azure's default content filter blocked nothing past the first 1,000 characters of a user message and nothing in tool results. When the agent did not act on an injected request to email an internal document to an outside address, nine of twelve models told the user to do it, as the next step, in 61% to 100% of their runs (1,440 in all, labelled by language models); only Claude Sonnet 5.5 and the two Opus models warned. One of two cross-family raters disagreed with the labels, so this generality claim missed its pre-registered criterion; the other agreed on 97% of items.
ER -