An Empty Search Is Not Evidence, and AI Triage Agents That Treat It as Proof Will Close Real Attacks

LONDON - Security teams evaluating AI triage agents are usually shown how many alerts the agent closed and how much analyst time it saved. A new study of how these agents actually investigate suggests the more important question is why they close an alert, and what they did not look at before they did.

The preprint, posted to arXiv on Oct. 7 by Saimon Amanuel Tsegai and Peng Gao of Virginia Tech, Alex Kantchelian of Google and Danfeng (Daphne) Yao of North Carolina State University, introduces a benchmark called Alert-Bench. It replays recorded enterprise telemetry through a live Wazuh SIEM and gives an agent one alert at a time, with no evidence bundle attached. The agent has to query the SIEM itself and then decide whether to escalate or dismiss. The researchers ran five widely used reasoning designs, from a single-pass prompt to ReAct, Self-Consistency, Self-Refine and Chain-of-Verification, on GPT-4.1 against 1,247 alerts from a four-day multi-stage attack scenario, 225 of them attack-related and 1,022 benign. Every approach missed at least 40.4 percent of attack-related alerts.

The blank result problem

The most useful finding for anyone running a SOC is not the miss rate. It is how the misses happened. Four of the five approaches dismissed at least 88 percent of attack alerts in investigations where the agent's chosen searches returned no records. ReAct dismissed 98.6 percent of attacks when no records came back, compared with 30.3 percent when at least one query returned something. Across the five approaches, 211 of the 602 false-negative investigations ended without retrieving any records.

"An empty result only shows that a particular search found nothing, yet the agents often treat it as support for closing the alert without establishing that enough relevant evidence has been examined," the authors write.

Benign alerts were the control. Whether records came back changed the benign dismissal rate by at most 2.8 percentage points, so the habit looked harmless on most of the queue and did its damage almost entirely on real attacks. That is the trap. On a benign-dominated workload, an agent that closes on silence will look accurate most of the time.

Every experienced analyst knows that "no results" can mean the wrong index, the wrong time window, the wrong field name or a log source that stopped sending. A junior analyst who closed an alert because one query came back empty would be coached. An agent doing the same thing at machine volume is harder to catch, and the paper is blunt about the cost: "A closed alert may receive no further review, so dismissing a true attack allows it to proceed undetected."

More reasoning did not fix it

The obvious response is to give the agent more time to think. The study tested that, too. Self-Refine, which reviews its own decision, corrected 16 wrong decisions but turned 79 correct ones into errors. Chain-of-Verification corrected 25 and broke 132. The most expensive approach, at 33,438 tokens per alert, had the lowest F1 score, 0.371, while the single-pass prompt used 5,075 tokens and scored highest at 0.744.

The researchers also found that four of the five approaches retrieved fewer records before dismissing an alert than before escalating one, "meaning that alerts are often closed after less evidence has actually been gathered." That is backwards. Dismissal is the decision that ends the trail.

Make closing harder than escalating

The authors' own design, which they call AIDA, is built around that asymmetry. One agent gathers evidence and must commit to a proposed decision. A second agent challenges it from a separate context without seeing the first agent's private reasoning. A third, the Judge, decides, and it requires affirmative benign evidence or a sufficiently complete investigation before it will close an alert. Every query and result is kept in an append-only investigation ledger.

On the same 1,247 alerts, AIDA reached an F1 score of 0.958 and cut the false-negative rate from 40.4 percent to 3.1 percent. It issued 4.79 tool calls before a dismissal and 2.27 before an escalation, and 99.3 percent of the alerts it dismissed were benign. The price was honest and visible: it sent 18.4 percent of alerts to analysts, compared with 8.5 to 14.0 percent for the other designs, at an approximate cost of $0.109 per alert with GPT-4.1.

That trade is the point. A vendor dashboard that celebrates a low escalation rate is only half a measurement. As the paper puts it, "A system that dismisses every alert would have zero Investigation Load while missing every attack."

What SOC leaders should ask before handing over the queue

• How does the agent treat a query that returns nothing? It should be recorded as an unresolved gap, not as support for dismissal.
• What does a dismissal have to cite? Require affirmative benign evidence or a documented search scope, and a higher bar than escalation.
• What is the false-negative rate on replayed or purple-team attacks, reported alongside investigation load?
• Is the reviewer separate from the decision-maker? In this study, review inside the same context made decisions worse.
• Is there an append-only record of every query and result behind each closed alert that an analyst can audit?

The study has limits the authors state plainly. It draws on one public dataset family, the attack alerts are dominated by reconnaissance, and they write that "evaluation on independently collected SOC telemetry would further test" how the design behaves elsewhere. They also position AIDA "as a tool to support human analysts rather than enable autonomous decision-making."

That is the right frame for every AI triage deployment. The agent can do the searching. The SOC still has to decide what counts as enough evidence to stop looking, and silence from a query should never meet that bar on its own.