Patterns for Building Cybersecurity Evals

Why Talk About Cybersecurity Evals Now? As AI models advance rapidly, a critical question emerges: what role will AI play in cybersecurity—helping defenders find vulnerabilities faster, or empowering attackers? To answer this, we need a rigorous evaluation framework to measure AI's cybersecurity capabilities. Eugene Yan's article breaks down the core patterns for building such evaluation systems.

The Four Pillars of Cybersecurity Evals The article outlines four core components that form the foundation of effective cybersecurity evaluations, creating a controlled experimental environment.

First is the sandboxed target, the vulnerable system under test. It typically runs in isolated Docker containers—either a codebase with known flaws or a network with various services and hosts. This ensures a safe, controllable testing environment without affecting the real world.

Second are inputs that influence task difficulty. This design cleverly simulates different attack scenarios. The simplest case provides vulnerability descriptions or even patches, akin to attackers reverse-engineering patches to build exploits (one-day exploitation). The most challenging scenario gives the model only the vulnerable code, with no prior knowledge of the flaw or patch (zero-day discovery). By adjusting the amount of input information, we can progressively test the model's upper limits.

Third is the toolset. The AI model can use various tools during tasks—bash shells, read/write tools, web search, debuggers, static analyzers, and more. This simulates the real workflow of security researchers and tests the model's ability to plan and use tools effectively.

Finally, the grader accepts the model's submitted work (e.g., a working exploit or captured flag) and provides immediate feedback. Grading is typically deterministic: for C/C++ memory bugs, success means triggering a sanitizer crash; for unauthorized code execution, it means retrieving a hidden flag string.

From Outcomes to Process: A Key Insight The article makes a crucial point: grading solely on final outcomes is too coarse. A model that successfully finds a vulnerability but fails to build an exploit (scoring 0) differs vastly from one that can't even locate the flaw (also scoring 0), yet their scores don't reflect this distinction.

Thus, more advanced evaluations introduce multi-level subtasks along the attack chain to award "partial credit." This forms a capability pyramid:

1. Find the vulnerability: Locate security flaws in the codebase.
2. Reproduce it: Construct a proof-of-concept that triggers the vulnerability.
3. Exploit it: Execute unauthorized code on the target system.
4. Achieve the goal: Fulfill the attacker's intent, such as data exfiltration or privilege escalation. This tiered assessment paints a clearer picture of a model's capability boundaries, showing where it struggles—is it "trying but unable" or "completely clueless"?

Practical Value for Developers and Companies For developers and companies focused on AI security, this evaluation model offers direct reference. First, it provides a blueprint for building internal security AI testbeds. Second, it reminds us that when purchasing or evaluating security-related AI tools, we shouldn't just look at binary outcomes like "can it find a vulnerability." Instead, we should assess performance across the entire attack chain—accuracy in vulnerability localization, ability to generate proof-of-concept code, etc. This leads to a more rational evaluation of AI's true utility in security workflows.

The Unseen Challenge: Evaluations in a Fast-Moving Field The article implies a deeper challenge: the timeliness of evaluation benchmarks. In cybersecurity, vulnerabilities and attack techniques evolve rapidly. A CTF challenge effective today might soon be "memorized" by new AI models. Thus, continuously updating and designing novel evaluation tasks is itself a formidable challenge. Assessing AI's security capabilities is inherently a dynamic, adversarial process.