Threat Model Suite: (Mostly) Autonomous Threat Modeling

Threat Model Suite is a security-assessment suite for Claude Code. You point it at a real system and a pipeline of specialist agents produces a full review: an architectural threat model with a diagram set, a privacy impact assessment, a compliance gap analysis, and prioritized findings. The last step is the one I care about most. An evaluation harness checks that the output is reliable before you trust it.

What it produces

The main skill is threat-model: an 8-phase run from reconnaissance to final report, in Solo mode (threat model plus report) or Team mode (adds privacy, compliance, and code-review agents). The pipeline is seven personas, plus two standalone reviewers you can call on their own. Each agent runs in a fresh context, writes a structured output file, and emits run events so you can watch what is happening.

On the visual side it renders Mermaid data-flow diagrams (a four-layer structural view plus a risk overlay) and a set of analytical visuals: attack trees, attack-flow and kill-chain graphs, auth sequences, a STRIDE-per-element matrix, a likelihood-by-impact risk heat map, a MITRE ATT&CK technique layer, an RBAC matrix, and an SBOM dependency graph. Output goes to HTML, Word, PDF, or an executive PPTX. A coverage ledger records, per item, what was found, what is absent, and what could not be determined from the sources, so gaps are recorded instead of dropped.

Example output

Here is the suite running end to end against a real target, a Terraform stack that stands up an ECS-hosted web application on AWS:

The live demo hosts the generated deliverables in full: the threat model report, the same report as a PDF and a Word document, and an executive summary deck. The structural data-flow diagram and its risk overlay came out of that run:

The phase-by-phase artifacts are committed in the repo, including recon.json, findings.json, coverage.json, the privacy assessment, and the compliance gap analysis.

Spec-first, with one hard rule

The whole thing is designed spec-first with OpenSpec. Every capability is written as Requirements and Scenarios (WHEN/THEN) before it is built, and those specs live in openspec/ as the starting point for understanding the system.

One principle runs through all of it: the agents do all the reasoning and generation, and the only determinism is in the templates and the evals. The deterministic code never scripts or judges the answer. When an eval needs to gate on something, the skill emits that thing as a neutral structured field (a declared kill chain, a trust-boundary kind, a dependency manifest, a coverage state) and the eval checks structure over that fact. It never tries to infer content in code. That boundary keeps the automated checks from re-deriving the analysis and getting it wrong.

Why the evaluation is reference-free

Verifying an agent’s security analysis is hard because there is no answer key for an arbitrary repo. An earlier version of the harness used hand-authored cases with golden “must-find” lists and exact-severity gates. That measured conformance to a fixed key rather than reliability. It forced answers: it once recorded a correctly reasoned MEDIUM finding as a miss because the key demanded HIGH. It could not credit a valid finding nobody had pre-listed, and it did not scale to new targets.

So the golden lists were dropped in favor of grading properties that should hold for any good threat model, with no per-target ground truth. Deterministic layers enforce structure and internal consistency (for example, severity == band(L×I)), grounding against the real repo, coverage of the system’s own discovered surface, and verification that the diagram layers and visuals are present and well-formed. LLM-judged layers assess the softer questions: reasoning quality, adversarial recall, recon completeness, diagram correctness, and cross-run stability. None of those need an answer key, so the harness works on whatever you point it at. The reliability harness also has committed runs against a vulnerable web API and a vulnerable Terraform target, each with its eval report.

How it fits a workflow

Install it as a Claude Code plugin and the skills, the specialist agents, and a validation hook all install together. Then: Run a threat model on <target>. Before the report is generated, a PreToolUse hook runs the deterministic validator over the emitted recon.json, findings.json, and coverage.json, and blocks report generation until they pass, feeding the specific defects back to the agent to fix. The harness enforces this validate-then-retry loop with specific feedback; it does not rely on the agent to check itself. Install the skill alone and you get the same check as a soft gate the orchestrator runs instead.

The suite is MIT-licensed. Code and the full spec set are in the repo.