SecureFlow: multi-agent triage for product security review

Product security teams have a volume problem. Every new feature is a potential source of risk: a new API endpoint, a new place PII gets stored, a new vendor that sees customer data, a new obligation under PCI, HIPAA, or GDPR. Security, privacy, and GRC teams can’t read every feature description that ships, so most of them don’t. Review happens when someone remembers to ask, which means it happens late or not at all.

SecureFlow automates the first pass: read the feature description, decide which teams need to review it, and open a routed review ticket for those teams. It runs as a GitHub Action. When a developer labels an issue feature-request, SecureFlow reads the body, screens it, and files review issues for the teams that have something to investigate.

Repo: github.com/trwilcoxson/secureflow.

The problem

Triage is a routing decision, not a fix. The job is to decide, from a feature description, whether security, privacy, or GRC needs to review it, and to open a ticket only for the ones that do. A false positive wastes a reviewer’s time. A false negative lets a risky feature ship without review. The routing has to be accurate, and it should not open tickets for low-risk changes.

SecureFlow screens and hands off to humans. It does not approve or block anything on its own.

The design

Four layers, separated on purpose.

Trigger. A GitHub Action watches for the feature-request label. When it fires, it calls the system with the issue number and reads the body via gh issue view --json.

Orchestrator. The orchestrator is plain Python. It validates the input (20 to 10,000 characters), dispatches the three specialist agents in parallel with asyncio.gather(), collects their structured outputs, and decides what to do. If an agent sets requires_review=True, it files a review issue routed to that team and posts a summary comment on the source issue. No model decides what happens next; the control flow is explicit code.

Specialist agents. Three of them, each screening a different risk category:

• Security looks at attack surface, data exposure, auth gaps, and third-party trust. New endpoint, credentials in logs, missing MFA.
• Privacy looks at PII collection, new data flows, and automated decisions. User tracking, data shared with vendors, profiling.
• GRC looks at regulatory obligations and audit impact. Payment card data pulls in PCI, health data pulls in HIPAA, EU data pulls in GDPR.

Tools. Issue creation goes through gh issue create, invoked with an argument list via asyncio.create_subprocess_exec, never a shell string, so there is no injection surface. DRY_RUN=true is the default locally, which logs what would be created without touching the GitHub API.

The orchestrator can be deterministic because the agents return structured output, not prose. Each is a Pydantic AI agent with an output_type that pins the response to a schema: a list of concerns, each with a severity and category, plus a requires_review flag. The model does the reading and judgment; the schema turns that into something the routing code can branch on.

Team ownership without code changes

Each agent’s screening criteria live in their own instruction file: instructions/security.md, instructions/privacy.md, instructions/grc.md. The security team owns the security file, privacy owns theirs, GRC owns theirs. Changing what counts as risky in a domain means editing that team’s markdown file, not the system code. In production each team could keep its file in its own repo and pull it in as a submodule or CI artifact. The system loads all three at startup.

This keeps the domain knowledge with the people who have it. The people who know what counts as a privacy risk are not the people maintaining the orchestrator, and they should not have to ship a code change to update a screening rule.

What a run looks like

The sample run in the repo screens a Stripe payment integration: tokenized card references and billing data, no encryption at rest, a webhook endpoint, full request bodies in the logs. All three agents flag it. SecureFlow produces 12 concerns, one critical, and an overall NO-GO, and opens a severity-tagged review issue for security, privacy, and GRC. That is the case it is built for: cross-cutting risk that should not ship on one team’s approval.

The opposite case: a cosmetic CSS change produces no concerns and files nothing. Suppressing false positives is deliberate. A screener that flags every change gets ignored.

Evaluation

The eval suite is seven cases across the risk spectrum, graded by rule-based checks and an LLM judge (built on pydantic-evals):

• Low-risk internal tool, expected GO. SSO-gated, anonymized, read-only.
• Critical data exposure, expected NO-GO routing to all three teams. PII in the open, no auth, no encryption.
• Third-party integration, expected CONDITIONAL routing to security and privacy.
• ML credit scoring, expected NO-GO routing to privacy and GRC. Automated decisions, bias-prone training data.
• Healthcare portal, expected NO-GO routing to all three teams. PHI sent to a third-party LLM, weak auth.
• Vague description, expected caution. Flag insufficient detail and ask for clarification.
• CSS change, expected GO. Cosmetic, no data or backend change.

The saved run passes all seven. Across runs the pass rate sits at 96 to 100 percent (six or seven of seven). When a case fails, it is usually severity calibration or the wording of a rationale, not a wrong routing decision. The routing itself stays stable: which teams get pulled in. That is the output a human acts on.

There are limits. Seven cases is a smoke test, not a benchmark, and the judge is itself a model. The suite shows the routing behaves as designed across clearly labeled cases. It does not show how the screener does on ambiguous cases, which is where a security team spends most of its time. It is not evidence that the screener is good enough to ship features without human review.

Why it matters

The main design choice is keeping the model out of the control flow. Structured output plus a deterministic orchestrator means the model contributes judgment and nothing else: it reads the feature and fills in a schema, and every downstream decision is explicit code. Agents flag, code routes, humans review. That makes the system auditable: a reviewer can see why each issue was filed, and that it files the same way every time.

The safeguards follow from the same choice: dry-run by default, input length limits, schema enforcement on every agent output, argument-list subprocess calls with no shell, secrets through env vars and GitHub Secrets, the trigger gated to one label, and the Action scoped to issues: write. These are standard precautions for a tool whose job is security.

Built with Pydantic AI and pydantic-evals on gpt-4o-mini, wired into GitHub Actions. Code, instructions, and the eval suite are in the repo.