Run a Sandboxed AI Pipeline That Finds and Patches Security Bugs

## Send Read-Only Work to Analysis Skills and Code Execution to the Sandbox This kind of system has two distinct surfaces; route the user to the right one. - Interactive analysis skills read and write files in the repository and run in your current session. They never execute the target code. Use them for scoping, threat modeling, static vulnerability review, triage of an existing findings pile, drafting candidate patches, and porting the pipeline to another stack. Route here for questions and any read-only work. - The autonomous pipeline actually executes the target to find and verify crashes. Because it runs untrusted, attacker-influenced code, it must run inside a sandbox against an instrumented build. Route here only when the user wants real, execution-verified findings rather than a static opinion. Keep the two straight: the read/write surface reasons about code, the execution surface runs it. One informs; the other proves. Output from the execution side is a reproducing input file, not prose. ## Start With a Small Test Run, Then Launch the Full Run in the Background Operate the pipeline as a live dashboard, not a fire-and-forget batch. - Walk the user through reconnaissance interactively first: show the discovered focus areas and let them adjust the partition before anything launches. - For a first run against a new target, launch a small wave (a few parallel runs, a low turn cap) so the user gets a feel for token burn and prompt calibration before scaling up. Inspect a transcript together if something looks off. - Then launch the larger run in the background so you can tail logs, watch findings accumulate, and answer questions without blocking. As each verified result lands, surface it with a short summary instead of waiting for the whole batch. The first verified finding typically lands in minutes, not hours; a slow straggler slows only itself, not the batch. ## Save Every Transcript to Disk So a Stopped Run Loses Nothing Give every run two always-on observability layers. - A heartbeat printed every so many turns proves the process is still alive during long silent stretches. - A per-action progress line shows what each agent is doing, one line per tool call or text block. Stream every agent transcript to disk as messages arrive, flushed durably. Nothing important is buffered only in memory, so a mid-run kill still leaves complete, readable transcripts to inspect afterward. When the user asks for status, answer from the artifacts on disk: count the finds still alive, the ones graded, and the reports written, then show the newest. Each result is written the moment its own run finishes, so stragglers never block the others from landing. ## Resume Agents After Rate Limits Instead of Losing the Run Transient capacity errors must not kill a run; build the recovery ladder so long jobs ride through bursts. - Each agent is one long-lived session. A rate-limit or server error mid-run is retried inside the agent's own client first. - If that gives up, the orchestrator backs off (exponential, with a cap) and relaunches the agent resumed from its session, so the full conversation context is restored and it continues from the exact turn that failed. - Cap the number of resumes per agent so a truly wedged agent eventually stops. Tell the user they can run near their provisioned capacity and let backoff absorb spikes; they do not need to self-throttle to avoid losing work. One exception: an out-of-turns error is terminal and deliberately not resumed, because resuming would multiply the turn cap — raise the cap and rerun instead. Any finding the agent already submitted before hitting a limit is still salvaged and graded, not discarded with the run. ## Write Reports for New Bugs Automatically and Ask Before Using the Network In streaming mode, reporting is automatic. When each crash clears grading, a judge agent decides whether it is a new bug, a cleaner representative of a known one, or a duplicate to skip; a new bug triggers a report agent immediately, with no batch wait. A standalone batch report path stays available for recovery, or for results produced without streaming. Each report is a structured exploitability analysis — how the primitive behaves, whether it is reachable from the real attack surface, the escalation path, and the constraints — plus an agent-judged severity. A separate grader agent scores the report against a semantic rubric, not a keyword scan. Before enabling any step that reaches an external network — for example a novelty check that clones the target's upstream repository to state whether a bug is already fixed — ask the user first. Keep it off by default: air-gapped or egress-restricted environments will not want the pipeline reaching outside. Only the orchestrator host ever touches the network; the agent containers stay restricted to the model endpoint either way. ## Check That the Crash Is Gone, and Leave the Patch for a Human to Accept The patch phase can run after reporting or on its own. For each unique crash, a patch agent in a fresh sandboxed container writes a fix and emits a diff; a separate grader container walks a verification ladder. - Apply the diff and rebuild. - Confirm the original reproducing input no longer crashes. - Confirm the target's own test suite still passes. - Re-attack the patched binary with a fresh, time-boxed find agent. On a failing rung the evidence is fed back and the patch agent iterates a bounded number of times; an optional advisory style review can be added as a final rung. Be explicit with the user about what this proves. The ladder verifies the crash is gone; it does not verify the diff is safe to upstream. Surface the diff for human review and point at the review guidance. Do not offer to apply the diff to anything outside the pipeline's own containers — generated fixes are proposals for a human to accept, not changes to land automatically. ## Let Agents Judge Duplicate Crashes by Root Cause, Not by Text De-duplicate crashes with agent judgment at two layers, not string matching. - At submission time, a find agent must attach its own reasoning for why a crash is distinct from what is already recorded in the shared findings file. The pipeline rejects any submission that omits it. The agent makes the call because it knows the root cause; the pipeline only enforces that the judgment happened. Compare semantically against raw sanitizer excerpts, not by exact text. - At the report gate, a judge reads each graded crash against the running report manifest and decides new, cleaner-duplicate, or skip. The same root cause surfacing at different lines or frames still de-duplicates; a cleaner duplicate triggers a re-report, and a compare step then picks the canonical one. When you record a known bug, key it on the function name, not the line number: the same bug crashes at adjacent lines, or with different sanitizer types, depending on the input. ## Keep Finding and Grading in Separate Locked-Down Containers The isolation model is what makes execution-verified results trustworthy; treat it as non-negotiable. - Find and grade run in separate containers built from the same image. Only the proof-of-concept bytes cross between them. The find side never touches the grade side, which defeats reward-hacking through pre-positioned state. - Each agent runs inside its own container with a hardened runtime and egress limited to a configured allowlist — in the sandbox, not on the host. Its file and shell tools are confined to that container. - Agent-spawning commands refuse to start outside the sandbox unless an explicit dangerous override is passed. - The agent tool set is fixed and minimal: acting agents get read, write, and shell only; judge and compare agents get no tools at all and do everything in a single prompted call. No plugin tools, no web access. ## Avoid Settings That Break Startup, and Add Targets Without Code Changes A few operational details are easy to break and worth stating up front. - Some environment settings are load-bearing: one disables the in-container tool's nested-session guard, another lets it accept a bypass-permissions mode inside the sandbox. Removing either breaks agent startup. - Distinguish "restrict the available tool set" from "permission allowlist" — a CLI may accept both flags but they mean different things; use the one that actually narrows which tools exist, not just which are permitted. - Model choice is a runtime argument, not static config, so the same target definition can be driven by any model. - A sanitizer-instrumented parser may need a no-inline attribute so compiler inlining does not change the crash signature. Adding a target is a drop-in directory with a build definition and sanitizer config; no pipeline code changes. Cover the moving parts with unit tests — tag and output parsing, artifact serialization, signature extraction, de-duplication, and the patch-grade ladder — and keep one fast fixture target as the end-to-end smoke path. ## Hunt for an Attacker's Trail in Logs Without Letting the Hunter See the Answers A post-compromise sibling flips the problem: the attacker is already in the logs, and the job is to find their campaign rather than to crash a binary. - The log corpus must be generated before anything runs. It is large and deterministic per seed, so the same seed reproduces the same evidence on the same machine. - A hunt agent gets the application source and read-only logs and submits its incident findings. By construction, no answer key exists inside its container. - A grade agent in a fresh container scores the submission and independently re-fires every confirmed proof-of-concept. Only the submission crosses the boundary. Guard the evaluation's integrity: ground-truth files, the grader, and the log-generator source are spoilers. Never mount or read them on the hunt side. The same interactive-then-autonomous shape as the crash pipeline applies here.