A security agent finds a weakness, writes a patch and opens a pull request. The tests pass. A reviewer merges it. The affected service is still running yesterday’s build.
That gap is where a continuous defense program can lose its meaning. The team has produced work, but has not yet established that the exposure is gone. As more investigation and remediation move to agents, each handoff needs a clear answer: what changed, who allowed it and what evidence shows that it worked?
OpenAI’s Defense Factory describes an agent-driven operation that connects inventory, discovery, validation, ownership and verified remediation. Its reference architecture includes isolated environments, policy enforcement, credential mediation and audit. Our focus at Oktsec is how to make those boundaries hold while agents do useful security work.
Follow the finding to the running service
OpenAI reports that follow-up checks exposed a gap between merged patches and fixes deployed across its fleet. It kept automatic issue reopening disabled while accounting for rollout delays. The described workflow retains human review and authorized deployment even when agents prepare the patches. OpenAI’s case study and remediation workflow.
Consider a hypothetical API that lets one tenant read another tenant’s export. A code review can identify the missing access check. An isolated reproduction can establish that the tested version returns the wrong tenant’s data. A patch test can establish that a proposed version rejects the same request. None of these observations, by itself, identifies the version serving a customer request now.
The final check needs the deployment identity and a safe way to test the affected path. Use synthetic tenants and records where possible. Record the environment, build, test identity and outcome. If rollout is partial, report which instances were checked and which remain outstanding. A ticket should make those states visible instead of compressing all of them into “fixed.”
- Isolated vulnerable buildReproduce the issue
The test account can read another synthetic tenant’s export.
- Proposed patchTest the correction
The same request is denied. Permitted exports still work.
- After approved deploymentVerify the service
The running build enforces the correction in the checked environment.
A passing patch test should not close an exposure that remains in a running version.
Validation needs a working environment
OpenAI reports reproducing 19.5% of findings at runtime and a 0.81% false-positive rate after dynamic validation. It also describes missing dependencies and configuration differences that prevented tests from running correctly. These are results from its own program, not a benchmark for another organization. Read the validation discussion.
The two percentages should not be treated as complements. Nor should every finding without a successful reproduction be labeled a false positive. If the application never started, the investigation has learned something about its test environment, not necessarily about the suspected vulnerability.
Keep separate outcomes for a reproduced issue, a finding disproved by relevant evidence and an inconclusive test. Include the reason for an inconclusive result: a missing dependency, unavailable identity provider, unrepresentative fixture or unresolved configuration difference. That record tells the next reviewer what must change before another run is useful.
For the tenant-export example, first establish that an authorized export succeeds. Then test the cross-tenant request. If both requests fail because the storage emulator is unavailable, the run cannot validate the access-control fix. This small baseline check prevents a broken environment from becoming evidence of successful remediation.
A repeatable environment also needs a recorded starting state. Pin the source revision and relevant dependencies, identify the fixtures and create a fresh workspace for each run. Preserve enough configuration to explain the result without placing live secrets in the evidence package.
More parallel work does not need more authority
OpenAI describes expanding the work agents could perform separately from the changes they were permitted to make. That is a useful distinction for a team deciding how much autonomy to introduce. Lessons from the defensive loop.
An investigator may need to inspect several repositories and reproduce a suspected issue. It does not follow that it should be able to change repository settings, merge its own patch or deploy a new version. Assign those operations deliberately. Where they belong to different stages, use identities and permissions that keep the separation enforceable.
Review what an approval actually covers. An approval for a patch should identify the change being reviewed. A deployment approval should identify the intended environment and artifact. If the patch changes afterward, the earlier review should not silently authorize the new content. The person reviewing it needs the relevant diff, test results and any unresolved limitations.
The security agent is part of the attack surface
An agent investigating a vulnerability will encounter material an attacker could have written: a repository file, an issue description, a service response or a test artifact. The content can help the investigation without acquiring authority over the investigator.
Suppose a repository instruction says that reproducing the tenant-export issue requires uploading an environment file to an external debugging service. The agent should be able to report that instruction. It should not gain the ability to read unrelated secrets or send them to that service merely because the instruction appears near relevant code.
A reproducible container is a useful starting point, but inspect what it can reach. A mounted host socket, broad cloud role or unrestricted credential store can give a short-lived environment lasting consequences. Define the runtime’s identity, network destinations, resource limits and teardown behavior. Where a credential broker is used, check what operations the broker will authorize for that runtime.
MCP tools, direct command-line clients and APIs may offer different routes to the same operation. If a gateway controls one connector while the runtime can make the request directly with the same credential, the permission review is incomplete. Test the paths the agent actually has, including delegated work.
Similarly, SECURITY.md and reusable skills provide valuable context, but text instructions alone do not enforce access. Version and review them as workflow dependencies. Enforce the resulting limits in the identity system, runtime, gateway or destination service that can actually reject the operation.
Measure the completed defense
Finding volume is useful for understanding workload, but a larger queue can also mean more duplication or more work waiting for an owner. Start with measures that make the bottleneck visible.
- Time to a validated result. Track the interval from a candidate finding to a reproduced, disproved or explicitly inconclusive result. Report environment failures separately.
- Time to accepted ownership. Distinguish assigning a team from that team accepting responsibility. Keep unresolved ownership visible.
- Time to verified remediation. Measure through authorized deployment and the relevant follow-up check, rather than stopping at merge.
- Verification coverage. State which affected environments and paths were checked. A passing check on one instance is not fleet-wide coverage.
Keep the definitions stable when comparing runs. A shorter queue after excluding difficult environments is not necessarily better defense. Record the exclusions and changes to the test process so that an apparent improvement can be evaluated.
These measures also help decide where automation is worth adding. If findings wait for ownership, another scanner may not address the delay. If validation repeatedly stalls on the same dependency, fixing the environment can be more valuable than increasing agent concurrency.
Start with one controlled loop
Choose a service with an accountable owner, an environment the team can reproduce and a clear release process. Give the agent a bounded investigation task. Establish the evidence required for a finding, the permissions needed to prepare a patch and the checks required before closing the issue.
At Oktsec, this is a useful scope for an agent security assessment: test whether untrusted material can redirect the investigator, whether its credentials reach beyond the assignment and whether another tool can bypass a restriction. The test should also demonstrate that the intended investigation still works.
Oktsec Control can govern actions routed through its configured enforcement point. Action evidence can support the review of those decisions. The destination system and deployment records are still needed to establish what was installed and whether the correction holds.
After one workflow is reliable, expand its coverage with the same permission and evidence requirements. Continuous defense becomes useful when each run leaves a better-understood system and a verified change. The agents doing that work need boundaries just as much as the agents the organization is trying to secure.
For the attacker’s side of the problem, read our analysis of Anthropic’s September threat report. For the evidence question, see AI agent incidents: what can you prove?
Source and scope
Based on OpenAI’s Defense Factory architecture and case study, reviewed September 10, 2026. Reported program results are OpenAI’s own figures. The tenant-export scenario, permission model and measurement recommendations are Oktsec’s analysis, not a reproduction of that program or a claim of equivalent results.