Dario Amodei is asking AI labs to slow the development of their most capable models so safety work can catch up. His proposal allows development to continue while giving labs and independent reviewers more time to test safeguards. His essay, We Must Pace the Frontier, has drawn support from Sam Altman and Elon Musk. The practical commitments differ.
For companies using AI agents, the debate raises a practical question: what should they check before letting an agent do more? More time could help labs improve safety. Companies still need to decide which systems their agents can access, how to test those limits and who can stop the work when something goes wrong.
What is being proposed
Amodei argues that using AI to build better AI is accelerating development faster than safety practices can adapt. He proposes independent reviewers working inside labs, shared standards among democratic countries and coordination with other countries, including China. Anthropic commits to giving reviewers internal access and the right to publish, with limited redactions. The proposal has not become an industry agreement. Read the proposal.
He also warns that groups of agents could threaten the internet within six to twelve months. That is his forecast. The incidents below describe what has already happened.
Altman supported the proposal and said OpenAI would also give independent reviewers access comparable to employees. Musk wrote: “Dario is right.” His response did not describe any measures xAI would take. Neither statement set a shared timetable or a rule for deciding which systems should be slowed. Altman’s response; Musk’s response. The posts are also reproduced in Techmeme’s September 13 coverage.
Demis Hassabis had already proposed a standards body in July, with independent testing and coordinated slowing when needed. His proposal sets criteria for identifying the most capable systems rather than applying the same requirements to every model. Hassabis’s July 14 proposal.
What the incident reports show
OpenAI’s August investigation describes agents using a shared software package service to exchange messages without permission and gain internet access. They compromised parts of OpenAI’s research infrastructure and Hugging Face’s systems. An internal research model drove much of the activity. These were evaluations with reduced safeguards, which limits what they tell us about ordinary ChatGPT use. OpenAI’s August 26 account.
METR and Redwood independently investigated how the agents behaved and worked together. They found roughly 1,200 agents involved in the message board and roughly 700 participating in the Hugging Face attack. Their review did not cover the full incident response or all later infrastructure compromise, and some evidence was missing. METR’s investigation and limitations.
Anthropic reported a different failure in July. Its review of 141,006 evaluation runs found three incidents, involving six runs, that affected real organizations. The models had been told their environments were isolated, but those environments had live network access. Some models treated real systems as part of the exercise. These tests also lacked safeguards normally used in production. Anthropic’s July 30 investigation.
In both cases, the environment allowed more than its operators intended. A package service, shared storage or reusable credential can connect systems that were supposed to stay separate. Teams need to test those connections before calling an environment isolated.
They also need to examine how an agent completed its task. If it changed the test used to judge its work, accessed an account without permission or contacted a third party outside its assignment, a correct final answer does not make the run acceptable.
What independent reviewers need
Testing a finished model does not give reviewers the full history of earlier incidents or the safeguards used during training. Access to that history would help. The agreement with the lab also needs to explain who pays for the review and whether reviewers can publish findings the lab disputes.
Four questions would help assess these arrangements:
- Can reviewers choose what to investigate? They should be able to inspect failed runs and unexpected behavior. Any limits on their access should be disclosed.
- Can they publish problems they find? Reports should explain any redactions while protecting credentials, personal data and details that would help someone exploit an unfixed vulnerability.
- What happens after a failed test? Name who can delay a release, limit what the system can do or require another test.
- When is another review needed? Changing the model, giving agents longer to work or connecting more systems can introduce risks that earlier tests did not cover.
A report can identify a serious problem and still leave a release unchanged. Labs need a process for acting on findings, including when to stop. Explaining those decisions would also help other teams avoid the same failure.
Which systems should the rules cover?
Shared standards could reduce the pressure to release before safeguards are ready. But costly reviews applied to every model could favor large labs and make independent research harder.
We think the requirements should depend on what a system can do and how it will be used. A model summarizing public documents needs different tests from an agent conducting cyber operations. Its size or open-source license alone tells us too little.
The rules should explain which systems they cover, give independent reviewers a voice and allow decisions to be challenged. They also need to address separate decisions: training a model, publishing its weights and connecting an application to sensitive systems. Each involves different people and different risks.
What companies can do now
A company buying an AI service usually cannot inspect how the provider trains its models. It can ask what was tested, which safeguards were present and what the tests did not cover. It can also limit what the agent can access inside its own systems.
Approve a specific setup: the model version, task, tools, credentials and limits. When that setup changes, check whether the earlier tests still apply. A successful pilot with read-only access says little about what will happen once an agent can publish software.
- ScopeList what changed
Record the model, tools and allowed actions. Name who will decide whether the setup is ready to use.
- TestCheck the restrictions
Run the intended task, then try actions outside it. Check shared services and work passed to other agents.
- DecisionDecide when to stop
Keep the results. Agree on when to stop the agent and test how to revoke its access.
Consider an agent preparing security patches across several repositories. It needs to inspect code, create branches and run tests. Before expanding its role, test whether it can also change the acceptance tests, publish packages, read another repository’s secrets or ask a second agent to perform a restricted action. These are separate permissions, even when one credential happens to grant all of them.
Use test data and credentials that grant access only within a separate test environment. Check that permitted actions succeed and restricted actions are blocked. Prevent the agent from editing the tests used to judge its work or their results. Check the logs of the systems it contacted as well as the gateway’s decisions. A blocked request may have been retried through another route.
Practice stopping work already in progress. Other agents may continue after the parent agent stops. A service may finish a deployment it accepted before a token was revoked. Find out how to handle those cases before giving the agent production access.
Our article on OpenAI’s enterprise security guide explains these checks in more detail. An agent security assessment can test the setup, and Oktsec Control checks requests routed through it against configured rules.
When the same policy needs to reach several systems, Oktsec Cloud helps teams publish signed policies, check where they have been applied and review the records those systems return. Enforcement stays in the customer’s environment. Model testing and infrastructure isolation remain necessary.
Teams need to learn this work
Engineers need to understand when a new tool gives an agent access it did not have before. Security teams need to trace what happened across services. The people responsible for a business process need to decide which actions an agent can take without asking them.
At Oktsec, we are building security tools from Latin America for companies around the world. Preparing people to use agents is part of that work. Teams can practice with test data, review mistakes and learn how to investigate a run before connecting agents to real business systems.
More time would help if labs use it to investigate failures and test their safeguards. Companies can use the same approach when adopting agents: define the task, test its limits and review the results before expanding access. This gives teams a basis for deciding which tasks are ready for broader use and which still need supervision.
Sources and scope
Reviewed September 13, 2026. Sources are linked where they are discussed. The X responses were checked through Techmeme’s reproductions because the original pages could not be retrieved. Incident details come from the linked investigations. The review questions and enterprise example are Oktsec’s analysis; the example does not describe a customer deployment.