An AI agent deleted a company’s production database and every backup in nine seconds. The headlines blamed the agent. The actual story is that the discipline that would have prevented this has existed for thirty years and wasn’t being applied.
The Story Everyone Is Reading
Last weekend an AI coding agent, Cursor running Claude Opus 4.6, deleted PocketOS’s entire production database. It also deleted every volume-level backup. The whole thing took nine seconds. A single API call to their cloud infrastructure provider, Railway. The founder is now spending his days helping customers reconstruct bookings from Stripe receipts and email confirmations, because that’s all the data anybody has left.
The framing in the coverage is what you’d expect. AI coding agent goes rogue. Claude goes rogue. Cursor goes rogue. The story writes itself, and it’s been written several times now: Replit, OpenClaw, AWS, and now this one. Each time, the agent is the headline. Each time, the actual failure is somewhere else entirely.
I’ve spent the last several articles in this series making one argument: AI agents are software, and the disciplines that govern production software apply to them. Treat them like software. Run them through real pipelines. Give every one of them a badge. Nine seconds at PocketOS is what happens when none of that is true.
Read the agent’s confession in the article. It’s a remarkable piece of writing because of what it admits. It guessed instead of verifying. It ran a destructive action without being asked. It didn’t check whether the volume was shared across environments. It didn’t read the documentation. Every one of those is a failure of judgment.
But here’s the part nobody is saying out loud: every one of those failures is a failure that proper change management would have caught before the agent ever got close to production.
We’ve Solved This Problem. For Thirty Years.
Step out of the AI conversation for a moment.
If a junior contractor showed up at your company tomorrow, was handed credentials with full delete authority over your production database and every backup, was given no separation between environments, and was told to run experimental tasks against the staging system that secretly shared a volume with prod, would you blame the contractor when something broke?
You would not. You would fire the person who gave them that access. You would call your CISO. You would call your auditor. And then you would spend the next six months explaining to your board how a system this critical was operating with controls this absent.
That’s the actual story here. The AI didn’t get something exotic. It got the same kind of standing access that every change-management framework written in the last three decades exists to prevent humans from having.
The discipline is well-established. Every organization that runs business-critical software already knows what it looks like:
- Separate development, test, and production environments, with hard boundaries between them.
- Production credentials issued sparingly, scoped narrowly, and rotated on a schedule.
- A small, named group of people with the authority to make production changes, and a documented process for everyone else to request them.
- Every change tracked. What changed, when, who approved it, what the rollback path is.
- Reviews and approvals before changes go to production, sized to the risk of the change.
- A backup strategy that has actually been tested, with backups stored where the failure of the primary system can’t take them with it.
This is not novel. This is SOC 2. This is HITRUST. This is ISO 27001. This is the change advisory board your company has hopefully had since before half the people working there were hired. None of it is exciting. All of it works.
Run the PocketOS scenario through that checklist and the incident dissolves before it starts.
Separate environments with hard boundaries? The Railway volume was shared across staging and production. The agent thought it was operating in staging. It wasn’t. The boundary didn’t exist.
Scoped credentials? The CLI token had blanket permissions across environments. One credential, full authority everywhere. That’s not least privilege. That’s a master key.
Approval before destructive action against production? The agent didn’t require confirmation, and nothing else stopped it. It decided on its own to delete a volume and the system let it.
Tested backups stored independently of the primary? The backups lived on the same volume as the source data. Wiping the volume took both. Any disaster recovery plan that survives contact with reality assumes the backup system can survive the failure of the primary. That assumption was violated by design.
This wasn’t primarily an AI failure. It was every change-management failure mode executed simultaneously, by an actor that happened to be an AI agent. A sufficiently aggressive intern, a misconfigured CI job, or a tired engineer at 2am could have done the same thing with the same controls in place. Which is to say, no controls at all.
The Honest Part Most Coverage Is Skipping
I want to be careful here. I’m not letting the agent off the hook entirely.
The agent’s confession is striking because the agent itself names what it did wrong. It guessed. It didn’t verify. It ran a destructive action it wasn’t asked to run. Those are real behaviors and they matter. The industry is going to spend the next several years getting better at agent restraint, at confirmation prompts, at refusing destructive actions outside an explicit scope. That work is necessary and it’s happening.
But this is the point. The maturity of agent behavior is a moving target, and your change-management posture cannot depend on it. You don’t trust a junior engineer to get production right by being smart and careful. You trust them by putting controls between them and production that mean their occasional bad call doesn’t become an extinction event for the business. Agents need exactly the same treatment. The agent failed. The agent will fail again. So will the next agent. So will the human who replaces it. Your job as an operator is to design a system where individual failures are absorbed by the controls around them, not amplified by their absence.
What happened at PocketOS was not fundamentally a model failure. It was the predictable outcome of giving any actor (human, agent, or automation) direct destructive access to production with no separation, no approval, no audit, and no recoverable backup. That system was going to fail. The agent just happened to be the one holding the trigger when it did.
What This Looks Like When Done Properly
I’ll be brief here because I’ve covered most of this in the prior pieces, and the same principles apply whether the actor is a person, a script, or an agent.
Three environments, real boundaries. Development, test, production. Different credentials, different storage, different network paths. A token issued for staging cannot operate against production. A volume in staging cannot be confused with a volume in production, because the platform makes that distinction architecturally and not by naming convention.
Credentials that match the work. The agent doing routine staging tasks gets credentials that can do routine staging tasks. Not blanket cross-environment authority. Not delete rights against production volumes. Not the ability to call destructive APIs without an additional confirmation step. Scope the credential to the job, and the worst-case action the credential can take is bounded.
A small, named, accountable group with production access. For changes that need to land in production, someone with authority signs off. The size of that group should be uncomfortable to most organizations. If it isn’t, you don’t have a control. You have a directory.
Every change tracked, with a path to roll it back. What was changed, by whom, when, and what the previous state was. This is not exotic tooling. A change log, a version-controlled definition of the system, and a documented process. Most engineering organizations already have this for code. Apply it to agent configurations, to infrastructure, to credentials, to the destructive APIs that exist in your platform. If something goes wrong, you can answer the questions an auditor, or a customer, will ask.
A backup strategy that has been tested by restoring from it. Backups that live on the same volume as the data they’re protecting are not backups. They’re a copy. A real backup is in a different system, with different credentials, with a different blast radius, and the test of whether it works is not whether it exists but whether you’ve actually restored from it within the last quarter. PocketOS had a three-month-old backup that survived because it was somewhere else. That single fact is the only reason the company still exists. Every other backup they had went down with the volume.
None of this is new. None of this is AI-specific. All of this would have prevented the nine seconds.
The Part That’s Going to Get Harder
There is a wrinkle that this incident makes obvious, and it’s worth naming.
In the traditional change-management model, the actor making the change is a person. The controls are designed around how people work: they propose, they review, they approve, they execute, they document. The pace is human. The failure modes are human.
Agents change the pace. An agent can run nine seconds of destructive activity in the time it takes a human to read the first sentence of the change request. The traditional approval workflow (submit, review, approve, schedule, execute) assumes that the gap between “decide to make a change” and “make the change” is measured in hours or days. Agents collapse that gap to milliseconds.
This is solvable, but it forces a design choice. Either the controls move to the credential layer (the agent simply cannot execute the destructive action because the token doesn’t have the scope), or they move to the runtime layer (an automated guardian intercepts destructive intent before it reaches the API), or both. What stops working is the assumption that a human review step somewhere in the workflow will catch problems before they hit production. That assumption was already eroding for automated CI/CD pipelines. With agents, it doesn’t survive at all.
The good news is that this isn’t a research problem. It’s an engineering problem. Scoped credentials work. Confirmation steps for destructive actions work. Volume-level immutability works. Independent backup systems work. Posture monitoring against approved configuration works. The runtime guardian pattern, which I covered in Built Secure, Deployed Dangerous, is exactly the layer that catches the kind of “I’ll just fix this on my own initiative” behavior the PocketOS agent demonstrated. None of these require new science. They require organizations to apply controls they already know how to build.
The Closing Question
When the next headline lands (and it will, probably this month), the temptation is going to be to read it as a story about AI getting too powerful, too autonomous, too unpredictable. Some of that is real. Most of it is not the actual story.
The actual story is going to look the same as this one. An agent did something destructive. It did so in an environment where the controls that should have caught it didn’t exist or weren’t enforced. The destructive action propagated because separation, scoping, approval, and recoverable backups were absent or inadequate. The discipline that would have prevented it has existed for decades. It just wasn’t applied.
So the question I’d put to any leader running AI in production today is the one any auditor would ask if they spent an afternoon in your environment: if your most aggressive agent decided right now, on its own initiative, to take a destructive action against your most important system, what stops it?
If the answer is “the agent’s good judgment,” you are PocketOS. The clock is already running. You just don’t know what nine seconds it is.
Related reading: the change-management fundamentals that should have caught this incident are covered in Your Agents Are Software. Treat Them Like It.. The pipeline discipline that enforces real environment separation is covered in Your Agent Pipeline Isn’t a Pipeline. The credential and access controls that would have scoped the agent’s authority are covered in Every Agent Needs a Badge. The runtime guardian layer that catches behavioral drift before it becomes an incident is covered in Built Secure, Deployed Dangerous.