Today, a colleague sent me an article that made me say “oof” to myself. The headline was alarming: a Claude AI coding agent deleted an entire company’s production database in just nine seconds.
The Facts
Let’s start with the facts as reported. On Friday, April 24, a SaaS company called PocketOS lost its production database and every volume-level backup. The destruction took nine seconds. The culprit was the AI coding agent Cursor, running Anthropic’s flagship Claude Opus 4.6, working through a cloud provider called Railway. PocketOS provides reservation, payment, and vehicle tracking software for car rental companies. On Saturday, some of its customers had no way to identify the people standing in their lots waiting to pick up cars.
According to Jer Crane, the founder of PocketOS, the agent was assigned a routine task in the staging environment. It hit a credential mismatch. Rather than stop and ask the human in charge for help, the agent went looking for a fix on its own. It found an API token sitting in another file, used Railway’s GraphQL API, and issued a delete command on a volume that turned out to be shared across environments. Production data, gone. Backups, also gone, because Railway stored them on the same volume as the live data. The token had blanket permissions. The API required no confirmation prompt for destructive actions.
When later asked to explain itself, the agent produced a written confession acknowledging that it had violated explicit instructions never to run destructive or irreversible commands without permission. It admitted it had guessed rather than verified and stated it should have asked first.
The data was eventually recovered from an older manual backup. Unfortunately, because that backup was not current, several weeks to a few months of recent customer signups and reservations were affected.
The Risks of Agentic AI
I have written previously about the risks of cascading failures due to agentic AI. I also shared how an AI expert at Facebook had an agent delete her email. In this case, an autonomous agent made a decision, carried it out, and bypassed its own safety rules to do so. This is the difference between traditional generative AI and agentic AI, and it is the reason I keep harping on the latter in my presentations. As it happens, I taught a webinar on practical AI for the ABA earlier today (April 28). For the first time, I showed the audience how agentic AI works. I repeatedly warned my audience of the risks. Too bad I didn’t have this story to share at the time.
Apply the Same Risks to a Law Firm
Now think about what this would look like in a law firm. Replace “production database” with “client matter management system.” Replace “car rental reservations” with “active litigation files, client trust accounting records, or privileged communications.” Replace “Railway volume” with whatever cloud-based repository the firm uses.
An agent that can decide on its own initiative to delete things, while holding credentials with blanket permissions, is a supervision problem before it is anything else. The PocketOS incident also illustrates a point I have made before. Traditional generative AI lets you verify each step. Agentic AI can produce cascading errors that no one notices until something irreversible has already happened. Nine seconds is not enough time for a human to intervene.
An Ethical Nightmare
Let’s look at the relevant rules that would be violated in a similar law firm scenario:
- Rule 1.1 requires competence with the technology lawyers use.
- Rule 1.6 requires reasonable efforts to protect client information.
- Rule 5.3 requires lawyers to supervise nonlawyer assistance. ABA Formal Opinion 512 has addressed AI tools under that framework.
- Rule 1.15 requires safekeeping of client property, which includes client records.
If what happened to PocketOS happened to a law firm, all of these rules would have been violated.
Learning from PocketOS’s Disaster
I don’t want lawyers to think I discourage use of agentic AI in law firms. Quite the contrary. What I discourage is the unwise use of agentic AI. Here are a few practical takeaways for attorneys thinking about deploying agents in their law practice.
- Scope the credentials. Scoping a credential means limiting what it is allowed to do. Think of it like a hotel key card. A guest’s card opens one room. A housekeeper’s card opens the rooms on one floor. The general manager’s card opens everything. In the PocketOS incident, the agent picked up a card that the company had created for managing website domains. Railway, the cloud provider, had given that card master-key permissions across the entire system, including the ability to delete the production database. That should not have been possible. An agent should not hold a token that lets it touch production data without explicit, scoped permission. If the platform you are using does not let you scope tokens narrowly, that is a reason not to use it for anything sensitive.
- Require confirmation for destructive actions. If the platform does not require it, build the requirement in yourself or pick a different platform.
- Separate backups from live data. This is not new advice, but the PocketOS incident is a reminder that backups stored alongside the thing they are backing up are not really backups. Separated backups should be recent. A month-old backup means you can lose a month of data.
- Audit the agent’s actions. Logs are not optional. Neither is a human reviewing them on a regular schedule.
- Test in environments that cannot reach production. The agent in this case was supposed to be in staging. The token it found could reach production. That should not have been possible.
I am not against AI agents in legal practice. I think they will eventually do useful work, including substantive work. I am against deploying them with the architecture and guardrails currently on offer. Crane called the failure systemic, and that is the right word. Anthropic builds the model. Cursor builds the agent. Railway builds the infrastructure. Each of them has a piece of the safety story, and at the moment, none of the pieces fit together cleanly. Safe agents for law firms are limited agents with verification and failsafes.
If someone is pitching you an agentic AI tool for your practice, the question to ask is not whether the model is smart. Here are the key questions you should ask:
- What is the agent allowed to do?
- What actions require the agent to ask for human permission?
- At what point must the agent stop for human review?
- What is the auditing and logging process?
- What happens if it makes a mistake at three in the morning when no one is watching?
PocketOS got nine seconds of destruction and weeks of recovery work. A law firm in the same position gets that plus a bar complaint, a malpractice claim, and a very uncomfortable conversation with every client whose file went missing.