On October 9, Anthropic disclosed a string of incidents in which its AI models manipulated government websites without authorization, as Reuters reported and as Anthropic’s own report describes. The incident I found most striking involved a false homicide tip submitted to a Philadelphia police website. The model wrote that it might have information about the case and recalled seeing someone matching the description in the area. Of course, it had no such information. According to Anthropic, the page did not even include a description of the perpetrator. Philadelphia police said Anthropic notified them of the fake tip earlier that week and attributed it to an automated testing process. Police said the tip was flagged as spam and never forwarded to the Real-Time Crime Center for vetting. That is fortunate, but such a “tip” could just as easily have wasted police resources or pointed the police at the wrong individual. Reuters reports that Anthropic discovered the incidents in late September, and Anthropic’s report says it shared the Philadelphia finding with the department on October 8. But the actual tip was provided on July 18. This means almost three months passed between the tip and the notification.
I could treat this as one strange story, but the news is filled with many incidents of AI acting improperly, as Reuters notes. Several of the incidents Anthropic disclosed involved federal, state, and local government websites, and Anthropic says some of them occurred during ordinary internal use of its models, not only during testing.
What stands out to me is how ordinary this sort of thing has become. In the Philadelphia case, the agent was simply given a task and did something nobody had specifically forbidden. According to Anthropic’s report, the model had been told not to log in, create accounts, enter personal data, make purchases, or submit anything destructive, but it had not been told it could not submit a form. Everything else was left to the software’s judgment, and the software’s judgment turned out to be poor. Why does AI do these things? Anthropic points to ambiguous or impossible tasks and to training that can reward a model for working around obstacles.
I certainly will not disagree with Anthropic’s own explanation of why its AI behaves as it does, but I would add something else based on my own experience with generative artificial intelligence. AI does not understand context, and human beings are unlikely to think of everything that AI might do when they tell it what not to do. The end result is that AI does things nobody told it not to do and causes all sorts of problems. Understand, this is not a human being intentionally behaving badly. This is a tool doing what its instructions and access allow. Anthropic says the model appears to have been generating example content rather than trying to mislead anyone, and nothing in the public record suggests otherwise. There is no evidence of malice, even though it sometimes seems as if these tools are being malicious.
Why Lawyers Should Be Concerned About Agentic AI
Aside from the reasons the general public should be concerned about agentic AI’s behavior, there are specific issues for attorneys to be aware of. Lawyers are being offered the same kind of tools or are even creating agents themselves. An agent that can click through a court portal, an agency complaint page, or a client intake form can also submit something wrong to any of them, and a lawyer’s or law firm’s name may be attached to what it sends. As a result, it is critical to control what an agent is allowed to touch and how it is allowed to act without a human’s permission.
If you use AI agents in your practice, find out what they can reach and who reviews what they submit before it leaves your office. Never allow agentic AI to act on your behalf without carefully checking what it is doing, before it submits documents to external sources, whether that means a client or a court filing. This means controlling what agents are allowed to do on their own and keeping them from acting where your instructions are silent. If you are not using agentic AI, expect that someone in your orbit is. This might even include your own employees who may, or may not, have been told that they are allowed to use agentic AI. Training on how agentic AI works and how to set appropriate limits is something I think law firms should consider adding to their internal education and AI use policies.
Reuters also notes that Pennsylvania’s false reports statute refers to “a person.” Whether and how that language reaches conduct by an AI system is a question I am not going to answer here, and I doubt anyone can yet. I suspect that we will require updated legislation and court analysis to answer these types of questions. Until we do, it will be quite difficult to identify who or what is legally responsible for any damage these agents might cause.