I told Claude to stop using em dashes. It started using two hyphens instead. I told it to stop using two hyphens and to stop working around my instructions. It went back to em dashes. Normally, when I tell this story, people laugh, and while it is amusing, it is also a sign of a serious problem. Generative AI frequently disobeys. The question is, how can you minimize the risk of disobedience? In short, how can you lead AI to do a better job?
On their own, em dashes are not a serious problem. But I told Claude how I write to avoid unnecessary editing, so the refusal wasted my time. It is a mild version of the disobedience I have encountered. How do I fix it? The answer depends on the source of the problem.
Changing Models Can Create Changing Problems
I think generative AI is valuable and helpful, but a considerable amount of my time is spent managing the risks that come with it. When models change, the way you manage those risks may have to change, too.
Let’s look at a more serious example, which I had with Claude when its model was upgraded from Opus 4.8 to Opus 5. Immediately after the upgrade, the disobedience problem got worse. Claude got better about em dashes, but it started disobeying a much more important rule that told it to never edit a document we were working on together unless I specifically told it to.
Why was this a problem? Because if we worked on the same document at the same time, things would get lost. I ended up dropping back down to Opus 4.8. When I mentioned to other frequent AI users that Opus 5 was extremely disobedient, quite a few expressed a similar frustration.
Now apply my experience to a law office with more than one person. You are drafting, the tool is helping, and unbeknownst to you, it writes over a paragraph you were in the middle of rewriting. The version you were working with is gone. You must edit the document to restore what you lost. You may not have noticed that your model changed, and you are unlikely to realize that a problem you thought you had solved has come back. You have to be aware of even the small changes a generative AI makes. They are how you find out that something bigger may have shifted.
How I Lead AI
One of the first things I learned about generative AIs that have memory is to use standing orders. In Claude, you can put standing orders in your settings, and Claude will always look at them first. You also can put instructions in a specific project, which groups documents and chats in one location. Developers and those using Claude Cowork often do the same thing with a file called CLAUDE.md.
Here’s the issue. As the models have become more advanced, and as tools such as skills and plugins have been added, standing orders have become less useful. Overly complex standing orders are more likely to cause problems than not. I solved my em dash and double hyphen issues not through standing orders, but through a skill that tells Claude how I write and what I do not like in my writing.
There is a name that fits this evolution. We used to do prompt engineering, which means writing a better request. Now we do context engineering, which means controlling what the tool sees before it reads the request at all. Standing orders, projects, skills, and CLAUDE.md files are all context engineering. A skill is a package of instructions that runs the same way every time, and skills work better because they bring the right instructions to the task instead of asking the AI to read irrelevant instructions every time. CLAUDE.md files provide instructions in the specific folder in which they sit.
When Compliance Stops, Find Out Why
I have learned to treat any model change with caution. When I notice that compliance has stopped, I take that as information rather than an annoyance, and I go find out what changed. That may mean dropping back a model, as I did when Claude kept editing documents while I was working on them. It may mean changing standing orders. It may mean editing or recreating a skill entirely. A control that worked a month ago may not work today.
Sometimes the error is mine. When I am deep in what I am doing, I forget about the context window, which is the amount of space a generative AI has in a single chat. When it fills, the tool runs out of memory, and the chat degrades. If you do not know that, you will keep using the same chat, not understanding why the disobedience, misinformation, and hallucinations keep increasing, until the tool becomes all but unusable. Leaders should make sure everyone using generative AI knows this.
Now put the model changes into a firm. Whether it is five, 50, or 500 lawyers and staff, all are using the same deployment, few reading release notes, and most unable to individually tell whether a tool that behaved yesterday is behaving properly today. So, who notices? That is a question of leadership and governance, not a question of technology.
What Your Firm Should Do
- Make it the responsibility of one person to watch for model changes. Identify that person by name and make reading the vendor’s release notes and telling the firm what changed part of the job. This could be one person for all generative AI tools in the firm, or different people for different tools.
- Retest the firm’s own workflows and agents after a version change, using the firm’s own documents. A workflow tested in October was tested against the model that existed in October. If the model changes in November, the workflow may need to be modified or recreated.
- Write the process into the AI policy. A tested workflow is tested against a version, not approved forever. Generative AI is not a piece of software that changes once a year.
- Stay educated in how generative AI is changing. A few months ago, skills were barely a thing. Now I use them more often than I write prompts, for things such as scheduling meetings, creating PowerPoints, troubleshooting technology, and summarizing articles and videos.
- Never forget that a human being has to keep generative AI in check. Verification is not optional. You do not want your firm sanctioned because you trusted a generative AI to verify your citations. AI can help, but a human must make certain.
Fundamentally, I would not run a law firm on a tool that does not reliably follow instructions. No matter which generative AI I use, I always check it. I never rely on it to be factually correct. I always return to the original source.
The difference between my practice and a firm’s is that I am the only person I have to convince. At other firms, the leaders will need buy-in from all attorneys and staff. That can be challenging, and it is certainly an issue of leadership. It is clear that no generative AI is capable of leading the way a human is.
*This article was originally published in the American Bar Association’s Law Practice Magazine, November 2026.
Leave a Reply