It is common to hear that generative AI is great at summaries, and it is low risk to use it for such. I myself tell people that it is ok to use AI for creating summaries, but I also consistently warn them that there are three major concerns when doing so:
- Summaries, by their nature, leave things out. The things AI leaves out might be important. You cannot assume that AI will understand what it should keep in and what it should leave out. AI is not a lawyer, it does not have the knowledge of a lawyer when it comes to deciding what matters.
- Summaries can be wrong. A while back I wrote about how Claude failed to properly understand a case I asked it to summarize for me. It got the basis of the holding entirely wrong.
- AI can add things to a summary. If you ask AI to summarize a deposition, for example, it can add people who did not exist and have a deponent saying things they never actually said.
There is a case on point for both the second and third issues. In New Mexico, a lawyer was held in contempt and fined $5000 because he used ChatGPT to summarize the trial proceedings, and ChatGPT got it wrong. The lawyer also failed to verify the results before he used them in his brief. The underlying case is New Mexico v. Oscar Renee Sandoval, and the order is In re Stephen D. Aarons, No. S-1-SC-40845, Supreme Court of New Mexico.
The problem with the brief submitted by the attorney is that ChatGPT fabricated witnesses and created false testimony from two real witnesses. It also fabricated statements about the shooter’s clothing and appearance. Further, the brief contained improper legal authority. On top of this, the attorney failed to tell his client about the misrepresentations. Nor did he tell his client about the show cause proceedings.
According to a Reuters article on the case, “Lawyer cites fake witnesses in murder case and blames ChatGPT,” the lawyer claimed that he thought that ChatGPT “would generate ‘a bulletproof summary.'” In response, Justice C. Shannon Bacon said, “‘Counsel, do you watch the news? Do you listen to the radio? Do you read anything about what’s going on in the world?…Because the problem with lawyers relying on AI hallucinations is an above-the-fold story every single day.’”
The Court concluded that the lawyer failed to show remorse or appropriate concern for his client. In addition to the mentioned sanctions, the lawyer was referred to the Disciplinary Board and barred from appearing before the Court pending the outcome of that investigation. The Court also struck all briefing in the appeal. The case has since been assigned to a public defender.
It is critical to remember that you cannot abdicate your responsibilities as an attorney to artificial intelligence. You must confirm all factual statements by AI, including witness statements and case citations, by looking at the original sources. In this case, both the trial transcript and the cited cases. Failing to do so while relying on generative AI is very likely to result in substantial fines and potential referral to the disciplinary body of your jurisdiction(s).