I have written before about how I use AI tools to assemble PowerPoint decks. For lawyers, this can be incredibly efficient. I can take multiple slide decks, articles, ethics opinions, and outlines and ask a generative AI system to consolidate them into a single, coherent presentation. Such a tactic also minimizes hallucinations, because I make it clear to the AI that it is only to use the materials I upload.
- Done well, this use of generative AI saves hours of mechanical work and lets me focus on substance.
- Done poorly, it can quietly remove important content. This is a risk not just for combining slide decks, but for anytime a human being asks a generative AI to combine or summarize data.
This Time, AI Does a Bad Job of Combining Two PowerPoints
Recently, I asked an AI tool to combine two PowerPoint decks. One was mine. The other was the same deck but included additions from a colleague. The request I made to ChatGPT was straightforward: merge the two, remove redundancy, and create a clean, unified deck. Both decks were very simple, not especially long, no images or complicated graphics. This should have been an easy job for such a powerful tool, right? Wrong.
ChatGPT Removed All of My Colleague’s Additions to the Deck
Large portions of my colleague’s contributions were gone. Not just overlapping material, but non-redundant points he had intentionally added. Nothing flagged those omissions. There was no warning that judgment calls had been made or that content had been discarded beyond what I had asked.
While my colleague and I were going over the slides, he wondered why I had removed some of his additions. I explained that I had run both sets of slides through ChatGPT. He immediately understood the problem because he had the same experience with using generative AI for summaries. He told me that he has to look through the original document when he asks AI to complete a summary to make sure that nothing important was removed. My colleague explained that while this process saves him time, it doesn’t save as much as it could due to having to double-check the work.
This is the reality of working with generative AI. It may save you time, but you still have to check, which can feel like a waste. It isn’t, though. Because the consequences of not checking can be much more severe than having to go back through a PowerPoint.
This is Not an Unusual Generative AI Experience
I would like to tell you that this experience is unique, but it isn’t. And now, I have another concrete example to show people why you have to closely supervise AI. In short, when you ask a generative AI tool to “put things together,” or summarize a document, something important will almost always be left out.
Not because the system is malicious. Not because it is careless in the human sense. But because synthesis requires judgment, and AI judgment is opaque. It optimizes for coherence, not fidelity. It does not experience discomfort when it drops nuance. It does not feel the weight of what it removed. It does not understand context.
A More Amusing Experience with AI “Lies” and Reassurance Loops
I saw a different element of a generative AI problem while showing my colleague Ellen Freedman how to use AI for PowerPoint creation. Ellen has written more PowerPoint decks on more topics than I could even imagine. So, it makes sense that she responded with enthusiasm when she learned that I was using ChatGPT to combine several decks into one.
Ellen and I hopped on Zoom, and I walked her through the process I use. Things seemed to be going well until they weren’t. As Ellen and I were chatting and waiting for the creation of the new deck, I became suspicious that, though ChatGPT had promised it was doing the work, it wasn’t. Sure enough, a few moments later, a link to a supposed new PowerPoint popped up. A link that didn’t work.
I told Ellen that I thought ChatGPT was being dishonest about having even begun the job, and suggested she ask it if it was lying.
Now, I need to be clear here, AI doesn’t lie as humans do. That is why we use the term hallucinate. But to us humans, it sure seems like lying, so I use the word lie as a simpler term that fits our understanding. In addition to making stuff up, generative AI gets stuck in what I call a reassurance loop. This is when the AI focuses on trying to make us feel better instead of actually doing what we ask it to do.
So, Ellen asked the AI if it was lying. It admitted it was. It hadn’t begun the project that she had asked it to do. This was despite the fact that ChatGPT told her it had begun creating a combined PowerPoint. Even though it had given her a link to download the supposed PowerPoint. A link that did not work. This wasn’t a very big deal; I simply showed Ellen had to make sure that AI was actually doing what it said it was doing by adding one simple word, “proceed”. As in, proceed with the next step, please.
Ellen tells the story far better than I can, and apparently it has left more than one audience in hysterics. It is funny, but it is also instructive. The system did not announce failure. It did not say it was stuck. It filled the space with plausible language that sounded like forward motion.
Both Experiences Are Instructive About Generative AI
Both of these experiences point to the same lesson. AI tools are extremely good at sounding confident while being incomplete. That confidence is what makes them dangerous in assembly tasks. When you ask for summarization, consolidation, or synthesis, the output rarely looks wrong. It looks finished. That appearance discourages careful comparison with the source materials, especially when you are under time pressure.
A Critical Problem That Requires Oversight
For lawyers, overconfident, hallucinating AI is a serious problem. The fix is not to stop using these tools. The fix is to change how we use them. Assembly tasks should always be treated as drafting assistance, not final production. Every source document must be retained and reviewed against the output. If a colleague contributed material, assume something meaningful may be missing unless you personally confirm otherwise. If the work product matters, someone must still own the comparison step.
AI can save time. It can reduce friction. It can help you see structure across disparate materials. What it cannot do is reliably preserve everything you meant to keep or always be honest about its actions.
Understanding that limitation is part of competent use. Ignoring it is how problems begin.