There were a lot of great questions during today’s webinar. One hour isn’t very long so we didn’t have enough time to provide detailed answers. Hans will provide answers to his questions on his blog at: https://www.deliberately.ai/blog
Here are the answers to my questions.
1. When conducting legal research for cases, which do you prefer? Westlaw or LexisNexis?
Lexis gave me access to their tools for free, so I use Lexis. I have reached out to Westlaw so I can give them the same opportunity to be shown in my courses, but they have not responded. I also use Claude and Perplexity for legal research. If I am using any AI tool, including Lexis’ AI, I confirm the cases through a non-AI tool.
2. Will you address when the use of AI could make matters otherwise privileged discoverable and how to prevent that?
3. Any instructions to clients about what they should/should not use tools like ChatGPT to protect against discoverability?
I will address these questions together. There are a number of cases across the United States that address the issue of AI chats and privilege/work product. The first two cases are Heppner and Warner. Heppner found that there is no reasonable expectation of confidentiality in an AI chat tool based upon a reading of the terms of service of the AI tool Heppner was using.
What the judge left open is the question of if the attorney had instructed the client to use AI, that would have created protection. There is not yet a case that answers this question. For pro se cases, we have only a few decisions, but in those cases the courts have found that using AI in anticipation of litigation is protected under the work product doctrine. There is a recent case out of Texas in which a judge found, for the first time, that a represented party’s use of AI is protected under work product, to a certain extent.
At this point, my recommendation is to warn clients not to use AI, not only because it can be wrong, but because their chats may well be discoverable. If you want to take the risk, I suggest in such a case that you document that you specifically told the client to use AI, but still, warn the client, in writing, that their chats may be discoverable.
I further recommend providing a written document with the engagement agreement (but separate from that agreement) that discourages your clients from using AI due to its potential ability to be discovered. Here is a sample clause you may feel free to modify for your own use.
Client Use of AI Tools: Confidentiality and Discoverability Risks. You should understand that information you enter into an AI tool (including ChatGPT, Gemini, Copilot, and similar) is not (or may not be) protected by attorney-client privilege or the work-product doctrine. Depending on the tool’s terms of service, your inputs may be stored, reviewed by the provider, used to train AI models, or produced in response to legal process. Communications, drafts, and notes you create with AI assistance may be discoverable in litigation to the same extent as any other document you create. We strongly advise against entering information about your legal matter into any AI tool that has not been vetted and approved by this firm. If you have used AI tools in connection with your matter before engaging us, please tell us so we can assess any resulting issues.
4. What is the 5th D as compared to the 4D model?
The four Ds are Dull, Dirty, Dangerous, and Dear. Dull means repetitive and monotonous. Dirty means unpleasant or unhygienic. Dangerous means physically risky. Dear means expensive or too costly to get wrong. There is not an agreed upon fifth D, but there are options. Decision-making/decision-autonomy, which is the most common. I have also seen difficult, demeaning, and demanding. The first four Ds describe work nobody minds handing off. A fifth D forces a more difficult conversation, especially if we settle on decision-making. In the end, the final decision must always rest with the attorney under our ethical obligations.
5. Interested in setting CLAUDE up at firm to do quality control and research. My server is based in the cloud. What are best practices in set up to secure privacy? In other words, how is it possible to set up Claude “as closed loop?” to clarify, my firm practices administrative law and is not a litigation firm.
First, a true closed loop, meaning an air-gapped model running entirely inside your own infrastructure with nothing leaving your walls, is not what Claude is. Claude is a hosted service. So, what you are asking for is not possible with Claude. What you can get is functionally close for confidentiality purposes, that is a setup where client data is not used to train the model, is not retained beyond a short and defined window, and is governed by a contract that makes the firm the data controller. That’s a realistic target, and for most administrative practices it’s enough.
Begin by choosing the correct account type for your needs. Stay away from the consumer plans (Free, Pro, and Max) which all default to training and do not give you the level of confidentiality you require. These can retain training data for up to five years unless you opt out. These terms do not apply to Claude for Work, which includes Team and Enterprise, or API, Amazon Bedrock or Google Vertex. Under these commercial accounts, Claude does not use the data you share to train its models, and the customer organization is the controller of the data while Anthropic acts as a processor that only handles the data to provide the service. You will want to read the terms of service of any tool you use in the chain to make certain of the terms of service and privacy policies as far as retaining and using data. Also, keep in mind that the Fable model keeps data for 30 days, even if you are on a commercial account.
For a greater level of protection, you want Zero data retention (ZDR), meaning Anthropic does not store inputs or outputs after the response, beyond what is needed for legal compliance and abuse prevention. This is available subject to approval by Anthropic. You will want a data processing addendum plus the usual enterprise controls, which include SSO, audit logs, and user provisioning, which give your firm the administrative oversight you need.
Since you practice administrative law, if you are dealing with medical records that are PHI, you want to go beyond the confidentiality terms. You want to make certain you have the Business Associate Agreement, which Anthropic does offer, but only under the first-party API and enterprise plan. The BAA comes with configuration requirements and certain feature limits. If you do not handle PHI, you do not need to worry about this.
Remember your ethical obligations under the jurisdiction(s) in which you practice. Model rules 1.6 for confidentiality, 5.3 on supervising non-lawyer assistance, and of course 1.1 which includes technological competence. I would recommend that you have your clients specifically consent via your engagement agreement and check your jurisdiction’s specific requirements.
Please also remember that none of this fixes the hallucinations, incorrect law, and disobedience issues that come with generative AI.
6. Other than reading the case, what other non AI tools are there to verify that the case exists and supports the position taken
There is really no tool that can take the place of reading a case to make sure it stands for the position taken. However, there are some tools that can help. First, as far as checking the citation alone, you can use any traditional tool such as Lexis or Westlaw. There are also tools you can purchase such as Benchly which will check the citations in your documents. You can also use Google Scholar, which is free, but does not have all cases.
As far as checking the content, your best bet is something like Shepard’s, KeyCite, BCite, or Authority Check. These can help somewhat for verification of content, but only to a certain extent. You can also look at Headnotes or digests, but those can be inaccurate.
I should mention that many of these tools incorporate AI. The actual treatment flag is a citator and it is not AI. As a result, you can trust it for whether a case exists and whether it is still good law (subject to ordinary editorial error). However, any AI answers or summaries still have the risk of hallucination, which includes citing a real case for something it doesn’t hold.
Given this reality, there is no solution I can suggest that replaces reading the case.
7. When you put the client letter in ChatGPT to summarize, did you use an enterprise version to protect the confidentiality of information?
For purposes of this course, no I did not use an enterprise tool nor did I anonymize. The reason for this is that when I present in courses such as this one, I do not use real clients. On occasion, I will include a sample that includes ethics opinions I have written, but the opinions are public and any confidential data is redacted.
8. Do you ever use GPTs or Claude Projects for these kinds of tasks/prompts?
Yes, whenever I show an example of how I use AI, it is something I have used in some way for work. Not necessarily the actual sample, since I do not like to use real data, but the underlying concept such as using AI to respond to a letter, adjust tone, perform legal research, or draft a document. Every caution and workflow I recommend, I follow in my own work. Currently, the generative AIs I use most frequently include Lexis, Claude, and Copilot. But I use others as well.
9. Can you discuss use of free AI (ChatGPT, Claude, etc) vs a paid version, such as pro (but not enterprise)?
The biggest consideration when choosing which type of account is not paid versus free, but the actual privacy policies and terms of service for the specific type of account. Otherwise, when you use free versions of these tools, you will find that they are limited in some way. Free versions often only allow a minimal number of chats.
Claude’s Pro and Max, which are paid, are still consumer level tools. As such, they are inappropriate for entering confidential data. The first level that I would consider using for confidential data is Team. That said, I still would not enter confidential data in such a tool. I suspect that the reason I feel this way is because I am an ethics lawyer. However, there are many lawyers I respect who are comfortable entering confidential data in Team. My suggestion is that you read the terms of service and privacy policy and decide what you think is reasonable under the rules. The reality is that for many law firms, Enterprise is simply cost prohibitive. Claude is aware of this concern and is looking at it, so I am hopeful Enterprise may become more affordable in the future.
10. Thank you, Jennifer, for this excellent presentation on how to effectively use AI. Although, I use primary sources and read the entire case for my students to make sure its factual information. As a Program Chair and Lead Paralegal Instructor, how can I incorporate this into my class?
You are very welcome. Thank you for attending. As I mentioned, if you would like to talk this through, please feel free to reach out to me.
The habit you already model, going to primary sources and reading the whole case, is exactly the skill your students need most in an AI world. AI gives you a live, undeniable reason to teach it due to the number of lawyers getting in trouble. There are quite a few bench slap videos you can show your students to help them understand what happens when hallucinated citations are included in documents filed in court.
Always read the full case can sound to students like an old-fashioned rule. Now you can show them why it is so important to read the original document. I would use this reality to sharpen the verification skills you already teach. You might also explore with your students the fact that Gemini has developed a loop in which it insists that in Pennsylvania, lawyers are mandated to disclose AI use in court filings, when they are not. I describe the situation in this blog post.
Here are some additional recommendations:
- Run a hallucination exercise. Give students a research question, have them get an answer with citations from an AI tool, then require them to verify each citation two ways: confirm the case exists and is still good law using a citator, and read the actual opinion to confirm it supports the stated proposition. Have them write up what they found. Some citations will be fine. Some will be fabricated. The most instructive ones will be real cases cited for something they don’t actually hold. This is very common in tools such as Lexis and Westlaw.
- Teach the error types, which include hallucinations, failure to follow the proposition stated, and disobedience explicitly, because they connect directly to paralegal cite-checking work. A nonexistent case is the easy catch; it won’t pull up. A real case that has been mischaracterized, or has been overruled, is the more dangerous issue. The only way to catch this kind of error is to read the case. That is the paralegal’s job, and AI raises the stakes on it rather than removing it. Disobedience can be both easy and challenging to catch. My simplest example, and one you might use, is that my standing orders tell AI not to use the em dash. In an effort to get around this, Claude started using two dashes, so I told it not to use those nor to try to get around my instructions. It still frequently uses em dashes. The reason for this is the amount of training it had on em dashes. It is difficult for it to get over its training.
- Require a verification log on assignments. Let students use AI, but make them submit a log showing every proposition and citation and how they confirmed it against the primary source. You reward the verification, not the drafting. That trains the exact professional habit. This is a habit I recommend to anyone who uses AI for legal research. It is a necessary step for potential malpractice or if called out by a judge for including a mistaken citation. Mistaken citations, of course, existed long before AI, it is just that now the assumption will be that the person used AI regardless of whether they did.
- Cover confidentiality and tool choice. Paralegals often do the actual data entry, so they need to know not to paste client information into a personal or consumer AI account, and to understand which firm-approved tools are safe. That’s a competence and confidentiality point they will need to use immediately. You might also have them review the privacy policies and terms of service. They can use AI to do the review (a different AI from the one they are asking about since the tools can be biased) and then compare the results to their reading of the policies and terms.
- Use the cautionary cases as discussion. There are now multiple sanctions decisions where lawyers filed briefs with AI-invented citations. Those make the risk concrete and give students a professional-consequences frame.
- Since you’re the Program Chair, you can do more than one class. You can write “verify AI output against primary authority” into a program-level competency, so it shows up across courses and in assessment, not just once.
- The fundamental thing I would remind them is that AI doesn’t change what a good paralegal does. It makes careful cite-checking and primary-source reading more valuable, not less. You’re not adding a new skill to teach. You’re giving the old one teeth.
11. It may be worth noting Google incorporated AI into their general search, so it is important to make sure the AI mode is turned off when using Google to verify the case outside of an AI.
This is a very good warning for people to keep in mind. If you use Google, you will want to look into what it is doing with your data. Also keep in mind that different AIs constantly change their terms of service and privacy policies. It is critical to stay on top of these terms of service. There are a number of places where you can find help on how to turn off Google’s training. Here is one example. If you use Google, you’re training its AI. Here’s how to opt out. | TechCrunch
Hans was specifically asked these two questions. You may find the answers on his Blog.
1. For Hans, how do we confirm that the client intelligence program has a data shield and is not sharing PII to an LLM, and that the AI learning on data is keeping client data separate from other client data contained in the entire database?
2. Hello Hans, does this keep track of process of service? jury selection?