Key Takeaways
- AI hallucinations arise because large‑language models generate text by predicting the next word, not by verifying factual accuracy.
- Heavy workloads and the allure of rapid, polished output tempt attorneys to skip verification, leading to fabricated case citations in filings.
- Even experienced lawyers can be caught off guard when they lack a systematic process to check AI‑generated content.
- Human verification remains the most reliable cure; firms that institute clear AI‑use policies and verification checkpoints save time overall.
- Courts cannot be counted on as an infallible backstop; judges and their staff are also susceptible to AI‑produced errors.
- Small businesses and non‑lawyer users face heightened risk because they lack access to authoritative legal databases and attorney‑client privilege protections.
- Emerging local court rules and potential model ethical guidelines aim to require disclosure of AI use and mandatory verification of AI‑sourced law.
- The ultimate responsibility for accuracy rests with the submitting attorney under the duty of candor to the tribunal and rules such as FRCP Rule 11.
Introduction: The Rise of AI‑Generated Fake Case Law
In recent months, federal judges have begun criticizing Justice Department attorneys for citing cases that never existed—citations that appear to have been produced by artificial intelligence. Terry Gerton opens the discussion by asking why this phenomenon is surfacing more frequently, noting that a judge’s rebuke highlighted a filing that relied on a non‑existent precedent. Crawford Appleby responds that the problem is not isolated to government lawyers; it can affect any attorney, from large‑firm partners to solo practitioners, because AI tools deliver confident‑sounding answers that look authoritative at first glance.
Why AI Produces Hallucinations: Large Language Models and Guesswork
Appleby explains that the underlying technology behind most AI assistants is a large language model (LLM). “What that means is they are AI that was trained on an immense amount of data that was created by human beings, text and images and everything else,” he says. LLMs are designed to predict the next word in a sentence as a human would write it, not to retrieve verifiable facts. When the model encounters a gap in its training data, it “will try to invent that answer because it’s trying to guess what the next words would be.” Consequently, it can fabricate case names, citations, or legal principles that sound plausible but are entirely invented.
Pressure on Lawyers and the Temptation to Cut Corners
The conversation then turns to the practical pressures that drive attorneys to rely on AI without scrutiny. Gerton notes that lawyers are under constant pressure to produce high‑quality work quickly. Appleby agrees, observing that “AI quickly provides confident sounding, polished answers to even the most complex legal questions.” The speed advantage creates a strong presumption to cut corners: if a tool can draft a paragraph in a couple of minutes that would take a lawyer thirty minutes to write from scratch, the incentive to accept the output without verification grows.
Why Experienced Attorneys Still Fall Victim
Even seasoned lawyers are not immune. Appleby points out that many attorneys lack formal systems to catch AI‑generated errors. “They’re not prepared and I think part of it also is a failure to understand what the technology is,” he says. For those unfamiliar with LLMs, the technology can appear almost magical, leading to misplaced trust. Without a clear understanding that the model is essentially guessing, experienced counsel may assume the output is accurate and skip the verification step.
Human Verification as the Essential Safeguard
When asked about solutions, Appleby stresses that “the cure for AI hallucinations is human verification.” He advises that any attorney using AI must verify every piece of output—facts, law, citations—before incorporating it into a legal document. The verification step need not be prohibitive; if a lawyer already knows the relevant law, checking AI‑generated text may take only five to ten minutes, preserving much of the time saved by the initial AI draft.
Practical Verification Protocols and Firm Policies
Appleby shares that his firm, Wisner Baum, has instituted concrete safeguards: approved AI tools, firm‑wide accounts, and an AI policy that all lawyers know. He likens the verification habit to a “long game” where the goal is to shave minutes off an hour, hours off a day, and weeks off a month. By embedding verification into the workflow, firms can reap the efficiency benefits of AI while mitigating the risk of hallucinated citations.
The Role of Courts as a Backstop
Gerton questions whether courts can serve as a reliable safety net, assuming judges will catch erroneous citations. Appleby tempers this optimism, noting that “we all hope so, that the courts would be held to a higher standard than the attorneys,” but adds that judges and their staff are also susceptible to mistakes, including AI hallucinations. Relying solely on judicial review is therefore insufficient; attorneys must assume primary responsibility for accuracy.
Implications for Small Businesses and Non‑Lawyer Users
The discussion highlights particular vulnerability among small businesses that lack in‑house counsel. Appleby warns strongly against using AI for any legal‑related tasks by non‑lawyers. He offers two reasons: first, AI may deliver “very polished, confident answers that are 100% wrong,” and without access to databases like Westlaw or LexisNexis, users have no way to detect the error. Second, conversations with AI are not protected by attorney‑client privilege; courts have already ruled that data shared with a chatbot can be disclosed, exposing sensitive business information.
Emerging Judicial and Ethical Responses
Looking ahead, Appleby anticipates that the legal profession will develop new standards. He expects “model ethical rules” addressing AI use to appear within the next year or two, building on the existing duty of competency. Already, some judges are issuing local rules that require attorneys to disclose whether AI was used in drafting documents and to confirm that they verified the AI‑sourced content. While a few courts might consider banning AI outright, Appleby argues that prohibitions would merely drive the practice underground, eliminating oversight. Effective regulation, therefore, hinges on supervision and clear verification requirements.
Conclusion: Responsibility Lies with the Submitting Attorney
The dialogue concludes with a reaffirmation of the attorney’s duty of candor to the tribunal. Appleby reminds listeners that attorneys are bound by ethical rules that prohibit making up facts or law for the court, and that violations can trigger sanctions under rules such as FRCP Rule 11. Ultimately, the responsibility for ensuring that filings contain accurate, verifiable legal authority rests with the submitting lawyer, not with the AI tool, the judge, or any external party. By combining a sound understanding of AI’s limitations with disciplined verification habits, the legal profession can harness the technology’s efficiency gains without sacrificing the integrity of the judicial process.
Increasingly, court filings are citing legal cases that don’t actually exist