In 2023, New York lawyer Steven Schwartz asked ChatGPT whether a court decision it had supplied was real. The chatbot assured him that it was. The sanctions opinion in Mata v. Avianca records the exchange, six fabricated decisions, and a $5,000 penalty imposed jointly and severally on two lawyers and their firm. [1]
Legal experience had not prevented the failure. The case exposes a mistake anyone using these tools needs to understand: asking the source of a claim to vouch for it does not independently verify the claim.
In What You Can Defend, I argued that generative AI weakens the signals we use to read expertise, because polished work can now exist without the understanding it once implied. That argument was about the person presenting the work. This article is about the system producing it.
Almost overnight, everyone seems to have become an AI expert. Using a tool every day can make it feel familiar long before we understand how it produces an answer. That understanding determines which checks we can ask the system to help with and where independent evidence must enter the process.
What happened in Mata v. Avianca
The lawyers submitted invented authorities in a personal injury case. Schwartz's screenshots showed ChatGPT confirming their existence. The court also recorded conflicting accounts of when he asked those questions and what he believed. It found subjective bad faith, including conscious avoidance of evidence that the cases did not exist. [1]
This was more than an innocent misunderstanding of software. My narrower point concerns the verification method: another generated assurance supplied no independent court record. Professional responsibility still required checking the authorities.
What a language model does
A large language model, or LLM, learns statistical relationships from training data. Its parameters encode patterns that support language generation and many other tasks. Those parameters are not a database of verified answers.
The autoregressive models commonly used in chat systems generate text one token at a time. A token may be a word, part of a word, punctuation, or another text unit. At each step, the model computes probabilities for possible next tokens from the available context. A decoding procedure selects the next token, adds it to the sequence, and repeats. Selection may involve sampling or choosing the highest-scoring token. The probabilities are about token continuation, not a direct measurement of factual truth. [2]
Most of these models use a Transformer architecture. Tokens become numerical representations, and attention layers help the model combine information from different positions in the context. Learned weights transform those representations into predictions. The instructions and conversation affect the computation, but an ordinary exchange does not itself retrain the model's weights. [2, 3]
That mechanism supports more than surface imitation. Models learn representations that enable useful generalization, including answering questions and solving some unfamiliar problems. Later training can improve instruction following, reasoning, and truthfulness. For example, Ouyang and colleagues trained models using demonstrations and human preference feedback, improving results while leaving errors unresolved. Fluency and accuracy remain separate properties. [2, 4]
The product surrounding a model matters too. A chat system may search the web, retrieve documents, run code, or use other tools. Retrieval introduces material from outside the model's learned parameters and can improve factual performance. It also introduces further questions: what was retrieved, whether it was reliable, and whether the answer represents it accurately. [5]
A correct explanation and an incorrect one can arrive in the same confident professional language. Knowing how generation works should make us more demanding about the evidence attached to an answer.
Four things that do not verify an answer
Models can critique their work, identify mistakes, and help perform checks. The problem is treating one of the following as sufficient verification.
The model confirming its own claim. A follow-up question becomes part of the context for another answer. The model may correct itself through further computation, but an assurance alone adds no independent evidence. In Mata, the missing step was checking the purported authority against an external record.
The model repeating the answer. Repetition shows that the answer recurs under the tested prompts and settings. It does not directly reveal the model's probability for the complete answer or establish its truth. There is a useful qualification here: Wang and colleagues showed that sampling multiple reasoning paths and selecting their most consistent answer improved accuracy on several benchmarks. That is a tested method for improving performance, not independent corroboration of every resulting claim. A repeated error can still be an error. [6]
The answer including a citation. A generated citation does not even guarantee that the source exists. Opening the source addresses existence. Reading the relevant passage against the claim addresses support. Those checks are different. Gao and colleagues evaluated citation quality separately from fluency and correctness, finding substantial gaps in support in the systems and tasks they tested. Retrieval helps, but a real source can still be misread or attached to a claim it does not establish. [7]
The model explaining its own reasoning. A generated explanation can help a reader assess an argument. It is not automatically a faithful record of the computation that produced the answer. In a 2023 study, Turpin and colleagues introduced biasing features into prompts, including arranging examples so that the correct option was always labelled A. GPT-3.5 and Claude 1.0 could follow those cues without mentioning them in their explanations. When biased toward incorrect answers, the models frequently rationalized those answers. The study establishes a failure mode in the tested systems, rather than proving that every explanation from every model is unfaithful. [8]
The last point will be familiar to any investigator. A model's account of its own reasoning is an artifact to be tested. The account does not establish its own accuracy.
Where domain knowledge fits
Domain knowledge and tool knowledge do different jobs. Domain knowledge helps you judge whether an answer is plausible and what evidence would confirm or refute it. Tool knowledge helps you understand what the system did to produce that answer and what remains unchecked.
Take a forensic examiner who asks a model to help interpret Windows event logs. Domain knowledge lets the examiner assess whether the proposed timeline fits the records. Tool knowledge prompts a further question: did the model retrieve the relevant documentation, or generate its account from information encoded in its parameters and supplied in the prompt?
The examiner then checks the event definition against the relevant Microsoft documentation, including the applicable provider and system context, and checks the proposed interpretation against the original records. Even a retrieved passage needs to fit the event being examined. Returning to the model for a reassuring explanation does not complete those steps.
A person without either kind of knowledge has fewer ways to detect a plausible error. A professional with substantial domain knowledge still needs a method that brings the relevant evidence into the assessment. Mata shows that expertise and responsibility do not transfer to a tool simply because its answer looks familiar.
What understanding the tool changes
I believe these tools can improve how we work, learn, and investigate. Using them well depends on sending verification to the right place.
In practice, that means identifying the claims in an output and deciding what would test them. The system may help run a calculation, retrieve a court decision, or quote a document. The resulting calculation, record, or passage must still be inspected, along with its relevance to the claim. A tool-assisted check can happen within the same interface while drawing on evidence outside the language model.
Other checks require access the system does not have: the original evidence, a restricted database, a physical observation, or a qualified reviewer. The model's confident wording, repeated answer, or explanation of itself cannot replace that missing access.
It also means being willing to say, "I do not understand how this output was produced well enough to know what it has verified." That is an accurate statement about the limits of your knowledge of the tool. It tells you what to examine next.
The level of scrutiny should match the cost of being wrong. A poor restaurant recommendation may cost an evening. An unsupported conclusion in an investigation can affect someone's career or liberty. A plausible paragraph deserves much closer examination when someone else bears those consequences.
The question we should be asking
Prompting is a useful skill. Professional AI literacy also requires understanding how training, context, generation, retrieval, and tool execution contribute to an answer. Without that distinction, a capable person can mistake another assurance for a completed check.
Knowing the architecture will not prevent every error. It gives professionals a basis for asking what the system consulted, what it computed, and which claims still require verification.
Before relying on an answer, we should be able to explain what supports it. If the only support is that the same model said it again, the verification is still unfinished.
References
- U.S. District Court, Southern District of New York (June 22, 2023). Mata v. Avianca, Inc., Opinion and Order on Sanctions, No. 22-cv-1461, Document 54, especially findings 45–47 and the sanctions order.
- Brown, T. B., et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020.
- Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017.
- Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022.
- Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020.
- Wang, X., et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
- Gao, T., Yen, H., Yu, J., and Chen, D. (2023). Enabling Large Language Models to Generate Text with Citations. EMNLP 2023.
- Turpin, M., Michael, J., Perez, E., and Bowman, S. R. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS 2023.