When someone says, "We don't understand how AI works," I want to know what, exactly, they tried to explain. The training process? A model's response to one prompt? The source of a fact it used? Its behaviour after gaining access to a tool? Those are different investigative questions. Treating them as one mystery makes it harder to find an answer.
As a digital forensic practitioner, I start with a distinction between an artifact and a finding. An output is an artifact. It tells me what the system produced under particular conditions. It does not, by itself, tell me which training material influenced the answer, which internal operations mattered, or whether the system would behave the same way again. Those claims require different evidence.
I understand why people are uneasy. We train models on volumes of material that no person can review item by item. We can describe neural network architectures and training procedures, yet often cannot give a tested account of the relationships a particular model learned or used in one response. The gap is real. The phrase "we don't understand AI" leaves its boundaries unclear.
I had been wondering whether we needed something like psychology for these systems: a discipline that investigates behaviour even when it cannot predict every response. Then I learned that researchers had already proposed machine behaviour and machine psychology. Rahwan and colleagues argued for an interdisciplinary study of AI behaviour. Hagendorff and colleagues proposed experiments inspired by psychology to examine language models beyond benchmark scores. CogBench applied cognitive psychology measures across 35 models. An idea can be new to me without being new to the field. What interests me now is how to connect this work to a defensible investigation. [1][2][3]
The analogy has limits. A language model is not a human mind. A response that sounds like a belief or emotion is not evidence of human experience. The useful inheritance from psychology is methodological: vary conditions, look for repeatable behaviour, test competing explanations, and examine where the explanation fails. With AI systems, we can add training records, internal measurements, and controlled interventions.
What did the model learn?
Consider an obscure research paper that enters a training dataset. Few investigators have read it. Later, a model produces an answer consistent with that paper and combines it with information from elsewhere. The answer may look astonishing to someone who has never encountered the research. In this hypothetical, part of the apparent mystery comes from unequal exposure to information. Controlled experiments on out-of-context reasoning show that models can infer and later use relationships distributed across training examples, even when the later question was not supplied during training. [4]
I would still resist saying that the paper caused a real answer merely because the two resemble one another. Other sources or a different route through the model could explain it. Training data attribution methods can help estimate influence, but their results depend on the method, model, and question being examined. They do not establish the provenance of each part of an output. This is the forensic problem: distinguish the observed artifact from a supported finding about its origin. [5]
There is a second problem after data exposure. Reading every document would not tell us which relationships a model actually learned or how it combines them. OpenAI researchers found that fine-tuning on wrong answers in one narrow domain could affect responses in unrelated domains. They then tested an internal pattern associated with that behaviour. In a separate distillation experiment, researchers found that numerical data appearing unrelated to a preference could pass that preference between models sharing a base model. These are bounded findings, not proof that any dataset can induce any behaviour. They do show why inspecting the visible content of training data is only one part of the investigation. [6][7]
Adding training material, changing a training procedure, and measuring performance on selected tasks are comparatively fast. Establishing what relationships formed, how they affect behaviour across contexts, and whether an audit can detect them is harder and slower. Data curation and interpretability research are active fields, so it would be inaccurate to say nobody is investigating. The documented limits of current methods support a narrower criticism: our ability to produce more capable systems can run ahead of our ability to account for their learned behaviour. [8][9]
Which kind of understanding is missing?
Dario Amodei has said that developers often cannot explain at a specific, precise level why a generative model makes a particular choice, and that interpretability has fallen behind AI capability. I take that admission seriously. It identifies a deficit in explaining particular mechanisms. It does not mean researchers lack an understanding of architecture, training, or observed behaviour. Conversely, knowing the training algorithm cannot establish why a model gave a consequential answer. We need to say which kind of understanding a claim concerns. [9]
Sam Altman made a different point in a 2024 World Economic Forum discussion. He suggested that we might ask an AI to explain its reasoning and judge whether it sounds reasonable without tracing every internal connection, much as we do with people. That can help us assess an answer's logic. It cannot, on its own, establish what caused the answer. Anthropic's experiments found that a model's written reasoning did not always faithfully report influences on its response. A stated explanation is evidence to test, not a substitute for investigating the process. [10][11]
The research also shows why investigation is possible without promising a complete map. Anthropic has traced portions of circuits involved in particular model responses. OpenAI has explored automated explanations of neural activity and smaller, deliberately sparse models whose circuits are easier to inspect. The explanations remain partial, and results from one setup do not automatically generalize to a frontier model. Yet the work offers ways to formulate and test causal hypotheses about behaviour. [12][13][14]
One example connects this work directly to auditing. Anthropic deliberately trained a model with a hidden objective, then asked investigators who did not know that objective to examine it using behavioural tests, training data, and internal evidence. In a later study, AuditBench tested investigative techniques on 56 models with hidden behaviours. Some behaviours were easy to surface through ordinary prompting. Others were not. Additional tools sometimes improved detection, and the training method affected auditability. These results come from experimental settings. They do not establish how reliably an audit will uncover every problem in a deployed model, but they identify which questions to test next. [15][16]
What would a defensible deployment account look like?
The forensic parallel gives me a practical standard, and machine psychology supplies part of the method: define the behaviour, vary the conditions, and record what can be reproduced. Before granting a system consequential authority, an operator should be able to identify the model and version tested, the relevant training and post-training information available for review, the behaviours sought, the conditions and tools used to elicit them, and the results that could be reproduced. The account should distinguish observed behaviour from an inference about its cause. It should state which tests were not performed or could not be performed.
Then it should describe the system as deployed: what information it may read, what actions it may take, which actions require a person's approval, what records of its actions are retained, and what technical boundary prevents an unauthorized action. A written instruction asking the model to behave is not the same as a permission boundary. Controls can be layered and tested even when an internal mechanism remains unclear. The limits of those tests belong in the report. [17]
An account like this needs a reader who can challenge it. At minimum, that is whoever in the deploying organization is accountable for granting the system its authority. Where a failure could reach customers or the public, the account should also be available to an independent auditor or regulator. The test is whether someone outside the development team could examine it and identify what was established, what was assumed, and what was never tested.
This standard has consequences for pace. If an agent is about to gain the ability to alter records, move money, execute code, or affect infrastructure, a developer may need to delay that access while behavioural testing and control verification catch up. A delay can also postpone useful capabilities, so the decision should depend on what the system can do and the consequences of failure. I doubt competition will produce a broad voluntary slowdown. That is a judgment about incentives, not a finding established by the interpretability studies.
Using a model does not require an explanation for every output. It does require that investigative capacity grow alongside capability, and that the authority granted to a model reflect what the investigation can actually support. When a developer says a behaviour is not understood, the next disclosure should identify the missing evidence and the actions the system can still take if the explanation is wrong.
Research and sources
- Iyad Rahwan et al., "Machine behaviour", Nature, 2019.
- Thilo Hagendorff et al., "Machine Psychology", 2023.
- Julian Coda-Forno et al., "CogBench: a large language model walks into a psychology lab", 2024.
- Peter Hase et al., "Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data", 2024; Atticus Wang et al., "Simple Mechanistic Explanations for Out-Of-Context Reasoning", 2025.
- Roger Grosse et al., "Tracing model outputs to the training data", Anthropic, 2023.
- OpenAI, "Toward understanding and preventing misalignment generalization", 2025.
- Alex Cloud et al., "Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data", 2025.
- Jeffrey Li et al., "DataComp-LM: In search of the next generation of training sets for language models", 2024; Lee Sharkey et al., "Open Problems in Mechanistic Interpretability", 2025.
- Dario Amodei, "The Urgency of Interpretability", 2025.
- World Economic Forum, "Davos 2024: Technology in a Turbulent World", transcript of Sam Altman's remarks, 2024.
- Anthropic, "Reasoning models don't always say what they think", 2025.
- Anthropic, "Tracing the thoughts of a large language model", 2025.
- OpenAI, "Language models can explain neurons in language models", 2023.
- OpenAI, "Understanding neural networks through sparse circuits", 2025.
- Anthropic, "Auditing language models for hidden objectives", 2025.
- Anthropic, "AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors", 2026.
- OpenAI, "How we think about safety and alignment", accessed September 24, 2026.