On 30 July, the Queensland Law Society's Proctor reported that Judge Douglas Humphreys of the Federal Circuit and Family Court, dealing with yet another AI-generated submission from a self-represented litigant, decided out of curiosity to ask ChatGPT about a judgment he had written himself and published on AustLII. The model produced a further judgment that does not exist, commentary on it from firms that never wrote any, and literary quotations he had never made. Pressed, it answered: "I completely fabricated those details."
The obvious reading is that AI invents citations and lawyers should check their work. That has been true and widely known since 2023, and saying it again changes nothing. The line that should hold a director's attention is a different one. Judge Humphreys observed that unrepresented litigants and AI systems "cannot be referred to a regulator when references are hallucinated and/or contain misleading or entirely incorrect propositions."
That sentence describes a structural failure, not a technical one. Every professional assurance system we have is enforced at the point of an accountable person. You strike off a solicitor, sanction an auditor, disqualify a director, refer a financial adviser. The whole edifice assumes that at the end of any piece of consequential work there is somebody who can be named, questioned and punished. Machine-generated work product arrives with that position vacant. Boards should recognise the shape of this immediately, because their own assurance frameworks rest on precisely the same assumption: every control we rely on terminates in a human being who can be held to account for it.
The second thing worth noticing is subtler and, for boards, more uncomfortable. The judge asked the system to explain itself, and it confessed. A confession from a system that has just demonstrated it fabricates is not evidence of anything at all. Yet this is the species of assurance boards accept routinely: the vendor demonstration, the model explaining its own reasoning, the management response to the AI risk question that was itself drafted with AI. We are collecting the system's account of its own behaviour and filing it as verification.
- When a paper reaches this board, who is the accountable human at the end of it, and what did that person verify personally rather than accept?
- Where in this organisation does machine-generated output become a signed, attributable deliverable, and at which step does a named person take responsibility for its contents?
- If a court, a regulator or a major customer found a fabricated fact in our work product tomorrow, who would be referred, and would our records show what they actually checked?
- We ask our AI vendors to explain how their systems reach conclusions. Whose account are we relying on, and what evidence do we hold that was not produced by the system being assessed?
- Which of our external advisers now use AI in preparing the advice this board relies on, and has anyone asked a single one of them what their verification process is?
- What is our policy on AI-assisted drafting of board papers, and has it been tested against practice, or only circulated?
- Assurance that terminates in the system rather than in a person. If your evidence that a model behaves correctly is the model's own explanation, you are holding no evidence.
- An AI policy that governs employees but is silent on advisers, contractors and outsourced providers, who between them produce a large share of what this board actually reads.
- Verification framed as a training or awareness problem. Awareness of hallucination has been close to universal for three years and has not stopped it. Awareness is not a control.
- Volume presented as a benefit. The judge's practical complaint was the burden of reviewing lengthy AI-generated submissions. Cheap production does not remove cost, it transfers it to whoever has to check the work.
- No record of what was checked. Under section 180 the defence is the enquiry you made, not the good faith you brought to it. Reliance on advice is a tool, never a shield.
- Nobody in the organisation has run the judge's experiment on your own material.
The upside is that this is one of the cheapest controls a board will ever ask for. Verification is not a technology problem; it is a process design question with a well-understood answer. Map the points where machine output becomes attributable work product, put a named person at each one, and require a short record of what that person actually checked. Organisations that do this get to use AI far more aggressively than their peers, because they have somewhere defensible to put the risk. The business judgment rule has always rewarded the director who can show the enquiry they made, and a verification record is exactly that enquiry in written form.
The downside is that this failure mode is silent. A fabricated citation is fluent, confident and formatted correctly; it does not announce itself the way a spreadsheet error does, and the people best placed to catch it are the people least likely to re-check work that reads well. So it surfaces late, and it surfaces in front of the worst possible audience: a court, a regulator, an acquirer in diligence, a journalist. And because the accountability gap the judge described is real, the consequence does not stop at the system that produced it. It travels up to the nearest person who can be held responsible.
Do three things this quarter. First, map where machine-generated material becomes attributable work product in your organisation, and put a named person and a recorded verification step at each of those points; the map itself will be more revealing than the control. Second, extend your AI policy beyond employees to advisers, contractors and outsourced providers, and put the verification question into the next engagement letter you sign, because a material share of what your board reads is written by people you do not employ. Third, stop accepting a system's account of itself as assurance, and require at least one piece of evidence about any AI deployment that was produced independently of the model. Then run the judge's experiment on your own material, and see what comes back. He did it out of curiosity and found something his whole profession needs to reckon with; the boards that do it deliberately will simply find out early what the rest will find out in front of an audience.
Researched and drafted by Brad's agentic AI team. Edited and published by Brad Ferris.