AI document review: can you trace the answer back to the files?
To check an AI document review, trace each material conclusion to the exact source passage and test whether that passage supports it. Check missing and conflicting material too: an accurate summary of the wrong set of files can still mislead.
Ger Perdisatt, Acuity AI Advisory
2 min read
Define the question and the file set
A request to “review these documents” leaves too much open. Say what you need to decide, which files are in scope and what a useful output contains. For a service complaint, that might be a dated account of events, unresolved questions and the evidence needed before replying.
Record which files were supplied and which were successfully read. Check page counts, attachments and version dates. If a key page is a faint scan, the review needs to say so. Do not assume that uploading a file means every page was understood.
Check the passage, not just the link
A link may lead to the right document while the answer overstates what it says. Open the cited passage. Does it name the same entity? Is the date an event date, a proposed date or the date the document was written? Is the statement conditional?
For an important conclusion, check the surrounding text and relevant attachments. A clause or sentence may be qualified elsewhere. Where consequences are significant, use a reviewer who understands the subject and can recognise an omission.
Ask what might be missing
A review can be internally consistent because an awkward document was never included. Compare the file set with the record you expected to receive. Keep absent documents, unreadable sections and unresolved contradictions in the output.
Separate extraction from interpretation. First establish what the documents say. Then label any proposed explanation or next step as judgement. In professional work, the appropriate qualified person remains responsible for conclusions and advice.
Revisit the answer when the record changes
If a new delivery note arrives, update the working file set and ask which earlier conclusions change. Do not simply append a new summary below the old one. Review any reply, tracker or recommendation that relied on the earlier position.
Save the checked output, the source references and the remaining questions together in the agreed location. The next person should be able to continue the work without relying on an inaccessible chat history.
Editorial guidance informed by Acuity's work with teams. Examples are fictional and client materials stay private. To record who owns an AI use and when it is reviewed, see AI Register.