A good first read, and nothing more
Summarising is probably the most useful thing AI does for lawyers, and also the easiest to misuse. A long agreement, a disclosure bundle, a board pack, a set of expert reports: the tool can turn any of them into two readable pages in under a minute. That is genuinely valuable. It tells you where to look, what the document is broadly about and which sections deserve your full attention.
What it does not do is read the document the way you would. A summary is a first read, prepared by something that does not know your client, your case theory or which single word in clause 14 changes everything. Treat it as a map drawn by a quick, well-read assistant who has never met the client. Useful for finding your way. Not something you would put your name to.
Before you summarise anything, the confidentiality question comes first. Only put client or matter documents into a tool your firm has approved for that material. Lesson four covers this properly.
Three ways a summary goes wrong
It misses things. Long documents can exceed what the tool holds in view at once, and even within that limit, attention is uneven. A proviso buried in a schedule, a carve-out in a definition, a notice requirement in a boilerplate clause: these are exactly the details that matter and exactly the details a summary drops. Nothing tells you that something was left out.
It flattens nuance. Legal drafting is precise because the precision matters. An absolute obligation and an obligation to use some level of effort are not the same thing. A qualifier, a proviso or a phrase such as "subject to clause 9" can change the effect of everything that follows. A summary written in smooth, general language tends to iron these distinctions out, because smoothing is what it is good at.
It can invent. Less often with a document in front of it than when answering from memory, but it happens. A summary may describe a clause that is not there, attribute an obligation to the wrong party, or state a figure or date that does not match the source. It will do so in the same even tone as everything else.
None of these failures announce themselves. There is no marker on the sentence that is wrong. The only reliable control is checking the summary against the source for anything you intend to rely on.
Checkpoint
An AI summary is a first read that can miss, flatten or invent, so check anything you will rely on against the source document itself.
Make the summary checkable
The fix is not to read harder. It is to ask for a summary that carries its own evidence. If every point comes with the exact words and the clause number it relies on, you can check each point in seconds, and a point with no quotation behind it stands out immediately.
โ Weak prompt
Prompt
Summarise this share purchase agreement and tell me the key risks for the buyer.
Output
The agreement is broadly standard. The seller gives the usual warranties, liability is capped at the purchase price, and there are no unusual provisions. The main risk for the buyer is the limited warranty period.
Smooth, reassuring and uncheckable. Broadly standard compared with what? Which clause sets the cap, and is it really the full price? It has also slipped into advice by assessing risk for a client it knows nothing about.
โ Good prompt
Prompt
Summarise this agreement under these headings: parties and structure, price and payment, warranties and indemnities, limitations on liability, conditions, termination, and governing law and disputes. For every point, quote the exact wording you are relying on and give the clause number. If you cannot find a quotation for a point, do not make the point. List separately any clause you found hard to interpret. Do not assess risk or give advice.
Output
A structured summary in which every statement is followed by a verbatim quotation and a clause reference, plus a short list of clauses flagged as difficult to interpret.
Now each line can be verified against the source in seconds, missing quotations are visible, and the hard clauses are pointed out for your own reading. The judgement about risk stays with you.
Two parts of that prompt do most of the work. "If you cannot find a quotation, do not make the point" discourages unsupported statements. "Do not assess risk or give advice" keeps the tool on retrieval, which it does reasonably well, and off judgement, which is yours.
It will not always follow those instructions. That is fine. The quotations are on the screen, and you are the one checking them.
How to check efficiently
You do not need to reread the whole document to check a summary, but you do need a method.
Check every quotation you rely on. Search the source for the quoted words. If they are not there, or are there but say something different in context, discard that point and read the clause yourself.
Read the definitions. Summaries often use a defined term as though it had its ordinary meaning. Look up every defined term that carries weight.
Read the flagged clauses in full. Anything the tool marked as hard to interpret is, by definition, the part you most need to read yourself.
Look for what is not there. Ask yourself what you would expect a document of this kind to contain. If the summary is silent on it, search the source. Silence in a summary is not evidence that the document is silent.
You have summarised the document above.
Now list, as questions for me to check, the things a careful reviewer
would expect a document of this type to address that your summary
does not mention. For each one, say where in the document I should look.
Do not assume the document is silent on them. Do not give advice.
I will paste this material in [number] parts.
After each part, give me: the main points, each with a quotation
and a page or paragraph reference, and anything that refers back
to or forward to another part.
Do not produce an overall summary until I say all parts are in.
If any part seems cut off or incomplete, tell me.
Splitting a very long bundle helps with the context problem, and asking for cross-references helps you see where one part depends on another.
Checkpoint
Ask for a quotation and a clause reference behind every point, then verify the ones you rely on and read the flagged and defined terms yourself.
๐ Quiz
Question 1 of 4What is the right way to think about an AI summary of a long agreement?