Why You Cannot Trust AI With Arithmetic

What is actually happening when it gives you a total

A confident total in a chat reply is not evidence that the right rows were calculated. A text-only answer may be generated without running a calculation. Code-backed analysis is different: ChatGPT can write and execute Python for some data-analysis tasks. Do not assume every reply uses that capability.

Even executed code can use the wrong range, omit a row, change a sign or apply the wrong assumptions. Review the source data, code or formula, row coverage and output. OpenAI's data-analysis guidance recommends reviewing generated code, outputs and assumptions.

Three ways it goes wrong

It gets simple sums wrong, confidently. Not always, and not predictably. That unpredictability is the point. If it failed every time you would catch it immediately. It fails occasionally, in the same steady tone it uses when it is right.

It drops rows. Give it ninety transactions to summarise and it may summarise eighty-four. Nothing announces this. The subtotal simply comes out lower and looks entirely normal, because a smaller number is still a number.

It doubles rows. The mirror image, and rarer, and worse. A duplicated line inflates a total, and an inflated total in a VAT calculation or a set of management accounts is the kind of thing that gets noticed by exactly the person you least want noticing it.

The rule

Here it is, and it is the spine of everything that follows:

Use an inspectable calculation and verify its inputs, assumptions and outputs, whether it runs in a spreadsheet, accounting system or code-backed AI tool.

Asking for a spreadsheet formula is one useful way to make the method visible. A formula is not automatically correct: its range, input types, exclusions and signs still need checking. The same checks apply to executed code.

Making it show its working

The point of asking for the method is that a method is checkable and a bare figure is not. You can read a rule, disagree with it, and test it on five rows. You cannot do any of that with the number 14,382.67.

Do not give me a total.
Instead give me, in this order:

Which rows you would include and which you would exclude, and the exact rule.
Any rows you are unsure about, listed individually.
A spreadsheet formula that applies the rule, referencing my actual columns.
A second formula that counts how many rows the first formula covers.
My columns are: [list them]. My data runs from row [2] to row [900].

That fourth item is the one people skip and the one that catches the quiet failures. If you expected 47 matching rows and the count says 41, you have found a real problem in ten seconds.

I gave you [90] transactions. Before summarising anything,
tell me how many transactions you can see, and list the first
and last reference number in the set.
Then group them by [category] and tell me the row count in each group.
The group counts must add back to the total count.
If they do not, say so plainly and stop.

It will not always obey that last instruction. That is fine. You are the one checking that the group counts add back, and now the counts are on screen where you can see them.

Try these checks on the synthetic reconciliation practice.

What good looks like

You asked for a rule. You read the rule and agreed with it. You put the formula in the sheet. You checked the row count against what you expected. You recalculated three awkward rows by hand: the biggest one, one that should have been excluded, and one credit note.

Total time: about four minutes. And now, when a partner or a client asks where the number came from, you have a real answer, which is the entire job.

I have calculated [describe the figure] as [amount] using this method:
[describe your method in a sentence].
Do not recalculate it and do not give me a number.
Instead list, most likely first, the ways this method could be wrong:
rows that might be wrongly included or excluded, sign errors,
date boundary problems, duplicates, and anything else.
For each one, tell me the specific check I should run in my spreadsheet.

This is the tool at its best. It is not being asked to be right about a number. It is being asked to be imaginative about how a number could be wrong, which is a language task, and one it is very good at.