What to Do When an Automation Breaks Quietly

Loud failures are the good ones

An automation that stops is a mild irritation. Something errors, a red mark appears, you spend twenty minutes fixing it and you carry on. Nobody has ever lost a customer to an automation that stopped.

The expensive failure is the one that keeps going. It runs on schedule, produces output, fills the spreadsheet, posts to the channel, and every one of those results is wrong in a way that looks completely normal. Nothing turns red. Nothing asks for your attention. You find out in month four, when somebody asks why an entire category of enquiries has gone unanswered since spring.

Every automation in the last six lessons can fail this way. This lesson decides whether you are still running any of them in a year, so read it twice.

The four silent failures

One: the source changed shape. A supplier redesigns their invoice and moves the total. A form gains a field, so the answers shift by one column. Somebody renames a folder. The automation reads the new thing with the old assumptions and returns something perfectly formatted and completely wrong. This is the most common of the four by a distance, and it is nobody's fault. Sources change because the world changes.

Two: an empty result treated as real. The step that fetches yesterday's orders returns nothing because a connection dropped for ninety seconds. The automation does not know that nothing is unusual. It writes zero into the report, or posts nothing to the channel, and the day simply disappears. Nothing errored. There is now a hole in your records with no marker on it.

Three: duplicate processing. The same item goes through twice. A retry after a timeout, a folder synced from two places, a trigger that fires on edit as well as on create. Two invoices for one delivery, two reminders to one customer, one number counted twice in a total. This one is particularly nasty because the output looks entirely legitimate. It is just there twice.

Four: confidently wrong extraction. The AI step reads a document, cannot quite make out a figure, and returns a plausible one in exactly the same calm tone it uses when it is right. A date that is a month out. A supplier name that is nearly correct. A category that is defensible but not what you meant. No flag, no hesitation, no difference in presentation between the answer it was certain of and the one it constructed.

Fix one: make it announce what it did

An automation that runs invisibly is an automation you cannot supervise. Make every one of them report, briefly, somewhere a person already looks.

ONE SHORT MESSAGE PER AUTOMATION PER DAY. FIVE LINES.
1  How many items it processed today.
2  How many yesterday, and the daily average this month.
3  How many were skipped, flagged, or marked unclear.
4  The oldest item still sitting unhandled, and its age.
5  Anything it did for the first time: a new supplier,
a new sender, a category it had not used before.
WHY EACH LINE EXISTS
Line 1 alone tells you nothing.
Line 2 is what makes line 1 mean something. Forty items
when the average is forty is health. Forty when the
average is four hundred is a silent failure in progress.
Line 3 catches the fallback that stopped firing.
Line 4 catches the pile nobody owns.
Line 5 catches the source that changed shape, usually
on the very first day it changes.
SEND IT WHERE PEOPLE ALREADY LOOK.
A digest in a channel nobody opens is not a digest.

Fix two: sample weekly, whether or not anything looks wrong

Ten minutes a week. Five results picked at random, checked properly against the thing they came from. Not the ones that look odd. Random ones, because the whole problem is that the broken ones do not look odd.

SAME DAY EVERY WEEK. FIVE RANDOM RESULTS. TEN MINUTES.
For each one, open the original and answer:

Is every extracted value actually on the document?
Is the category the one I would have chosen?
Did it go to the right person or place?
Was anything invented that was not in the source?
Write down every disagreement, however small.
READ THE PATTERN, NOT THE COUNT
Same field wrong every time    your definition is loose
Same source wrong every time   that source changed shape
Same item appearing twice      duplicate processing
Wrong but plausible, scattered confident extraction error
Errors rising week on week     stop it and rebuild
Nothing wrong for six weeks    sample fortnightly instead
KEEP THE SHEET. Six weeks of samples is the only honest
record of whether this thing works.

Fix three: fail loudly rather than guess

The default behaviour of every AI step is to produce something. Left alone it will always give you an answer, because that is what it is for. Your job is to write the instructions that make silence a permitted outcome.

Give every step a way out and a place to put the things it could not handle. UNCERTAIN, NEEDS HUMAN, SKIP, NOT FOUND, UNCLEAR. Then check that the exit actually fires by feeding it something impossible on purpose. An empty document. A page in a language you do not handle. A form with the fields deliberately jumbled. If it returns a confident answer to any of those, your fallback is decoration and you have not tested it.

The recovery checklist

You have found one. Something has been wrong for a while. Work through this in order and resist the urge to start with the fix.

1  TURN IT OFF. Not pause and investigate. Off. Every
minute it runs adds to what you have to clean up.
2  FIND THE FIRST BAD ONE. Work backwards through the
output until you reach results you trust. That date
is your blast radius. Write it down.
3  LIST THE DAMAGE. Which records, which customers, which
totals, which decisions were made on this output.
Everything downstream is suspect until checked.
4  TELL ANYONE AFFECTED. Before they find it. A short,
factual note beats a discovered mistake every time,
and this includes your own team.
5  FIX THE RECORDS FIRST, THE AUTOMATION SECOND. Wrong
data in your systems keeps causing harm while you
sit rewriting a prompt.
6  FIND THE CAUSE, NOT THE SYMPTOM. Which of the four
was it? Source changed, empty treated as real,
duplicate, or confident guess? Name it out loud.
7  ADD THE CHECK THAT WOULD HAVE CAUGHT IT. Not a
promise to be more careful. A line in the digest,
a test in the reconciliation, a new exit condition.
8  RUN IT ALONGSIDE FOR A WEEK. Output going nowhere,
checked by you daily, before it touches anything
real again.
9  WRITE DOWN WHAT HAPPENED. Four lines in a shared
note. In six months this is the only reason anyone
will understand why that odd looking check exists.