LearnAI home

Building ยท Lesson 4

What to Do When an Automation Breaks Silently

The dangerous failure is the one that keeps running.

The failure you are worried about is not the dangerous one

When people imagine an automation breaking, they picture it stopping. An error, a red mark, a message saying something went wrong. That is the good outcome. A stopped automation announces itself within a day, because the posts stop appearing and somebody asks where they went.

The failure that costs you is the other one. The automation keeps running. It fires on schedule, produces output in the right shape, posts it in the right place, and has been quietly wrong since a Tuesday in March. Nothing looks unusual, because nothing is unusual except the content, and nobody reads content that arrives reliably.

This is the most important lesson in the course. Everything before it was about building. This is about the six months after you build, which is where automations actually live.

In plain English

Silent failure:
The automation carries on running and producing confident output that is wrong, with nothing to signal it.
Empty result:
A step that found nothing, which the next step treats as though it were a real answer.
Duplicate run:
The same item processed twice, so it gets posted, counted or actioned twice.
Sample check:
Opening a few real originals each week and comparing them with what the automation said about them.
Fail loudly:
Designing a step so that when it cannot do its job it stops and says so, rather than producing its best guess.

The four ways it goes quiet

The source changed shape. Somebody adds a column to the spreadsheet, renames a form field, or switches the enquiry form from one name box to two. Your automation carries on reading position three, which used to be the phone number and is now the postcode. Every record from that day forward is subtly wrong and every record looks perfectly normal.

An empty result is treated as a real one. A step looks something up and finds nothing. Instead of stopping, it hands nothing along, and the next step does its best with nothing. You get summaries of messages that were not there, notifications about a customer whose name is blank, or a report confidently stating that this week had zero orders because the lookup pointed at a folder that was moved.

The same item is processed twice. A trigger fires again because something was edited, a connection reconnected and replayed the backlog, or two automations turned out to be watching the same folder. The same enquiry appears twice in the channel, which is merely annoying, or the same figure lands twice in the monthly total, which is not.

The AI step is confidently wrong. The worst of the four, because it is invisible by design. An exact rule that does not match fails and tells you. An AI step presented with something it does not understand never says so. It produces a category, correctly formatted, with the same assurance as every other run. It has been putting refund requests in the sales pile for five weeks and the output has looked immaculate throughout.

Notice that all four produce output that passes inspection. The shape is right, the timing is right, the tone is right. If your only check is does this look like it worked, all four sail through it. That is the whole problem in one sentence.

Fix one: make it announce what it did

The cheapest defence is to make every run report on its own inputs. Not a log buried in a settings screen that nobody will ever open, but a line at the top of the thing your team already reads.

Instruction you can copy: add a header line to any automation

Begin every post with one line, before anything else.

SOURCE ...... which source you read, by name DATED ....... when that source was last changed COUNT ....... how many items you read NEW ......... how many of those were not seen in the previous run

Then these rules.

If the source cannot be found or opened, post the words SOURCE MISSING plus the name, and produce nothing else.

If COUNT is zero, post NOTHING FOUND and stop. Do not produce a summary of nothing.

If an item has already been processed, post ALREADY DONE plus its reference and stop.

If any field you needed was blank, list it under MISSING FIELDS. Never fill a gap with your best guess.

That header does your noticing for you. A stale date, a count of zero, or a NEW figure of zero on a busy Monday is visible in a glance, and a glance is all anybody will ever give it.

Fix two: check a sample every week

You will not audit your automations. Nobody audits their automations. So make the check small enough that it survives a bad week.

Instruction you can copy: the ten minute weekly check

Same slot every week. Ten minutes. One person, named.

  1. Pick three items at random from the last seven days. Random matters. Not the three at the top.

  2. Open the original for each one. Read it yourself.

  3. Ask: is what the automation said actually true of this item?

  4. Look at the needs a human pile. Is it growing? Is anyone emptying it?

  5. Look at the LOW confidence count. If it has crept up, something about the incoming items has changed.

  6. Write one line in a shared note: date, three checked, what you found. One line. Every week.

If two weeks in a row come back clean and dull, you have built something trustworthy. That is what good looks like.

Checkpoint

The dangerous failure keeps running and looks normal. Make every run announce its source and its counts, check three real items a week, and design every step so it stops rather than guesses.

Fix three: design it to stop rather than guess

The first two fixes help you catch problems. This one prevents a whole class of them, and it is a decision you make while building.

At every step where something could be missing, you have a choice between carrying on with a gap and halting. Carrying on feels helpful. It is the reason a report can confidently state zero orders this week. Halting feels obstructive, right up until the first time it saves you, at which point it becomes the thing you build into everything.

โŒ Weak prompt

Prompt

Summarise the enquiries from this week's folder and post the totals in the team channel.

Output

This week saw a quieter period with no new enquiries recorded. Totals: 0 sales, 0 support, 0 billing.

The folder was renamed on Tuesday. The automation found nothing, treated nothing as a real answer, and produced a plausible quiet week. Two months of quiet weeks later, someone finds four hundred unread enquiries.

โœ… Good prompt

Prompt

Read the enquiries in the named folder. Begin with SOURCE, DATED and COUNT. If the folder cannot be found, reply SOURCE MISSING and stop. If COUNT is zero, reply NOTHING FOUND and stop. Only if COUNT is above zero, summarise and post the totals.

Output

SOURCE MISSING: enquiries-inbox. No summary produced.

Six words that stop two months of invisible damage. The automation refused to describe something it could not see, which is the entire discipline.

The day you find one

You will eventually open an original, compare it with what the automation said, and find they do not match. That is the system working, not the system failing. What matters is the next hour.

Instruction you can copy: what to do when you find a silent failure
  1. TURN IT OFF FIRST. Before investigating, before telling anyone. Every minute it runs adds to what you have to unpick later.

  2. FIND WHEN IT STARTED. Work backwards through the output until you reach the last run that was definitely right. That date is the size of your problem.

  3. LIST WHAT WAS TOUCHED between then and now. Posts, rows, labels, notifications, anything that left the building.

  4. FIX THE REAL WORLD BEFORE THE AUTOMATION. Wrong items get put right, and anyone who acted on bad output gets told. The automation can wait an afternoon.

  5. ASK WHAT CHANGED, not who got it wrong. A field renamed, a folder moved, a new kind of item arriving. It is almost never the tool and almost always the world moving.

  6. ADD ONE CHECK that would have caught it a week earlier. One. Then switch it back on and watch it for three days.

Give every automation an off switch you have actually used. Turn it off once, on purpose, on a quiet afternoon, and turn it back on. Finding the switch for the first time during a real problem is how a small mess becomes a long one.

๐Ÿ“ Quiz

Question 1 of 4

Which kind of automation failure does this lesson call the dangerous one?

Found this useful? Pass it on.