The expensive failures in an outbound system happen between two systems you own: the sender and the counter, the stop list and the provider, the rule and the gate that enforces it. They stay quiet by construction, because the part that faces the customer keeps working while the part that reports on it goes dark.
We run this engine on our own pipeline and on client workspaces: 5,345 cold LinkedIn invitations since 29 July 2026, out of 35 sending profiles across 15 workspaces, watched by 314 named checks that run every hour. Below are the failure families that came out of that, the numbers each produced, and the shape of check that catches it.
The worst failure sits between two systems you own
On 4 November our attribution step stopped marking sent invitations. Invitations kept going out, the panel kept working, the channel was healthy. What died was the one line that writes "this person was invited on this date" back onto the lead record, and 364 invitations went out over five days carrying no mark.
The cause was a single row. Attribution updates a lead in one statement that sets the platform id, the invite date and the stage together. A unique index refused exactly one conflicting row: a dead duplicate created on 11 September, holding the platform id of a live person invited on 4 November. Postgres rolls back the whole statement, so one duplicate took the update down for everybody.
It surfaced only in the weekly review, the single consumer that reads those marks. The first pass counted 77, 89 and 65 invitations for three workspaces over seven days, one step away from the conclusion "volume is down." After the fix the same seven days read 176, 155 and 144. The week had been understated by roughly half, and the natural response would have been to rewrite copy because of an artefact.
The lesson is in the shape of the question. A check that asks "did attribution run" sees a healthy process. A check that asks "an invitation left two hours ago and still carries no mark" sees the failure in two hours, because it measures the corpus the recipient already lives in. The same trap ruins weekly outbound reporting in the ordinary case, where a cohort gets read before it matures. Here the broken link was ours, which is what made it invisible.
The narrower lesson: a bulk update where one row can violate a constraint has to defend itself inside the statement, so one duplicate loses one field and the batch finishes. Today 0 of the last 3,106 sent invitations are missing their mark.
A fifth of our checks have never fired
Every check carries a class, because an unclassified check wakes everyone for everything. Of 314 named checks, 155 are breakage and wake a person, 155 are debt and wake the loop that clears them, escalating to breakage after five days, and 4 are outside, describing someone else's outage and waking nobody. Each gets a three-day quarantine from birth, so a new check cannot page anyone on the day it is written.
250 of the 314 have been red at least once. 64 have never fired, and that silence carries three meanings worth separating: the rule moved into a database constraint and the check became a receipt; the condition genuinely does not occur; or the check itself is broken, reading a column that stopped filling, which looks exactly like good news.
Right now 73 of the 314 are red. Most of that is weeks-old debt, and we publish the number: a dashboard that is always green has stopped asking.
The text checks that keep firing, and the ones that went quiet
Every outbound text passes a fixed set of rules before it can leave. Since 7 August that is 168,378 checks across 13 rules and 6 surfaces, with 7,126 findings, roughly one in twenty four.
The split inside that number is the useful part. Over the last 30 days the rules that still fire need judgment: a stamped phrase in 1,269 of 8,701 texts (14.6%), the same text aimed at two different people in 473 of 1,908 (24.8%), no direct question at the end in 498 of 2,656 (18.8%), assembled from a template rather than for a person in 74 of 616 (12.0%). The rules that went quiet are mechanical: the long dash appeared once in 9,280 texts, hype wording once in 8,701, the formulaic opener zero times.
The cause is structural. Mechanical rules moved out of the audit and into the door, where a database constraint and a send-time gate refuse the text outright, so the audit downstream finds nothing and the count stays honest. Judgment-shaped rules stay in the audit and produce work forever, so budget for them permanently. The same asymmetry decides which of the five steps of outbound an automated system can close on its own.
Stops outnumber sends ten to one, and that is fine
Over the last 60 days the engine sent 3,106 invitations, 2,035 direct messages and 89 InMails, and queued 1,574 follow-ups. In the same window its own gates logged 48,889 refusals of a prepared follow-up: 19,649 for an employer no longer confirmed by a second live source, 8,113 for a draft that repeated the invitation note the person had already read, 8,981 deferred on timing, 4,124 waiting on a fact.
Read naively that is ten refusals per outward touch. Those counters measure check events, not people. 369 people account for all 48,889, the median person carries 54, and 126 people carry 42,076, with a worst single case at 2,161. The loop re-examines the same stuck row on every tick, so one row that can never pass logs refusals forever.
Two things follow. Count events and people separately, or your alarm measures how often your own loop ran instead of how often something is wrong. And a row refused a hundred times has been abandoned, so it needs a cap that parks it and names it for a human. Most of those refusals come from the employer and staleness checks, the maintenance cost of insisting on one checkable fact per message.
Outside systems fail loudly, your own vocabulary fails quietly
Transport failures are cheap by comparison: 683 in 60 days, 633 of them a single not-found family, plus 94 invitations refused outright by the platform. Those belong to a retry policy and a strike ladder.
The quieter external failure is a rule that lives only in your own database. A client exclusion list stops our own sending the moment it is written, and email audiences get assembled on the provider's side, so the list has to travel into the provider's own suppression file to have any effect there. We hold 5,116 exclusion entries, 329 resolved to domains, and 7 company names that could not be turned into a domain safely. Right now 320 of 321 domains have arrived at the provider, and the check exists to name the one that has not, because partial arrival is the dangerous state. What belongs on such a list is its own discipline.
Then there is your own vocabulary. Every candidate we drop before contact is supposed to carry a reason code from a fixed set. Over 60 days there were 900 drops carrying 447 distinct reason strings, in two languages, most written once. That field became open prose, so every analysis of why the funnel narrows gets regrouped by hand, and validation passes on prose every time. The check compares distinct values against volume: once the reasons approach half the events, the field has stopped being a vocabulary.
What the machine repairs, and what it only records
Twelve repair recipes have applied themselves 929 times since 5 August. The most frequent one is infrastructure: 484, all since 19 October, reclaimed disk space from old logs. Then 156 restarts of a service that had died or gone stale, 94 installs of an edit written and never deployed, 81 reconciliations of a summary that had drifted from its source records.
Predictions are cheaper to open than to close. Every change the engine makes records three fields first: why, what we expect, and what we rejected. 991 decisions since 29 July, all 991 with all three filled, 62 made by a human. 423 carry a recorded outcome, and 303 are older than three weeks with none. That last number is the honest cost of the discipline, and it is the one to watch.
What to do on Monday
- For each number on your weekly report, write down which column it reads and when that column fills. The gap between the send and the mark is where a week of statistics disappears.
- Give every check a class: wakes a person, wakes a loop, wakes nobody. A check without one turns into noise inside a month.
- List the checks that have never fired and decide which of the three reasons applies. Treat "never fired" as unverified.
- Separate event counts from people counts in every alarm, and cap repeat refusals so a stuck row gets parked and named.
- Compare distinct values against volume in your reason and status fields. Two spellings of one state, or hundreds of one-off reasons, are both silent.
- Date every prediction you record and review the oldest ten each week.
The checks are as much the product as the sending is. For the cycle itself rather than its failure modes, how it works walks the same ground forwards.
Questions buyers ask
How do I know if my outbound automation is silently broken?
Watch the processes between your own systems rather than the outward channel. Our worst case ran five days: 364 invitations went out correctly while the step that marked them did nothing, and the weekly numbers came in at roughly half the truth. The check that catches it asks about the external fact, in the form "a message left two hours ago and still has no record."
How many checks does an outbound system actually need?
Ours runs 314 named checks hourly, each with a class: 155 wake a person, 155 wake a repair loop and escalate after five days, 4 wake nobody. Classification matters more than the count. 250 have fired at least once, 64 never have, and 73 are red today, mostly aged debt.
Do quality checks on outbound copy keep paying off?
The ones that need judgment do. Across 168,378 text checks on 13 rules we get 7,126 findings, and the live ones are stamped phrasing at 14.6%, the same text to two people at 24.8%, and no direct question at 18.8%. Mechanical rules stop firing once they move into a hard constraint at the point of sending, which is where they belong.
Why do automated follow-ups get blocked so often?
Because the gate re-examines the same queued row on every pass. In 60 days ours logged 48,889 refusals against 5,230 outward touches, and all of them belong to 369 people, median 54 each, worst case 2,161. The answer is a cap that parks a row after a set number of refusals and names it for a human.