Agents ·
The gap between an AI demo and something that runs unattended
A demo has a human absorbing every ambiguity in real time. Unattended software has none, so each judgement that human made silently becomes a rule, a guard or an escalation.
A demo has a human in the loop absorbing every ambiguity in real time — choosing the input, re-running the awkward case, interpreting a half-right answer charitably. Unattended software has none of that, so every judgement the demonstrator made silently has to become an explicit rule, a guard, or a route to a person. The work between the two is mostly not model work, which is why the estimate that felt right after the demo is usually wrong by a wide margin. Barnett et al. put it plainly for retrieval systems, and it generalises: robustness evolves rather than being designed in at the start (arXiv 2401.05856, January 2024).
A demo is a sampling artefact
You ran it a dozen times and showed the best one. That is not dishonest — it is how demonstrations work — but it means the demo measured the best case and production is priced on the tail.
Do the arithmetic before committing to a volume. A failure that occurs once in five hundred runs is invisible across an afternoon of testing. At ten thousand runs a day it is twenty incidents a day, every day, and each one may require a human to notice, diagnose and undo it. Whether that is acceptable depends entirely on what the failure does, which is why the first question about any unattended system is not how accurate it is but what the worst individual failure costs.
So begin by classifying actions by reversibility. Reading a document is free to get wrong. Drafting a reply is cheap. Sending an email is awkward. Issuing a refund, changing a record of account or filing a return is expensive and sometimes not undoable at all. The design work differs for each class, and lumping them under one accuracy threshold is how organisations end up ninety-five per cent right about things that needed to be certain.
We do not ship an agent that can take an irreversible action in the first phase. That is a standing position rather than a negotiating one.
- Classify every action by reversibility before setting any accuracy target
- Multiply the per-run failure rate by real volume and read the daily number out loud
- Cheap-to-reverse actions can run unattended much earlier than expensive ones
Retries are the most common production defect in agent systems
An agent that sends an email, times out waiting for the confirmation, and retries has sent two emails. Every side-effecting action needs an idempotency key and a deduplication window, and almost none of them have one in a prototype.
This is the ordinary at-least-once delivery problem from distributed systems, with an additional complication: the actor is non-deterministic, so on the retry it may not attempt the same action at all. Replaying the input does not reliably reproduce the behaviour, which breaks the usual mental model of a retry.
The fix is to make the decision the durable artefact rather than the input. When the agent decides to act, write the intended action to a store with a deterministic key derived from the business event — this ticket, this customer, this action type, this window — and then execute against that record. A retry re-reads the record and finds the action already committed. Recovery becomes a matter of replaying decisions rather than re-deriving them, which also makes the system explicable after the fact.
Partial failure needs the same treatment. An agent that completed three of five steps and then died has left the world in a state nobody designed. Either every step is individually idempotent and safe to re-run, or each has a compensating action that undoes it, and the sequence is orchestrated so that recovery is a defined path rather than an incident. Long-running agent workflows are transactions without a transaction manager, and they need the same discipline that any saga does.
- Idempotency key on every side-effecting tool call, derived from the business event
- Persist the decision, not just the input, so retries converge
- A compensating action for each step, or a re-runnable step
- Deduplication window sized to your real retry behaviour, then tested by killing the process
Authority is granted once and forgotten
An unattended agent needs credentials, and credentials get provisioned in the hurry before launch. What it can reach at three in the morning three months later is a question nobody asks until something goes wrong.
Scope per action rather than per system. An agent that needs to read tickets and write a status field should not hold a token that can also close accounts, because the difference between the two is one prompt injection or one confused reasoning step. Least privilege is standard practice everywhere else in engineering and it is routinely abandoned when the caller is a model, on the implicit theory that the model will behave.
Add quantitative limits, not only categorical ones. A spend cap per run and per day. A rate cap on messages. An allowlist of recipients for the first weeks of operation. A maximum number of records modified in one run, above which the agent stops and asks. These caps convert an unbounded failure into a bounded one, and they cost almost nothing to implement compared with what they prevent.
Define an approval threshold in business terms and put a human above it. Refunds under an agreed amount proceed; above it, a person signs off. The threshold is a business decision, it should be written down, and it should be adjustable without a deploy — because the first thing that happens after launch is that someone wants it changed.
- Per-action credentials, not per-system
- Spend, rate and blast-radius caps with hard stops
- A recipient allowlist for the first weeks
- An approval threshold expressed in business units and changeable without a deploy
The inputs will change and nobody will tell you
The demo used a clean PDF exported that morning. Production receives a photograph of a printout taken at an angle, a scan with a staple through the total, and next quarter a supplier who changed their invoice layout without mentioning it.
Input drift is continuous and mostly silent. Upstream systems change field names in a minor release. A partner starts sending a new document variant. Seasonal traffic brings a different mix of requests than the one you tuned on. None of these produce an error; they produce a slow decline in quality that nobody attributes to the cause.
Monitor the inputs, not only the outputs. Track document types, field presence rates, input lengths and source shares, and alert on a step change. A supplier that stops appearing in the input mix has usually changed something, and finding out from a dashboard is cheaper than finding out from a reconciliation three weeks later.
The model drifts too. Hosted models get updated behind a stable API name, and behaviour changes without any deploy on your side. Pin versions where the provider allows it, and re-run the evaluation suite on a version change rather than on a complaint — see how to evaluate a RAG system before you trust it in production.
Alert on the signals that actually move
Error rate is a poor smoke alarm for an AI system, because the characteristic failure is not an exception. It is a confident, well-formatted, wrong result that returns HTTP 200 and looks exactly like a success.
The signals that change before anyone complains are behavioural. Refusal rate: a sudden drop means the system has started answering things it used to decline, which is usually a retrieval or filter regression. Escalation rate: a rise means the agent is meeting cases it cannot handle, and a fall may mean it has stopped recognising them. Tool error rate per tool, which localises breakage to an integration. Output length distribution, which shifts noticeably when a prompt or model changes. Cost per run, which moves when retrieval starts returning more or the agent starts looping. And step count per run, which catches loops before the invoice does.
Each is cheap to emit and each has caught a real regression for us before a user did. Alert on the change rather than an absolute value, because the absolute values are domain-specific and drift legitimately.
Underneath all of it sits the trace: one record per run with the trigger, the inputs, the retrieved context, every tool call with arguments and results, the outputs, the model and prompt versions, the latency and the cost. Three weeks after an incident somebody will ask why the system did that, and there are two possible situations — you answer from the trace, or you reconstruct from logs that were never designed for the question.
- Refusal rate, escalation rate, tool error rate per tool
- Output length distribution, cost per run, steps per run
- Alert on step changes rather than absolute thresholds
- One durable trace per run, retained deliberately and versioned
Roll out in stages, and build the kill switch first
The reliable path from demo to unattended runs through shadow mode, then suggestion, then action with approval, then unattended on a narrow slice. Each stage produces evidence for the next, and each is cheap to reverse.
In shadow mode the agent runs on real traffic and its output is recorded and compared with what the humans did, but never used. This is the most informative and least risky week of the whole project, and it is the one most often skipped because it produces no visible progress. It gives you a baseline against actual human performance, which is the comparison that matters — not the agent's accuracy in the abstract, but whether it is better or worse than the process it replaces, on the same cases.
Then suggestion, where a person sees the agent's proposal and accepts or edits it, and every edit is training signal about where the gaps are. Then action with approval above a threshold. Then unattended, on a defined slice: one region, one product, one category of request, with the rest still going through the previous stage.
Build the kill switch before the first stage. A feature flag that stops the agent taking actions while leaving it drafting them, changeable by an on-call engineer in under a minute, with no deploy. You will use it. Every unattended system we have run has needed it at least once, usually because of something upstream rather than the agent itself.
One last thing, which belongs in the business case rather than the architecture. If an agent handles the routine share of a queue, the humans are left with the residual, which takes longer per case. Modelling the saving as a flat percentage of volume overstates it. The honest model prices the escalation path as a cost of the system, alongside monitoring, the on-call rota and periodic human review of a random sample. Those are not overheads on the project. They are the project.
- Shadow, suggest, act-with-approval, unattended on a narrow slice
- Compare against what humans actually did, on the same cases
- A kill switch that pauses actions without pausing drafting, usable without a deploy
- Escalation, monitoring and review costed as part of the system, not as overhead
Sources
Every regulatory or measured claim above is attributed here. Where we could not find a source we trust, the sentence says so instead of guessing.
- Seven Failure Points When Engineering a Retrieval Augmented Generation System — Barnett, Kurniawan, Thudumu, Brannelly, Abdelrazek — arXiv 2401.05856, January 2024 · robustness evolves rather than being designed in at the start
Questions people ask about this
How long does it take to get from a working demo to unattended production?
It depends almost entirely on the reversibility of the actions and the state of the surrounding systems, so a number given before seeing both would be invented. What we can say is what the phases are — hardening the integrations, idempotency and authority scoping, observability, shadow running, staged rollout — and we scope them as separate fixed-scope phases you can stop after.
Our prototype is 90 per cent accurate. Is that good enough to launch?
The question cannot be answered without knowing what the other 10 per cent does. If the failure is a wrong draft a human reviews, 90 per cent may be excellent. If it is a payment sent to the wrong recipient, 99.9 per cent is not good enough either. Classify the actions by reversibility first, then set a threshold per class.
Do we need a human in the loop forever?
No, but you need one until the evidence says otherwise, and you need one permanently on the actions where a single failure is expensive. The staged rollout exists to move categories of work out of human review as they earn it, individually, rather than making one all-or-nothing decision at launch.
What is the most common thing teams forget?
Idempotency on side-effecting tool calls. It is invisible in a demo, because a demo never times out and retries, and it is the defect we most often find first when we are called in to stabilise an agent that is already live.
Related
Where this connects
AI Agent Development
Agents that take actions, with approval gates, run traces and idempotent retries.
AI Workflow Automation (n8n)
n8n and custom workflows engineered so they do not fail silently at 3am.
AI PoC & MVP Development
A working AI product in weeks, scoped to prove one thing to investors or users.
Read next: How to evaluate a RAG system before you trust it in production · Why voice agents fail on Hindi-English code-switching
More notes on the blog index, and the practical questions about cost, IP and data residency are on the FAQ.
The demo worked. Now tell us what happens at three in the morning
We scope hardening as fixed-scope phases with a decision point at the end of each. Bring the prototype and the volume you expect, and we will tell you which phases you actually need.