AI Reliability: Lessons From 140M Miles

AI reliability is the probability that a system delivers a usable result under real conditions, including when parts of it break. It is not the same as model accuracy, and the distinction is the whole subject. A model that is right 95 percent of the time inside a system with no defined behavior for the other 5 percent is a less reliable product than a weaker model wrapped in retries, fallbacks, and a human escalation path.

Two companies have more evidence on this than almost anyone, because they have been shipping autonomous systems into the physical world for a decade while most of us were shipping web apps.

Zipline says it has flown more than 140 million autonomous commercial miles and completed more than 2.5 million deliveries with no safety incidents, figures the company restated publicly this month. Waymo has published data covering more than 220 million fully autonomous miles.

Both companies did improve their models, continuously and at considerable expense. The point is that the models were necessary and nowhere near sufficient. What produced those numbers was assuming every component would fail and designing for the day it did.

What follows is what transfers from that work to a software startup, including the parts that do not.

What AI reliability actually means

Four things get called reliability and they are different problems, which matters because they have different owners and different fixes:

• Evals ask whether the output is good. Covered in LLM evals for founders.
• Testing asks whether the software still works after a change. Covered in AI testing in the AI coding era.
• Observability asks what actually happened in production. Covered in LLM observability.
• Reliability asks whether the system keeps producing a usable result when something in it fails.

The first three are inputs to the fourth. You can pass all of them and still have an unreliable product, because reliability is decided in the parts of the system nobody demos: what happens when the model API returns a 503, when a tool call times out, when a retrieval index is stale, when a downstream service rate limits you at exactly the wrong moment.

Waymo co-CEO Dmitri Dolgov made the point directly in a talk at Y Combinator's Startup School, speaking about systems that act in the world: "for physical AI agents, model-level evaluation is not enough." His team evaluates every component of the system, on board and off, plus the operational processes around it. For a software startup that translates almost exactly. Your eval suite measures the model. Your reliability work measures everything else.

Count your nines before you count your demo views

The most useful framing in Dolgov's talk is an argument about arithmetic.

"A working demo is 1% at best of the work that you have to do," he said. The reason is that "reliability and performance lives on this exponential ladder of nines. So, getting to that first 90% or 99%, that's the easy part. But then every next nine that you want to add, that takes about 10 times more effort."

Which produces the instruction most teams skip: "you need to know up front exactly how many nines your product actually needs."

That is a scoping decision, not an engineering one, and it belongs at the start because it dictates the architecture. Dolgov's own rough scale: a demo might need one nine, a copilot a few, and a fully autonomous agent operating in public "needs a whole stack of them."

The reason the target has to come first is that each nine is a different technique, not more of the last one. "Getting those next nines means doing something different every time," he said, and his own escalation runs from ordinary engineering and bug fixes at the start to fully redundant systems and tiered backup architectures later. Translated to a software stack, my own reading is that the early nines come from careful engineering and retries, and the later ones force you into fallback models, degraded modes, and redesigning the flow so failures are survivable. Each of those is a different piece of work. Picking the architecture for two nines and later needing four means rebuilding, not tuning.

Zipline's founders describe the same curve in money. Co-founder and CEO Keller Rinaudo Cliffton says the company signed its first Rwandan contract at 30 dollars per delivery and launched at an actual cost of 300 dollars, then spent years walking it down through 120, 75, 40, 28, 18, to about 12 today outside the US. His advice to hardware founders is to take your estimate and "assume it's going to cost 10 times that," then ask whether the business still works. The reliability equivalent is the same question asked about effort.

Dolgov's warning about what happens when teams skip this is worth quoting in full, because it describes the current market: "the recurring mistake of every cycle is spending on the demo when you should be saving for the nines." Every capability jump makes demos dramatically easier and moves the tail much less. His line for it: "count your nines before you count your demo views."

Pick a benchmark that embarrasses you

Zipline's original safety goal was to be 10 times safer than cars. In a board meeting, Sequoia partner and Zipline board member Alfred Lin told them that was a bad goal and the right target was two times safer than Waymo. Rinaudo Cliffton's summary of the reasoning: cars are archaic technology. The team reset the target, and Zipline's stated goal for the end of this year is to be twice as safe as Waymo.

Two things are worth copying here, and one is worth checking.

The mechanism is the copyable part. They benchmarked against the best system currently operating rather than against the baseline they were replacing. Beating the incumbent baseline is the goal that lets you stop early, because the incumbent is usually bad. In AI products the equivalent baseline is the manual process you are replacing, which is slow and error prone and therefore easy to beat. Benchmark against the best automated system in your category instead, and the target stays uncomfortable long enough to be useful.

The second is that the target was specific, external, and dated. Not a vague instruction to improve reliability, but a multiple of a named system by a named date, which is something an engineering team can be held to.

The part worth checking is the number itself, because an external benchmark is only as good as your reading of it. In the conversation, someone puts Waymo at roughly 10 to 12 times safer than human drivers. Waymo's own published safety data, through March 2026 and covering 220.6 million rider-only miles, reports 94 percent fewer crashes causing serious injury or worse than human drivers in the same areas, which works out closer to 17 times, and Dolgov used the 17 times figure in his own talk.

Both numbers are defensible, which is the actual lesson. Waymo's multiple depends entirely on which metric and which release you pick: 94 percent fewer serious injury crashes is about 17 times, 82 percent fewer airbag deployment crashes is about 6 times, and Waymo's own earlier report of 92 percent at 170 million miles is about 12. A target pegged to a multiple of an incumbent is therefore only meaningful if you say which of the incumbent's numbers you mean and when you last checked it. Otherwise the goal drifts without anyone deciding to move it.

Assume every component fails, including the one watching for failures

Eric Watson, who leads systems engineering, certification, and system safety at Zipline and designed satellites before that, states the design principle plainly: "you need to assume that any part of the system can have a fault, can have a hiccup, something can go wrong. And that's how you really design something to be robust, reliable, and safe."

The implementation is instructive because of how far it goes. Zipline's aircraft carry two flight computers, both of which behave as though they are flying the aircraft, both receiving sensor data and both sending commands to the actuators. A third small computer arbitrates, monitoring the health of the two and telling every other node on the aircraft which one to obey.

Then the question that separates real redundancy design from the diagram version: what happens when the arbiter fails? Watson's answer is that the aircraft keeps flying on whichever computer was already in charge and completes its mission. The monitor failing does not take down the thing being monitored.

This is not exotic. As Rinaudo Cliffton points out, a second flight computer is how a Boeing 777 works. What Zipline had to do was rebuild it out of smartphone supply chain components: "You can achieve similar levels of safety to traditional aerospace, but you can move 100 times as fast at 1/100 of the cost."

The software translation is direct, and most AI products fail it:

• Two independent paths for anything that must not stop. A primary model and a fallback from a different provider, with the switch tested rather than assumed. Not a retry against the same endpoint that is already down.
• A health check that cannot take the system down when it fails. Circuit breakers that fail open, monitoring that degrades to permissive rather than blocking, a rate limiter that lets traffic through if its own backing store is unavailable. The monitor going down should not be an outage.
• A defined degraded mode. Cached results, a smaller model, a simpler answer, or an honest message and a human handoff. Anything except an unhandled exception.
• A last resort that is safe rather than optimal. Zipline's is a parachute. Yours is the state the user is left in when everything else fails, which should be recoverable and legible rather than a spinner.

Watson mentions a failover firing in production shortly before the recording: after a delivery, the main flight computer hiccuped, the backup took over, and the aircraft flew itself home and landed normally. That is what a reliability investment looks like when it pays out. Nothing happens, and almost nobody notices.

Test to failure, not to pass

The testing philosophy is the part I would push hardest on any founder, because it inverts the default.

Zipline runs highly accelerated life testing on components, putting a part such as a motor through heat, humidity, corrosion, and UV while it is running, and separately puts full aircraft through vibration tables, wind tunnels, and thermal chambers. They test at 49 degrees Celsius because they fly in Phoenix in summer, and at minus 25 because they fly in the northern US.

The goal is not a pass. Watson describes it as wanting "to understand is how is this going to break, right? More than just is it good enough? Like, we want to know how it's going to break." And explicitly: "we don't just want to say, 'We ran the test campaign and nothing failed. We're done.' It's like, 'No, no, let's take this thing to failure, right? Let's see where the limits are.'"

A green test suite tells you the system works in the cases you thought of. Knowing where it breaks tells you how much margin you have, which is the number that predicts production behavior.

For an AI product, testing to failure means deliberately running the conditions you are hoping not to hit. Send inputs ten times longer than your longest real one and find where quality falls off a cliff rather than degrades. Run at five times peak concurrency until something queues or drops. Kill the vector database mid request. Return malformed JSON from a tool call. Feed the ambiguous, adversarial, and deliberately contradictory prompts you have been steering around in demos. Then write down the limit you found, because that number is your actual capacity, not the one in your load test summary.

There is a related point about suppliers that lands hard for anyone building on model APIs. Watson says that when he asks people in other industries how they think about reliability, a common answer is that they ask the supplier what the reliability of the part is. His response: "what if we're the supplier, you know?" You are the supplier now. Your customers experience your provider's outage as your outage, and a provider incident is an explanation rather than an excuse. Reliability you have not measured yourself is reliability you are taking on faith.

The model is not the system

The single most transferable number in the Zipline conversation has nothing to do with drones.

The company launched in 2016 convinced the aircraft was the product. It signed a contract to serve 21 hospitals and served one for the first nine months. What they learned, in Rinaudo Cliffton's words, is that "the drone is 15% of the complexity of the solution." The other 85 percent was inventory management, integration with a national civil aviation authority, integration with a national healthcare system, ordering, demand management, and maintenance. Watson's framing of the company is the same point: "we're not a software company, right? We're a real world AI robotics company."

Substitute your own noun. Zipline's 15 percent is a figure about drones and I am not claiming it measures software, but the shape holds for every AI product I have seen: the model is the small visible fraction, and the rest is data plumbing, retrieval quality, permissions, error handling, observability, evaluation infrastructure, the human review path, and the operational work of running it. That is where reliability is won and lost, and it is consistently the part that founders defer because it is less interesting than the model.

There is a counterweight in the same conversation that keeps this from becoming an argument for building everything. Rinaudo Cliffton's other principle is deletion: "The most reliable part on an aircraft is the part that is not on the aircraft at all because you deleted it in the last design. That part will never fail." Paired with questioning every requirement, this is the discipline that stops redundancy from becoming sprawl. Every component you add is a component that can fail, and the cheapest reliability work available to most AI startups is removing a step from the pipeline rather than adding a monitor to it.

Both things are true at once: the system is much bigger than the model, and it should be as small as it can be while still meeting its number of nines.

Stage autonomy by ratio, not by feeling

Zipline did not switch from supervised to autonomous. It moved a ratio.

Its first US regulatory permission required one remote pilot per aircraft. Rinaudo Cliffton describes the progression: "we went from one to one to one to three, one to six, one to 20, one to 40, and now operating one to 100, and have plans to go well beyond that." The people doing it are no longer called pilots. They are fleet commanders, and the term appears in Zipline's FAA documentation.

This is the best available answer to a question every agent startup is arguing about internally, which is when to take the human out of the loop. The answer is that you do not take them out. You change the ratio, and the ratio is the metric.

One reviewer per output is where you start. As your measured intervention rate falls, one reviewer covers five, then twenty, then a hundred. The rate at which humans have to intervene is your reliability number expressed in a form the business can act on, and it tells you your unit economics at the same time. Rinaudo Cliffton is blunt that one human watching one aircraft was "not great for unit economics," which is exactly the position a startup is in when a person checks every agent output.

Two conditions make the ratio move honestly. Interventions have to be logged and categorized, or you cannot tell whether the rate is falling because the system improved or because reviewers stopped looking. And the humans have to keep a real ability to intervene at the higher ratio, which means the system surfaces the cases that need attention rather than requiring someone to find them. Designing for agent operators is the same problem viewed from the interface side.

The scale trap sits underneath this, and Watson names it: "if you have a one in a million situation, it's going to happen every single day" once you are doing a million deliveries a day. Dolgov says the same thing about miles: "a rare event that might happen once in a million miles, that just becomes your daily reality." Every failure mode you have dismissed as too rare to handle is a scheduled event at scale. Growth does not just add load. It promotes your entire tail into routine operations.

Design the failure case to be boring, and then talk about it

On September 4 this year, a Zipline aircraft on its way to pick up an order in northwest Harris County, Texas, hit what the company called a midflight issue and deployed its parachute, coming down in the southbound lanes of Highway 6. KPRC 2 in Houston reported that local law enforcement received no calls about it, and that according to the company nobody was hurt and no property was damaged. The cause had not been determined, and the company said it did not believe weather was a factor despite storms in the area that day. Zipline's statement noted that each aircraft carries multiple layers of safety backups and "runs more than 500 safety checks each second while in flight."

I include this because it is the honest version of everything above, ten days before this was written and reported by a local news station rather than a press release.

The last resort fired, in public, and the aircraft came down under a parachute rather than falling. That is the system working as designed. It also came down in a live traffic lane, which is not a good outcome, and the underlying fault had not been explained. Both readings are correct, and a post that reported only the first would be marketing.

It is also worth being precise about what the zero in "zero safety incidents" counts. That is Zipline's own figure under Zipline's own definition, and the company classes a paraland as the redundant safety system doing its job rather than as an incident. That is a reasonable engineering position and it is not a neutral one. When you publish a reliability number, you are also publishing a definition, and the definition is the part a careful reader will check first. Say what yours excludes before someone else does.

The founder lesson is the one people skip: your failure case will eventually run in front of customers, so design what it looks like from outside. When your agent fails, does the user see a clear message and a path to a human, or a silent wrong answer they act on? Does your team find out from monitoring or from a customer? Is there a log good enough to explain the cause afterward, or will you be saying that the cause has not been determined?

Dolgov's version of this is about what you do afterward: "in the physical world, trust is everything," and you earn it by publishing your safety data rather than proving it to yourself behind closed doors. "You earn it gradually, day by day," he said, by proving in the field that the system works. Waymo publishes its crash data, which is what makes its numbers checkable at all, and is why a section of this article could compare its figures against what was said about them on a podcast. Published reliability numbers are still rare among AI startups. The first ones in a category to publish real ones, including the bad quarters, will own a trust position competitors cannot copy with a landing page.

The security half of reliability

Reliability and security are the same discipline pointed at different causes. Both start from the assumption that components fail and that the system must hold anyway. The difference is that one failure is random and the other is chosen, and an attacker will find your unhandled path faster than your users will.

This is where redundancy stops being automatically good. Every fallback is another path into your system, and fallback paths are consistently the least reviewed code you own:

• A fallback model is another data processor. If your primary provider has a no-training agreement and your fallback does not, your data protection posture silently changes during exactly the incidents when nobody is watching.
• Degraded modes are where authorization gets skipped. The classic incident is a cache or permissive mode that fails open on data access rather than on rate limiting. Decide deliberately which checks may fail open and which must fail closed, and write it down. Availability checks can usually fail open. Authorization never should.
• Retries multiply side effects. Any retried operation that is not idempotent can double charge, double send, or double write. This is a correctness bug, a security finding, and a support burden at once.
• Failure messages leak. The error path is where internal hostnames, stack traces, prompts, and occasionally other customers' data surface, because it is the path nobody designed.
• Rare paths get stale. Code that runs once a quarter does not get patched, reviewed, or tested on the same schedule as the hot path. Exercise your fallbacks on a schedule, in production, deliberately.

The test that catches most of this is simple to run and uncomfortable to answer. Take each failure mode and ask what an attacker who could trigger it on demand would gain. If the answer is anything other than a slower or simpler response, that failure mode is a security control rather than just a reliability one, and it needs the same review.

What this looks like in a software startup

The honest limit of these lessons is that hardware forces a rigor software does not. A drone that fails falls out of the sky, so the regulator, the physics, and the customer all enforce the discipline. A failed API call returns a 500 and someone retries. Nothing forces you to do any of this, which is precisely why it does not get done until a customer is lost.

The techniques that do transfer, in the order I would do them:

1. Role separation. The component producing an output should not be the component judging it. Factory's Matan Grinberg describes routing by task and giving reliability-critical work a split pipeline: "we really care about reliability, so let's generate the code with OpenAI, test it with Anthropic, review it with like Gemini." Different models fail differently, which is the entire value of the arrangement. One model checking its own work correlates its errors.
2. Explicit failure behavior on every step. Every model call, tool call, and retrieval gets a timeout, a defined number of retries with backoff, and a defined result when retries are exhausted. No step may hang.
3. Reversibility on anything consequential. Actions that move money, send communications, or change production state get a confirmation, a reversal path, or a human gate. A wrong output should cost a correction, not an incident.
4. A measured intervention rate. Log every human correction with a category. That number is your reliability metric and the input to your autonomy ratio.
5. Scheduled failure exercises. Turn off the primary provider in production on purpose, during business hours, with the team watching. An untested fallback is a hypothesis.

What to do this week

1. Write down how many nines your product actually needs, and what the consequence of one failure is. If you cannot state the consequence, you cannot justify the target.
2. List every external dependency in your critical path: model APIs, vector stores, tool integrations, your own services. For each one, write what happens when it fails. Every blank is an outage you have not designed.
3. Pick your single worst dependency and give it a defined failure behavior this week: a timeout, a fallback, or an honest degraded response. One is enough to start.
4. Run one test to failure rather than to pass. Ten times your longest input, or five times peak concurrency, and record the limit you find.
5. Instrument your human intervention rate, with categories. Without it you are guessing at both reliability and unit economics.
6. Audit your fallback paths for the security questions above, especially anything that fails open on authorization.
7. Schedule the first deliberate failure exercise and put a name against it.

The pattern under all of it is that reliability is an architecture decision made early, not a quality you add later by tuning a model. Building that architecture, and the operating discipline that keeps it honest as you scale, is what the AI Operating System for Startups course is about. The AI for startups guide is the free overview of how evaluation, agents, and operations fit together, and starting a robotics company covers the cost structure when your reliability problem is physical.

Sources

• Inside Zipline's Autonomous System: 140M Miles, Zero Incidents, Sequoia Capital's Training Data podcast, hosted by Alfred Lin and Pat Grady. The interview this article distills, and the source for the compute failover architecture, the testing philosophy, the 15 percent figure, the autonomy ratio, and the Waymo benchmark story. Guests: Zipline co-founder and CEO Keller Rinaudo Cliffton and Eric Watson, who leads systems engineering, certification, and system safety.
• Waymo Co-CEO Dmitri Dolgov: The Demo Is Only 1% Of The Work, Y Combinator. The ladder of nines, the demo to product gap, the long tail at scale, and the argument for publishing safety data.
• Waymo's Safety Impact hub, for the published figure of 220.6 million rider-only miles through March 2026 and 94 percent fewer serious injury crashes than human drivers in the same areas.
• Food delivery drone crashes onto Highway 6 in NW Harris County, KPRC 2 Houston, September 4 2026, for the paraland event, Zipline's statement, and the reported absence of injuries or damage.
• Every CIO will have to answer for every token, Sequoia Capital, with Matan Grinberg of Factory, for the split generation, testing, and review pipeline.
• Zipline's safety fact sheet, the company's own account of its redundancy and safety systems. Note that it carries an earlier figure, more than 135 million commercial autonomous miles as of November 2025, so the 140 million used above comes from Zipline's September 2026 statement to KPRC rather than from this page.