Waymo's 220M Miles Prove Evaluation Is Worth More Than the Model

Source: Y Combinator | Published: 2026-08-03T22:42:52Z

With a traffic death occurring every 26 seconds today, Waymo's current scale is preventing roughly one serious casualty every eight days.

Recounting the moment at YC Startup School, Dolgov said it crystallized something for him: the best AI moments in the physical world look like nothing happened at all. Task completed. Everyone safe. Nobody noticed.

Today, Waymo completes roughly 500,000 fully autonomous rides per week, covering over 4 million miles weekly across 15 cities in the US. Drawing on more than 220 million miles of autonomous driving data, Waymo's serious injury and fatality rate is approximately 92% lower than that of human drivers. Behind that number sits nearly two decades of engineering — and a methodology forged in the physical world.

18 Months to Demo, 15 Years to Product

In 2009, the project that would become Waymo launched with a team of roughly a dozen engineers and two goals: drive 100,000 miles autonomously, and complete 10 fixed routes of 100 miles each — every one of them without human intervention from start to finish.

Eighteen months later, they had done it. Day and night, traffic lights, pedestrians, construction zones, highways — all covered. This was 2010, well before the deep learning boom, before Transformers and large language models. The team felt like autonomous driving was "solved."

Then reality showed up.

Getting from "it works" to "it's commercially viable" took another 10 years. Then expanding from commercial launch to 500,000 rides per week took 5 more. The first 100 million miles took 15 years. The second 100 million miles took 7 months.

That curve reveals a core truth about physical AI products: the barrier to entry keeps falling, but the difficulty of the long tail barely moves. Every AI wave — deep learning, Transformers, VLMs — makes it easier to build a demo, and easier for founders to misjudge how far they are from a real product.

Count Your Nines First

Reliability lives on an exponential staircase. The first nine — going from 0% to 90% — is the easiest. Going from 99% to 99.9% is ten times harder. From 99.9% to 99.99%, ten times harder again.

Dolgov's question: how many nines does your product actually need?

A language model assistant might need two or three nines — if it makes a mistake, the user tries again. An advanced driver assistance system needs more. A fully autonomous system operating on public roads with children inside, where the cost of failure is human lives rather than tokens — the number of nines it requires fundamentally determines what technology path you should choose and how much you need to invest.

The problem is that every AI breakthrough sharply lowers the floor for demos while barely touching the difficulty of the tail. This pattern plays out every AI cycle: a wave of breathtaking demos, almost no products that actually work. Dolgov's advice is blunt: count your nines before you count demo views.

Camera-Only Is Possible, But Not Enough

There's a long-running debate in autonomous driving: what sensors do you actually need? Humans drive with their eyes, so in theory cameras should be sufficient.

Dolgov's answer: it depends on how many nines you're aiming for.

If the goal is roughly matching human performance, or building an assistance system, cameras alone are a reasonable starting point. But if the target is full autonomy and meaningfully superhuman safety, the performance curve of weaker sensor stacks flattens out well before it reaches the required threshold.

Waymo uses cameras, lidar, and radar — not as backups for each other, but as complementary physical sensing modalities. Cameras deliver high resolution and color, but degrade in darkness and glare. Lidar directly measures the 3D structure of the surrounding world, generates its own light, and remains sharp in complete darkness. Radar penetrates fog, rain, and snow, and directly measures velocity.

Dolgov showed several scenarios: in a Phoenix dust storm, cameras could barely see anything while lidar clearly revealed a pedestrian standing by the roadside. In a nighttime construction zone, the camera feed was nearly black while lidar picked up two people climbing over a concrete barrier. In another scene — dogs chasing a cow, children chasing the dogs — in a completely unlit environment, only lidar saw the kids.

On hardware cost, he offered a line worth remembering: don't treat today's hardware prices as constants. Waymo is now on its sixth hardware generation, with each generation delivering better performance at significantly lower cost. Betting your company on today's hardware prices means betting on a number that's about to expire.

Rebuilding From Scratch, Every Few Years

Waymo has been through several rounds of core technology replacement: embracing convolutional neural networks for perception around 2013, betting on Transformers for perception and behavior prediction in 2017, and now working with VLMs and frontier world models.

Each time, it wasn't just a new module bolted on — the entire driving system was rebuilt around the new technology wave.

Two traps come with this. First: a new technology proves out but nobody thinks through how to integrate it into the larger system, and it dead-ends — wasted effort, wasted morale. Second: only asking "what new capabilities does this give us," without asking "does it make our system simpler or more fragmented?"

Dolgov's advice is to set the bar for adoption higher: demand not just performance breakthroughs, but radical simplification and unification at the same time. If a new technology makes your system more complex, it's not time to adopt it yet.

The Bitter Lesson, and the Right Use of Structure

There's a concept in AI called the "bitter lesson," from Richard Sutton's 2019 essay: general methods that leverage large-scale compute and data will always beat methods that rely on human-designed structure. Waymo has validated this firsthand through every technology transition.

But Dolgov raises a nuance: structure itself isn't the problem. The issue is the relationship between structure and scale. Structure that suppresses scale always loses. Structure that amplifies scale always wins.

He used Go-playing AI as an example: you could train an end-to-end system that goes directly from pixels to moves, but if your goal is world-champion play, the intermediate representation — the 19×19 board state — doesn't constrain the model. It gives it an extremely efficient lever for scaling.

In the real physical world, no such clean intermediate representation exists, which is exactly why end-to-end learning is necessary. But the physical world has its own structure: the laws of physics, traffic rules, predictable object behavior. Waymo's approach is what they call "structure-augmented end-to-end" — layering concrete structural representations on top of learned representations.

This yields three benefits: you can validate model behavior at inference time instead of operating a complete black box; large-scale training doesn't require running the full sensor-to-decision chain every time, since parts of it can happen in the structural representation space; and you get stronger verifiable feedback signals for reinforcement learning.

Closed-Loop Simulation: As Hard as Building the AI Itself

There are two ways to evaluate physical AI: open-loop and closed-loop. Open-loop gives the system a scenario and asks "what would you do," then scores the answer. Closed-loop actually takes the action, observes what it does to the world, and decides what to do next.

For safety-critical systems in the physical world, closed-loop simulation is mandatory, not optional.

Building a simulation system that's actually useful is roughly as hard as building the AI agent itself. It needs to understand physics, traffic dynamics, and weather. It needs to be realistic enough that capabilities learned in simulation transfer to the real world — and that alone is an extraordinarily difficult problem.

Waymo began building behavioral world models years before that term became fashionable. Now they build for both behavioral fidelity and sensor fidelity, using Google DeepMind's Genie 3 to generate controllable, high-quality scenarios — not just videos, but full simulations capable of running closed-loop driving. They've simulated cars stopped in the middle of a highway, a plane landing on a road, an elephant crossing an intersection, a snow-covered Golden Gate Bridge, and a dinosaur.

This isn't for show. Scenarios that almost never occur in the real world can now be generated at scale in simulation for training and evaluation.

Three AIs and a Flywheel

To make physical AI work at scale, you need to build three AIs simultaneously.

The first is the Agent — the system that acts in the physical world. The second is the Simulator — the virtual environment where the Agent learns and is tested. The third is the Critic — the system that rigorously evaluates the Agent's performance and tells it how to improve.

The good news is that the core reasoning and generation capabilities of all three are shared. In Waymo's architecture, all three are built on the same underlying world model.

Once all three are in place, the flywheel can turn: real-world deployment generates data, data makes the Simulator more realistic, the Simulator generates harder edge cases, the Critic scores the Agent, the Agent gets smarter on harder scenarios, then gets deployed back into the real world to generate more data.

The flywheel can spin in any direction — or just spin in place. What determines whether it accelerates in the right direction is your evaluation metrics.

Evaluation Is the Moat, Not the Model

Dolgov closed his talk with a pointed claim: models are the price of admission; evaluation is the strategic moat.

The best model architectures today are fairly public, and new ideas spread fast. Data matters, but without a strong evaluation framework, you don't know which data is valuable or whether a change actually represents progress.

For physical AI, model-level evaluation isn't enough. You need coverage from the physical layer to the behavioral layer, spanning both onboard and offline systems, along with all operational processes. Waymo calls their framework the "Safety and Readiness Framework." They've spent years building and refining it, and consider it one of their most important assets.

More fundamentally: in the physical world, trust is everything. You can't win trust through flashy demos, slick presentations, or papers. You can only win it by proving, day after day in the real world, that the system is safe. That's why Waymo publishes its safety data and safety research.

Model architectures can be copied. Algorithms can be reverse-engineered. But 200 million miles of real-world autonomous operations, documented evaluation data, and publicly audited safety evidence — those are very hard to replicate.

Every 26 seconds, someone dies in a traffic accident. At Waymo's current scale, they prevent a serious injury or death approximately every 8 days. Dolgov ended by saying this isn't just a number on a dashboard — it means someone's family member walked through the door safely that night.