A Robot Fails, Then Invents a Fix Nobody Trained It On

Source: Y Combinator | Published: 2026-08-12T15:41:58Z

Physical Intelligence's π0.7 general-purpose model folded a shirt on a dual-arm robot it had never trained on, reaching near human teleoperation performance — a task absent from any of its training data.

Before Self-Driving Cars, Every AI Let Humans Make the Final Call

Look back at the past decade of ML commercialization and a pattern emerges that rarely gets discussed: from recommendation algorithms to ad ranking, from deep learning to ChatGPT, every one of these systems has been in the business of advising humans. The final decision always stayed with a person. You can ignore a recommendation. You can review Claude's code before running it. When the system fails, a human catches it.

That means these systems don't need to be anywhere near perfect — just useful enough.

Robots are different. They operate directly in the physical world, where mistakes mean spilled drinks, scrapped parts, and injury risk. Making a robot genuinely useful isn't about getting it right most of the time — it's about keeping it highly reliable, unsupervised, over extended periods. Waymo showed this is achievable: over 250,000 driverless rides per week, setting a reference point for the entire industry that autonomous AI in the physical world can actually work.

Let the Robot Find Its Own Failure Modes

The standard ML loop — collect data, train model, evaluate — requires humans to drive every iteration. Humans get tired. They cut corners. Pushing reliability past 99% this way is genuinely hard.

PI's approach is to let the robot drive the loop itself.

In practice, they built a modified reinforcement learning system: the robot attempts tasks repeatedly in real environments, and when problems arise, the system automatically identifies which scenarios need more data. Humans step in at key decision points, not to supervise every moment. Because most iterations are automated, you can run far more of them.

RL is mature territory for language models — PPO, GRPO, and similar algorithms have run hundreds of millions of attempts on LLMs. But porting that logic to robots hits a fundamental cost wall: running an LLM in a data center burns compute. Running a robot in the real world burns time, hardware, and labor. A one-minute task, run a million times, equals 700 robot-days.

Don't Let Robots Waste Time in Dead Ends

PI made two key improvements to address this cost problem.

The first is active trajectory truncation. In a box-folding task, robots sometimes pick up two stacked boxes and try to fold them as if they were one — a guaranteed failure. Letting the robot keep trying is pure waste. PI's solution is to have a human step in remotely at that exact moment, reset the robot's state, or terminate the episode and restart. Not one second wasted on a dead end.

The second is training a general-purpose "value function" — a model that judges whether the current state is good or bad. Traditional RL runs 10 to 50 attempts on the same task to estimate which approach is better; that cost is unacceptable with physical robots. PI instead trains a general evaluation model on large amounts of video data, teaching it to recognize whether a given action is moving the task forward or backward, then transferring that judgment to new tasks. Accidentally unfolding a shirt while trying to fold it? The value function flags it. Smoothly retrieving something from the fridge? Value function approves. One model, two completely different contexts.

Combined, these improvements produced a 2x throughput gain from the RL phase on the box-folding task. Espresso-making success rate: above 90%.

Thirteen Hours Next to an Espresso Machine

The latte task is harder than it sounds. Inserting the portafilter requires precise force control. Carrying a nearly full cup demands extreme stability. Timing — how long to wait for extraction, when to pour the milk — doesn't have the kind of clean solutions that visual recognition does.

PI didn't run this once to check if it worked. They ran the same policy for 13 hours, measuring how long the robot could keep going without an error. The conclusion: this robot is reliable enough for real deployment. No one needs to stand next to it.

Then they took the same algorithm to Dandelion Chocolate, a chocolate factory a few blocks from their office. Workers there fold boxes, apply labels, and stack pallets as part of their daily routine. PI's robot replicated that real-world workflow directly, trained with the same RL recipe — and hit the same industrial-grade reliability.

Remembering the Last Step Is What Makes a Complete Task Possible

Nearly every mainstream robot foundation model today has no memory. They only see the present — whatever the camera captures right now, with no relationship to what happened a second ago.

For repetitive tasks, that's fine. Making one espresso after another, each action nearly identical, there's nothing to remember. But for a multi-step task where order matters, no memory means no direction.

PI designed a multi-timescale memory system for this. Short-term memory stores roughly 10 seconds of video, but uses efficient compression rather than feeding hundreds of thousands of tokens into the model — at 50Hz sampling with four cameras and 256 tokens per frame, 10 seconds of raw video amounts to roughly 500,000 tokens, which is impossible to process in real time. Long-term memory uses text: the past 10 to 15 minutes get summarized into language, compressed, and fed into the model.

With this system, they had the robot complete a 10-to-15-minute kitchen cleaning task: wipe the counter with a sponge, dry it with a paper towel, throw the towel away, return the mustard to the fridge, put dishes back in the cabinet, wash the dirty bowls. Every step different, order non-negotiable, fully autonomous from start to finish.

One Model Does Everything — and Matches Specialists

Integrating all of these capabilities into a single generalist model is PI's primary focus right now. They call it π0.7.

Training data includes everything usable: high-quality teleoperated robot demonstrations, lower-quality autonomous robot rollouts, human operation videos, and image-text data from the web — extremely diverse sources, wildly varying quality.

The key to making the model digest this heterogeneous data is a detailed prompting mechanism. Beyond telling the model "what to do," it also receives the current subtask instruction ("specifically what to do in this step"), metadata about data quality and episode length, and a subgoal image of what the state should look like a few seconds ahead.

Adding metadata prompts produced a counterintuitive finding: without prompts, adding low-quality data to the training set degraded performance. With metadata prompts, the same low-quality data improved it. The metadata gives the model a frame of reference — it knows "this data is lower quality, weight it accordingly" — rather than treating good and bad examples identically.

The benchmark results surprised even the team: π0.7, a generalist pretrained model, matched or exceeded the throughput of task-specific models fine-tuned with RL for a single task — on both the latte and box-folding benchmarks. All tests were run without any post-training.

The Robot Folded a Shirt with Arms It Had Never Used

The most valuable capability of a generalist model isn't doing known things well. It's handling novel combinations well.

PI ran two tests. The first: an air fryer. The training data had almost no air fryer content — they later found three clips in the dataset, essentially nothing. The robot could open it, place a sweet potato inside, and close it. It combined "operating a rotational container" with "handling food items" on a device it had never encountered.

The second test was more extreme. PI has a dual-arm UR5e industrial robot system. They had accumulated extensive cloth-folding data on a different robot, but none at all on the UR5e. The arm dimensions, joint configurations, and link lengths are completely different.

When the team first saw the dual-arm UR5e fold a shirt on its own, everyone stopped and stared.

The robot wasn't fast, and it needed a few tries at times — but it completed the fold. Capability transferred from one platform to another that had never been trained on it. Quantitative data shows π0.7 on the dual-arm UR5e folding towels and shirts performs close to the level of human teleoperation.

The Robot Picked Up the Wrong Part — and Figured Out the Fix

The cross-platform generalization was already surprising. But there's a detail worth closer attention.

While training a windmill assembly task, all training data was recorded with a fixed strategy: right hand picks up the pin, left hand holds the paper, pin goes into the hole. The robot was learning this pattern. Then one attempt went wrong — the paper ended up on the right, the pin on the left.

The robot didn't freeze. Didn't restart. Didn't ask for help. It picked up the pin with its left hand and inserted it into the paper held in its right.

This action appeared in no training data — not in the task-specific fine-tuning data, not in pretraining. The model inferred the left-right symmetry on its own and applied it in the moment, improvising under error conditions. This wasn't trained behavior. It emerged.

The Robot World's "ChatGPT Moment" Won't Arrive on a Single Day

Someone asked: how far away is the ChatGPT moment for robotics?

Sergey's answer: the capability threshold is approaching, but the deployment model means it won't happen overnight. ChatGPT reached a million users in five days because all it needed was a computer and a browser. Robots need physical hardware, and deployment speed is fundamentally constrained by how fast hardware can be manufactured and installed. Waymo has been expanding quickly — and it still took years.

Two YC companies — Ultra and Weave — are already deploying PI's models in real environments: folding clothes, packing in warehouses. The same model foundation has been adapted to drones, quadrotors, surgical robots, and tractors.

Someone asked whether it's possible to break into robotics from a software engineering background. Sergey gave a concrete example: Jenny, who had worked in algorithmic trading and then legal tech at Harvey. She got interested in robots, bought a cheap one, tinkered with it in her bedroom, fine-tuned open-source models on it, and tried to get it to do something useful. Then she sent Sergey a cold email. She now works at Physical Intelligence.

PI's π0 and π0.5 models are already open-source and available for fine-tuning. Sergey's advice: start building. You don't need to wait until you feel ready.