Memory Gives Robots the Ability to Change Strategy After Failure

Source: Y Combinator | Published: 2026-08-08T14:00:16Z

When a robot fails to grasp with chopsticks, a memory-equipped system will proactively adjust its target position and retry. Stanford researchers say this adaptive recovery is the rarest capability in today's robotics.

When AlphaGo dropped, people said: we have the algorithms now, next year it'll happen. When MuJoCo arrived, you could train a walking robot in 3,000 iterations — next year. When Aloha came along with teleoperation data — next year. In early 2026, the diffusion policy paper landed, with multi-step reasoning on top — still next year.

It is now July 2026. You still cannot buy a Figure robot. Rosie still does not exist.

Four Walls

The host distills the core obstacles blocking robot deployment into four categories. Not one has been solved.

Physical world modeling. Current video world models don't obey physics. Drive a car into a supermarket in simulation and the car magically becomes a road and keeps moving — no crash. The sim-to-real gap is still very much there.

Sensorimotor problems. Human skin is blanketed in nerve endings that sense normal force, shear force, humidity, temperature, vibration, and friction coefficients. You can reach into a backpack and identify your charger without looking. When your fingers go numb in the cold, your ability to unlace ice skates collapses — that's sensory degradation causing policy failure. The best robots today have one force sensor per fingertip and a wrist camera. That's it.

Embodiment drift. Actuators accumulate dust and corrosion over time — especially rust in coastal environments. Press the throttle and the actual output force is a variable, one that drifts continuously. Battery discharge curves shift too. Data was collected from a specific physical state, but that state keeps changing, making teleoperation data progressively stale.

Memory Is What Enables Long Tasks

Stanford PhD student Marcel's internship work at Physical Intelligence points to the central missing piece for long-horizon tasks: memory.

A robot without memory will wash the same dish forever, or burn a grilled cheese to charcoal, because it has no concept of time — no concept of what it has already done. Pi0, GR00T, and other mainstream policy models have no memory mechanism. At every timestep, the robot receives a new observation and completely forgets what just happened.

MEM (Multiscale Embodied Memory) addresses this with a two-tier architecture. Short-term memory handles fine manipulation: a ViT-based architecture with temporal attention layers that compress recent frames into tokens fed into the low-level policy. Long-term memory handles task progress tracking: a high-level policy continuously predicts a text "memory note" describing what has happened so far, then feeds that text back into itself, forming a loop.

Text is far more token-efficient than images and easier to debug — you can literally read what the robot is "thinking."

With memory came a property no prior policy had exhibited: in-context adaptation. A robot without memory fails to grasp with chopsticks and repeats the exact same wrong position every time. With memory, it fails once and adjusts lower on the next attempt. Fails to open a fridge and tries the other side next. Marcel considers this among the scarcest capabilities in robot policy today.

What Should Robots Reason About

Stanford PhD student Milan, working at Waymo, asked a precise question: if you make a robot "think" before acting, what exactly should it think about?

There's no obvious answer. List every object in the scene? Predict the next plan? Track gripper position? Some of these help. Some actively hurt. The key insight: not all sensible reasoning helps with action.

Their method, R&B Encore, treats reasoning content as a latent variable and uses a variational inference framework to score and filter it. Three criteria: conciseness, non-triviality (encouraging generalization rather than rote repetition), and most importantly — action predictiveness (does this reasoning actually help the model predict better actions?).

On robot arm manipulation, they found movement reasoning and gripper position reasoning useful, but perceptual reasoning (listing scene objects) harmful — too many distractors in the scene. On autonomous driving, plan reasoning and meta-action reasoning were useful, while counterfactual reasoning was almost always redundant — there's nothing worth reasoning about counterfactually when you're driving straight.

On the latency problem, they introduced "action forcing": reasoning is used only during training. At inference, the reasoning steps are dropped and only the action head is kept. No latency at deployment, but the gains from text annotation during training remain.

Dexterous Hands Without Teleoperation Data

Tyler Lum's system sidesteps the teleoperation dependency that every other approach relies on.

SimTool Real trains entirely in simulation using massive parallel RL — tens of thousands of robots running simultaneously, roughly 1,000× real-time. Days of compute, equivalent to decades of human operational experience.

The key design: reduce all dexterous tool manipulation to a target pose tracking problem. The policy doesn't need to know what task it's doing. It only needs to move an object from its current pose to a target pose. Training uses randomly generated geometries (cylinders, cuboids) with random target poses. Inference uses human video — FoundationPose and SAM automatically extract the target pose sequence, and the policy tracks it.

Human video provides no robot actions, is not used for training or fine-tuning, and serves only as the source of target pose sequences.

The system simultaneously controls a 22-DOF hand and a 7-DOF arm at 60Hz. Tested zero-shot on 12 unseen tools, it showed significant progress across the board, particularly on long-handled tools. About 60% of failures came from pose tracking failure, not the policy itself.

Recovery behaviors — like immediately re-grasping a dropped hammer and completing the task — were not explicitly trained. Random external forces were applied to objects during simulation training to keep the robot on-task through disturbances. Recovery emerged naturally.

LSTM Beats Transformer?

Someone asked Tyler why he was using "ancient LSTM."

His answer: RL is extremely brittle. Once a version works, you don't touch the codebase. The existing codebase had LSTM, they tried it, it worked better, so they kept it.

He also noted that at larger dataset scales, Transformers may have the edge — they're better "data sponges." But in an RL setting where the policy updates continuously, dataset sizes aren't large enough for that advantage to show up yet.

The diffusion policy paper's results table shows CNN outperforming Transformer on more than half of tasks. Transformers are more sensitive to hyperparameters and, without careful tuning, aren't necessarily better.

Robot Application Companies — The Next SaaS

Rerun CEO Nico gave the most pragmatic advice of the evening.

The pattern he's actually seen work: not building a general-purpose robot, but taking end-to-end ownership of a specific business problem. Industries currently getting traction: data center management, warehousing, manufacturing, food.

Start from the simplest possible point — one customer, teleoperation, an off-the-shelf arm. Don't worry about not having a foundation model. Don't worry about generalization. If you can do it with teleoperation, you can usually train a model to do it.

He used a hypothetical "paper airplane factory" to illustrate what you'd actually learn: how many planes per day it takes to be commercially viable; whether defective planes can be detected and rejected; that what customers actually care most about is how fast you can launch new designs; that adding a paper tray reduces failure rate by 50%; that teleoperation operators need 20 hours to reach customer-required proficiency — which means you have to recruit and train them yourself, no outsourcing; and that your cheap robot arm will break and you'll need to switch suppliers.

None of this is discoverable in a lab. All of it requires actual deployment.

His argument: the timing to build these companies is far better now than two years ago. More arm vendors on the market, foundation models meaningfully stronger, and you don't need nine figures to get started. Friction in the physical world is real — which means moats are real. General model companies won't go out and learn what you learned at a customer site.

Starting the Data Flywheel

One trait Nico has observed across every robot company that's actually shipped: they all have replicas of customer environments in their offices for high-volume testing.

The most common early data collection mistake: collect a lot first, then start training. The right approach is to train and collect in parallel. Problems in the data surface early in training. Wait until later and you've already logged hundreds of hours of bad data.

When researchers at large companies debug data, the most common conclusion isn't "change the model" — it's "send new instructions to the data collection team and have them operate differently." That feedback loop has to exist during collection, not after.

World-Action Models vs. VLAs

General Instinct's Bill and Guanming, both with DeepMind backgrounds, are building infrastructure to accelerate world-action model inference.

VLAs (Vision-Action models) predict current actions from current observations — no explicit future dynamics learning. World-Action Models (WAMs) add a diffusion transformer that simultaneously predicts future frames and future kinematics. The core hypothesis: letting the model see "if I take this action, what will the world look like" helps it predict actions more accurately — because it's teaching the model physics at the pixel level.

The problem is speed. DreamZero requires two GB200 GPUs to run. About $70,000 each.

Their optimization path: knowledge distillation on the VLA encoder-decoder, distillation on the diffusion transformer, splitting the original shared DiT into two independent transformers (a video transformer and an action transformer connected via cross-attention), skipping full future frame decoding and using only the video transformer's hidden states as the conditioning signal. The most critical step: autoregressive flow matching distillation, compressing from 50–100 steps down to 1–2, yielding roughly 50× speedup.

After optimization, the world-action model runs on a Jetson Orin at 500ms per chunk (16 action frames).