NeurIPS Called It 'Negligible Impact' — Now Every AI Lab Uses It

Source: Y Combinator | Published: 2026-07-30T21:03:53Z

Jeff Dean's distillation paper was rejected by NeurIPS for having negligible impact. It's now standard practice at virtually every major AI lab.

He's already looking ahead to 2027: ML systems will begin autonomously improving their own capabilities. Not by having humans run experiments, review results, and iterate — but by systems that decompose problems themselves, run experiments, and synthesize results into a closed-loop, automated scientific method.

The Big 2027 Prediction: Machine Learning Starts Improving Itself

Jeff Dean's thesis for 2027: ML systems will increasingly use automated experimentation to enhance their own capabilities.

The mechanism works like this — a high-level goal comes in, the system breaks it into subproblems, runs massive experiments on each, finds optimal solutions, then composes those solutions into a complete answer. He describes the process as having "extremely low latency in the experiment loop" — not months per cycle, but potentially hours.

This logic isn't limited to machine learning. He believes any domain where you can define a measurable objective — materials science, chip design, quantum chemistry — can be accelerated with this framework. The prerequisite is building a fast-enough validator.

A colleague demonstrated this a decade ago. A density functional theory simulator that took an entire night to produce results for a single molecule was used to train a neural network approximation on its inputs and outputs. Speed improved by 300,000x with almost no loss in accuracy. Screening 10 million candidate molecules that would have taken six months could now be done over a lunch break.

The Napkin Calculation That Built the TPU

In 2013, Google's speech recognition system had just switched from older algorithms to deep learning, cutting error rates in half — compressing roughly 20 years of traditional progress into a few months. Jeff Dean and his team saw the implication clearly: if speech recognition worked well, people would use it more.

So he did the math on a napkin: if every Google user used speech recognition for just three minutes a day, how much would compute requirements increase?

The answer: Google would need to double its entire server fleet. That was simply not feasible.

The only way out was to build a dedicated chip that did one thing — the core operation of ML inference: low-precision dense linear algebra. Nothing else. No browser, no office apps. But precisely because of that focus, it delivered 30 to 80x better energy efficiency and 20 to 30x lower latency than the CPUs and GPUs of the time.

That was the TPU's origin. Worth noting: the Transformer architecture was invented after the TPU project was already underway. Dean's choice to build a general linear algebra accelerator rather than over-specializing meant the TPU stayed relevant in the Transformer era.

The "Fit in Memory" Moment for Inference Hardware

In 2001, Jeff Dean and Sanjay Ghemawat realized the entire Google search index could finally fit into RAM across their machines. They rewrote the search system in a few days, and Google Search became dramatically faster. It was the kind of moment where a system constraint disappears and the entire product changes with it.

He believes a similar inflection point is approaching: inference-specific hardware.

Everyone today is accustomed to waiting for model responses. His question: what happens if latency drops by 50x?

Current GPUs and TPUs are relatively general-purpose compute devices. He thinks aggressively specializing for inference — minimizing data movement, supporting only the precision formats inference actually needs, nothing else — could produce systems with far superior efficiency and latency compared to existing hardware. And inference latency improvements directly determine how widely agent systems can be deployed and how many users they can serve.

Moving Data Costs 1,000x More Than Computing — and That Shapes What Gets Built

When discussing the fundamental constraints of AI systems, Dean cited a concrete comparison: a single multiply operation costs roughly 1 picojoule; moving data from an accelerator's high-bandwidth memory into on-chip compute units costs 1,000 times more.

That gap quietly shapes a great deal of system design.

Batching exists to amortize data movement costs — rather than computing one sample at a time, you load a batch together and spread that 1,000x penalty across the batch size. But batching trades away latency. For inference scenarios that demand extremely low latency, batching isn't the right answer.

His view: many bottlenecks people attribute to "model problems" are actually system and data IO problems. The design choices in training — large batch sizes, multiple epochs — are fundamentally motivated by hardware energy economics, not model behavior.

Agents Can Run for Weeks — Most People Haven't Realized It Yet

Jeff Dean called out what he suspected was a widespread misconception among the 6,000 attendees: that agents are only suited for tasks taking an hour or two.

His take: for the right problem domains, with a sufficiently capable underlying model, agents can run continuously for days or even weeks, completing genuinely complex work. One example: fully rewriting a software codebase from one programming language to another, including migrating tests, comparing behavioral differences, and eliminating all inconsistencies. He says models are "quite capable of this already."

When asked why agents tend to go off the rails around step 30 or 50, his answer: models are trained on a particular distribution. Once a task drifts outside the familiar range, performance degrades. The further the drift, the higher the failure rate.

Two approaches can help: first, define sufficiently specific "skills" for the model so it stays in territory where it performs well; second, run multiple agents in parallel exploring different paths, use another agent or model to evaluate which paths look most promising, discard the ones going sideways, and retain the ones with potential. The underlying mechanism is search at inference time.

Context Engineering: The Model Is Just One Component

A few weeks before this talk, Jeff Dean and Sanjay were working on optimizing the performance of one of Google's internal microbenchmarking libraries — used by millions of processes, where even small improvements have real impact.

The old workflow: run benchmark → modify code → re-run benchmark → check cache utilization → iterate. They formalized this as a "skill" and handed it to an agent: measure current performance, modify code, re-test, see what improved, keep going.

He described it as "giving the model the same methods we'd use as humans, but in a form the model can actually use."

His definition of context engineering: model, tool calls, history, retrieval systems, task decomposition logic — all of it working together is what actually solves the problem. The training data baked into a model is a soup of trillions of tokens; it's far less precise than information placed directly into context.

He also published a 30-page "performance tips" document, co-authored with Sanjay, covering low-level performance optimization techniques. Someone summarized the document and fed it to a model, and the model's performance on code optimization reasoning improved noticeably. He says it's freely available — anyone can use it.

Looking for a Startup Idea: 0% Success Rate Is a Good Sign

Dean's filtering logic is counterintuitively specific: test your target task against the best general-purpose models available. If the success rate is 0% to 1%, that's a good sign. If it's already at 20%, it's probably not a good market to enter.

The reason: 20% means capability is already emerging. In another 6 to 12 months, a general-purpose model will likely do it better.

He outlined two paths:

The first is proprietary data. General-purpose models don't have your users' personal data, internal data, or private knowledge bases. If your product gives models access to information they'd never otherwise see, that's a moat. His example: helping users organize their personal information. Google organizes the world's information — but nobody is managing your information.

The second is specialized models. AlphaFold is the reference case: it only does protein folding, it's not a general-purpose model, but it achieves accuracy a general model simply cannot match. Comparable white space exists in materials science, chip design, and any engineering domain where you can define clear evaluation criteria.

When AI Writes All the Code, Taste Becomes the Scarce Resource

When all code can be generated by agents, what does the person managing a hundred agents actually need?

Jeff Dean's answer: knowing what to ask the agents to do.

He drew the analogy to research: the tools and methods are available, but the critical skill is still picking the right problem. A researcher who brilliantly executes on a boring question is far less valuable than one who identifies the right question in the first place.

He acknowledged that "taste" is hard to quantify, but said it can be calibrated through experience: look back at the list of things you thought might matter 12 months ago, and see what actually landed and what quietly faded away. That builds your sample set. Another method is running "crazy thought experiments" — refusing to treat assumptions most people take for granted as fixed constraints.

His example: for 60 years, the central assumption of chip design has been "every chip must perfectly match the design — no errors allowed." But large-scale distributed systems have long operated on a different logic: building reliable systems from unreliable components. What if you abandoned the "zero errors" assumption at the transistor level and allowed 20 bit flips per day? System design would look completely different, but it could unlock enormous manufacturing advantages. He wasn't proposing anyone go build this now — it's a way of asking: what happens if we drop this assumption?

The Paper Rejected for "Unlikely to Have Significant Impact" Became Industry Standard

In 2014, Jeff Dean, Geoffrey Hinton, and Oriol Vinyals wrote a paper on "distillation" — using a large model as a "teacher" to train a smaller, cheaper, equally capable "student" model. The technique is now standard practice at virtually every major AI lab.

The paper was rejected by NeurIPS. The review: "unlikely to have significant impact."

Dean said he didn't blame the committee — reviewers may not have had experience deploying large-scale AI systems in production, and likely evaluated the work through the lens of "is this a fundamental advance?" rather than "how urgently do you need this when running at scale?"

They put the paper on arXiv. The right people read it. The right people used it. Today, Gemini Flash's strong performance-per-parameter ratio is partly attributable to distillation.

His attitude: if you get rejected, keep going. If you can see why the problem matters, that judgment is more reliable than a reviewer's opinion.