Diffusion Language Models Finally Match Autoregressive Quality—at 10x the Speed

Source: No Priors | Published: 2026-09-18T10:00:10Z

Inception has demonstrated for the first time that diffusion language models can match autoregressive models on text quality while generating output 10x faster.

Now Ermon is doing something even stranger: applying diffusion models to text and code generation, and using that to challenge the big model labs — OpenAI, Anthropic, Google. He founded Inception, now around 50 people, already serving real customers.

Inference efficiency is the entry point for the next paradigm shift

To understand Inception's central thesis, you need to understand a specific problem with compute utilization.

Ermon offered an analogy on a recent podcast: in 2017, researchers switched from RNNs to Transformers — not because Transformers were smarter, but because they were better suited for training. They could process all tokens in parallel, GPU utilization was high, training was fast, and you could ultimately run larger models. That was a training-side paradigm shift.

But the inference side has never been properly solved.

Autoregressive models — the architecture behind the GPT family — must generate one token at a time, sequentially. You can't start generating token 10 until you've finished token 9. This serial computation is deeply hostile to GPUs: GPUs are fundamentally designed for massive parallel computation, and serial autoregressive inference means they spend most of their time shuttling weight matrices around rather than doing matrix multiplication. This is a fundamental architectural limitation, not something optimization can fully fix.

Diffusion models address this directly: they work through iterative denoising from coarse to fine, processing all tokens in the sequence at every step — inherently parallel. The computational structure of diffusion inference matches diffusion training. The GPU is doing what it does best: high utilization, high throughput.

Ermon's conclusion: if the 2017 Transformer solved the parallelization problem on the training side, diffusion models are the Transformer moment for inference.

Going from images to text means reinventing the science

The logic sounds clean, but applying diffusion models to text runs into a fundamental obstacle: text is discrete.

Images can be interpolated. There's always a color between two colors. You can add Gaussian noise to pixels and learn to remove it — the whole process is mathematically coherent. Text isn't like that. There's no word that sits halfway between "apple" and "banana." You can't do continuous noise interpolation over a vocabulary.

This kept discrete diffusion models stuck in academic prototype territory for a long time. Ermon's team spent years on R&D, developing a new mathematical framework for diffusion processes in discrete spaces.

In 2024, they published a paper showing the first key proof-of-concept: at GPT-2 scale (under one billion parameters), their diffusion language model matched autoregressive models on perplexity — a measure of how well the model understands data structure — while generating text 10x faster.

This was the first demonstration that a diffusion model could match an autoregressive model on text generation quality. Not approach it — match it. Ermon then founded Inception with the goal of pushing that result to commercial scale.

Mercury: getting custom-chip speed on Nvidia GPUs

Inception's commercial product is called Mercury. On benchmarks, it's comparable to Claude Haiku, GPT-4o mini/nano, and Gemini Flash — while being significantly faster.

Ermon cited a specific customer case: OpenCall, a company building voice AI agents. Voice AI has brutal latency requirements. The ASR → LLM → TTS pipeline collapses if any single leg introduces too much delay. OpenCall had been running its LLM inference on Cerebras custom silicon to hit latency targets. Custom chips offer high memory bandwidth and low latency — but they're scarce and expensive.

After switching to Inception's Mercury, they achieved equivalent latency on standard Nvidia GPUs. The reason: diffusion models' parallel inference characteristics produce far higher GPU utilization than autoregressive models, and that software-level acceleration closes the hardware gap entirely.

What does this mean for voice AI companies? Same latency, but GPU supply is orders of magnitude greater than custom silicon — lower cost, higher availability.

20–30% of the market is already highly latency-sensitive

A natural question: where does the speed advantage actually matter?

Ermon made a rough estimate. He referenced OpenRouter's taxonomy of LLM use cases — research, conversation, coding, software engineering, log processing, and so on. By his estimate, 20–30% of these scenarios are extremely latency-sensitive: in this segment, users will consistently choose the faster model at equal quality and are willing to pay a premium for speed.

He used broadband as an analogy: once people experience high-speed internet, they won't go back to dial-up. Model latency has a similar ratchet effect — users who've experienced fast models can't tolerate slow ones.

20–30% doesn't sound like a majority. But given the scale of the LLM inference market, that share represents a substantial addressable market. Ermon explicitly calls this his current "lower bound" — at minimum, this portion of the market is up for grabs.

Building everything from scratch: moat and burden at once

The cost of starting from a new architecture is that the entire ecosystem doesn't exist yet. You have to build it all yourself.

Ermon listed what they've had to build in-house: an inference engine (Mercury can't run on vLLM or SGLang, both designed for autoregressive models), an SFT training stack, an RLHF training stack, RL infrastructure. All of this exists as mature, open-source tooling for autoregressive models. For diffusion language models, it's essentially a blank slate.

They've chosen to keep their core code closed-source, keeping IP inside the company. That decision has costs: reduced community contributions, higher adoption friction, more difficult enterprise on-prem deployments. But Ermon believes that at this stage, keeping the core technology proprietary is necessary to maintain a defensible competitive position.

A second moat comes from being forced to ship real, end-to-end production deployments from the start. His reasoning: if you only do research, you publish papers — but any big lab can read the paper and reproduce it. But if you actually have to run the model in production, you hit a class of problems that never appear in papers: inference engine tuning, evaluation against real customer data, handling edge cases. That accumulated operational knowledge becomes something that's genuinely harder to replicate.

Another potential advantage: diffusion models are more controllable

Ermon flagged a particularly interesting property that matters in voice AI: diffusion models are inherently easier to steer.

The reason lies in the structural difference in generation. Autoregressive models generate a complete sequence left-to-right in one pass. You have to wait until the sequence is complete before evaluating whether it satisfies a constraint — a format requirement, a safety constraint, some structural rule. If it doesn't, you regenerate.

Diffusion models work iteratively from coarse to fine. From the very first step, you can see the direction the generation is heading and apply external reward functions or constraints at each iteration to steer it. There's extensive support for this in the academic literature — from controllable synthesis in image generation to protein design, diffusion models have consistently outperformed autoregressive methods on controllability.

Ermon says the question they're working through is: given this property, what product experiences become possible that autoregressive models simply can't deliver? There's no public answer yet, but he believes this has more long-term value than the speed advantage.

Data efficiency: may matter most once you scale

Ermon mentioned another finding from the literature: diffusion models may be more data-efficient than autoregressive models.

The intuition holds up. Training a diffusion model means taking the same image or text and adding different levels of noise, then learning to remove it. The same data appears in training under many different noise perturbations — it's essentially built-in data augmentation.

If this property holds at scale, diffusion language models would have a structural advantage in data-constrained tasks — specialized domains where high-quality training data is scarce.

This remains an unverified direction, and Ermon is appropriately cautious about it. But it's the second differentiation point he thinks could emerge within two years — first speed, then potentially data efficiency.

Research at 50 people: compute is scarcer than headcount

Inception is about 50 people and two years old. Ermon describes the team structure simply: one team owns the best current version of the model and serves existing customers; another is building the next version, covering training, RL post-training, and inference research.

He offered an interesting observation: at 50 people, the bottleneck feels more like compute than headcount. AI coding tools have dramatically expanded what small teams can accomplish, but compute constraints are real and hard.

On the kind of researchers they attract: not people who want to make incremental progress in a mature direction, but people who want to invent new things and explore greenfield territory. Diffusion language models don't yet have a standard training recipe, a reusable inference engine, or an established RL post-training pipeline — all of it is still being invented. For one type of researcher, that's an opportunity. For another, it's too much uncertainty.

Flash Attention and DPO (Direct Preference Optimization, one of the most widely used algorithms for LLM alignment) both originated in Ermon's Stanford lab. That history goes a long way toward explaining why he's willing to bet a commercial company on a new architecture direction before large-scale validation results exist.