Have Beliefs

When I started learning AI research last year, there was one question I could not get past. If the lesson of the last decade is that scale wins, what is research for? Why not buy more GPUs, train a bigger model on more data, and wait?

It is an uncomfortable question because the people who ask it are mostly right. Scale is the most fundamental thing in this field, and most clever ideas do not survive it.

I have since spent time interning at Moonshot AI, the lab behind Kimi, close enough to pretraining to watch ideas get tested, shipped and killed. I now think the question has a plain answer. Research does not make a model smarter than scale would. It makes the same model arrive sooner. What research buys is time, and time is what every lab is short of.

This essay is about that answer and what follows from it: how to judge an idea, how to read a paper, and what is worth learning if you are starting now.

It is all about scaling

Rich Sutton, The Bitter Lesson, 2019.Rich Sutton's essay The Bitter Lessonis seven years old and still the best description of the field. His observation is that building what we know into a system “always helps in the short term, and is personally satisfying to the researcher”. Then it plateaus, and the breakthrough comes from the opposite direction: general methods that scale with computation.

The word that matters there is satisfying. Say you believe an agent should stop every so often and reread its whole context. It makes sense. It matches how you work. That feeling is the warning. The ideas that feel most like research are often the ones that matter least once the model is ten times bigger.

Kaplan et al., Scaling Laws for Neural Language Models, 2020.In 2020 a group at OpenAI turned this into numbers. Compute is two things multiplied together: how many parameters the model has and how many tokens it is trained on. As compute grows, loss falls along a power law, a straight line on log axes, across many orders of magnitude. A lower loss means the model compresses its training data better. So if the line holds, what is left is a data problem: give the model good data and it will learn it.

Academia mostly cannot think this way, because it does not have the GPUs. Industry thinks about almost nothing else. Put like that, it sounds like the end of research. I think it is where research starts, because of one property of that line.

The line is a gift

Research needs something you can check. Pretraining has it: one number, the loss, that behaves the same way at every scale and in every lab. Reinforcement learning has no such thing. Every lab has its own environments and its own data, results jump instead of sliding, and a slightly bigger model may learn to game the reward instead of improving. This is a large part of why so much careful research happens in pretraining.

The scaling line turns an architecture change into a measurement. Train the baseline at several sizes and you get a line. Train the new architecture at the same sizes and you get a second line. The horizontal gap between them is the result: how much less compute the new architecture needs to reach the same loss.

Larry Dial, Improving our LLM Pretraining Efficiency, OpenAthena, 2026. The speedups there are theoretical: they count model FLOPs only.You see this framing everywhere once you look for it. When OpenAthena wrote up their pretraining recipe this year, every change came with one number of this kind. Going from 64 to 256 experts, for instance, was worth 1.4× in compute.

1.25× means the same loss for 1 ÷ 1.25 = 80% of the compute. 20 days × 24 hours × 1,000 GPUs = 480,000 GPU hours.What does a number like 1.25× buy? A frontier training run can occupy thousands of GPUs for months. If a run takes 100 days on 1,000 GPUs, a 1.25× gain gives back 20 of those days, about 480,000 GPU hours. And it is not paid once. Every later model built on that architecture gets the same discount, so the gains stack.

The figure also shows the more common ending. Most ideas help a small model and help a large one less, until the two lines meet. One training run cannot tell these two cases apart. Only the line can.

How one idea made it

Kimi Team, Attention Residuals, arXiv:2603.15031, 2026.In March, the Kimi team published Attention Residuals. I wrote about the idea when it came out, and I still think it is a beautiful one. Here I want to tell the other half: how an idea like this gets from a sketch to a production model. That process taught me more than the idea did.

The idea is natural. A standard Transformer stacks attention and MoE layers and adds each layer's output into one running sum. Attention Residuals lets each layer look back at the outputs of all earlier layers and choose its own mix.

First, a line, not a point. The first experiment was not one training run. It was the full scaling curve: baseline and new architecture at several sizes. Plenty of ideas win at one small size, so a single win says very little.

Then the expensive point. The next configuration cost about ten times the compute of the one before it. Its only job was to answer whether the gap was still there after the model grew tenfold. It was. The two lines ran parallel, about 1.25× apart.

Then infrastructure said no. A large model is spread across many GPUs. Full AttnRes needs the output of every earlier layer kept in memory and moved to wherever the current layer runs. With 54 layers that is a lot of memory and a lot of traffic, and it was too slow to consider. In its original form the idea was dead, with a good scaling curve.

The largest model has 54 layers. With 6 layers per block that is 9 blocks plus the token embedding, 10 sources in all. The two losses are read off the fitted curves at 5.6 PFLOP/s-days.Then the version that shipped. Block AttnRes groups layers into blocks. Inside a block, outputs are summed as before. Attention happens only across blocks. In the paper's largest model that is 10 sources in place of 54, and almost none of the gain is lost: at the top of the fitted range, Block reaches a loss of 1.692 against the baseline's 1.714. It went into a 48B-parameter model trained on 1.4 trillion tokens.

One more thing counts as infrastructure: inference. A change that trains fast and serves slowly has only moved the bill.

The lesson I took from this is that far too many ideas work on paper. “Does it work?” is rarely the question that decides anything. The question is whether it saves time once everything is counted.

Time is the currency

100 × (1 − 0.3) ÷ (1 − 0.4) ≈ 117 days.Put the two halves of that story together. Suppose an idea saves 30% of the compute needed to reach a given loss, and costs 40% of your training throughput. On a slide it is a 30% win. In practice a 100-day run now takes 117. A better loss curve and a faster training step are the same kind of thing. Both are ways to save time.

But why should time matter this much? A model that finishes training 20 days early is not smarter for it. Users get the same weights either way.

It matters because of what happens in those 20 days. The model is out. People use it. That usage is data, the real kind, and it feeds the next round of pretraining and reinforcement learning. A better next model brings more users, more funding, more GPUs and more people, and the loop turns again, a little faster.

This also answers the obvious objection: if data decides how good a model is, why not put everyone on data? Because returns on data flatten. The first people you add make a large difference and later ones much less. Architecture research starts with smaller gains, but it keeps paying roughly in proportion to the people doing it. Past a certain size, a lab's next researcher is worth more on the architecture side.

Research does not make the model smarter. It makes the same model arrive sooner.

Read the experiments

This changed how I read papers. I used to skip the introduction, read the method carefully, and skip the experiments. Every paper's experiments say the same thing: we beat the baseline. And benchmarks from different labs cannot be compared anyway.

People in industry read the other way round. The method you can learn in an evening with a model to answer your questions. The experiments are where a lab shows, sometimes without meaning to, what it checks before it trusts an idea.

Two of these rows need a word.

Loss over benchmarks. Say an architecture improves eight benchmarks out of ten and the two that drop are both about code. Is it worse at code? You cannot tell. Every lab mixes its training data differently, and a smaller share of code would do the same thing. Loss has no such problem. If the loss is lower, the model has fit its training data better, and from there benchmark scores follow the quality of the data. At large enough scale I trust the loss first.

Xie et al., mHC: Manifold-Constrained Hyper-Connections, DeepSeek-AI, 2025.The metric nobody asked for. DeepSeek's mHC paper plots gradient norm over training for a 27B model. You will not care about that plot unless you have trained something very large. Then you care a great deal, because one spike, perhaps from one bad batch, can damage the whole network. A figure like that tells you what the lab is afraid of, which no method section will.

The numbers themselves take practice. The first time I read “1.4T tokens” I had to ask a model what that meant. After enough papers you get a feel for it. An idea tested only at a few hundred million parameters can usually be skipped. One tested above ten billion is worth reproducing, because someone spent real compute to find out. It is also why people who train these models rarely ask about a new model's architecture. They ask how big it is and how many tokens it saw.

What is fundamental depends on the question

A lot of what counts as innovation in the LLM era is older than it looks. The intuitions often go back to computer vision or further. What is new is making them work at scale, which is why so much of the best recent work is co-design: an old intuition plus the engineering that makes it affordable.

Yuan et al., Native Sparse Attention, 2025. Schlag et al., Linear Transformers Are Secretly Fast Weight Programmers, 2021. Yang et al., Parallelizing Linear Transformers with the Delta Rule over Sequence Length, 2024.Native Sparse Attention is a good example. Everyone already agreed that sparse attention should behave the same in training and inference. The contribution was achieving that in a way that suits the hardware. DeltaNet, now a widely used linear attention variant, is another: the idea is years old, and it spread only after a hardware-friendly way to compute it was worked out. Attention Residuals fits the same pattern. The idea was the easy part, and the block structure is what made it affordable.

So is architecture research fundamental, or is data? I think it depends on what you are asking.

Jeremy Bernstein, Modular Manifolds, Thinking Machines Lab, 2025.Data is itself running short. As models grow, the tokens needed to train them grow faster than the supply of good text, which is why labs now generate synthetic data and train on the same data more than once. It is also why some people treat the optimizer as a research direction of its own: one that learns more from each token is another way to stretch the data you have.

Long context is the clearest case where data cannot help.

DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, 2026.At one million tokens, the DeepSeek-V4 paper reports 3.7× to 9.8× less compute per token than its predecessor, and a cache 9.5× to 13.7× smaller. That is what makes an agent that works for hours affordable. Newer benchmarks increasingly measure exactly that: how long a model can keep going on its own.

So “just scale it” is true under one definition of the goal. If the goal fits inside the context we already have, it is mostly a data problem. If the goal is a model that works for days and improves itself, research is squarely on the path.

Experience, then beliefs

If methods are cheap to learn, what is worth having? Two things, I think.

The first is experience. I mean the kind you only get from seeing ideas pushed through a real pipeline: a fixed ladder of model sizes that every idea has to climb, knowing which metric to watch at each rung, knowing when infrastructure gets a veto. The process has to be principled. You do not ship an idea because it feels right, and you do not drop one that works because you dislike it. Now that the model is the product, a lab's research is only as good as this process.

You can see what this experience is worth. Several new labs have raised very large rounds in the past two years before shipping anything. Their investors were not buying an idea, since ideas are everywhere. They were buying people who have already taken a model from small to very large and know what breaks on the way.

The second is beliefs. Methods expire. In ten years nobody will start a company around today's attention variants. What carries over is judgment: knowing what to care about. Does quality matter most, or cost, or context? Each answer is a bet, and nobody knows which is right.

Taste is the small, daily form of this. Suppose someone proposes that attention should read important tokens twice. My first thought is not whether it helps. It is that attention is already the bottleneck at long context, so running it twice is not an option. Asking what an idea costs before asking what it gains is most of what I mean by taste. The rest is being willing to back a design that looks wrong today because of where you think costs are heading.

My own bet, for what it is worth: the line that matters is the day a model can ask a good research question and then improve itself. There are benchmarks pointed at this now, but so far they mostly measure how much compute you can spend at test time. I do not think they capture it yet.

I could be wrong about that. When people ask whether architecture research still matters, whether it can raise the ceiling, whether it is saturated, the honest reply is that I do not know and that the answers keep changing. But ask me today and I will tell you what I believe and why. I think that is the standard to hold yourself to.

Where to start

Tan et al., Scaling Stick-Breaking Attention, ICLR 2025.Learn backwards. Do not begin with ten-year-old fundamentals and work forward. Pick a recent paper and read it with a model beside you. The first paper I read was on stick-breaking attention, which almost nobody uses now. It did not matter. I pasted in screenshot after screenshot and asked about everything I did not understand. A question you had to ask is knowledge you keep, because you already know why it matters.

Get the map before the details. Early on, it is worth knowing what each direction is for and why it beats the default, before going deep on any formula. Most of your time should still go into real work. The difference is in the goal: “I must publish in two months” produces different research from “I want to understand why this works”.

Listen to people who have done it. Blogs and long interviews with people at frontier labs are worth more than another tutorial. Dwarkesh Patel's interviews are a good place to start.

Write in public. Not only summaries of how something works, though that is a fine place to begin. Write what you think. I have seen people with no papers get research offers on the strength of one good blog post.

Then get into a room where ideas are tested. Reading is passive. Watching one idea go through a real pipeline taught me more than a year of papers would have. Every time I have changed environment I have grown the most, and what gets you into those rooms is agency far more than brilliance.

Is it too late to start? I think that is the wrong question. The better one is whether anything else you could be doing would teach you more about where all of this is going. For me the answer was no.

Nobody knows which bet is right, and anyone who says the picture will look the same in three months is guessing. That is not a reason to wait for the answer. Learn how ideas are tested. Decide what you think matters. Be ready to say it out loud, and to change it.

Have beliefs.