Skip to main content

How Models Learn

What were frontier language models actually trained to do?

One thing: predict the next token. Every summary, translation, and working program a model produces today came out of a system optimised for that single narrow objective. Nobody wrote a translation objective. Nobody wrote a coding objective. The distance between what was asked for and what showed up is the most important fact in modern AI — and the honest way to reason about it splits cleanly into what is settled and what is still a bet.

This page is the predecessor to How Models Think: that page covers what a trained model does internally; this one covers how it got that way.

The objective is narrow​

In 2017, Google researchers published the transformer architecture ("Attention Is All You Need"). Its core mechanism, self-attention, lets every part of an input sequence relate to every other part simultaneously. Earlier models processed text sequentially, word by word, which capped how much context they could use at once. The transformer removed that cap — and, just as importantly, made training parallelisable enough that scale became an engineering decision rather than a research problem.

The training objective stayed almost embarrassingly simple: given the tokens so far, predict the next one. For the mechanism-level detail — the attention formula, token embeddings, and the compute costs that follow from them — the canonical mechanics live in Data Tokenization § The Transformer Architecture.

The capabilities were byproducts​

Here is the settled part. Models trained only on next-token prediction went on to summarise documents, answer novel questions, translate between languages, and generate functional code. None of these were the stated goal; each emerged as a byproduct of pursuing the narrow objective at scale. Several arrived undesigned and, in documented cases, unanticipated — GPT-3's few-shot learning was reported by its own builders as an observed result, not a design target.

Calibration stamp: 2026-07-29. The list of capabilities and where they sit relative to human performance moves constantly — check current model pages, not this paragraph, for today's frontier. What does not move is the empirical pattern: optimise a narrow objective at sufficient scale, and capabilities appear that nobody specified.

The scaling hypothesis is a live bet​

Here is the contested part — an informed inference, not a demonstrated outcome. The scaling hypothesis proposes that increasing model size, data, and compute will keep yielding new capability, and that at sufficient scale something resembling general intelligence may emerge.

The case for: the trend has been consistent. Capabilities absent at one scale have reliably appeared at the next, and tasks that needed specialised systems a decade ago are handled today by general-purpose models with no task-specific training.

The two strongest rebuttals:

  • Prediction is not comprehension. David Deutsch's version is the sharpest: scaled models produce unanticipated capabilities, but they do not originate unanticipated ends — no new preferences, no self-generated explanations, no genuine disobedience. On this view, closing the remaining gap requires architectural or conceptual innovation, not more scale.
  • Some emergence is a measurement artifact. Schaeffer, Miranda, and Koyejo (2023) showed that several celebrated "abrupt" capability jumps flatten into smooth, predictable improvement when scored with continuous instead of all-or-nothing metrics. Emergence is real as an observation; how much of it is a property of the model versus the yardstick is genuinely open.

Both positions remain defensible on current evidence. Anyone who tells you the scaling question is settled — in either direction — is selling something.

What did not emerge​

Capability emerged. Intention did not.

This is the part of the story that gets lost in the scaling argument. What the transformer scaled was attention — a mechanism for weighting which parts of a sequence relate to which. Attention has no view on what is worth doing. No volume of next-token prediction produces a northstar, because no northstar was ever in the objective. Deutsch's rebuttal states this negatively: the models originate no ends of their own. The positive reading is the one that matters if you are building: direction is the part humans still supply, and now have to supply deliberately.

That is the shift from an economy that harvests attention to one that serves declared intent — the Internet of Intent, where good intentions are not decoration but the load-bearing input. As capability gets cheaper and broader, the scarce input moves from can it be done to should it be, and toward what.

The engineering consequence is the yin and yang of this stack. AI is probabilistic invention: it generates what nobody specified, which is exactly why it cannot bound itself. Blockchain and its rails are deterministic proof: incentive, settlement, and verifiable consequence that do not drift. Emergence is the strongest argument for coupling the two — a strand that invents beyond its specification needs a strand that cannot. That spiral is Pikorua, and this page is the evidence under its AI half.

It is also why feedback loops shape our destiny is a claim about mechanism, not motivation: you do not get capabilities by specifying them, you get them by running a tight objective through an enormous loop. What the loop optimises toward is the whole game.

Principles​

  • Separate the settled from the contested. Emergence happened — empirical fact. Scale reaching general intelligence — live hypothesis. Claims that blur the line inherit the confidence of the first and the uncertainty of the second.
  • Narrow objectives, broad byproducts. Capability breadth came from objective narrowness at scale, not from enumerating capabilities.
  • Capability emerges; direction does not. Scaling produced abilities nobody specified and no intentions at all. Good intentions stay a human input, and an unstated one defaults to whatever the loop already rewards.
  • Probabilistic invention needs deterministic bounds. A strand that generates beyond its specification cannot also be the strand that constrains it.
  • The trend is evidence, not proof. Consistent past scaling gains raise the prior on the hypothesis; they cannot settle it.
  • Check the yardstick before the miracle. Before treating a capability jump as emergent, ask whether the metric is discontinuous.

Inversion​

Conventional wisdom says: to get a capability, design for it — specify the behaviour, train for the behaviour. The inversion: the largest capability gains in software history came from not designing the capabilities, and the designers did not fully anticipate what they built. Specification is how you get reliability; scale on a narrow objective is how the field got reach.

How to apply​

  1. Split any AI-capability claim in two. When a vendor, paper, or thread asserts what models will do, mark each clause settled (documented, dated) or contested (scaling-dependent). Fund, build, or repeat only what survives the split honestly labelled.
  2. Date your capability assumptions. Any plan resting on "models can't do X" inherits the emergence risk that X appears at the next scale jump. Write the assumption down with a date and a recheck trigger.
  3. Judge emergence claims by their metric. Ask whether the reported jump survives a continuous scoring rule before citing it as evidence of a discontinuity.

Proof of done: one live plan or pitch in your operation has its AI assumptions explicitly split into settled versus scaling-dependent, each with a date.

Changes my mind: a frontier training run at materially larger scale that produces no new capability class would weaken the hypothesis side; a scaled model originating verifiably self-generated goals or explanations would weaken the Deutsch rebuttal.

Context​

  • pairs-with How Models Think — how the model got its capabilities; that page covers what it does with them internally
  • depends-on Data Tokenization — canonical home of the transformer mechanics: attention formula, embeddings, compute costs
  • pairs-with LLM Models — dated capability calibration across providers; the moving frontier this page deliberately does not track
  • pairs-with AI Compute Industry Players — who is betting billions on which side of the scaling question
  • pairs-with Pikorua — the belief-layer claim this page evidences: probabilistic invention coupled to deterministic proof
  • pairs-with Intentions — what scaling never supplied, and the Internet of Intent it points to
  • depends-on AI Principles — the principles hub this page belongs to

Questions​

If a system optimised for one narrow objective produced abilities its designers did not anticipate, what does your operation do differently on the day the next one appears?

  • Which of your current plans quietly assume today's capability frontier is permanent?
  • If the Deutsch rebuttal is right, what class of work stays durably human — and are you investing in it?
  • What would count, for you specifically, as decisive evidence that the scaling hypothesis had succeeded or failed?

Next question: if scale supplies capability and never supplies direction, what is your operation doing to make intent explicit enough that a system which invents beyond its specification still moves toward something you chose?