Parameters, Training, Inference: Three Words That Explain AI's Cost Structure

A classic paradox in AI news: training a model burns hundreds of millions of dollars, yet asking it a question costs a fraction of a cent. How do these numbers coexist?

The answer lies in three words: parameters, training, inference.

Parameters: The Model's Knobs

① Parameters: the model's "knobs."

A large model is one enormous mathematical function: tokens go in, massive multiplication and addition happen, the next token comes out. Every adjustable coefficient is a parameter. Picture a giant mixing console covered in knobs — their positions collectively determine the output.

7B, 70B, 1.6T means 7 billion, 70 billion, 1.6 trillion knobs. More knobs, more computation per token, higher cost — the physical root of why flagships cost 10x more than lightweight models.

Training: A One-Time Fixed Cost

② Training: setting all the knobs — once.

Training feeds the model oceans of text, comparing output to correct answers and nudging billions of knobs until the answers are good enough. It takes tens of thousands of GPUs running for months; a frontier training run costs tens to hundreds of millions of dollars.

Key feature: for a single model, a one-time fixed cost. Whether 1 user or 100 million follow, the money is spent once. (Though vendors must burn another round every few months to keep up — a story for the loss-ledger article.)

Inference: Recurring Cost Per Request

③ Inference: every use costs money.

After training, the knobs freeze. Each question you ask runs your input through billions of parameters to produce the first word; that word is appended to the input and the whole computation repeats for the second word. A 300-word answer means hundreds of full rounds. This is inference: cheap per request, but a recurring variable cost — every call occupies GPUs and burns electricity. At scale, cumulative inference spending dwarfs the training bill.

AI vs. Traditional Internet Economics

The structure resembles moviemaking. Training is production: hundreds of millions in, one film out. Inference is screening: each showing costs a little power and rent. But note — inference is also when the vendor gets paid. Every call brings in revenue and burns compute at the same time; profit or loss depends on the gap between the two. Traditional internet software has near-zero marginal cost, so "more usage, more profit" is an iron law — which is why Netflix can sell truly unlimited subscriptions. AI performs tens of trillions of operations per answer, so its subscriptions must hide message caps and rate limits inside. Marginal cost is non-negligible — the deepest economic difference between AI and the traditional internet: for the first time, it turns "scale means profit" into a question mark.

Remember the division of labor: training cost sets the barrier — why only giants remain; inference cost sets the price — why bills are counted in tokens.