What's Actually Scarce in the "AI Compute Shortage"?

"Compute" has become the buzzword at every business dinner.

Ask a simple follow-up — what exactly is compute, and what exactly is short? — and the room goes quiet. Then someone says "compute means GPUs, the Nvidia thing." Someone else pushes back: "No, it's data centers — look at what Microsoft and Google are building."

Both are right. Both are only half right. And the missing half is where this business gets interesting.

What AI Compute Actually Is

At its core, AI compute is matrix multiplication per second. Whether it's training a frontier model or answering your ChatGPT question, 99% of the work is multiplying giant matrices. GPUs dominate because they were born for this: an Nvidia H100 packs tens of thousands of small cores working in parallel, versus a few dozen on a CPU.

So yes — the atomic unit of compute is the chip. But if compute equaled GPUs, this would be easy: just buy cards. It's not.

The Networking Layer Nobody Mentions

Overlooked layer #1: turning 10,000 cards into one machine. No single GPU can hold a large model. Training at GPT-4 scale takes 10,000+ GPUs working as one computer for months. That demands ultra-fast networking between cards — NVLink, InfiniBand — and this gear quietly eats 15–25% of total cluster cost. If the network is slow, your expensive GPUs sit waiting for data instead of computing. The industry metric is MFU (utilization) — in real large-scale training, 40–50% counts as excellent. Read that again: buying 10,000 GPUs does not mean owning 10,000 GPUs of compute. Same hardware, different networking and scheduling — output can differ by 50%.

Power Is the Deeper Constraint

Overlooked layer #2: electricity. One H100 draws ~700 watts. A 100,000-GPU cluster with networking, storage, and cooling runs to hundreds of megawatts — a mid-sized city. Why is Microsoft restarting nuclear plants? Because chips can be bought with money; grid connections and substations take years, and no amount of money compresses that.

Where the Shortage Stands Now

So what's actually short? The answer keeps moving. - 2023–24: chips. Not design — manufacturing. The world's most expensive processors were bottlenecked by memory (HBM) and packaging. - Now: power. Data centers stand ready with no grid to plug into, and the queue is measured in years. - Always: time. A data center takes 2–4 years to build; a fab, 3–5. AI demand grows monthly. Demand outruns construction, so the gap persists.

The Full Chain That Must Hold

The takeaway: GPUs are the engine, data centers are the chassis and fuel tank — and real compute also needs a driver (software and operations). Break any link and the chain doesn't run.

When Big Tech says "we're building compute," they never mean just buying cards. They mean locking chip supply, building data centers, signing decade-long power deals, and hiring teams who can push utilization up.

Which bottleneck bites hardest by 2028 — chips, power, or time? What's your take?