At scale, teams don’t win by owning more FLOPs; they win by shrinking the distance between hypothesis and measurement. I learned that the expensive way: running large training pipelines where iteration speed was the difference between “we think this works” and “we know” - building some of the most capable open-weights models available while leading the OpenOrca team in 2023. So I took Karpathy’s microgpt - a Transformer small enough to hold in your head - and made it fast enough that you can also throw it around and learn its behavior by feel: change a learning rate, flip a batch size, tweak a layout, rerun, and immediately see what moved; full sweeps at interactive speed.
In this toy regime, performance is set by granularity. When the work is a pile of tiny matrix multiplies and elementwise kernels, overhead and launch/scheduling costs can dominate peak throughput. Laptop CPUs can be faster than Blackwell GPUs. That’s a regime inversion: the “faster” machine can lose because it spends too much time on ceremony per step, while a simpler execution path spends a higher fraction of wall time doing useful math. In that corner of the world, a laptop CPU can beat a datacenter GPU for this workload - not because it’s a better chip, but because it’s spending less time dispatching and more time learning. That inversion reshapes the early-time Pareto frontier, loss versus wall-clock, where you’re trading model capacity against steps-per-second under a fixed time budget.
Early-time is where most iteration happens. It’s where you decide whether an idea is promising, where you map stability boundaries, where you learn which knobs matter and which are placebo. If you can push the frontier down and left in the first few seconds, you don’t just finish runs faster.. you change what you can notice. You turn “training” into feedback.
Inside, I take you on a tour of the AI engine room: how scalar autograd explodes into tens of thousands of tiny ops, how rewriting it as a handful of tight loops collapses overhead, how caches and SIMD lanes dictate what “fast” even means, why skipping useless work beats clever math, and how ISA-specific accelerators like Neon/SME2 shift the cost model again. The result is a ~19,000× speedup on a toy problem - not as a parlor trick, but as a microcosm of the same compounding process that drives real progress: better execution buys more experiments, more experiments buy better understanding, and better understanding buys better execution.
At scale, teams don’t win by owning more FLOPs; they win by shrinking the distance between hypothesis and measurement. I learned that the expensive way: running large training pipelines where iteration speed was the difference between “we think this works” and “we know” - building some of the most capable open-weights models available while leading the OpenOrca team in 2023. So I took Karpathy’s microgpt - a Transformer small enough to hold in your head - and made it fast enough that you can also throw it around and learn its behavior by feel: change a learning rate, flip a batch size, tweak a layout, rerun, and immediately see what moved; full sweeps at interactive speed.
In this toy regime, performance is set by granularity. When the work is a pile of tiny matrix multiplies and elementwise kernels, overhead and launch/scheduling costs can dominate peak throughput. Laptop CPUs can be faster than Blackwell GPUs. That’s a regime inversion: the “faster” machine can lose because it spends too much time on ceremony per step, while a simpler execution path spends a higher fraction of wall time doing useful math. In that corner of the world, a laptop CPU can beat a datacenter GPU for this workload - not because it’s a better chip, but because it’s spending less time dispatching and more time learning. That inversion reshapes the early-time Pareto frontier, loss versus wall-clock, where you’re trading model capacity against steps-per-second under a fixed time budget.
Early-time is where most iteration happens. It’s where you decide whether an idea is promising, where you map stability boundaries, where you learn which knobs matter and which are placebo. If you can push the frontier down and left in the first few seconds, you don’t just finish runs faster.. you change what you can notice. You turn “training” into feedback.
Inside, I take you on a tour of the AI engine room: how scalar autograd explodes into tens of thousands of tiny ops, how rewriting it as a handful of tight loops collapses overhead, how caches and SIMD lanes dictate what “fast” even means, why skipping useless work beats clever math, and how ISA-specific accelerators like Neon/SME2 shift the cost model again. The result is a ~19,000× speedup on a toy problem - not as a parlor trick, but as a microcosm of the same compounding process that drives real progress: better execution buys more experiments, more experiments buy better understanding, and better understanding buys better execution.
Technically interesting post, but the LLM writing style is really grating these days, couldn't bear to finish reading it...