learnings/

concept · · week 8

Why LLM decoding is memory-bound

To write each new token, the GPU has to read every weight of the model. That read, not the math, sets the speed limit.

The short version: an LLM writes one token at a time. For every token, the GPU has to pull the entire model out of memory and run it once. GPUs can do math far faster than they can move data, so most of the time the math units sit idle, waiting for weights to arrive.

One token, the whole model

A model like Llama 7B has about 7 billion parameters. Stored in fp16 (2 bytes each), that’s about 14 GB of weights. To produce the next token, every one of those weights has to be used once, so all 14 GB travel from the GPU’s memory (HBM) to its compute units. Then the next token starts, and the whole trip happens again.

one decode stepdiagram
GPU memory (HBM)layer 1 weightslayer 2 weightslayer 3 weightslayer 4 weightslayer 5 weightslayer N weightsread every weightonce per stepcomputematrix multipliesmostly waitingnext token
Every step reads all the weights once. With one sequence, that whole read buys a single token.

Moving bytes is slower than doing math

Two numbers describe a GPU here. An H100 can move roughly 3.35 TB of data per second from memory. It can do roughly 989 trillion fp16 operations per second. That’s about 300 operations for every byte it reads.

Decoding one token with one sequence does about 2 operations per parameter (one multiply, one add), and each fp16 parameter is 2 bytes. So it does about 1 operation per byte. The GPU could do 300 per byte, so it spends almost all of its time waiting on memory. That’s what “memory-bound” means.

The simple rule that follows:

max tokens/sec ≈ memory bandwidth / bytes of weights

For a 7B model in fp16 on an H100, that’s 3.35 TB/s ÷ 14 GB ≈ 240 tokens per second, at best. Real numbers come in lower, because the KV cache also has to be read and nothing runs at peak.

Batching: share the read

If the weights are already on their way to the compute units, you can use them for several sequences at once. A batch of 4 reads the weights once and produces 4 tokens, one for each sequence. The memory time stays the same while the useful work goes up four times.

one decode step, batch of 4diagram
GPU memory (HBM)layer 1 weightslayer 2 weightslayer 3 weightslayer 4 weightslayer 5 weightslayer N weightsread every weightonce per stepcomputematrix multipliesmostly waitingseq 1seq 2seq 3seq 4
Same read, four tokens. This is why serving systems work so hard to batch requests together.

That works until the math catches up. At some batch size, the compute time grows past the memory time and the GPU becomes compute-bound. On an H100 in fp16, that happens around a batch of 300.

Try it

Change the model, the precision, the GPU and the batch size. The longer bar is the one that limits speed.

decode speed ceilingplayground
read weights
do the math

weights
tokens/s per sequence
tokens/s total
flips to compute-bound at

A rough upper bound. It ignores the KV cache, attention and overheads, and uses approximate peak specs, so real speeds are lower.

Things to notice:

  • At batch 1, the “read weights” bar is huge and the “do the math” bar is a sliver. That’s the whole idea.
  • Switching to 4-bit cuts the weights to a quarter, so tokens per second roughly quadruple. This is the main reason people quantize models for inference.
  • Raising the batch barely changes the step time at first, so total tokens per second climbs almost for free. Then the math bar overtakes, and the gains stop.

What I still want to measure

  • How close does my own model get to the ceiling, with and without a KV cache?
  • Where does the flip to compute-bound actually happen on a free T4?