Concepts
One idea per page, in the order the path covers them.
- week 8Why LLM decoding is memory-bound
To write each new token, the GPU has to read every weight of the model. That read, not the math, sets the speed limit.
One idea per page, in the order the path covers them.
To write each new token, the GPU has to read every weight of the model. That read, not the math, sets the speed limit.