Path
From a tiny GPT I train myself, to serving models at scale, to contributing and writing.
Weeks 1–4
Build a tiny GPT
A transformer I wrote, trained, and can explain.
w1Let's build GPTnow0/4
- Karpathy, "Let's build GPT: from scratch, in code" (code along)
- Train a char-level transformer on tiny Shakespeare
- Change layers, heads, context length and note what changes
- Explain self-attention in 5 lines without looking
w2Tokenizers0/3
- Karpathy, "Let's build the GPT Tokenizer"
- Implement byte-pair encoding (BPE)
- Note why tokenization affects cost and speed
w3Reproduce GPT-2 (124M)0/3
- Karpathy, "Let's reproduce GPT-2 (124M)"
- Mixed precision, gradient accumulation, torch.compile, flash attention
- Run training on a free GPU (Colab or Kaggle)
w4Read and consolidate0/4
- Read "Attention Is All You Need"
- Skim the GPT-2 paper
- Optional: Jay Alammar, "The Illustrated Transformer"
- Explainer: how a transformer generates one token
Weeks 5–8
Inference fundamentals
Memory bandwidth is the bottleneck. Learn to see it in my own measurements.
w5KV cache0/3
- Add a generation loop to my model, with and without a KV cache
- Measure tokens/sec vs sequence length and plot it
- Note where the cache saves work and where it costs memory
w6Inference arithmetic0/3
- Read kipply, "Transformer Inference Arithmetic"
- For a 7B model, compute weight bytes and KV-cache bytes per token
- Estimate max tokens/sec as bandwidth / weight bytes, compare with a measurement
w7Batching and the roofline0/3
- Read Horace He, "Making Deep Learning Go Brrrr From First Principles"
- Measure throughput vs batch size on my model
- Find where it flips from memory-bound to compute-bound
w8Attention variants and quantization0/4
- Read the multi-query and grouped-query attention papers
- Run a 7B-class model in fp16 vs int8 vs 4-bit
- Compare speed, memory, and output quality
- Explainer: why LLM decoding is memory-bound
Weeks 9–16
Serving at scale
KV-cache memory, scheduling, and GPU-to-GPU traffic.
w9vLLM hands-on0/3
- Install vLLM and serve a small model
- Benchmark throughput, time-to-first-token, inter-token latency
- Read the vLLM docs overview
w10PagedAttention and continuous batching0/3
- Read the PagedAttention paper
- Read the Orca paper (continuous batching)
- Draw how paged KV blocks work; skim the vLLM scheduler
w11FlashAttention and GPU memory0/4
- Read FlashAttention and FlashAttention-2
- Learn the HBM vs SRAM hierarchy
- Implement tiled / online softmax in NumPy
- Optional: a GPU MODE lecture
w12Prefix caching and scheduling0/3
- Read about SGLang's RadixAttention
- Learn chunked prefill
- Compare vLLM vs SGLang on a shared-prefix workload
w13Parallelism0/3
- Read Megatron-LM (tensor) and GPipe (pipeline parallelism)
- Read the sharding and inference chapters of "How to Scale Your Model"
- Rent 2+ GPUs and run tensor-parallel serving
w14Mixture of Experts0/3
- Read Switch Transformer and Mixtral
- Read DeepSeek-V2/V3 (latent attention, expert parallelism)
- Explain why MoE is cheap in compute but heavy in memory
w15Speculative decoding0/2
- Read Leviathan et al. and Chen et al.
- Try it in vLLM; measure speedup vs acceptance rate
w16Putting it together0/3
- Read DistServe and Splitwise (prefill/decode disaggregation)
- Explainer: life of a request in an LLM serving cluster
- Publish all explainers here
Week 17 onward
Contribute and publish
Open source fixes, a reproduction, and a first paper draft.
w17+Open-ended work0/5
- Read one systems paper a week
- Fix a "good first issue" in vLLM or SGLang
- Reproduce one paper's result and write it up
- Pick a narrow question from my own benchmarks
- Draft an arXiv note or workshop paper on it