Parallel & Distributed Computing — Learning Roadmap

A ~3-month path built for a CS grad who already codes and has trained models. Read left to right. Each milestone has a concrete goal, the exact materials to use, and how to learn them so the ideas actually stick. The through-line: at scale, communication is the bottleneck and failure is normal.

Free / online Book Paper Tip: this scrolls sideways — drag or shift-scroll. Tick a milestone to mark it done.
Progress
0 / 6 complete
0
Warm-up · optional
~1 week

Foundations refresher

Goal: Be fluent in concurrency (threads, locks, races) and TCP/IP basics before scaling anything.
Materials
How to learn it
  • Skip this if threads & sockets are already comfortable.
  • Write one program with a deliberate data race, then fix it with a lock — feel the bug, don't just read about it.
1
Parallel fundamentals
Weeks 1–3

Parallelism & the cost of talking

Goal: Understand Amdahl's law and all-reduce — the collective that powers every distributed training run.
Materials
How to learn it
  • Don't aim to master MPI — aim to understand why adding nodes stops helping.
  • Build: matrix-multiply or N-body kernel in OpenMP, then MPI.
  • Plot speedup vs. core count. Watch Amdahl's law appear in your own graph.
2
Distributed systems core
Weeks 3–6

Failure, consistency, consensus

Goal: Implement Raft. If you finish one hard thing this quarter, make it this — it's what makes distributed systems click.
Materials
How to learn it
  • DDIA is non-negotiable — it's the bridge from "I train models" to "I get the systems under them".
  • Do the labs, don't just watch: MapReduce lab → Raft lab.
  • Read the Raft paper before coding it; keep Figure 2 open the whole time.
3
Distributed ML training
Weeks 6–10

Where your background pays off

Goal: Train a model across 2 GPUs/nodes with DDP and measure scaling efficiency. See communication eat your speedup.
Materials
How to learn it
  • No GPUs? Use gloo backend across 2 CPU processes to learn the mechanics, then rent spot multi-GPU by the hour.
  • For each method, answer: exactly what is communicated, and when?
  • Connect it back: DDP's gradient sync is the all-reduce from Milestone 1.
4
Advanced scaling
Weeks 10–12

Splitting the model itself

Goal: Explain tensor / pipeline / sharded parallelism and pick the right one for a given model + hardware budget.
Materials
How to learn it
  • Build a mental decision tree: fits on one GPU? → DDP. Too big? → FSDP/ZeRO. Huge? → tensor + pipeline.
  • Optional breadth: run PageRank or a feature job in Spark to see the MapReduce lineage in production form.
5
Capstone · portfolio
Weeks 12+

Prove it & write it up

Goal: Ship one project too big for a single GPU, plus a write-up of the tradeoffs — portfolio gold for ML-infra interviews.
Pick one to ship
  • Train a model that won't fit on 1 GPU using FSDP / ZeROflagship
  • Hand-roll a mini all-reduce / parameter server over socketsdeep
  • Your Raft KV-store extended to sharding (MIT labs)systems
How to finish strong
  • Measure everything: throughput, memory, scaling efficiency at 1 / 2 / 4 workers.
  • Write a short report on what you observed and why — the analysis is the differentiator, not the code.
  • Learn Docker so multi-node experiments are reproducible.

The two ideas that tie it all together

1. Communication is the bottleneck, not computation. Every scaling limit you hit — Amdahl's law, DDP overhead, why tensor parallelism needs fast interconnects — is really about the cost of moving data between workers.   2. Failure is normal, not exceptional. Raft, replication, and consensus exist because at scale a machine is always down somewhere. Keep asking "what breaks when a node dies mid-operation?"

How to use this: Do it in order — read a chunk, then immediately write code that scales and measure it. The intuition for communication cost, contention, and failure only comes from watching your own program not speed up the way you hoped. Checkboxes track progress within this view (they reset if you reload).
Roadmap generated Jul 2026 · links verified at build time