01 Foundations

The base training loop is still forward pass, loss, backward pass, optimizer step. The report does not replace this loop. It stresses it with more scale, more precision constraints, and more routing complexity.

• Cross-entropy and perplexity remain the base reading tools for model quality.
• AdamW and stable accumulation remain central because novelty lives elsewhere in the stack.
• Low precision works only when sensitive states are protected instead of compressed blindly.
• Multi-token prediction matters because it adds denser supervision and later helps inference.

02 Architecture

The key architectural moves are not cosmetic. RoPE changes how position enters attention. Latent KV compression changes what must stay resident in memory. Sparse experts change how compute is activated and where traffic appears.

• RoPE encodes position through rotation, which helps relative behavior but creates long-context tension.
• Latent KV compression is an architecture choice with direct inference consequences.
• Sparse experts only pay off if routing is constrained enough to keep communication sane.
• Balancing strategy is part of the model design, not a pure training afterthought.

03 Infrastructure

At scale, the system stops looking like a pure matrix multiplication story. It starts looking like a topology and overlap problem. The report is strong when it makes that explicit.

• Cluster shape matters because tokens, activations, and expert traffic have to travel somewhere.
• DualPipe-style scheduling matters because the goal is to hide communication behind useful work.
• FP8 is not one datatype choice. It is a policy across storage, compute, scaling, and accumulation.
• Memory savings mostly come from refusing to keep every intermediate state resident.

04 Pre-Training

A model cannot specialize into capabilities that its data mixture never makes available. Pre-training is therefore not just about token count. It is about data shape, schedule, and context regime.

• Mixture design decides what domains become easy, hard, or absent.
• Hyper-parameters reveal what the system is optimized to tolerate.
• Long-context extension is usually staged because long windows are not free.
• Ablations matter because they separate structural wins from incidental ones.

05 Inference

Prefill and decode are not the same operating regime. One likes larger compute chunks. The other is dominated by cache behavior, memory movement, and latency.

• Prefill is easier to batch and more compute-dense.
• Decode is more exposed to cache size, cache bandwidth, and routing overhead.
• Latent KV compression pays off here because resident state becomes the cost center.
• Sparse serving introduces hot-expert problems that look like systems load balancing.

06 Post-Training

Post-training is where raw capability is turned into usable assistant behavior. The cleanest way to read this stage is as a sequence: supervised shaping, reward design, relative preference optimization, and filtered self-improvement loops.

• SFT establishes answer format, tone, and baseline usefulness.
• Rule-based rewards dominate when correctness can be checked mechanically.
• Model-based rewards fill the gap for open-ended judgments.
• Generated data becomes valuable only after filtering, rejection, and re-weighting.

07 Evaluation and Limits

Benchmarks show that the final system works. They do not prove that every engineering decision was necessary or optimal. The best evidence usually lives in the appendices and the ablations, not in the headline tables alone.

• Benchmarks are outcome evidence, not perfect causal attribution.
• Open-ended review is useful but weaker because judgment quality varies.
• Low-precision and specialization appendices often carry the strongest technical signal.
• The right final takeaway is systems-level: each layer narrows what the next layer can do.