Course: Course 3 — LLM Fine-Tuning Masterclass Module: FTDD-07 — DeepSeek-R1 Duration: 45 minutes Level: Senior Engineer and above Prerequisites: FT14 (GRPO), FT15 (CoT Distillation)
After completing this module, you will be able to:
If you internalize one sub-section from this deep-dive, make it this one. R1-Zero is the clearest empirical demonstration of the course thesis.
DeepSeek started with DeepSeek-V3-Base — a 671B-parameter Mixture-of-Experts model (37B active per token), a standard pretrained base with no instruction tuning and no supervised reasoning data. They applied GRPO (Group Relative Policy Optimization, the algorithm you met in Module FT14) directly to the base, with reward functions that were verifiable — math correctness, code execution pass/fail, LeetCode verdicts. No human preference model. No rubric judge. Just: did the answer check out.
The prompt template was deliberately minimal. The model was given a user question and told to reason between <think> and </think> tags before answering. That is the entire scaffold. No demonstration chains. No seed examples of good reasoning.
Long, structured chain-of-thought emerged. Over training, the average response length grew substantially as the model learned that spending more tokens on verification, backtracking, and self-correction improved its reward. The chains began to show behaviors nobody had put in the data: "Wait, let me reconsider," spontaneous re-derivation of formulas, checking an answer by substituting it back. The paper famously highlighted an "aha moment" — a checkpoint where the model, unprompted, started re-evaluating its own intermediate steps. This was not coached. The reward signal made deliberation pay off, and deliberation is what grew.
This is the steering thesis (Module FT00) demonstrated at scale. The base already had the capability — it saw billions of tokens of mathematical and programmatic reasoning during pretraining. RL did not teach it to reason. RL redirected its probability mass so that the reasoning pathways it already had became the reliably chosen behavior under the verifiable-reward regime. Steering, not teaching. If reasoning were a knowledge-injection problem, GRPO on a base model with no reasoning traces could not have produced it.
R1-Zero was a proof, and it showed. The chains were often hard to read. The model mixed languages mid-chain (Chinese and English tangled together). It produced run-on chains that were correct but ugly. It sometimes failed to converge to a clean final answer despite correct intermediate work. On raw benchmark scores it was competitive with the leading reasoning models of the time — but you would not ship it to a user. R1-Zero answered the can-reasoning-emerge question. R1 was built to answer the can-we-actually-ship-this question.
R1-Zero proved emergence. R1 made it usable. The pipeline is four stages, each fixing a specific failure of the one before.
You cannot go straight from base to RL and get a readable model (R1-Zero proved that). So R1 begins with a small amount of high-quality supervised reasoning data — on the order of thousands of curated long-CoT examples — used to cold-start the model into the right format: readable chains, a clean <think> structure, a single language, a properly formatted final answer. This anchors tone and template. It is not trying to teach reasoning (the base has that). It is steering the model into the output shape RL will then amplify.
The cold-start seed is exactly what R1-Zero lacked, and its absence is why R1-Zero was messy. Stage 1 is small in data volume but large in consequence: it determines the readability of everything downstream.
With the format anchored, run GRPO on verifiable-reward reasoning tasks — the same mechanism as R1-Zero, but now starting from a model that already produces clean chains. This stage scales capability. Math, code, and structured-logic benchmarks rise sharply here. The reward functions are still verifiable (execution, correctness checks), not model-judged.
This is the stage that does the heavy lifting on reasoning quality. The cold-start gave the model the right shape; Stage 2 fills that shape with genuine, reward-driven deliberation. By the end of Stage 2 you have a strong reasoning model.
Here is the move that separates R1 from a one-shot RL recipe. Take the Stage-2 RL'd model and use it to generate a large synthetic dataset of reasoning traces — but do not keep all of them. Use rejection sampling: generate many candidates per problem, filter for correct final answers, and further filter for quality (no language mixing, no repetition, readable structure). The result is a large, clean SFT dataset of reasoning traces that are correct and readable, produced by the model's own best behavior under RL.
This dataset serves two purposes. First, it produces the polished reasoning model: SFT on this clean data gives you a model that reliably produces the high-quality chains the RL'd model produced only sometimes. Second — and this is the load-bearing consequence for the rest of the field — this curated trace set is the raw material for the distillation. The 800K traces that trained R1-Distill-Qwen-32B came out of this stage.
Stage 2's RL optimized only reasoning tasks. Stage 4 runs a second RL pass across the full distribution — reasoning and general assistant tasks — to align the model as a usable product. The reward mix now includes both verifiable reasoning rewards and preference/policy signals for general helpfulness, safety, and formatting. This is the stage that turns a strong reasoner into a shippable assistant.
The part of R1 that changed practice the most. You can inherit reasoning by SFT alone.
DeepSeek took the curated ~800K reasoning traces from Stage 3 (plus additional non-reasoning SFT data for general capability) and performed plain SFT on six dense base models: Qwen-1.5B, Qwen-7B, Qwen-14B, Qwen-32B, Llama-8B, and Llama-70B. No RL on the students. No reward model. No GRPO loop. Just supervised fine-tuning on the teacher's best traces.
R1-Distill-Qwen-32B beat OpenAI o1-mini on AIME 2024 (the competition-math benchmark that became the standard reasoning yardstick). A 32B dense model, trained by SFT only, outperformed a frontier proprietary reasoning model. That single result is why chain-of-thought distillation went from "a technique some labs used" to "the default way to get a reasoning model." You no longer needed the RL supercomputer. You needed a strong teacher's traces and a standard SFT run.
This is the course thesis again, from the other direction. If reasoning were a capability the small models lacked — a knowledge gap — distilling traces could not have transferred it. The fact that SFT on traces did transfer reasoning behavior is evidence that the small bases already had latent reasoning capability; the traces steered them to express it. The student inherits the behavior, not new ability.
DeepSeek explicitly reports that applying RL on top of the distilled students yielded only marginal gains over the SFT-only distillation. This is counterintuitive if you assume RL is always necessary for reasoning. The interpretation: the teacher's curated traces were already a near-optimal steering signal. They showed the student exactly the high-reward reasoning trajectories. RL on top could only refine what the traces had already installed — and the marginal refinement was small relative to the compute cost. For practitioners, the takeaway is concrete: if you can distill from a strong reasoning teacher, SFT-first; reach for RL on the student only if you have a verifiable-reward domain the traces under-cover and the compute to spend.
Two canonical reasoning references exist in the open ecosystem: R1 and Qwen3 (the next deep-dive). They prove different things and complement each other.
R1 was released as arXiv:2501.12948 in January 2025, then published in Nature (s41586-025-09422-z). The Nature publication matters: it passed peer review and gave the field a citable, refereed account of the emergence result and the pipeline. The weights and the distilled models are open under the DeepSeek License (commercially usable with conditions). The GitHub repository is deepseek-ai/deepseek-r1, hosting the model weights and the technical report; the distilled students live under the same org.
Three things became standard after R1:
CoT distillation is the practical path to a reasoning model. Before R1, the assumption was that you needed a full RL pipeline. After R1-Distill-Qwen-32B beat o1-mini, the default became: get a strong teacher, curate its traces, SFT your student. RL is optional, reserved for when you have verifiable rewards and the compute.
Verifiable rewards beat reward models for reasoning RL. R1-Zero and R1's reasoning stages used execution and correctness checks, not a learned reward model. The field took this as the cleaner signal: if you can verify, do not approximate with a judge.
RL-on-base proves the capability is latent. R1-Zero is the reference everyone cites when arguing that reasoning is steering. It is the strongest single empirical argument for the course thesis.
R1 and Qwen3 (FTDD-08) are the two references, and they teach different lessons. R1 is the distillation reference: the teacher pipeline plus the SFT-only transfer that made CoT distillation standard. Qwen3 is the hybrid reference: a single model that fuses thinking and non-thinking modes via a thinking-budget mechanism, so you get a reasoner and a fast assistant in one set of weights. If you want to build a reasoning model by distilling from a teacher, R1 is your blueprint. If you want a production model that adapts its compute to the question, Qwen3 is your blueprint. You should know both.
R1's own ablation shows SFT-only distillation captures most of the gain; RL on the student is marginal. Reaching for GRPO before trying clean trace-distillation burns compute for little return. Distill first; RL the student only with a verifiable domain and a clear gap.
R1-Zero mixed languages and produced ugly chains. That is irrelevant to what it proved. The emergence of self-correcting CoT from pure RL on a base is the finding. R1's cold-start stage is exactly the fix for the messiness — it does not contradict the emergence, it builds on it.
The whole mechanism depends on the teacher producing high-reward trajectories. A weak or misaligned teacher produces traces that steer the student toward mediocre reasoning. Garbage traces in, mediocre reasoner out. The teacher's quality is the ceiling.
| Term | Definition |
|---|---|
| R1-Zero | Pure RL (GRPO) on DeepSeek-V3-Base with verifiable rewards and zero supervised reasoning traces. Proved reasoning emerges from RL alone. |
| R1 | The four-stage production pipeline: cold-start SFT → reasoning RL → rejection-sampling SFT → final RL. |
| GRPO | Group Relative Policy Optimization. The RL algorithm (no learned value function; group-relative advantages) used in R1's reasoning stages. |
| Verifiable rewards | Reward signals computed by checking correctness (math, code execution, LeetCode) rather than a learned reward model. R1's defining reward choice. |
| Rejection sampling | Generate many candidates, keep the correct, high-quality ones. Stage 3 of R1; the source of the 800K distillation traces. |
| CoT distillation | Transferring reasoning behavior from a teacher to a student via SFT on the teacher's chain-of-thought traces. Made standard by R1. |
| R1-Distill-Qwen-32B | The 32B dense student distilled from R1. Beat OpenAI o1-mini on AIME 2024. The result that made CoT distillation the default. |
See 07-lab-spec.md. The "Distill and Observe" lab: load R1-Distill-Qwen-1.5B, run it against a non-distilled Qwen-1.5B base on the same math problem, and witness the transferred reasoning — the distilled model produces a structured <think> chain and arrives at the correct answer where the base flails. The steering thesis, felt on your own machine in under 45 minutes.
github.com/deepseek-ai/deepseek-r1. Model weights, distilled students, and the technical report.# Module FTDD-07 — DeepSeek-R1 **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FTDD-07 — DeepSeek-R1 **Duration**: 45 minutes **Level**: Senior Engineer and above **Prerequisites**: FT14 (GRPO), FT15 (CoT Distillation) --- ## Learning Objectives After completing this module, you will be able to: 1. Explain why R1-Zero (pure RL on a base model, zero supervised traces) is the single strongest published evidence that reasoning is a *steering* problem, not a *knowledge* problem. 2. Draw the four-stage R1 pipeline (cold-start SFT → reasoning RL → rejection-sampling SFT → final RL) and state exactly what each stage fixes that the one before could not. 3. Distinguish the distillation (SFT-only on ~800K curated traces, six dense models) from the teacher pipeline, and explain why R1-Distill-Qwen-32B beating OpenAI o1-mini on AIME made chain-of-thought distillation the default it is today. 4. Defend the design decision to publish both the messy proof (R1-Zero) and the polished product (R1), and explain why applying RL on top of the distilled students gave marginal returns. --- # 7.1 — R1-Zero: The Proof That Reasoning Emerges From RL *If you internalize one sub-section from this deep-dive, make it this one. R1-Zero is the clearest empirical demonstration of the course thesis.* ## The experiment DeepSeek started with DeepSeek-V3-Base — a 671B-parameter Mixture-of-Experts model (37B active per token), a standard pretrained base with no instruction tuning and no supervised reasoning data. They applied GRPO (Group Relative Policy Optimization, the algorithm you met in Module FT14) directly to the base, with reward functions that were *verifiable* — math correctness, code execution pass/fail, LeetCode verdicts. No human preference model. No rubric judge. Just: did the answer check out. The prompt template was deliberately minimal. The model was given a user question and told to reason between `<think>` and `</think>` tags before answering. That is the entire scaffold. No demonstration chains. No seed examples of good reasoning. ## What happened Long, structured chain-of-thought emerged. Over training, the average response length grew substantially as the model learned that spending more tokens on verification, backtracking, and self-correction improved its reward. The chains began to show behaviors nobody had put in the data: "Wait, let me reconsider," spontaneous re-derivation of formulas, checking an answer by substituting it back. The paper famously highlighted an "aha moment" — a checkpoint where the model, unprompted, started re-evaluating its own intermediate steps. This was not coached. The reward signal made deliberation pay off, and deliberation is what grew. This is the steering thesis (Module FT00) demonstrated at scale. The base already had the capability — it saw billions of tokens of mathematical and programmatic reasoning during pretraining. RL did not *teach* it to reason. RL *redirected* its probability mass so that the reasoning pathways it already had became the reliably chosen behavior under the verifiable-reward regime. Steering, not teaching. If reasoning were a knowledge-injection problem, GRPO on a base model with no reasoning traces could not have produced it. ## Why R1-Zero was not a product R1-Zero was a proof, and it showed. The chains were often hard to read. The model mixed languages mid-chain (Chinese and English tangled together). It produced run-on chains that were correct but ugly. It sometimes failed to converge to a clean final answer despite correct intermediate work. On raw benchmark scores it was competitive with the leading reasoning models of the time — but you would not ship it to a user. R1-Zero answered the *can-reasoning-emerge* question. R1 was built to answer the *can-we-actually-ship-this* question. --- # 7.2 — R1: The Four-Stage Pipeline *R1-Zero proved emergence. R1 made it usable. The pipeline is four stages, each fixing a specific failure of the one before.* ## Stage 1 — Cold-start SFT You cannot go straight from base to RL and get a readable model (R1-Zero proved that). So R1 begins with a small amount of high-quality supervised reasoning data — on the order of thousands of curated long-CoT examples — used to cold-start the model into the right *format*: readable chains, a clean `<think>` structure, a single language, a properly formatted final answer. This anchors tone and template. It is not trying to teach reasoning (the base has that). It is steering the model into the output shape RL will then amplify. The cold-start seed is exactly what R1-Zero lacked, and its absence is why R1-Zero was messy. Stage 1 is small in data volume but large in consequence: it determines the readability of everything downstream. ## Stage 2 — Reasoning RL With the format anchored, run GRPO on verifiable-reward reasoning tasks — the same mechanism as R1-Zero, but now starting from a model that already produces clean chains. This stage scales capability. Math, code, and structured-logic benchmarks rise sharply here. The reward functions are still verifiable (execution, correctness checks), not model-judged. This is the stage that does the heavy lifting on reasoning quality. The cold-start gave the model the right *shape*; Stage 2 fills that shape with genuine, reward-driven deliberation. By the end of Stage 2 you have a strong reasoning model. ## Stage 3 — Rejection-sampling SFT (the data cleanup) Here is the move that separates R1 from a one-shot RL recipe. Take the Stage-2 RL'd model and use it to *generate* a large synthetic dataset of reasoning traces — but do not keep all of them. Use rejection sampling: generate many candidates per problem, filter for correct final answers, and further filter for quality (no language mixing, no repetition, readable structure). The result is a large, clean SFT dataset of reasoning traces that are correct *and* readable, produced by the model's own best behavior under RL. This dataset serves two purposes. First, it produces the polished reasoning model: SFT on this clean data gives you a model that reliably produces the high-quality chains the RL'd model produced only sometimes. Second — and this is the load-bearing consequence for the rest of the field — this curated trace set is the raw material for the distillation. The 800K traces that trained R1-Distill-Qwen-32B came out of this stage. ## Stage 4 — Final RL (whole-distribution alignment) Stage 2's RL optimized only reasoning tasks. Stage 4 runs a second RL pass across the full distribution — reasoning *and* general assistant tasks — to align the model as a usable product. The reward mix now includes both verifiable reasoning rewards and preference/policy signals for general helpfulness, safety, and formatting. This is the stage that turns a strong reasoner into a shippable assistant. --- # 7.3 — The Distillation: 800K Traces Into Six Dense Models *The part of R1 that changed practice the most. You can inherit reasoning by SFT alone.* ## The setup DeepSeek took the curated ~800K reasoning traces from Stage 3 (plus additional non-reasoning SFT data for general capability) and performed plain SFT on six dense base models: Qwen-1.5B, Qwen-7B, Qwen-14B, Qwen-32B, Llama-8B, and Llama-70B. No RL on the students. No reward model. No GRPO loop. Just supervised fine-tuning on the teacher's best traces. ## The result that made the field notice R1-Distill-Qwen-32B beat OpenAI o1-mini on AIME 2024 (the competition-math benchmark that became the standard reasoning yardstick). A 32B dense model, trained by SFT only, outperformed a frontier proprietary reasoning model. That single result is why chain-of-thought distillation went from "a technique some labs used" to "the default way to get a reasoning model." You no longer needed the RL supercomputer. You needed a strong teacher's traces and a standard SFT run. This is the course thesis again, from the other direction. If reasoning were a capability the small models lacked — a knowledge gap — distilling traces could not have transferred it. The fact that SFT on traces *did* transfer reasoning behavior is evidence that the small bases already had latent reasoning capability; the traces steered them to express it. The student inherits the *behavior*, not new *ability*. ## Why RL on the students gave marginal returns DeepSeek explicitly reports that applying RL on top of the distilled students yielded only marginal gains over the SFT-only distillation. This is counterintuitive if you assume RL is always necessary for reasoning. The interpretation: the teacher's curated traces were already a near-optimal steering signal. They showed the student exactly the high-reward reasoning trajectories. RL on top could only refine what the traces had already installed — and the marginal refinement was small relative to the compute cost. For practitioners, the takeaway is concrete: if you can distill from a strong reasoning teacher, SFT-first; reach for RL on the student only if you have a verifiable-reward domain the traces under-cover and the compute to spend. --- # 7.4 — Why R1 Is The Reference *Two canonical reasoning references exist in the open ecosystem: R1 and Qwen3 (the next deep-dive). They prove different things and complement each other.* ## The publication R1 was released as arXiv:2501.12948 in January 2025, then published in Nature (s41586-025-09422-z). The Nature publication matters: it passed peer review and gave the field a citable, refereed account of the emergence result and the pipeline. The weights and the distilled models are open under the DeepSeek License (commercially usable with conditions). The GitHub repository is `deepseek-ai/deepseek-r1`, hosting the model weights and the technical report; the distilled students live under the same org. ## What the field took from R1 Three things became standard after R1: 1. **CoT distillation is the practical path to a reasoning model.** Before R1, the assumption was that you needed a full RL pipeline. After R1-Distill-Qwen-32B beat o1-mini, the default became: get a strong teacher, curate its traces, SFT your student. RL is optional, reserved for when you have verifiable rewards and the compute. 2. **Verifiable rewards beat reward models for reasoning RL.** R1-Zero and R1's reasoning stages used execution and correctness checks, not a learned reward model. The field took this as the cleaner signal: if you can verify, do not approximate with a judge. 3. **RL-on-base proves the capability is latent.** R1-Zero is the reference everyone cites when arguing that reasoning is steering. It is the strongest single empirical argument for the course thesis. ## R1 versus Qwen3 R1 and Qwen3 (FTDD-08) are the two references, and they teach different lessons. R1 is the *distillation* reference: the teacher pipeline plus the SFT-only transfer that made CoT distillation standard. Qwen3 is the *hybrid* reference: a single model that fuses thinking and non-thinking modes via a thinking-budget mechanism, so you get a reasoner and a fast assistant in one set of weights. If you want to build a reasoning model by distilling from a teacher, R1 is your blueprint. If you want a production model that adapts its compute to the question, Qwen3 is your blueprint. You should know both. --- ## Anti-Patterns ### Assuming RL is required for a reasoning student R1's own ablation shows SFT-only distillation captures most of the gain; RL on the student is marginal. Reaching for GRPO before trying clean trace-distillation burns compute for little return. Distill first; RL the student only with a verifiable domain and a clear gap. ### Treating R1-Zero's messiness as a reason to dismiss the result R1-Zero mixed languages and produced ugly chains. That is irrelevant to what it proved. The emergence of self-correcting CoT from pure RL on a base is the finding. R1's cold-start stage is exactly the fix for the messiness — it does not contradict the emergence, it builds on it. ### Distilling from a weak teacher The whole mechanism depends on the teacher producing high-reward trajectories. A weak or misaligned teacher produces traces that steer the student toward mediocre reasoning. Garbage traces in, mediocre reasoner out. The teacher's quality is the ceiling. --- ## Key Terms | Term | Definition | | --- | --- | | **R1-Zero** | Pure RL (GRPO) on DeepSeek-V3-Base with verifiable rewards and zero supervised reasoning traces. Proved reasoning emerges from RL alone. | | **R1** | The four-stage production pipeline: cold-start SFT → reasoning RL → rejection-sampling SFT → final RL. | | **GRPO** | Group Relative Policy Optimization. The RL algorithm (no learned value function; group-relative advantages) used in R1's reasoning stages. | | **Verifiable rewards** | Reward signals computed by checking correctness (math, code execution, LeetCode) rather than a learned reward model. R1's defining reward choice. | | **Rejection sampling** | Generate many candidates, keep the correct, high-quality ones. Stage 3 of R1; the source of the 800K distillation traces. | | **CoT distillation** | Transferring reasoning behavior from a teacher to a student via SFT on the teacher's chain-of-thought traces. Made standard by R1. | | **R1-Distill-Qwen-32B** | The 32B dense student distilled from R1. Beat OpenAI o1-mini on AIME 2024. The result that made CoT distillation the default. | --- ## Lab Exercise See `07-lab-spec.md`. The "Distill and Observe" lab: load R1-Distill-Qwen-1.5B, run it against a non-distilled Qwen-1.5B base on the same math problem, and witness the transferred reasoning — the distilled model produces a structured `<think>` chain and arrives at the correct answer where the base flails. The steering thesis, felt on your own machine in under 45 minutes. --- ## References 1. **DeepSeek-AI (2025)** — *DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*. arXiv:2501.12948. The R1-Zero and R1 pipeline, the distillation, the R1-Distill-Qwen-32B result. 2. **DeepSeek-AI (2025)** — *DeepSeek-R1 (Nature publication)*. Nature s41586-025-09422-z. The peer-reviewed account. 3. **DeepSeek-AI** — `github.com/deepseek-ai/deepseek-r1`. Model weights, distilled students, and the technical report. 4. **Shao et al. (2024)** — *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models*. arXiv:2402.03300. The origin of GRPO. 5. **Course 3, Module FT14** — GRPO and RL on verifiable rewards. The algorithm R1 uses. 6. **Course 3, Module FT15** — Chain-of-thought distillation. The technique R1 made standard.