Course: Course 3 — LLM Fine-Tuning Masterclass
Module: FTDD-07 — DeepSeek-R1
Duration: ~45 minutes (spoken at ~140 wpm)
Format: Verbatim transcript with [SLIDE N] cues. Read aloud or use as speaker notes.
[SLIDE 1 — Title]
This is deep-dive seven: DeepSeek-R1. If you took module fourteen, GRPO, and module fifteen, chain-of-thought distillation, this is the model that made both of those the center of the field. R1 is the reasoning distillation reference. It is also, in my view, the single strongest published evidence for the thesis of this entire course: that reasoning is a steering problem, not a knowledge problem.
We are going to look at three things. R1-Zero, the proof. R1, the product. And the distillation, the result that changed how everyone builds reasoning models.
[SLIDE 2 — The thesis, demonstrated]
Here is the thesis, demonstrated. Reasoning is a steering problem. DeepSeek-R1 is the evidence. Three pieces. R1-Zero: pure reinforcement learning on a base model, with zero supervised reasoning traces, and reasoning emerged. R1: the four-stage pipeline that made that emergence shippable. And the distillation: SFT-only on eight-hundred-thousand curated traces, where R1-Distill-Qwen-thirty-two-B beat OpenAI's o-one-mini.
If reasoning were knowledge, none of this would work. It is capability the base already had. Let's see why.
[SLIDE 3 — R1-Zero, the proof]
R1-Zero. The experiment. Start with DeepSeek-V-three-Base — a six-hundred-seventy-one-billion-parameter mixture-of-experts model, thirty-seven billion active per token. A standard pretrained base. No instruction tuning. No supervised reasoning data.
Apply GRPO directly to that base. The reward functions are verifiable: math correctness, code execution pass or fail, LeetCode verdicts. No reward model. No judge. No human preferences. Just: did the answer check out.
The prompt template is deliberately minimal. A user question and a scaffold that says reason between think tags before answering. No demonstration chains. No seed examples. That is the entire setup.
[SLIDE 4 — Why R1-Zero proves the thesis]
What happened. Long chain-of-thought emerged. Over training, the average response length grew substantially. The model learned that spending more tokens on verification, backtracking, and self-correction improved its reward. The chains began to show behaviors nobody had put in the data. Wait, let me reconsider. Spontaneous re-derivation of formulas. Checking an answer by substituting it back. The paper highlighted an aha moment — a checkpoint where the model, unprompted, started re-evaluating its own intermediate steps. This was not coached.
Now the thesis. If reasoning were a knowledge problem — if the base did not already have the capability — then GRPO on a base with no reasoning traces could not have produced it. You would need supervised demonstrations or continued pretraining on reasoning corpora. But if reasoning is steering — if the base already saw billions of reasoning tokens during pretraining and only needs to be redirected — then this is exactly what you would expect. RL redirects probability mass so the reasoning pathways become the chosen behavior. That is what happened.
R1-Zero was a proof, and it showed. Mixed languages mid-chain. Run-on chains. Correct but ugly. You would not ship it. R1 exists to fix this.
[SLIDE 5 — R1, the four-stage pipeline]
R1 is four stages. Stage one, cold-start SFT. A small amount — thousands — of high-quality curated long-CoT examples. This anchors format: readable chains, a clean think structure, a single language, a properly formatted final answer. It is not teaching reasoning. The base has that. It is steering the model into the output shape that RL will then amplify. This is exactly what R1-Zero lacked.
Stage two, reasoning RL. With the format anchored, run GRPO on verifiable-reward reasoning tasks — the same mechanism as R1-Zero, but now starting from a model that already produces clean chains. This stage scales capability. Math, code, logic benchmarks rise sharply. The cold-start gave the model the right shape; stage two fills that shape with reward-driven deliberation.
Stage three, rejection-sampling SFT. This is the move that separates R1 from a one-shot recipe. Take the stage-two RL'd model, use it to generate a large synthetic dataset of reasoning traces. But do not keep all of them. Rejection sampling: generate many candidates per problem, filter for correct final answers, further filter for quality — no language mixing, no repetition, readable structure. The result is a large, clean SFT dataset of correct and readable traces. This produces the polished reasoning model. And — this is load-bearing for the rest of the field — this curated trace set is the raw material for the distillation.
Stage four, final RL. Stage two optimized only reasoning. Stage four runs a second RL pass across the full distribution — reasoning and general assistant tasks — to align the model as a usable product. This is the stage that turns a strong reasoner into a shippable assistant.
[SLIDE 6 — Why each stage exists]
Read the table. Each stage fixes a specific failure. Cold-start fixes R1-Zero's unreadable chains. Reasoning RL fills the anchored shape with capability. Rejection sampling makes the RL'd model's best behavior reliable. Final RL aligns the whole model, not just reasoning.
[SLIDE 7 — The distillation]
Now the part that changed practice the most. The distillation. DeepSeek took the curated eight-hundred-thousand reasoning traces from stage three, plus non-reasoning SFT data for general capability, and performed plain SFT on six dense base models. Qwen at one-point-five, seven, fourteen, and thirty-two billion. Llama at eight and seventy billion. No GRPO on the students. No reward model. No RL loop. Just supervised fine-tuning on the teacher's best traces.
The result that made the field notice. R1-Distill-Qwen-thirty-two-B beat OpenAI o-one-mini on AIME twenty-twenty-four — the competition-math benchmark that became the standard reasoning yardstick. A thirty-two-billion dense model, trained by SFT only, outperformed a frontier proprietary reasoning model. That single result is why chain-of-thought distillation went from a technique some labs used to the default way to build a reasoning model. You no longer needed the RL supercomputer. You needed a strong teacher's traces and a standard SFT run.
[SLIDE 8 — Why RL on the students gave marginal returns]
Here is the counterintuitive part. DeepSeek explicitly reports that applying RL on top of the distilled students yielded only marginal gains over the SFT-only distillation. If you assume RL is always necessary for reasoning, this surprises you.
The interpretation. The teacher's curated traces were already a near-optimal steering signal. They showed the student the high-reward reasoning trajectories directly. RL on top could only refine what the traces had already installed, and the marginal refinement was small relative to the compute cost.
The practitioner takeaway is concrete. If you can distill from a strong reasoning teacher, do SFT first. Reach for RL on the student only if you have a verifiable-reward domain the traces under-cover and the compute to spend.
[SLIDE 9 — Distillation equals the thesis, other direction]
Notice this works from the other direction too. If reasoning were a capability the small models lacked — a knowledge gap — then distilling traces could not have transferred it. The fact that SFT on traces did transfer reasoning behavior is evidence that the small bases already had latent capability. The traces steered them to express it. The student inherits the behavior, not new ability.
[SLIDE 10 — R1 vs Qwen3]
R1 and Qwen3 are the two canonical reasoning references, and they teach different lessons. R1 is the distillation reference. Teacher pipeline plus SFT-only transfer. Use it when you are building a reasoner by distilling from a teacher. Qwen3, the next deep-dive, is the hybrid reference. One model that fuses thinking and non-thinking modes via a thinking-budget mechanism, so you get a reasoner and a fast assistant in one set of weights. Use it when you are shipping a production model that adapts its compute to the question. Know both.
[SLIDE 11 — What the field took from R1]
Three things became standard after R1. First, CoT distillation is the practical path. Before R1, the assumption was you needed a full RL pipeline. After R1-Distill-Qwen-thirty-two-B beat o-one-mini, the default became: get a strong teacher, curate its traces, SFT your student. Second, verifiable rewards beat reward models for reasoning RL. If you can verify, do not approximate with a judge. Third, RL-on-base proves the capability is latent. R1-Zero is the reference everyone cites when arguing reasoning is steering.
The paper is published in Nature. The weights are open at deepseek-ai slash deepseek-r-one.
[SLIDE 12 — Anti-patterns]
Three anti-patterns. First, assuming RL is required for a reasoning student. R1's own ablation shows SFT-only captures most of the gain. Distill first. Second, dismissing R1-Zero for its messiness. The mixed languages are irrelevant to what it proved. R1's cold-start stage is the fix. Third, distilling from a weak teacher. The teacher's quality is the ceiling.
[SLIDE 13 — What you can now do]
You can now explain why R1-Zero is the strongest evidence that reasoning is steering. You can draw the four-stage pipeline and what each stage fixes. You can distinguish the distillation from the teacher pipeline and why it beat o-one-mini. And you can defend publishing both the proof and the product.
Next, deep-dive eight: Qwen3. The hybrid reference. Let's see how a single set of weights serves both reasoning and fast responses.
End of module FTDD-07. Duration: approximately forty-five minutes at one-hundred-forty words per minute.
# Teaching Script — Module FTDD-07: DeepSeek-R1 **Course**: Course 3 — LLM Fine-Tuning Masterclass **Module**: FTDD-07 — DeepSeek-R1 **Duration**: ~45 minutes (spoken at ~140 wpm) **Format**: Verbatim transcript with `[SLIDE N]` cues. Read aloud or use as speaker notes. --- [SLIDE 1 — Title] This is deep-dive seven: DeepSeek-R1. If you took module fourteen, GRPO, and module fifteen, chain-of-thought distillation, this is the model that made both of those the center of the field. R1 is the reasoning distillation reference. It is also, in my view, the single strongest published evidence for the thesis of this entire course: that reasoning is a steering problem, not a knowledge problem. We are going to look at three things. R1-Zero, the proof. R1, the product. And the distillation, the result that changed how everyone builds reasoning models. [SLIDE 2 — The thesis, demonstrated] Here is the thesis, demonstrated. Reasoning is a steering problem. DeepSeek-R1 is the evidence. Three pieces. R1-Zero: pure reinforcement learning on a base model, with zero supervised reasoning traces, and reasoning emerged. R1: the four-stage pipeline that made that emergence shippable. And the distillation: SFT-only on eight-hundred-thousand curated traces, where R1-Distill-Qwen-thirty-two-B beat OpenAI's o-one-mini. If reasoning were knowledge, none of this would work. It is capability the base already had. Let's see why. [SLIDE 3 — R1-Zero, the proof] R1-Zero. The experiment. Start with DeepSeek-V-three-Base — a six-hundred-seventy-one-billion-parameter mixture-of-experts model, thirty-seven billion active per token. A standard pretrained base. No instruction tuning. No supervised reasoning data. Apply GRPO directly to that base. The reward functions are verifiable: math correctness, code execution pass or fail, LeetCode verdicts. No reward model. No judge. No human preferences. Just: did the answer check out. The prompt template is deliberately minimal. A user question and a scaffold that says reason between think tags before answering. No demonstration chains. No seed examples. That is the entire setup. [SLIDE 4 — Why R1-Zero proves the thesis] What happened. Long chain-of-thought emerged. Over training, the average response length grew substantially. The model learned that spending more tokens on verification, backtracking, and self-correction improved its reward. The chains began to show behaviors nobody had put in the data. Wait, let me reconsider. Spontaneous re-derivation of formulas. Checking an answer by substituting it back. The paper highlighted an aha moment — a checkpoint where the model, unprompted, started re-evaluating its own intermediate steps. This was not coached. Now the thesis. If reasoning were a knowledge problem — if the base did not already have the capability — then GRPO on a base with no reasoning traces could not have produced it. You would need supervised demonstrations or continued pretraining on reasoning corpora. But if reasoning is steering — if the base already saw billions of reasoning tokens during pretraining and only needs to be redirected — then this is exactly what you would expect. RL redirects probability mass so the reasoning pathways become the chosen behavior. That is what happened. R1-Zero was a proof, and it showed. Mixed languages mid-chain. Run-on chains. Correct but ugly. You would not ship it. R1 exists to fix this. [SLIDE 5 — R1, the four-stage pipeline] R1 is four stages. Stage one, cold-start SFT. A small amount — thousands — of high-quality curated long-CoT examples. This anchors format: readable chains, a clean think structure, a single language, a properly formatted final answer. It is not teaching reasoning. The base has that. It is steering the model into the output shape that RL will then amplify. This is exactly what R1-Zero lacked. Stage two, reasoning RL. With the format anchored, run GRPO on verifiable-reward reasoning tasks — the same mechanism as R1-Zero, but now starting from a model that already produces clean chains. This stage scales capability. Math, code, logic benchmarks rise sharply. The cold-start gave the model the right shape; stage two fills that shape with reward-driven deliberation. Stage three, rejection-sampling SFT. This is the move that separates R1 from a one-shot recipe. Take the stage-two RL'd model, use it to generate a large synthetic dataset of reasoning traces. But do not keep all of them. Rejection sampling: generate many candidates per problem, filter for correct final answers, further filter for quality — no language mixing, no repetition, readable structure. The result is a large, clean SFT dataset of correct and readable traces. This produces the polished reasoning model. And — this is load-bearing for the rest of the field — this curated trace set is the raw material for the distillation. Stage four, final RL. Stage two optimized only reasoning. Stage four runs a second RL pass across the full distribution — reasoning and general assistant tasks — to align the model as a usable product. This is the stage that turns a strong reasoner into a shippable assistant. [SLIDE 6 — Why each stage exists] Read the table. Each stage fixes a specific failure. Cold-start fixes R1-Zero's unreadable chains. Reasoning RL fills the anchored shape with capability. Rejection sampling makes the RL'd model's best behavior reliable. Final RL aligns the whole model, not just reasoning. [SLIDE 7 — The distillation] Now the part that changed practice the most. The distillation. DeepSeek took the curated eight-hundred-thousand reasoning traces from stage three, plus non-reasoning SFT data for general capability, and performed plain SFT on six dense base models. Qwen at one-point-five, seven, fourteen, and thirty-two billion. Llama at eight and seventy billion. No GRPO on the students. No reward model. No RL loop. Just supervised fine-tuning on the teacher's best traces. The result that made the field notice. R1-Distill-Qwen-thirty-two-B beat OpenAI o-one-mini on AIME twenty-twenty-four — the competition-math benchmark that became the standard reasoning yardstick. A thirty-two-billion dense model, trained by SFT only, outperformed a frontier proprietary reasoning model. That single result is why chain-of-thought distillation went from a technique some labs used to the default way to build a reasoning model. You no longer needed the RL supercomputer. You needed a strong teacher's traces and a standard SFT run. [SLIDE 8 — Why RL on the students gave marginal returns] Here is the counterintuitive part. DeepSeek explicitly reports that applying RL on top of the distilled students yielded only marginal gains over the SFT-only distillation. If you assume RL is always necessary for reasoning, this surprises you. The interpretation. The teacher's curated traces were already a near-optimal steering signal. They showed the student the high-reward reasoning trajectories directly. RL on top could only refine what the traces had already installed, and the marginal refinement was small relative to the compute cost. The practitioner takeaway is concrete. If you can distill from a strong reasoning teacher, do SFT first. Reach for RL on the student only if you have a verifiable-reward domain the traces under-cover and the compute to spend. [SLIDE 9 — Distillation equals the thesis, other direction] Notice this works from the other direction too. If reasoning were a capability the small models lacked — a knowledge gap — then distilling traces could not have transferred it. The fact that SFT on traces did transfer reasoning behavior is evidence that the small bases already had latent capability. The traces steered them to express it. The student inherits the behavior, not new ability. [SLIDE 10 — R1 vs Qwen3] R1 and Qwen3 are the two canonical reasoning references, and they teach different lessons. R1 is the distillation reference. Teacher pipeline plus SFT-only transfer. Use it when you are building a reasoner by distilling from a teacher. Qwen3, the next deep-dive, is the hybrid reference. One model that fuses thinking and non-thinking modes via a thinking-budget mechanism, so you get a reasoner and a fast assistant in one set of weights. Use it when you are shipping a production model that adapts its compute to the question. Know both. [SLIDE 11 — What the field took from R1] Three things became standard after R1. First, CoT distillation is the practical path. Before R1, the assumption was you needed a full RL pipeline. After R1-Distill-Qwen-thirty-two-B beat o-one-mini, the default became: get a strong teacher, curate its traces, SFT your student. Second, verifiable rewards beat reward models for reasoning RL. If you can verify, do not approximate with a judge. Third, RL-on-base proves the capability is latent. R1-Zero is the reference everyone cites when arguing reasoning is steering. The paper is published in Nature. The weights are open at deepseek-ai slash deepseek-r-one. [SLIDE 12 — Anti-patterns] Three anti-patterns. First, assuming RL is required for a reasoning student. R1's own ablation shows SFT-only captures most of the gain. Distill first. Second, dismissing R1-Zero for its messiness. The mixed languages are irrelevant to what it proved. R1's cold-start stage is the fix. Third, distilling from a weak teacher. The teacher's quality is the ceiling. [SLIDE 13 — What you can now do] You can now explain why R1-Zero is the strongest evidence that reasoning is steering. You can draw the four-stage pipeline and what each stage fixes. You can distinguish the distillation from the teacher pipeline and why it beat o-one-mini. And you can defend publishing both the proof and the product. Next, deep-dive eight: Qwen3. The hybrid reference. Let's see how a single set of weights serves both reasoning and fast responses. --- *End of module FTDD-07. Duration: approximately forty-five minutes at one-hundred-forty words per minute.*