Lab Specification — Module FTDD-07: DeepSeek-R1

python3.11 -m venv ftdd07-env && source ftdd07-env/bin/activate
pip install -q transformers accelerate torch
# On Apple Silicon, torch uses MPS automatically. On CUDA, it uses CUDA.

import torch
print(f"PyTorch: {torch.__version__}")
print(f"MPS available: {torch.backends.mps.is_available()}")
print(f"CUDA available: {torch.cuda.is_available()}")

from transformers import AutoModelForCausalLM, AutoTokenizer

STUDENT_ID = "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B"  # the distilled student

tokenizer = AutoTokenizer.from_pretrained(STUDENT_ID)
student = AutoModelForCausalLM.from_pretrained(
    STUDENT_ID,
    torch_dtype=torch.float16,
    device_map="auto",
)
student.eval()

def generate(model, prompt, max_new_tokens=1024):
    messages = [{"role": "user", "content": prompt}]
    text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False, temperature=1.0)
    return tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

# A problem that rewards deliberation
MATH_PROBLEM = "A train travels 60 km in 45 minutes. What is its average speed in km/h? Show your reasoning."

print("=== R1-DISTILL-QWEN-1.5B (the distilled student) ===")
student_output = generate(student, MATH_PROBLEM)
print(student_output)

BASE_ID = "Qwen/Qwen2.5-1.5B"  # a non-distilled instruction-tuned Qwen of the same size

base_tokenizer = AutoTokenizer.from_pretrained(BASE_ID)
base = AutoModelForCausalLM.from_pretrained(
    BASE_ID,
    torch_dtype=torch.float16,
    device_map="auto",
)
base.eval()

def generate_base(model, prompt, max_new_tokens=512):
    messages = [{"role": "user", "content": prompt}]
    text = base_tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = base_tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False)
    return base_tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

print("=== QWEN2.5-1.5B (non-distilled, same size) ===")
base_output = generate_base(base, MATH_PROBLEM)
print(base_output)

def analyze_chain(text):
    has_think = "<think>" in text or text.strip().startswith("Think") or "**reasoning**" in text.lower()
    has_self_correct = any(w in text.lower() for w in ["wait", "let me", "actually", "reconsider", "check"])
    length = len(text.split())  # rough token proxy
    # crude final-answer extraction
    lines = [l.strip() for l in text.split("\n") if l.strip()]
    final_line = lines[-1] if lines else ""
    return {
        "visible_chain": has_think,
        "self_correction": has_self_correct,
        "word_count": length,
        "final_line": final_line[:120],
    }

print("=== STUDENT (distilled) ===")
print(analyze_chain(student_output))
print("\n=== BASE (non-distilled) ===")
print(analyze_chain(base_output))

# Lab Specification — Module FTDD-07: DeepSeek-R1

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FTDD-07 — DeepSeek-R1
**Duration**: ~45 minutes
**Environment**: Python 3.11+. A consumer GPU (RTX 4090 / 16–24GB) OR Apple Silicon (M-series, 16GB+ unified) OR free Google Colab T4. ~5GB free disk.

---

## Learning objectives

By the end of this lab you will have:

1. **Loaded R1-Distill-Qwen-1.5B** and observed its structured `<think>` chain-of-thought on a math problem — felt the distilled reasoning behavior.
2. **Compared it against a non-distilled Qwen-1.5B base** on the same problem — witnessed what the distillation transferred (the chain) and what it did not (new capability — the base flails because it was never steered to deliberate).
3. **Measured the chain-of-thought structure** — token length, presence of self-correction, presence of a verifiable final answer — and stated, in your own words, what the distillation installed.
4. **Connected the result to the thesis**: the distilled model inherits the *behavior* of reasoning (the trace shape), not new *ability* — which is why SFT-only distillation works at all.

This lab is about *observing* transferred reasoning, not training. You witness what 800K curated traces install in a 1.5B student.

---

## Phase 0 — Environment setup (5 min)

```bash
python3.11 -m venv ftdd07-env && source ftdd07-env/bin/activate
pip install -q transformers accelerate torch
# On Apple Silicon, torch uses MPS automatically. On CUDA, it uses CUDA.
```

Verify the stack:

```python
import torch
print(f"PyTorch: {torch.__version__}")
print(f"MPS available: {torch.backends.mps.is_available()}")
print(f"CUDA available: {torch.cuda.is_available()}")
```

The 1.5B model in FP16 needs ~3GB VRAM. CPU works but is slow (~3–8 tok/s); MPS/CUDA is comfortable.

---

## Phase 1 — Load R1-Distill-Qwen-1.5B (the student) (8 min)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

STUDENT_ID = "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B"  # the distilled student

tokenizer = AutoTokenizer.from_pretrained(STUDENT_ID)
student = AutoModelForCausalLM.from_pretrained(
    STUDENT_ID,
    torch_dtype=torch.float16,
    device_map="auto",
)
student.eval()

def generate(model, prompt, max_new_tokens=1024):
    messages = [{"role": "user", "content": prompt}]
    text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False, temperature=1.0)
    return tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

# A problem that rewards deliberation
MATH_PROBLEM = "A train travels 60 km in 45 minutes. What is its average speed in km/h? Show your reasoning."

print("=== R1-DISTILL-QWEN-1.5B (the distilled student) ===")
student_output = generate(student, MATH_PROBLEM)
print(student_output)
```

**Record**: the full output. Look specifically for: (a) a `<think>` block or visible chain-of-thought, (b) self-correction ("wait," "let me check"), (c) a clearly stated final answer with units.

> **What just happened (the teaching moment):** This 1.5B model produces a structured reasoning chain. It deliberates. It was *never* trained with RL — it inherited this behavior via SFT on R1's curated traces. The behavior was transferred; the 1.5B base always had latent capability. The traces steered it to the surface.

---

## Phase 2 — Load a non-distilled Qwen-1.5B base for comparison (8 min)

```python
BASE_ID = "Qwen/Qwen2.5-1.5B"  # a non-distilled instruction-tuned Qwen of the same size

base_tokenizer = AutoTokenizer.from_pretrained(BASE_ID)
base = AutoModelForCausalLM.from_pretrained(
    BASE_ID,
    torch_dtype=torch.float16,
    device_map="auto",
)
base.eval()

def generate_base(model, prompt, max_new_tokens=512):
    messages = [{"role": "user", "content": prompt}]
    text = base_tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = base_tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False)
    return base_tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

print("=== QWEN2.5-1.5B (non-distilled, same size) ===")
base_output = generate_base(base, MATH_PROBLEM)
print(base_output)
```

**Record**: the base's output. It will typically answer more directly, with shorter or no visible deliberation, and may get the arithmetic wrong (45 minutes = 0.75 hours; 60/0.75 = 80 km/h — a common error is treating 45 min as 0.45 h).

---

## Phase 3 — Measure what the distillation installed (8 min)

```python
def analyze_chain(text):
    has_think = "<think>" in text or text.strip().startswith("Think") or "**reasoning**" in text.lower()
    has_self_correct = any(w in text.lower() for w in ["wait", "let me", "actually", "reconsider", "check"])
    length = len(text.split())  # rough token proxy
    # crude final-answer extraction
    lines = [l.strip() for l in text.split("\n") if l.strip()]
    final_line = lines[-1] if lines else ""
    return {
        "visible_chain": has_think,
        "self_correction": has_self_correct,
        "word_count": length,
        "final_line": final_line[:120],
    }

print("=== STUDENT (distilled) ===")
print(analyze_chain(student_output))
print("\n=== BASE (non-distilled) ===")
print(analyze_chain(base_output))
```

**Record**: both analyses side by side. Expect the distilled student to show: longer output, self-correction markers, a more explicit final answer. The non-distilled base will typically be shorter and more direct.

---

## Phase 4 — The thesis, in your own words (5 min)

No code. Write 3–5 sentences answering:

1. What behavior did the distillation install in the 1.5B student? Did it install *new capability*, or steer *existing* capability to the surface? What is your evidence?
2. R1's distillation used SFT-only (no RL on the student). Why does SFT-only suffice if reasoning is a steering problem? What would you expect if reasoning were a knowledge problem instead?
3. The student and the base are the same size (1.5B). What does the difference in their output tell you about what fine-tuning does — and does not — change?

---

## Deliverables

Submit `ftdd07-lab-report.md`:

- [ ] Phase 1: the distilled student's full output on the math problem (note the chain structure)
- [ ] Phase 2: the non-distilled base's output on the same problem
- [ ] Phase 3: the `analyze_chain` results for both, side by side
- [ ] Phase 4: your 3–5 sentence thesis statement

---

## Solution key

- **Phase 1**: the distilled student produces a visible chain-of-thought. It typically restates the problem, converts 45 minutes to 0.75 hours, computes 60 / 0.75 = 80 km/h, and may self-correct or verify. The `<think>` structure (or visible deliberation) is the transferred behavior.
- **Phase 2**: the non-distilled base often answers more directly. A common failure: treating 45 minutes as 0.45 hours (yielding ~133 km/h, wrong). Or correct but without visible deliberation. The point is the *difference in process*, not always the final answer.
- **Phase 3**: the student typically shows higher word count, self-correction markers, and a more explicit final answer. The base is shorter and more direct. (Exact numbers vary by run; the *contrast* is the lesson.)
- **Phase 4**: a correct thesis statement names (a) the distillation installed the *behavior* of deliberation (chain structure, self-correction), steering existing capability to the surface — not new ability; (b) SFT-only suffices because reasoning is steering — the 1.5B base already had latent capability from pretraining, and the traces redirect probability mass to express it. If reasoning were knowledge, SFT on traces could not transfer it; (c) the difference in output between same-size models shows fine-tuning changes *behavior* (how the model reasons), not *capability ceiling* (what it can in principle do).

---

## Stretch goals

1. **Try a harder problem.** Swap the math problem for one requiring multi-step reasoning (e.g., a probability question or an AIME-style problem). Observe whether the student's chain grows longer and whether it still self-corrects. This is the adaptive compute behavior distillation transfers.
2. **Vary the base.** Repeat Phase 2 with a larger non-distilled model (e.g., Qwen2.5-7B). Does the larger base produce deliberation without distillation? This tests whether the *capability* was latent (larger base shows it spontaneously) versus truly absent.
3. **Inspect the chat template.** Print `tokenizer.chat_template` for the distilled student and note the `<think>` scaffolding baked into the template. This is the format anchor from R1's cold-start stage, inherited by the student. (Connects to Module FT07 — tokenizers and chat templates.)