It earns a place when
- Use fine-tuning when the task is stable, examples are strong and prompting or retrieval is not enough.
- You can evaluate it against a credible baseline.
- Its output fits the decision and data constraints.
Machine learning models · Phase 5 · Lesson 40
Fine-tuning, LoRA, adapters and distillation
Compare weight updates, parameter-efficient adaptation and teacher-student compression.
Lesson goal
Compare weight updates, parameter-efficient adaptation and teacher-student compression.
The core idea
Fine-tuning changes the model for a task. LoRA and adapters add a small trainable path. Distillation asks a smaller student to imitate a larger teacher.
Mental model
Picture it this way. Fine-tuning changes the model for a task. LoRA and adapters add a small trainable path. Distillation asks a smaller student to imitate a larger teacher. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
LoRA learns a low-rank update Delta W = A B while freezing the base W. Distillation can minimise a student loss against teacher logits, often softened by temperature.
Worked example
A support model can be adapted to follow an internal style using a curated dataset. LoRA reduces trainable parameters, while distillation can reduce serving cost.
When to use it
Do not make it the default when
- Do not fine-tune to inject rapidly changing facts, and do not assume a small adapter removes privacy, bias or evaluation obligations.
- A simpler model has not been tested.
- The data or target definition is still unclear.
Failure modes
Watch for this. Overfitting, catastrophic forgetting, data contamination and training-serving mismatch are common. A cheap model can inherit a teacher's errors.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Do this. Design a small adaptation experiment: define a held-out set, choose a base model, decide between prompting, LoRA and distillation, and state what success would mean.
Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What does LoRA usually freeze?
2. What is distillation?
3. When is fine-tuning a poor tool?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
Hugging Face LLM Course. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.