It earns a place when
- Use prompting for fast task changes, prototyping and low-volume workflows where examples can define the desired output.
- You can evaluate it against a credible baseline.
- Its output fits the decision and data constraints.
Machine learning models · Phase 5 · Lesson 39
Pretraining, prompting and in-context learning
Distinguish pretraining from prompting and understand why examples in a prompt can change behaviour.
Lesson goal
Distinguish pretraining from prompting and understand why examples in a prompt can change behaviour.
The core idea
Pretraining changes the model's parameters. Prompting changes the input context. In-context learning changes the current computation without updating weights.
Mental model
Picture it this way. Pretraining changes the model's parameters. Prompting changes the input context. In-context learning changes the current computation without updating weights. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
Pretraining minimises a corpus objective over many parameter updates. At inference, the prompt conditions the model's token distribution. Few-shot examples act as temporary context.
Worked example
A classifier prompt can show three labelled examples and ask for a fourth. The model may infer the task format, but performance depends on example quality, ordering and ambiguity.
When to use it
Do not make it the default when
- Do not assume a prompt permanently teaches the model or that a longer prompt is always better.
- A simpler model has not been tested.
- The data or target definition is still unclear.
Failure modes
Watch for this. Instruction conflict, irrelevant context, positional effects and prompt injection can change outputs. Few-shot examples can accidentally reveal the answer.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Do this. Create a small prompt test set. Vary only the order, number and wording of examples, and record output consistency rather than one impressive result.
Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What changes during prompting?
2. What is in-context learning?
3. Why can prompt order matter?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
Hugging Face LLM Course. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.