It earns a place when
- Use these models when global relationships or cross-modal retrieval and reasoning matter and suitable data and compute exist.
- You can evaluate it against a credible baseline.
- Its output fits the decision and data constraints.
Machine learning models · Phase 5 · Lesson 42
Vision transformers and multimodal models
Understand how transformers process image patches and how multimodal systems align different data types.
Lesson goal
Understand how transformers process image patches and how multimodal systems align different data types.
The core idea
A vision transformer turns image patches into tokens. A multimodal model learns relationships between representations from text, images, audio or other modalities.
Mental model
Picture it this way. A vision transformer turns image patches into tokens. A multimodal model learns relationships between representations from text, images, audio or other modalities. The important question is what assumption this picture makes, and whether that assumption fits the data.
Mathematical core
Patch embeddings produce a token sequence, positional information is added, and self-attention mixes patch representations. Contrastive objectives can align image and text embeddings.
Worked example
An image-text model can retrieve the caption most similar to a photo. A multimodal assistant can answer questions about an image, but its answer still needs visual and factual evaluation.
When to use it
Do not make it the default when
- Do not assume a multimodal answer proves the model saw the relevant detail, and do not ignore image resolution or modality-specific bias.
- A simpler model has not been tested.
- The data or target definition is still unclear.
Failure modes
Watch for this. Small objects, OCR, spatial relations, data imbalance and adversarial inputs can produce confident errors.
When a result looks surprisingly good, inspect the split, target timing, error slices and data-generating process before celebrating.
Practice
Use this as a small experiment rather than a recipe to copy blindly. Change one thing, record the result and explain the change.
Do this. Compare a CNN and a small vision transformer on a dataset matched to their data regime. For a multimodal example, test spatial, counting and text-reading questions separately.
Retrieval check
Answer from memory first. The buttons reveal feedback, but the durable step is explaining why.
1. What becomes a token in a vision transformer?
2. What does contrastive alignment try to do?
3. Why test OCR and counting separately?
Transfer prompt. Describe one real problem where this model or idea would be a sensible candidate. Name the target, the main risk and the metric you would inspect.
Primary source
Dive into Deep Learning. Use the source for the deeper treatment after you can explain the lesson's core idea without looking.