Advanced Topics / Transfer Learning & Fine-tuning
Transfer Learning & Fine-tuning
IntermediateReuse knowledge from a model pre-trained on a large dataset to solve a new task with less data.
Strategies
- Feature extraction — freeze the backbone, train a new head.
- Fine-tuning — unfreeze some or all layers with a small learning rate.
- Parameter-efficient fine-tuning — LoRA, adapters, prompt tuning.
Example
from torchvision.models import resnet50
model = resnet50(weights="IMAGENET1K_V2")
for p in model.parameters():
p.requires_grad = False
model.fc = nn.Linear(model.fc.in_features, num_classes)Interview questions & answers
Q1. What is the difference between feature extraction and fine-tuning?
Answer: Feature extraction freezes a pretrained backbone and trains a task-specific head. Fine-tuning updates some or all pretrained weights for the new task, often with a smaller learning rate. Freezing is cheaper, while fine-tuning offers more adaptation but increases overfitting risk.
Q2. When can transfer learning hurt performance?
Answer: Negative transfer can occur when the source task or data differs substantially from the target, so learned features or biases are unsuitable. Compare against a relevant baseline, inspect domain differences and validate which layers to freeze or adapt.
Q3. How does LoRA make fine-tuning parameter-efficient?
Answer: LoRA freezes selected pretrained weight matrices and learns low-rank updates through smaller trainable matrices. This reduces trainable parameter count and optimizer-state memory. It does not remove the cost of running the base model, and the chosen rank limits adaptation capacity.