Advanced Topics / Training Best Practices
Training Best Practices
IntermediateA practical checklist for training, debugging and evaluating deep networks.
Workflow
- Split data into train / validation / test sets.
- Start with a simple baseline; overfit a single batch first.
- Normalise inputs and use proper weight initialisation.
- Tune learning rate first, then batch size and regularisation.
- Track metrics with tools like TensorBoard or Weights & Biases.
Evaluation metrics
| Task | Metrics |
|---|---|
| Classification | Accuracy, Precision, Recall, F1, ROC-AUC |
| Regression | MSE, RMSE, MAE, R² |
| Detection | mAP, IoU |
| Generation | BLEU, ROUGE, Perplexity, FID |
Scaling up
- Mixed-precision training (FP16/BF16)
- Gradient accumulation
- Data and model parallelism
- Checkpointing
Interview questions & answers
Q1. How do you prevent data leakage during evaluation?
Answer: Create splits before fitting preprocessing, and learn transformations only from training data. Keep related samples together when needed, and use time-ordered splits for forecasting. Use validation data for model selection and reserve the test set for final evaluation.
Q2. Why try to overfit one small batch before full training?
Answer: A sufficiently capable model should usually be able to reduce loss substantially on a tiny fixed batch. Failure can expose incorrect labels, loss configuration, missing gradients or update bugs. Temporarily disable augmentation and strong regularization to make this diagnostic easier.
Q3. How does gradient accumulation differ from using a larger batch directly?
Answer: Accumulation sums gradients across several micro-batches before one optimizer step, allowing a larger effective batch with less activation memory. Normalize the loss consistently and step schedules at the intended update frequency. It is not always identical to a large batch, especially with batch-dependent layers such as Batch Normalization.