Loss, Metrics & Evaluation

Loss drives training. Metrics judge usefulness. They are not always the same.

Classification

• Accuracy fails on imbalanced classes
• Precision / recall / F1 depend on the cost of false positives vs negatives
• ROC-AUC vs PR-AUC: prefer PR when positives are rare

Regression

• MAE is robust to outliers; MSE penalizes large errors more
• Report units stakeholders understand

Protocol

Never tune on the test set. Use validation (or cross-validation) for model selection.