HEM: a margin-based loss for visual categorisation tasks

Training deep neural networks (DNNs) on classification tasks can be performed with a number of different losses, but cross-entropy (CE) loss is the de-facto standard. Here, we propose an alternative loss, high error margin (HEM), which is a margin based loss modified to improve the training dynamics of neural networks. HEM loss is evaluated extensively using a wide range of DNN architectures and benchmark datasets with all experimental settings and training hyper-parameters taken from the literature, and hence, optimised for CE loss. HEM is found to be more effective than CE loss across a range of image-based tasks: unknown class rejection, adversarial robustness, learning with imbalanced data, continual learning, and semantic segmentation (a pixel-wise classification task). HEM is inferior to CE only in terms of clean and corrupt image classification with balanced training data, and this difference is small. We also compare HEM to specialised losses that have previously been proposed to improve performance for specific vision tasks. LogitNorm, a loss achieving state-of-the-art performance on unknown class rejection, produces similar performance to HEM for this task, but is much poorer for continual learning and semantic segmentation. Logit-adjusted loss, designed for imbalanced data, has superior results to HEM for that task, but performs worse on unknown class rejection and semantic segmentation. DICE, a popular loss for semantic segmentation, is inferior to HEM loss on all tasks, including semantic segmentation. Overall, HEM is competitive with the best alternative loss for all the tasks we have used and performs better than all other tested losses in terms of rejecting out-of-distribution examples, for continual learning, and by a substantial margin for semantic segmentation.