Abstract
ABSTRACT This paper introduces a Gaussian mixture model designed to explore the implications of long‐tailedness on classification performance. Our study reveals that simple under‐specified classifiers are inherently limited in reducing generalization error within this framework, a limitation overcome by well‐specified and even more complex over‐specified classifiers capable of fitting rare examples. Through theoretical analysis and empirical evaluation on both synthetic and real datasets, we demonstrate how the performance gap between under‐specified and well/over‐specified classifiers varies with changes in the tail length of the distribution. Our results validate the necessity of fitting rare instances in training datasets to achieve enhanced generalization capabilities in models faced with long‐tail distributed data.