When high-quality data is lacking, small-model pretraining can solve problems such as “poor model generalization”...

The performance of pretrained models heavily depends on thequantity and quality of available data. Small models have fewer parameters and limited representational capacity, so with insufficient data, they struggle to learn generalized features. This often leads to underfitting or overfitting. As noted in technical documentation (e.g., Hugging Face), local adaptation typically requires large, diverse datasets—something small models handle poorly.