Hugging Face has released mmBERT, a new multilingual encoder trained on more than 3 trillion tokens across 1,833 languages. The model builds on the ModernBERT architecture and is the first to significantly improve upon XLM-R, a long-time baseline for multilingual understanding tasks.
mmBERT uses a progressive training schedule instead of training on all languages at once. It starts with 60 high-resource languages, expands to 110, and finally includes all 1,833 languages. The model reduces its masking ratio from 30% to 5% and adjusts the sampling distribution to represent smaller languages better.
This “progressive language addition” approach proved critical for coverage without overfitting. For example, Faroese and Tigrinya — introduced only in the final 100B-token decay phase — still showed substantial performance gains thanks to this strategy.
Community members were curious about this balancing act. Yasir Altaf, an enterprise AI practitioner, asked:
How was it ensured that the low-resource languages don’t drown in the 1,833-language phase? Was there a threshold for ‘minimum viable signal’ per language? Also, how confident are we that the model isn’t dominated by the top 50 languages, even if technically ‘trained’ on 1,833?
In response, Tom Aarsen, Hugging Face engineer and maintainer of Sentence Transformers, explained:
This was checked by evaluating on some of the low-resource languages that are only introduced in the final 100B tokens, such as Tigrinya and Faroese. They observed substantial improvements when these languages were included in the last phase.
mmBERT builds on the ModernBERT architecture, inheriting its fast, memory-efficient backbone with Flash Attention 2 and unpadded sequence processing, allowing for 8,192-token contexts.
While the base model has just 110M non-embedding parameters, it still rivals much larger multilingual models. A smaller 140M-parameter variant is also available for lighter workloads.
mmBERT follows the ModernBERT design, with a 22-layer encoder and support for sequences up to 8,192 tokens. The base model has 110M non-embedding parameters (307M total), while a smaller 140M variant is available for efficiency.
A distinctive element is the use of model merging. Instead of relying on a single trained model, the team combined three variants — English-focused, 110-language, and all-language — using TIES merging. This helped preserve performance across domains.
In evaluations, mmBERT consistently outperformed earlier multilingual encoders. On GLUE, it matched English-only baselines despite less than a quarter of its training data being English. On XTREME, it showed clear gains in cross-lingual tasks like XNLI and TyDiQA, while maintaining competitive results on structured prediction. For retrieval, mmBERT set new highs on the MTEB v2 multilingual benchmark, and even tied English-only models on the English track.
mmBERT demonstrates that scaling multilingual encoders does not have to come at the cost of efficiency. By balancing coverage with targeted improvements, it sets a new baseline for retrieval, classification, and cross-lingual tasks.
