ModernBERT

ModernBERT is a highly optimized, encoder-only transformer designed by Answer.AI and LightOn to bring BERT-style models into the modern era with native 8192-token context windows and blistering speed.

ModernBERT represents a complete architectural overhaul of the classic encoder-only transformer (originally limited to 512 tokens). Trained on 2 trillion tokens of English and code, this model integrates modern LLM breakthroughs: Rotary Positional Embeddings (RoPE), GeGLU activations, unpadding, and FlashAttention-3. Available in base (149 million parameters) and large (395 million parameters) configurations, it serves as a drop-in replacement for legacy BERT models, delivering up to a 4x speedup on GPUs while setting new performance benchmarks for semantic search, retrieval-augmented generation (RAG) pipelines, and classification tasks.

https://github.com/AnswerDotAI/ModernBERT