paper7Atualizado em 21/11/2026 · English
← Todos os papers

moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT

Thiago Laitz, Thales Sales Almeida, João Guilherme Alves Santos, Giovana Kerche Bonás

Resumo

Encoder-only transformer models remain essential for production NLP pipelines. We introduce moBERTo, a Portuguese adaptation of ModernBERT obtained through continued pretraining of the ModernBERT-base checkpoint on 60 billion tokens (5 epochs over a 12-billion-token corpus curated from FineWeb2 and filtered with educational and STEM classifiers). We preserve the original architecture, including rotary positional embeddings, alternating local-global attention, flash attention, and unpadding. We evaluate moBERTo across information retrieval (including long-context retrieval at up to 8,192 tokens…

De quem é

Outros papers das mesmas organizações