Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders
Lucas Hideki Takeuchi Okamura, Alexandre Alcoforado, Anna Helena Reali Costa
Abstract
Portuguese remains underrepresented in text embedding evaluation, despite being one of the most widely spoken languages in the world. As a result, embedding models are often selected based on English or multilingual metrics, while their effectiveness in Portuguese remains unclear. We present MTEB-PT, a Portuguese benchmark constructed from a subset of MMTEB, comprising 14 existing datasets across Semantic Textual Similarity (STS), classification, retrieval, and reranking. We use this benchmark to evaluate 17 open- and closed-source embedding models under a unified protocol. Our results show th…
Who it is from
IAra Projectautor: Alexandre Alcoforado (membro da organização no Hugging Face)