BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning
João Guilherme Alves Santos, Giovana Kerche Bonás, Thales Sales Almeida
Abstract
With the growing capabilities of Large Language Models (LLMs), there is an increasing need for robust evaluation methods, especially in multilingual and non-English contexts. We present an updated version of the BLUEX dataset, now including 2024-2025 exams and automatically generated image captions using state-of-the-art models, enhancing its relevance for data contamination studies in LLM pretraining. Captioning strategies increase accessibility to text-only models by more than 40%, producing 1,422 usable questions, more than doubling the number in the original BLUEX. We evaluated commercial …
Who it is from
Maritaca AIautor: Thales Almeida (cofundador)
Other papers from the same organizations
- PRISM: Transfer and Limits of Activation Interventions in Tool-Calling Language ModelsMaritaca AI
- Lumen-bench: Cross-Lingual Safety Gaps and the Say-Do Problem in Tool-Calling Large Language ModelsMaritaca AI
- Eleven Years of BRACIS: A Meta-Scientific Study of the Brazilian Conference on Intelligent SystemsMaritaca AI
- BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance ExamsTropic-AI
- moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERTTropic-AI
- LLM-Based Persuasion Enables Guardrail Override in Frontier LLMsMaritaca AI