Curió-Edu 7B: Examining Data Selection Impacts in LLM Continued Pretraining
Thales Sales Almeida, Rodrigo Nogueira, Hélio Pedrini
Abstract
Continued pretraining extends a language model's capabilities by further exposing it to additional data, often tailored to a specific linguistic or domain context. This strategy has emerged as an efficient alternative to full retraining when adapting general-purpose models to new settings. In this work, we investigate this paradigm through Curió 7B, a 7-billion-parameter model derived from LLaMA-2 and trained on 100 billion Portuguese tokens from the ClassiCC-PT corpus - the most extensive Portuguese-specific continued-pretraining effort above the three-billion-parameter scale to date. Beyond …
Who it is from
Maritaca AIautor: Rodrigo Nogueira (fundador e CEO)
Other papers from the same organizations
- PRISM: Transfer and Limits of Activation Interventions in Tool-Calling Language ModelsMaritaca AI
- Lumen-bench: Cross-Lingual Safety Gaps and the Say-Do Problem in Tool-Calling Large Language ModelsMaritaca AI
- Eleven Years of BRACIS: A Meta-Scientific Study of the Brazilian Conference on Intelligent SystemsMaritaca AI
- BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance ExamsTropic-AI
- moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERTTropic-AI
- LLM-Based Persuasion Enables Guardrail Override in Frontier LLMsMaritaca AI