Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regime
Hugo Abonizio, Thales Almeida, Roberto Lotufo, Rodrigo Nogueira
Resumo
Large language models (LLMs) often require vast amounts of text to effectively acquire new knowledge. While continuing pre-training on large corpora or employing retrieval-augmented generation (RAG) has proven successful, updating an LLM with only a few thousand or million tokens remains challenging. In this work, we investigate the task of injecting small, unstructured information into LLMs and its relation to the catastrophic forgetting phenomenon. We use a dataset of recent news -- ensuring no overlap with the model's pre-training data -- to evaluate the knowledge acquisition by probing the…
De quem é
Maritaca AIautor: Hugo Abonizio (cofundador)
NeuralMindautor: Roberto Lotufo (cofundador e CTO)
Outros papers das mesmas organizações
- PRISM: Transfer and Limits of Activation Interventions in Tool-Calling Language ModelsMaritaca AI
- Structural Segmentation for Brazilian Legal Document RetrievalNeuralMind
- Lumen-bench: Cross-Lingual Safety Gaps and the Say-Do Problem in Tool-Calling Large Language ModelsMaritaca AI
- Eleven Years of BRACIS: A Meta-Scientific Study of the Brazilian Conference on Intelligent SystemsMaritaca AI
- Legal Nugget Extraction for Granular Retrieval over Long Jurisprudential TextsNeuralMind
- BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance ExamsTropic-AI