paper7Updated Nov 21, 2026 · Português
← All papers

Automatic Legal Writing Evaluation of LLMs

Ramon Pires, Roseval Malaquias Junior, Rodrigo Nogueira

Abstract

Despite the recent advances in Large Language Models, benchmarks for evaluating legal writing remain scarce due to the inherent complexity of assessing open-ended responses in this domain. One of the key challenges in evaluating language models on domain-specific tasks is finding test datasets that are public, frequently updated, and contain comprehensive evaluation guidelines. The Brazilian Bar Examination meets these requirements. We introduce oab-bench, a benchmark comprising 105 questions across seven areas of law from recent editions of the exam. The benchmark includes comprehensive evalu…

Who it is from

Other papers from the same organizations