Pith. sign in

REVIEW 4 major objections 7 minor 18 references

The impact of fine tuning in LLaMA on hallucinations for named entity extraction in legal documentation

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning LLaMA models on a small human-corrected dataset substantially reduces hallucinations in legal entity extraction, lifting complete-pipeline accuracy from 39.5% with regex to 79.4%.

desk verdict Useful applied benchmark for Spanish legal NER, but the headline accuracy gain over regex is overstated because the test set conditions on successful segmentation. read the letter →

arxiv 2506.08827 v1 pith:MBQLCIIK submitted 2025-06-10 cs.CL cs.AI

Francisco Vargas , Alejandro González Coene , Gaston Escalante , Exequiel Lobón , Manuel Pulido This is my paper
classification cs.CLcs.AI
keywords namedentityrecognitionlegaldocumentsLLaMAfine-tuninghallucinationsretrieval-augmentedgenerationLoRASpanishcourtrulings
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a two-stage pipeline for extracting disability percentages and compensation amounts from Spanish court rulings on traffic accidents: a vector-similarity segmenter narrows each long judgment to relevant blocks, and a large language model then extracts the entities. The central claim is that fine-tuning open LLaMA models with a small, human-corrected dataset substantially reduces the hallucinations the base models produce when the target entity is absent from the selected segment. The fine-tuned LLaMA-2 70B reaches 79.4% accuracy on the complete pipeline, versus 39.5% for the existing regular-expression method, while the base LLaMA-3 8B already reaches 76.6% and GPT-4 Turbo 86.1%. If correct, this makes an open-source, fine-tuned model a practical replacement for manual and regex-based extraction in this domain.

What carries the argument

The load-bearing mechanism is a retrieval-augmented segmentation stage followed by prompted, optionally fine-tuned, LLM extraction. Documents are split into 120-token blocks, embedded with a multilingual transformer (paraphrase-multilingual-MiniLM-L12-v2 or the OpenAI embedding model text-embedding-ada-002), stored in a similarity-search index, and retrieved by cosine similarity against a query; each retrieved block is expanded to 360 tokens with its neighbours. This segmenter determines whether the extraction model ever sees the entity. The second stage is an LLM prompted to output the entity fields; for LLaMA models, fine-tuning uses QLoRA, a quantized low-rank adaptation method, applied to the attention layers and trained for three epochs on the cleaned dataset. The hallucination comparison runs on segments that contain none of the entities, isolating how often each model invents an answer rather than extracting one.

What would settle it

Run the entire pipeline on a random sample of court rulings without filtering by segmenter retrieval, counting any entity the segmenter fails to put into a candidate segment as a miss; if segmenter recall is below 100%, the end-to-end accuracy will fall below the reported 79.4%.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that parameter-efficient fine-tuning (QLoRA) of LLaMA models on roughly 861 human-verified samples turns an unreliable extractor into a production-viable one. The accuracy of LLaMA-2 7B improves substantially when fine-tuned, and on segments that deliberately contain none of the target entities fine-tuning cuts its hallucinated extractions by 47.78 percentage points. LLaMA-2 70B fine-tuned reaches 79.4% complete-pipeline accuracy, a 17.7-point gain over its base version; LLaMA-3 8B's base version is already at 76.6%, nearly matching the fine-tuned LLaMA-2 70B. GPT-4 Turbo remains the best extractor at 86.1%. The paper also reports that recall is lower than accuracy, so the fine-tuned models are conservative, and that a hallucination-detection rule based on the minimum token probability fails to separate correct from incorrect extractions.

Load-bearing premise

The test set was built by discarding entity samples that the vector segmenter could not retrieve into candidate segments (Section 2.5), so the reported accuracy implicitly assumes those misses are rare or unrepresentative of real legal documents.

Editorial extensions

If this is right

  • A retrieval-augmented segmenter plus a fine-tuned open LLM can replace regular-expression pipelines for disability and compensation extraction, roughly doubling end-to-end accuracy.
  • Fine-tuning's main benefit for smaller LLaMA-2 models is suppressing invented answers when the entity is not present in context, not just improving recognition when it is present.
  • Newer base models such as LLaMA-3 8B narrow or eliminate the fine-tuning gap, so the incremental value of task-specific training should shrink as base models improve.
  • Because recall trails accuracy, production use of the fine-tuned models will intentionally omit some correct extractions in exchange for fewer wrong ones.
  • Automatic extraction makes it feasible to compute monthly point values and disability distributions from a full year of rulings in about half an hour of compute, where manual analysis took over a week per month.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the test set was constructed by discarding entity samples the segmenter failed to retrieve, the reported 79.4% likely overstates the accuracy the full pipeline would achieve on arbitrarily formatted rulings; the true ceiling is set by segmenter recall.
  • The failure of the minimum-token-probability rule suggests hallucination detection needs answers that carry an explicit 'not found' option, such as fine-tuning on empty extractions, which would also close part of the recall gap.
  • If the segmenter is the bottleneck, the same fine-tuned extractor could be evaluated against a stronger or sliding-window segmenter, or with queries specialized per entity type, to measure how much of the 20-point gap to GPT-4 Turbo comes from segmentation rather than extraction.
  • The paper's assumption that a sample of one entity type helps the model recognize similar entity types could be tested directly with few-shot extraction experiments that vary the number of samples per entity type.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes a two-stage pipeline for extracting disability percentages and compensation amounts from Argentine traffic-accident judicial rulings: a segmenter (regex-based or semantic vector search) selects relevant text blocks, and an LLM extracts the entities. The authors evaluate LLaMA-2 7B/70B and LLaMA-3 8B, with and without QLoRA fine-tuning, plus GPT-3.5/4 Turbo, on a test set of 30 documents (120 samples). They report that fine-tuning substantially reduces hallucinations in LLaMA models and that the proposed vector-segmentation pipeline reaches 79.4% accuracy, far exceeding the 39.5% regex baseline, with GPT-4 Turbo at 86.1%. They also use the extracted entities to compute a monthly 'point value' statistic and disability distributions for insurance applications.

Significance. If the reported results are robust, this is a useful applied contribution: it shows that a fine-tuned open-source LLaMA model can approach a commercial model on a narrow Spanish legal NER task, and it provides a concrete demonstration that fine-tuning reduces false extractions on negative segments. The human-corrected labeling effort and the dedicated negative-segment hallucination experiment are strengths. The main limitations are the filtered test set, the small sample size with no uncertainty quantification, and the partially self-referential labeling procedure, so the headline quantitative claims need revision before the conclusions can be accepted.

major comments (4)
  1. [Section 2.5 and Section 3.2]
  2. [Section 3.2 and Section 3.3]
  3. [Section 3.3]
  4. [Section 2.5]
minor comments (7)
  1. [Abstract, Section 3.2, Conclusions] Several reported numbers are inconsistent across the manuscript. LLaMA-3 8B base accuracy is given as 76.6% in the abstract but 77.2% in Section 3.2, and LLaMA-2 70B base accuracy is 61.7% in the abstract but 58.90% in the Conclusions. Please harmonize these values and state whether all gains are absolute percentage points or relative improvements.
  2. [Section 3.1] The segmentation quality assessment is said to be performed on 'a test set of 30 sentences,' while the rest of the paper refers to 30 documents. Please correct the typo and describe how the 30 segmentation evaluation items were selected.
  3. [Section 2.2 and Section 3.2] The proposed segmentation method is introduced using paraphrase-multilingual-MiniLM-L12-v2, but the extraction results in Section 3.2 are reported using segments from text-embedding-ada-002. Please clarify which segmenter feeds the final pipeline and why the MiniLM variant is not used there, or report both.
  4. [Section 3.2] GPT-3.5 Turbo is mentioned as a baseline in the text, but no accuracy number is reported for it. Please either add the number or remove the comparison.
  5. [Section 2.6 and Appendix] Hyperparameters are listed (learning rate 5e-5, 3 epochs, LoRA rank 8), but the prompt templates are only referenced through an external GitHub link. Including the exact prompts in an appendix would substantially improve reproducibility.
  6. [Section 3.4, Eq. (1)] The point-value formula is difficult to parse as typeset. Please provide an unambiguous LaTeX rendering and explain the units and economic reasoning for dividing each amount by the relevant disability percentage.
  7. [Section 3.3] There is a duplicated paragraph describing the minimum-probability hallucination detection method; please remove the duplicate.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported 79.4% complete-pipeline accuracy is conditional on segmenter success by test-set construction; fine-tuning and hallucination findings remain independent.

  1. self definitional [Section 2.5 (test set construction) and Section 3.2 (complete pipeline results)]
    "There are labels that would be impossible to extract as they are not found in the segments provided by the segmenter. In these cases, the label should be empty. Therefore, a second dataset was generated. ... If the entities were not found in the segment, those samples were discarded. ... As a test database, 30 documents within the same domain were used, resulting in 120 samples. These documents do not belong to the training set, and their preparation was similar to the manual clean-up performed for the training Dataset 2. (Section 2.5) ..."

    The 'complete pipeline' accuracy is computed on a test set from which every sample that the vector segmenter failed to retrieve was removed, using the same cleanup as Dataset 2. By construction, the denominator contains only cases where the segmenter already succeeded, so the 79.4% is a conditional extraction accuracy, not an end-to-end pipeline accuracy. The RegEx baseline's 39.5% is reported for its full pipeline with no equivalent retrieval filter, making the comparison asymmetric. The paper's own segmentation QA (80.91% for text-embedding-ada-002, Section 3.1) shows that the excluded retrieval failures are substantial; counting them as errors would lower the unconditional end-to-end accuracy.

full rationale

The core findings about fine-tuning and hallucination reduction are measured directly and are not circular: the finetuned-versus-base comparisons use the same segments, and the hallucination test on negative segments (Section 3.3) does not depend on the test-set filtering. The central practical claim (79.4% vs 39.5% 'complete pipeline') is, however, partly defined by the test set: retrieval failures are excluded from the proposed method's denominator while the RegEx baseline is scored unconditionally. This is a self-definitional reduction of the headline comparison rather than a forced mathematical tautology; the independent hallucination and fine-tuning results keep the paper from being wholly circular. No self-citation chain or imported uniqueness theorem is involved, and the human-corrected labels provide independent grounding for the entity-extraction measurements.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central pipeline depends on several hand-chosen hyperparameters (block size, context length, LoRA settings, prompt). The reliability of the gold labels is a domain assumption, and the point-value formula is an ad hoc statistical construction.

free parameters (8)
  • Segment block size = 120 tokens
    Chosen due to embedding model token limit; affects which text is retrieved and thus what the LLM sees.
  • Context expansion length = 360 tokens (three 120-token blocks)
    Concatenation of retrieved block with neighbors; manually chosen by the authors.
  • Regex context window = 500 characters before and after the '%' match
    Manual choice for the baseline segmentation method.
  • LoRA rank = 8
    LoRA hyperparameter selected by the authors for fine-tuning.
  • Learning rate = 5e-5
    Training hyperparameter used in QLoRA fine-tuning.
  • Epochs = 3
    Training hyperparameter used in QLoRA fine-tuning.
  • Validation fraction = 10%
    Split used to select the best checkpoint during training.
  • Prompt template = optimized prompt on GitHub
    Prompt engineering is a free parameter; the exact prompt is not fully specified in the paper text.
assumptions (5)
  • domain assumption Legal documents in the corpus are sufficiently structured that fixed-size token blocks and semantic search can isolate segments containing the target entities.
    The whole segmentation approach relies on this assumption; described in Section 2.2.
  • domain assumption The manually corrected GPT-4-generated labels are a valid gold standard.
    Used to build both training and test sets; described in Section 2.5.
  • domain assumption Entity extraction accuracy measured on the 120-sample test set is representative of real-world performance.
    No error bars, repeated runs, or significance tests are provided; Sections 2.5 and 3.2.
  • standard math TF-IDF and cosine similarity provide useful retrieval signals for segment selection.
    Standard information retrieval techniques used without modification in Section 2.2.
  • ad hoc to paper The point value formula Eq. (1) assumes moral damage is divided by the physical disability percentage.
    This formula is introduced in Section 3.4 for the statistical application and is not derived from any external standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The impact of fine tuning in LLaMA on hallucinations for named entity extraction in legal documentation." pith.science (2026). https://pith.science/paper/MBQLCIIK

@misc{pith2026250608827,
  author       = {Pith},
  title        = {Pith review of: The impact of fine tuning in LLaMA on hallucinations for named entity extraction in legal documentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MBQLCIIK}},
  note         = {Machine review of arXiv:2506.08827}
}
read the original abstract

The extraction of information about traffic accidents from legal documents is crucial for quantifying insurance company costs. Extracting entities such as percentages of physical and/or psychological disability and the involved compensation amounts is a challenging process, even for experts, due to the subtle arguments and reasoning in the court decision. A two-step procedure is proposed: first, segmenting the document identifying the most relevant segments, and then extracting the entities. For text segmentation, two methodologies are compared: a classic method based on regular expressions and a second approach that divides the document into blocks of n-tokens, which are then vectorized using multilingual models for semantic searches (text-embedding-ada-002/MiniLM-L12-v2 ). Subsequently, large language models (LLaMA-2 7b, 70b, LLaMA-3 8b, and GPT-4 Turbo) are applied with prompting to the selected segments for entity extraction. For the LLaMA models, fine-tuning is performed using LoRA. LLaMA-2 7b, even with zero temperature, shows a significant number of hallucinations in extractions which are an important contention point for named entity extraction. This work shows that these hallucinations are substantially reduced after finetuning the model. The performance of the methodology based on segment vectorization and subsequent use of LLMs significantly surpasses the classic method which achieves an accuracy of 39.5%. Among open-source models, LLaMA-2 70B with finetuning achieves the highest accuracy 79.4%, surpassing its base version 61.7%. Notably, the base LLaMA-3 8B model already performs comparably to the finetuned LLaMA-2 70B model, achieving 76.6%, highlighting the rapid progress in model development. Meanwhile, GPT-4 Turbo achieves the highest accuracy at 86.1%.

Figures

Figures reproduced from arXiv: 2506.08827 by the authors.

Figure 1
Figure 1. Complete extraction pipeline. 2.5 Training dataset There are 278 filtered and preprocessed documents available (see Section 2.1), which are useful for training the model. To perform the training of the model, a [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Segmentation results by model. 3.2 Extraction results The results obtained using the segments from the text-embedding-ada-002 model with the finetuned models of LLaMA-2 7b, LLaMA-2 70b, and LLaMA-3 8b, are shown in Graph 3 a). Significant improvements in performance were observed when tuning with the specialized datasets. On the other hand, there is a gain of 9.4% in accuracy from Dataset 1 to Dataset 2. The latter … view at source ↗
Figure 3
Figure 3. a) Finetuning impact on accuracy for each model considering a perfect segmen￾tation. b) Accuracy of the complete model including the segmentation and the entity extraction language model. c) Recall from the entire pipeline taking into account seg￾menter and model. The accuracies shown in [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance of the models with segments that do not contain the entities to extract. To reduce hallucinations and their impact, a hallucination detection method was evaluated using the probabilities associated with the tokens generated by the model. The mitigation meth…
Figure 5
Figure 5. Figure 5: Variation of the point e value CPI during 2023, obtained through the proce￾dure described in [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 6 canonical work pages

  1. [1]

    Alemi, A.A., Ginsparg, P.: Text segmentation based on semantic word embeddings (2015), https://arxiv.org/abs/1503.05543

    arXiv 2015

  2. [2]

    In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

    Barrow, J., Jain, R., Morariu, V., Manjunatha, V., Oard, D., Resnik, P.: A joint model for document segmentation and segment labeling. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 313– 322 (Jul 2020). https://doi.org/10.18653/v1/2020.acl-main.29

    doi:10.18653/v1/2020.acl-main.29 2020

  3. [3]

    Bayomi, M., Lawless, S.: C-hts: A concept-based hierarchical text segmentation approach (may 2018)

    work page 2018

  4. [4]

    Chen, L., Zaharia, M., Zou, J.: How is chatgpt’s behavior changing over time? (2023), arXiv 2307.09009

    arXiv 2023

  5. [5]

    Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: Efficient finetun- ing of quantized llms (2023), arXiv 2305.14314

    arXiv 2023

  6. [6]

    Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.E., Lomeli, M., Hosseini, L., Jégou, H.: The faiss library (2024), arXiv 2401.08281

    arXiv 2024

  7. [7]

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models (2021), arXiv 2106.09685

    arXiv 2021

  8. [8]

    Jin, R., Du, J., Huang, W., Liu, W., Luan, J., Wang, B., Xiong, D.: A comprehen- sive evaluation of quantization strategies for large language models (2024), arXiv 2402.16775

    arXiv 2024

Show all 18 references
  1. [9]

    MatSciRN: Other Electronic (2011), https://api.semanticscholar.org/CorpusID:9633141

    Karaa, W.B.A.: Named entity recognition using web document corpus. MatSciRN: Other Electronic (2011), https://api.semanticscholar.org/CorpusID:9633141

    2011

  2. [10]

    Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners (2023), arXiv 2205.11916

    2023 arXiv

  3. [11]

    Poder judicial de la nación argentina, https://www.pjn.gov.ar/

  4. [12]

    Reimers, N., Gurevych, I.: Código fuente paraphrase-multilingual-minilm- l12-v2 huggingface., https://huggingface.co/sentence-transformers/paraphrase- multilingual-MiniLM-L12-v2

  5. [13]

    Natural Language Processing Journal6, 100056 (2024)

    Roumeliotis, K.I., Tselikas, N.D., Nasiopoulos, D.K.: Llms in e-commerce: A comparative analysis of gpt and llama models in product review evaluation. Natural Language Processing Journal6, 100056 (2024). https://doi.org/10.1016/j.nlp.2024.100056

    2024

  6. [14]

    Thoma, M.: Pypdf2, https://pypi.org/project/PyPDF2/

  7. [15]

    Touvron, H., Martin, L., Stone, K., et al.: Llama 2: Open foundation and fine-tuned chat models (2023), arXiv 2307.09288

    2023 arXiv

  8. [16]

    In: Hassanien, A.E., Bhatnagar, R., Darwish, A

    Vardhan, H., Surana, N., Tripathy, B.K.: Named-entity recognition for legal docu- ments. In: Hassanien, A.E., Bhatnagar, R., Darwish, A. (eds.) Advanced Machine Learning Technologies and Applications. pp. 469–479. Springer Singapore, Singa- pore (2021)

    2021

  9. [17]

    Varshney, N., Yao, W., Zhang, H., Chen, J., Yu, D.: A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence gen- eration (2023), arXiv 2307.03987

    2023 arXiv

  10. [18]

    Xu, L., Xie, H., Qin, S.Z.J., Tao, X., Wang, F.L.: Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment (2023), arXiv 2312.12148

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.