Pith. sign in

REVIEW 3 major objections 5 minor 29 references

Are manual annotations necessary for statutory interpretations retrieval?

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read A 72-billion-parameter LLM, used without fine-tuning, ranks statutory-interpretation sentences as well as or better than models trained on thousands of manually annotated examples.

desk verdict Solid empirical study of annotation budgets for statutory interpretation retrieval, but the headline LLM claim needs a contamination check before it carries the weight the authors put on it. read the letter →

arxiv 2506.13965 v1 pith:XOLOZ2AN submitted 2025-06-16 cs.CL

classification cs.CL
keywords statutoryinterpretationlegalinformationretrievalannotationoptimizationlargelanguagemodelsQwenDeBERTaNDCGactivelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the costly, expert-driven annotation of sentences in court rulings is actually needed to build systems that retrieve judicial interpretations of legal concepts. It reports three experiments on a benchmark of 42 legal concepts and over 27,000 sentences: how many annotations per concept are enough, whether selecting which sentences to annotate matters, and whether a large open-source LLM can replace human annotators entirely. The central finding is that Qwen 2.5 with 72 billion parameters, prompted with annotation guidelines and no fine-tuning, achieves NDCG@10 of 0.777 and NDCG@100 of 0.853 on the full test set, beating every manually trained DeBERTa model on NDCG@100 and coming within 1.4 percentage points of the best manual model on NDCG@10. If correct, this means manual annotation is largely unnecessary for the ranking task, though a small ground-truth set is still needed to evaluate any model.

What carries the argument

The load-bearing machinery is the pairing of a cross-encoder fine-tuned DeBERTa v.3 model (base and large) with the NDCG ranking metric, and, for the LLM experiment, a prompted Qwen 2.5 Instruct 72B whose per-label token probabilities are converted into a real-valued sentence score via a weighted sum of label values. The dataset, built by Savelka, contains 42 legal concepts and over 27,000 sentences from case law, each annotated by law students into four usefulness classes (high, certain, potential, no value). The LLM is constrained by guided decoding to emit only the four valid labels, and the paper computes NDCG@10 and NDCG@100 directly from the resulting ranked lists, which allows direct comparison with the fine-tuned models.

What would settle it

Train a DeBERTa-large or a fine-tuned LLM on a manually annotated set of, say, 200 sorted sentences per concept from the same dataset and compare its NDCG@10 on the test set against the Qwen 2.5 score of 0.777; if the fine-tuned model clearly exceeds that score, the claim that manual annotation is unnecessary would be weakened. A second direct test is to run the same prompt on a different jurisdiction's statutory interpretation corpus and check whether the NDCG gap between LLM and manually trained models persists.

Watch

Extended reading notes

Core claim

The paper claims that manual annotation is not required to obtain a high-quality ranker for statutory interpretation retrieval. Using the dataset and setup introduced by Savelka and extended by Smywiński-Pohl and Libal, the authors show that a prompted open-source LLM, Qwen 2.5 Instruct 72B, can annotate the full test set of more than 11,000 sentences in under 18 minutes on four GH200 nodes, and that the probability-weighted label scores from this model rank sentences better than all DeBERTa-base models and all but the best DeBERTa-large model on NDCG@10, and better than all manual models on NDCG@100. The paper further concludes that for large fine-tuned models, randomly sampling up to 1000 sentences per concept gives results nearly identical to training on the full dataset, and that for base models, sorting candidate sentences with a large model before annotation recovers most of the gap. The authors recommend fully automated LLM annotation as the default, with a small manually annotated set retained only for evaluation.

Load-bearing premise

The paper's main conclusion assumes that the LLM's label probabilities, derived purely from a prompt, capture enough of the true relevance ordering to serve as a full substitute for human annotations, and that this holds across concepts without any concept-specific manual examples.

Editorial extensions

If this is right

  • For large models, annotating about 1000 randomly chosen sentences per legal concept yields performance essentially equal to annotating the full dataset, so annotation budgets can be cut substantially.
  • For base models, first sorting candidate sentences with a large model and then annotating the top 1000 per concept recovers nearly all the performance of full-dataset training, enabling a two-stage pipeline.
  • Fully automated annotation with an open-source LLM is fast and cheap: the paper estimates that annotating even hundreds of thousands of sentences costs very little in rented GPU time.
  • A small manually annotated set, on the order of tens of examples per concept, remains necessary for evaluating model quality, but not for training the ranker.
  • The comparison with GPT-4 reported in prior work suggests that open-source models now match closed models on this specialized annotation task, making the approach reproducible without API costs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that the probability-weighted scoring method could be used directly as a production retrieval signal for statutory interpretation, not just as a labeling step, since the paper shows it ranks sentences better than fine-tuned models on NDCG@100.
  • I infer that the same prompt-and-score pipeline may transfer to other jurisdictions, languages, or specialized annotation tasks with ordinal labels, because the LLM requires no concept-specific training data; this is testable on non-English case law.
  • I infer that the finding puts pressure on the value of collecting large concept-specific annotation sets for ranking tasks, but only if the LLM's ranking quality survives domain shifts, which the paper does not test.
  • I infer that fine-tuning an LLM on even a small manual set could push performance beyond what the paper measured, since the paper explicitly leaves this as an open question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper asks whether manual annotation is necessary for statutory-interpretation sentence retrieval, using the Savelka dataset of 42 legal concepts and over 27,000 sentences. Three experiments are reported. RQ1 samples up to k sentences per concept uniformly at random (k = 100 to 1000) and trains DeBERTa v3 base and large models, finding that the large model reaches near-full-data performance with about 1000 sentences per concept. RQ2 replaces random selection with the top-k sentences ranked by a model trained on the other splits, treating this as an optimistic upper bound for active or transfer learning. RQ3 uses Qwen2.5-72B-Instruct to annotate the full 11,000-sentence test set and computes NDCG@10 and NDCG@100, reporting values of 0.777 and 0.853 for the original prompt, which are competitive with or better than the manually trained DeBERTa models. The paper concludes that full manual annotation is not required for this task, although a small ground-truth set is still needed for evaluation and prompt development.

Significance. If the RQ3 result is valid, the paper makes a strong practical contribution: it suggests that open-source instruction-tuned LLMs can replace most of the costly expert annotation effort for statutory-interpretation retrieval, extending earlier GPT-4 results from 256 sentences to the full test set. The study has notable strengths: the DeBERTa baselines are trained with five random seeds, the RQ2 oracle limitation is explicitly acknowledged, and the use of an open-source model makes the automatic-annotation pipeline more reproducible than closed-API alternatives. However, the central claim rests on the RQ3 NDCG values, and those values are not yet supported by a contamination analysis or any variance estimate. The practical significance of the paper therefore depends on an additional, targeted evaluation that rules out memorization of the public case-law test sentences.

major comments (3)
  1. [§5.3, Table 5] The central conclusion that a 72B LLM makes manual annotation unnecessary depends on the NDCG values in Table 5, but the paper provides no contamination check. The test sentences are drawn from public case law (case.law and EPO Boards of Appeal), and Qwen 2.5 is pretrained on public web data, so it is plausible that the model has seen these decisions during pretraining. The high NDCG scores could reflect memorized case-law knowledge rather than a general ability to annotate novel statutory-interpretation sentences, and the DeBERTa baselines are not exposed to the same risk. The paper should report a contamination-controlled evaluation, such as comparing performance on sentences from decisions published after the model's knowledge cutoff, using a held-out set of concepts not plausibly present in pretraining, or testing on a newly collected corpus of recent decisions. Without such a check, the headline claim is not yet established.
  2. [§5.3, Table 5] The LLM results are reported as a single run with no variance or confidence interval. The paper concludes that 'there is very low chance that the model trained on the manual annotation of the dataset will yield better results,' but the NDCG@10 gap between Qwen (0.777) and the best manually trained large DeBERTa model (0.791) is only 1.4 percentage points, which is smaller than the standard deviations visible for several k values in Figure 5. The authors should repeat the LLM annotation with different inference seeds or prompt-order permutations and report the dispersion of the NDCG scores, or otherwise justify why the single-run comparison is stable.
  3. [§4.2 and §5.2, Tables 4 and 6] RQ2 is explicitly an oracle setup because the ranker used to select top-k sentences is trained on annotated examples from the same corpus, and the paper acknowledges that 'this approach cannot be reproduced in practice.' This is acceptable as an optimistic upper bound, but the practical recommendations in §5.2 go beyond that framing: the suggested pipeline of first training a large model on random annotations, using it to sort sentences, and then training a base model assumes that the oracle gains transfer to a realistic setting. The paper should either soften these recommendations or validate them with a non-oracle experiment, such as using a model trained on a different jurisdiction or a genuinely separate set of concepts.
minor comments (5)
  1. [Abstract and throughout] There are several typos and grammatical errors, e.g., 'This allow legal professionals' in the abstract, 'The we should use that model' in §5.2, 'we have no tried' and 'compere these models' in the conclusions.
  2. [§2 and §4.1] The references to Savelka's PhD thesis are inconsistent: §2 says 'the Ph.D. of Jaromir Savelka [19]' but reference [19] is a 2021 EMNLP Findings paper, while the thesis is reference [17]. The same ambiguity appears in §4.1, where 'not addressed in [19]' should likely be [17].
  3. [Figure 5 and Figure 6] The captions do not explain what the shaded contour represents; the text says it is the standard deviation across the five seeds of the split-averaged scores, but this should also be stated in the captions for clarity.
  4. [§3.2, Figure 1] Figure 1 shows only 'text' and 'concept' fields, but the experimental setup in §5 states that the provision is also passed to the model. Please clarify how the provision is represented in the data format or why it is absent from the example.
  5. [§5.3] The conclusion correctly notes that a small ground-truth set is still needed for evaluation and prompt development, but this nuance is somewhat buried; consider stating in the abstract or introduction that 'manual annotation' refers to annotation for training the ranker, not for validation of the LLM.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the LLM-vs-manual-annotation comparison rests on held-out gold-label evaluation, and the only self-citation supplies a baseline rather than a load-bearing premise.

full rationale

The paper does not derive its main claims from its own inputs. RQ1 and RQ2 train DeBERTa models on sampled subsets of the training folds and evaluate NDCG on a separately held-out test subset (2 of 6 folds) against gold human annotations; the metric is the standard NDCG of Eq. (1). RQ2's sentence-selection procedure uses a ranker trained on the other three training splits (Sec. 4.2: 'a model is created by training on the other annotated three splits'), and the paper explicitly labels the setup as an optimistic, non-reproducible upper bound: 'we have taken an optimal setting by considering all examples of the other splits as training data points' and 'this approach cannot be reproduced in practice.' An oracle-bound experiment is not a disguised fit or a self-definitional prediction, since the reported NDCG is still computed on unseen test sentences. RQ3 evaluates Qwen 2.5 on the full held-out test set against the same gold labels, with few-shot examples drawn from the training set; Table 5 is therefore an independent measurement, not a renaming of the model's own outputs. The self-citation to [24] (same authors) supplies the baseline DeBERTa setup and prior state-of-the-art numbers, but the central conclusion that a 72B LLM suffices does not reduce to that prior work; it is a direct empirical comparison on the test set. A missing contamination check regarding whether Qwen 2.5 saw the public case-law test sentences during pretraining is a legitimate validity risk, but contamination is a correctness concern, not circularity. Overall, the derivation chain is self-contained against external test labels, with at most one minor non-load-bearing self-citation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central conclusions rest on a few inherited assumptions: the human labels are treated as ground truth; NDCG with a fixed relevance mapping is the evaluation metric; the RQ2 oracle ranker is a valid upper bound; and the LLM's token probabilities are treated as a ranking score. These are stated or acknowledged in the paper, but the RQ2 assumption is the weakest because practical recommendations depend on it.

assumptions (4)
  • domain assumption The Savelka dataset's human annotations (four usefulness categories assigned by law students) constitute a valid ground truth for statutory interpretation sentence relevance.
    The entire evaluation and comparison in Sections 5.1, 5.2, and 5.3 treats these labels as the reference for NDCG and accuracy. Section 3.1 describes the dataset construction but does not validate inter-annotator reliability beyond noting two annotators per sentence.
  • standard math NDCG with relevance values 3, 2, 1, 0 for the four labels is the appropriate evaluation measure for the ranking task.
    The NDCG formula (Eq. 1) is standard, but the choice of equal spacing (3,2,1,0) is an assumption inherited from Savelka [17] and Smywiński-Pohl and Libal [24] without justification.
  • ad hoc to paper A model trained on all examples of the three other splits can serve as an oracle ranker for selecting top-k sentences in RQ2, providing a valid optimistic upper bound.
    Section 4.2 states this setup 'cannot be reproduced in practice' and is used as an ideal limit. Practical recommendations in Section 5.2 and the Conclusion depend on this oracle assumption, which may not hold for realistic active-learning budgets.
  • domain assumption The probability-weighted scores obtained from Qwen's guided decoding provide a meaningful continuous ranking signal for NDCG computation.
    Section 5.3 describes computing a weighted sum of label probabilities as the sentence score, but does not validate that these probabilities are well-calibrated for ranking.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are manual annotations necessary for statutory interpretations retrieval?." pith.science (2026). https://pith.science/paper/XOLOZ2AN

@misc{pith2026250613965,
  author       = {Pith},
  title        = {Pith review of: Are manual annotations necessary for statutory interpretations retrieval?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XOLOZ2AN}},
  note         = {Machine review of arXiv:2506.13965}
}
read the original abstract

One of the elements of legal research is looking for cases where judges have extended the meaning of a legal concept by providing interpretations of what a concept means or does not mean. This allow legal professionals to use such interpretations as precedents as well as laymen to better understand the legal concept. The state-of-the-art approach for retrieving the most relevant interpretations for these concepts currently depends on the ranking of sentences and the training of language models over annotated examples. That manual annotation process can be quite expensive and need to be repeated for each such concept, which prompted recent research in trying to automate this process. In this paper, we highlight the results of various experiments conducted to determine the volume, scope and even the need for manual annotation. First of all, we check what is the optimal number of annotations per a legal concept. Second, we check if we can draw the sentences for annotation randomly or there is a gain in the performance of the model, when only the best candidates are annotated. As the last question we check what is the outcome of automating the annotation process with the help of an LLM.

Figures

Figures reproduced from arXiv: 2506.13965 by the authors.

Figure 1
Figure 1. Example of a data point in JSON format used for [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Example of a data point in JSON format after anno [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Example of a data point in JSON format after model [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The total number of sentences in the training subset [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: NDCG@10 and NDCG@100 results for the DeBERTa [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: NDCG@10 and NDCG@100 results for the DeBERTa [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 26 canonical work pages

  1. [1]

    Frontiers in Artificial Intelligence and Applications 259 (01 2013), 29–38

    Ashley, K., and W alker, V.From information retrieval (ir) to argument retrieval (ar) for legal cases: Report on a baseline study. Frontiers in Artificial Intelligence and Applications 259 (01 2013), 29–38

  2. [2]

    Guiding llms the right way: Fast, non-invasive constrained generation, 2024

    Beurer-Kellner, L., Fischer, M., and Vechev, M. Guiding llms the right way: Fast, non-invasive constrained generation, 2024

  3. [3]

    J., and Katz, D

    Bommarito, M. J., and Katz, D. 2022. arXiv Preprint

  4. [4]

    LEGAL-BERT: The muppets straight out of law school

    Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., and Androut- sopoulos, I. LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020 (Online, Nov. 2020), T. Cohn, Y. He, and Y. Liu, Eds., Association for Computational Linguistics, pp. 2898–2904

  5. [5]

    Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024

    Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2024

  6. [6]

    de Oliveira Lima, J. A. Unlocking legal knowledge with multi-layered embedding-based retrieval. CoRR abs/2411.07739 (2024)

  7. [7]

    Active Learning for BERT: An Empirical Study

    Ein-Dor, L., Halfon, A., Gera, A., Shnarch, E., Dankin, L., Choshen, L., Danilevsky, M., Aharonov, R., Katz, Y., and Slonim, N. Active Learning for BERT: An Empirical Study. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (Online, Nov. 2020), B. Webber, T. Cohn, Y. He, and Y. Liu, Eds., Association for Com...

  8. [8]

    Can gpt alleviate the burden of annotation? In Legal Knowledge and Information Systems

    Gray, M., Savelka, J., Oliver, W., and Ashley, K. Can gpt alleviate the burden of annotation? In Legal Knowledge and Information Systems . IOS Press, 2023, pp. 157–166

Show all 29 references
  1. [9]

    Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, 2021

    He, P., Gao, J., and Chen, W. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, 2021

  2. [10]

    Cumulated gain-based evaluation of ir tech- niques

    Järvelin, K., and Kekäläinen, J. Cumulated gain-based evaluation of ir tech- niques. ACM Transactions on Information Systems (TOIS) 20 , 4 (2002), 422–446

  3. [11]

    Gpt-4 passes the bar exam

    Katz, D., Bommarito, M., Gao, S., and Arredondo, P. Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A 382 (02 2024)

  4. [12]

    T., Moazam, H., Miller, H., Zaharia, M., and Potts, C

    Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., V ard- hamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., Miller, H., Zaharia, M., and Potts, C. Dspy: Compiling declarative language model calls into self- improving pipelines, 2023

  5. [13]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles (2023)

  6. [14]

    Automating legal concept interpre- tation with llms: Retrieval, generation, and evaluation, 01 2025

    Luo, K., Huang, Q., Jiang, C., and Feng, Y. Automating legal concept interpre- tation with llms: Retrieval, generation, and evaluation, 01 2025

  7. [15]

    An empirical study on cross-x transfer for legal judgment prediction

    Niklaus, J., Stürmer, M., and Chalkidis, I. An empirical study on cross-x transfer for legal judgment prediction. arXiv preprint arXiv:2209.12325 (2022)

  8. [16]

    Efficiently scaling transformer inference, 2022

    Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference, 2022

  9. [17]

    Discovering sentences for argumentation about the meaning of statutory terms

    Savelka, J. Discovering sentences for argumentation about the meaning of statutory terms. PhD thesis, University of Pittsburgh, 2020

  10. [18]

    Šavelka, J., and Ashley, K. D. Transfer of predictive models for classification of statutory texts in multi-jurisdictional settings. In Proceedings of the 15th International Conference on Artificial Intelligence and Law (2015), pp. 216–220

  11. [19]

    Savelka, J., and Ashley, K. D. Discovering explanatory sentences in legal case decisions using pre-trained language models. In: Findings of the Association for Computational Linguistics: EMNLP 2021. pp. 4273–4283 (2021)

  12. [20]

    Šavelka, J., and Ashley, K. D. Legal information retrieval for understanding statutory terms. Artificial Intelligence and Law (2022), 1–45

  13. [21]

    D., Gray, M

    Savelka, J., Ashley, K. D., Gray, M. A., Westermann, H., and Xu, H. Can gpt-4 support analysis of textual data in tasks requiring highly specialized domain expertise? arXiv preprint arXiv:2306.13906 (2023)

  14. [22]

    Cross-domain generalization and knowledge transfer in transformers trained on legal data

    Savelka, J., Westermann, H., and Benyekhlef, K. Cross-domain generalization and knowledge transfer in transformers trained on legal data. arXiv preprint arXiv:2112.07870 (2021)

  15. [23]

    S., Grant, J

    Savelka, J., Westermann, H., Benyekhlef, K., Alexander, C. S., Grant, J. C., Amariles, D. R., Hamdani, R. E., Meeùs, S., Troussel, A., Araszkiewicz, M., Ashley, K. D., Ashley, A., Branting, K., Falduti, M., Grabmair, M., Harašta, J., Novotná, T., Tippett, E., and Johnson, S. L...

  16. [24]

    Enhancing legal argument retrieval with optimized language model techniques

    Smywiński-Pohl, A., and Libal, T. Enhancing legal argument retrieval with optimized language model techniques. In JSAI International Symposium on Artificial Intelligence (2024), Springer, pp. 93–108

  17. [25]

    Qwen2.5: A party of foundation models, September 2024

    Team, Q. Qwen2.5: A party of foundation models, September 2024

  18. [26]

    Beyond borders: Investigating cross-jurisdiction transfer in legal case summarization

    Tyss, S., Venkatkrishna, V., Ghosh, S., and Grabmair, M. Beyond borders: Investigating cross-jurisdiction transfer in legal case summarization. In Proceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag...

  19. [27]

    Qwen2 technical report

    Y ang, A., Y ang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., W ang, J., Y ang, J., Tu, J., Zhang, J., Ma, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Y ang, K., Li, M., Xue...

  20. [28]

    Chunkattention: Efficient self-attention with prefix-aware kv cache and two-phase partition, 2024

    Ye, L., Tao, Z., Huang, Y., and Li, Y. Chunkattention: Efficient self-attention with prefix-aware kv cache and two-phase partition, 2024

  21. [29]

    Zheng, L., Guha, N., Anderson, B., Henderson, P., and Ho, D. When does pretraining help?: Assessing self-supervised learning for law and the casehold Are Manual Annotations Necessary for Statutory Interpretations Retrieval? ICAIL ’25, June 03–05, 2018, Woodstock, NY dataset of...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.