Pith. sign in

REVIEW 3 major objections 6 minor 30 references

Zero-shot prompting with expected-value scoring ranks statutory explanatory sentences better than fine-tuned encoders, beating prior state of the art.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 05:28 UTC pith:KDR2UCNC

load-bearing objection Solid empirical legal-IR update: EV prompting of frontier decoders beats ModernBERT and prior SOTA on the established statutory-term sentence ranking task, with useful context ablations. the 3 major comments →

arxiv 2607.05582 v1 pith:KDR2UCNC submitted 2026-07-06 cs.IR

Prompting Beats Fine-Tuning: Generative Expected Value Scoring for Statutory Term Retrieval

Alvin Wang , Jaromir Savelka This is my paper
classification cs.IR
keywords information retrievalstatutory interpretationcase-law analysisrelevant sentencesexpected-value scoringzero-shot promptingencoder-only fine-tuningNDCG ranking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Lawyers often need case-law sentences that explain vague statutory terms, but most retrieved sentences add little interpretive value. This paper shows that zero-shot prompting of large decoder-only models, when they emit class probabilities that are turned into an expected-value score over four explanatory-value labels, ranks those sentences more effectively than supervised fine-tuning of modern encoder models. On a fixed dataset of 26,959 sentences covering 42 U.S. Code concepts, the best prompted system surpasses all previously reported results at standard NDCG cutoffs. ModernBERT variants largely only match older BERT-family baselines, and adding paragraph-level context often hurts rather than helps, especially on large sparse queries. The practical claim is that instruction-tuned generative models can deliver stronger ranking of explanatory legal sentences without any task-specific fine-tuning, while expanded context is not a free performance boost.

Core claim

Across all 42 concepts and standard NDCG cutoffs, zero-shot decoder-only prompting with generative expected-value scoring achieves the strongest overall effectiveness on ranking case-law sentences by explanatory usefulness for statutory terms, with the best system surpassing all previously reported state-of-the-art results, while ModernBERT fine-tuning largely matches earlier BERT-family baselines and context expansion does not reliably help.

What carries the argument

Generative expected-value scoring: the model outputs a probability distribution over the four annotation classes (high, certain, potential, no value), and the ranking score is the expected value of that distribution under temperature 0.0.

Load-bearing premise

The model-emitted class probabilities at temperature zero are well enough calibrated, and survive format repair, that their expected value is a fair, stable ranking signal of explanatory usefulness across models.

What would settle it

Re-score the same 42-term collection with an independent, non-probability ranking method or with explicitly calibrated probabilities; if the prompted EV systems no longer lead at NDCG@10 and NDCG@100, especially on large sparse queries, the central ranking claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies ranking of case-law sentences by explanatory usefulness for statutory terms on an established dataset (26,959 sentences, 42 U.S. Code concepts, four graded labels). It compares (i) supervised fine-tuning of ModernBERT under six input encodings (sentence-only through provision+context) with six-fold stratified CV, and (ii) zero-shot prompting of proprietary and open-weight decoder-only models that emit a four-class probability distribution whose expected value is used as the ranking score. Main empirical claims: ModernBERT largely matches earlier BERT-family baselines; expanded context often hurts, especially on large-sparse queries; and generative EV prompting, led by GPT-5.4 (overall NDCG@10/100 of 0.82/0.87), surpasses previously reported SOTA on this task.

Significance. If the results hold, the work is a clear, practically relevant contribution to legal IR and statutory interpretation support: it shows that carefully prompted decoder-only models can outperform task-specific fine-tuned encoders on graded explanatory-sentence ranking without labeled training, and it documents that simply adding more statutory or paragraph context is not a free win. Strengths include use of a public labeled benchmark, stratified multi-fold evaluation with small/large and sparse/dense query slices, explicit comparison of open-weight and proprietary models, and a transparent expected-value scoring recipe. The context-degradation analysis and the engineering trade-off discussion (training cost vs. API latency) are useful for practitioners. The work is an honest extension of the authors’ prior demo and of prior BERT/DeBERTa/Qwen baselines rather than a closed-loop self-evaluation.

major comments (3)
  1. [Section 4.2; Table 2; Section 5.2] Section 4.2 and Table 2: The headline SOTA ordering rests on generative expected-value scores ∑ i·P(class=i) from temperature-0.0 model outputs. The manuscript does not report calibration diagnostics (e.g., reliability diagrams, ECE, or rank correlation of EV vs. hard argmax labels), nor does it quantify how the Regex/JSON repair pipeline in §5.2 shifts probability mass across the four classes. Without that, it is hard to know whether GPT-5.4’s lead over GPT-OSS-120B, Qwen-3-32B, and ModernBERT sp2snt is driven by better semantic ranking or by better-calibrated / repair-friendly probability formats. A short ablation (EV vs. discrete label; repaired vs. raw valid JSON only; optional temperature/logit-bias check) would make the central claim much more robust.
  2. [Table 2; Section 6; Related Work] Table 2 and Related Work / Results: Prior SOTA numbers from [13] (DeBERTa-large overall .79/.79; Qwen2.5-72B .78/.85) are reported only as overall aggregates with no subset breakdowns and were obtained under different prompting/fine-tuning protocols. The paper’s claim that GPT-5.4 “surpasses all previously reported state-of-the-art results” is therefore only partially controlled. Either re-evaluate the [13] systems under the same EV protocol and fold structure, or clearly frame the comparison as cross-paper rather than same-protocol SOTA and report confidence intervals / paired tests on the 42-term means.
  3. [Section 7; Table 2; Table 1] Section 7 (Context Expansion) and Table 2: The finding that sp2ctx and sp2snt+ctx degrade large-sparse performance (e.g., ModernBERT sp2snt LgSp NDCG@10 0.53 → 0.28 for sp2snt+ctx) is important but under-analyzed. It is unclear whether the drop is due to noise dilution, tokenizer truncation of the marked sentence, or the non-pretrained special markers used in sp+qry2snt. A brief error analysis or attention/saliency check on a few large-sparse terms would turn this from an observation into a transferable design lesson.
minor comments (6)
  1. [Table 2] Table 2: Prior-work rows leave subset cells blank (“–”). Either fill them from the original papers or state explicitly that subset metrics were not available, so readers do not misread blanks as zeros.
  2. [Section 4.2] Section 4.2: The class index mapping for EV (i = 0..3) should be stated once with the label order (No / Potential / Certain / High) so the formula is unambiguous without consulting the annotation guidelines.
  3. [Section 5.1; Section 7] Section 5.1: Batch size is given as 16 in the training recipe and as 8 in the Colab timing note in §7; reconcile or explain the difference.
  4. [Figure 1] Figure 1 is reproduced from [6]; ensure the caption and permission note remain clear in the camera-ready version and that axis labels remain legible at print size.
  5. [References; Section 4.2] Several model cards and blog posts (GPT-5.2, GPT-5.4, Llama 4) are cited via URLs with future-dated years; pin model versions/API dates used for the runs to aid reproducibility as proprietary endpoints change.
  6. [Title; Abstract; Section 8] Minor wording: abstract and conclusion say “prompting beats fine-tuning” as a general slogan; the body correctly scopes this to the present task and dataset—keep that scope in the title/abstract claims.

Circularity Check

0 steps flagged

No significant circularity: empirical IR ranking on an external labeled dataset with a fixed EV transform of model outputs.

full rationale

The paper’s central claim is an empirical comparison of fine-tuned ModernBERT variants versus zero-shot decoder-only prompting with generative expected-value scoring on the established 26,959-sentence statutory-term dataset of [6], evaluated by NDCG@10/100 against prior reported numbers (including [13]). The EV score is defined as a fixed, non-fitted transform ∑ i·P(class=i) of the model’s four-class distribution at temperature 0.0; it does not re-use evaluation labels as method parameters. Fine-tuning is ordinary supervised learning on the same external labels used for evaluation (standard stratified folds), not a prediction forced by construction. Self-extension of the authors’ demo [7] and co-author prior work [6] is disclosed and supplies the task/dataset, not a uniqueness theorem or ansatz that forces the SOTA ordering. No equation or claim reduces by definition to its inputs; the headline ranking is an experimental outcome on held-out folds and published baselines. Score 0 is therefore appropriate.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The central claim is an empirical ranking comparison. It rests on standard IR metrics, the pre-existing four-class annotation scheme, fixed training/prompt hyperparameters, and the modeling choice that EV of class probabilities is the ranking score. No new physical entities; free parameters are ordinary ML/prompt settings that affect absolute scores but are shared across fair comparisons when held fixed.

free parameters (4)
  • ModernBERT learning rate
    Set to 2e-5 with AdamW, weight decay 0.01, linear schedule, 10 epochs, batch 16; checkpoint chosen by val NDCG@100.
  • Prompt batch size and temperature
    25 sentences per user message, temperature 0.0; affects probability estimates used for EV.
  • Class index mapping for EV
    EV = sum i * P(class=i) for i in 0..3; the numeric coding of High/Certain/Potential/No value is a design choice that defines the ranking score.
  • Context token limits
    512 / 840 / 1280 / 1440 token caps for different ModernBERT input formats; chosen by hand relative to provision and paragraph lengths.
axioms (4)
  • domain assumption The four explanatory-value categories and student annotations (Krippendorff alpha 0.79) are a valid ground truth for ranking usefulness.
    Entire evaluation (NDCG) is defined against these labels from [6]; Section 3.
  • domain assumption NDCG@10 and NDCG@100 are appropriate primary metrics for this ranking task.
    Section 5.3; standard IR choice but still an assumption about what practitioners need.
  • ad hoc to paper Compacted annotation guidelines in the prompt are a faithful operationalization of the original labeling rules.
    Section 4.2; zero-shot performance depends on this prompt compression.
  • standard math Stratified six-fold splits by concept yield unbiased estimates of generalization across statutory terms.
    Section 5.1 experimental design inherited from [6].
invented entities (1)
  • Generative Expected Value Scoring (EV of four-class probability distribution) no independent evidence
    purpose: Turn decoder-only class probabilities into a continuous ranking score without fine-tuning.
    Named contribution in the title/method; it is a standard expectation transform, not a new physical object, and has no independent evidence requirement beyond ranking performance on this dataset.

pith-pipeline@v1.1.0-grok45 · 13347 in / 3091 out tokens · 30042 ms · 2026-07-11T05:28:42.611417+00:00 · methodology

0 comments
read the original abstract

Legal concepts in statutes are often expressed using vague terms, and practitioners frequently turn to case law to interpret them. We study the task of ranking case-law sentences by their usefulness for explaining a concept or target statutory term, using an established dataset of 26,959 sentences covering 42 U.S. Code concepts labeled into four explanatory-value categories. We compare two families of methods: (i) supervised fine-tuning of encoder-only models (ModernBERT) and (ii) zero-shot prompting of decoder-only models. We show that across all concepts and standard NDCG cutoffs, ModernBERT largely matches earlier BERT-family baselines. In contrast, prompting decoder-only models achieves the strongest overall effectiveness, with our best system surpassing all previously reported state-of-the-art results on this task.

Figures

Figures reproduced from arXiv: 2607.05582 by Alvin Wang, Jaromir Savelka.

Figure 1
Figure 1. Figure 1: The graph on the left shows the distribution of the labels. The graph on the right presents the distribution of the number of sentences retrieved for each term. The graph is reproduced from [6] with the permission of the authors. 2. Related Work In prior work, researchers employed a variety of traditional information retrieval measures and their combinations, e.g., BM25, novelty, topic modeling [8, 9, 10].… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 8 linked inside Pith

  1. [1]

    H. L. Hart, The Concept of Law, 2nd ed., Clarendon Press, 1994

    1994

  2. [2]

    Endicott, Vagueness in Law, Oxford University Press, 2000

    T. Endicott, Vagueness in Law, Oxford University Press, 2000

    2000

  3. [3]

    Endicott, Law and Language the stanford encyclopedia of philosophy, http://plato.stanford.edu/,

    T. Endicott, Law and Language the stanford encyclopedia of philosophy, http://plato.stanford.edu/,

  4. [4]

    Accessed: 2016-02-03

    2016

  5. [5]

    Daci, Legal principles, legal values and legal norms: are they the same or different?, Academicus International Scientific Journal 02 (2010) 109–115

    J. Daci, Legal principles, legal values and legal norms: are they the same or different?, Academicus International Scientific Journal 02 (2010) 109–115

    2010

  6. [6]

    D. N. MacCormick, R. S. Summers, Interpreting Statutes, Darmouth, 1991

    1991

  7. [7]

    Šavelka, K

    J. Šavelka, K. D. Ashley, Discovering explanatory sentences in legal case decisions using pre-trained language models, in: Findings of the association for computational linguistics: EMNLP 2021, 2021, pp. 4273–4283

    2021

  8. [8]

    A. Wang, J. Savelka, Explanatory case-law retrieval with encoder and decoder language models, in: Proceedings of the 21st International Conference on Artificial Intelligence and Law, 2026

    2026

  9. [9]

    Savelka, H

    J. Savelka, H. Xu, K. D. Ashley, Improving sentence retrieval from case law for statutory interpre- tation, in: Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law, 2019, pp. 113–122

    2019

  10. [10]

    Savelka, Discovering sentences for argumentation about the meaning of statutory terms, Ph.D

    J. Savelka, Discovering sentences for argumentation about the meaning of statutory terms, Ph.D. thesis, University of Pittsburgh, 2020

    2020

  11. [11]

    Šavelka, K

    J. Šavelka, K. D. Ashley, Legal information retrieval for understanding statutory terms, Artificial Intelligence and Law (2021) 1–45

    2021

  12. [12]

    Šavelka, K

    J. Šavelka, K. D. Ashley, Extracting case law sentences for argumentation about the meaning of statutory terms, in: Proceedings of the Third Workshop on Argument Mining (ArgMining2016), 2016, pp. 50–59

    2016

  13. [13]

    Savelka, K

    J. Savelka, K. D. Ashley, Learning to rank sentences for explaining statutory terms., in: ASAIL@ JURIX, 2020

    2020

  14. [14]

    Libal, A

    T. Libal, A. Smywiński-Pohl, A. Kaczmarczyk, M. Król, Are manual annotations necessary for statutory interpretations retrieval?, in: Proceedings of the Twentieth International Conference on Artificial Intelligence and Law, 2025, pp. 414–419

    2025

  15. [15]

    Krippendorff, Computing krippendorff’s alpha-reliability, Computing 1 (2011) 25

    K. Krippendorff, Computing krippendorff’s alpha-reliability, Computing 1 (2011) 25

    2011

  16. [16]

    Warner, A

    B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, S. Taghadouini, A. Gallagher, R. Biswas, F. Ladhak, T. Aarsen, et al., Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, in: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics ...

    2025

  17. [17]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4171–4186

    2019

  18. [18]

    Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019)

    Pith/arXiv arXiv 1907

  19. [19]

    J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, Y. Liu, Roformer: Enhanced transformer with rotary position embedding, Neurocomputing 568 (2024) 127063

    2024

  20. [20]

    Hurst, A

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al., Gpt-4o system card, arXiv preprint arXiv:2410.21276 (2024)

    Pith/arXiv arXiv 2024

  21. [21]

    URL: https://arxiv.org/abs/2508.10925

    OpenAI, gpt-oss-120b & gpt-oss-20b model card, 2025. URL: https://arxiv.org/abs/2508.10925. arXiv:2508.10925

    Pith/arXiv arXiv 2025

  22. [22]

    Addendum to the GPT-5 System Card

    OpenAI, Introducing GPT-5.2, https://openai.com/index/introducing-gpt-5-2/, 2025. Addendum to the GPT-5 System Card

    2025

  23. [23]

    OpenAI, GPT-5.4 Thinking system card, https://openai.com/index/gpt-5-4-thinking-system-card/, 2026

    2026

  24. [24]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, et al., The Llama 3 herd of models, 2024. URL: https://arxiv.org/abs/2407.21783.arXiv:2407.21783

    Pith/arXiv arXiv 2024

  25. [25]

    Meta AI, The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation, https://ai.meta.com/blog/llama-4-multimodal-intelligence/, 2025

    2025

  26. [26]

    URL: https://arxiv.org/abs/2505.09388

    Qwen Team, Qwen3 technical report, 2025. URL: https://arxiv.org/abs/2505.09388. arXiv:2505.09388

    Pith/arXiv arXiv 2025

  27. [27]

    Loshchilov, F

    I. Loshchilov, F. Hutter, Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101 (2017)

    Pith/arXiv arXiv 2017

  28. [28]

    J. Lin, R. Nogueira, A. Yates, Pretrained transformers for text ranking: Bert and beyond, arXiv preprint arXiv:2010.06467 (2020)

    Pith/arXiv arXiv 2010

  29. [29]

    Y. Wang, L. Wang, Y. Li, D. He, T.-Y. Liu, A theoretical analysis of ndcg type ranking measures, in: Conference on learning theory, PMLR, 2013, pp. 25–54

    2013

  30. [30]

    P. J. Sager, A. Kamaraj, B. F. Grewe, T. Stadelmann, Deep retrieval at checkthat! 2025: Identifying scientific papers from implicit social media mentions via hybrid retrieval and re-ranking, 2025. URL: https://arxiv.org/abs/2505.23250.arXiv:2505.23250

    Pith/arXiv arXiv 2025