Pith. sign in

REVIEW 6 major objections 6 minor 49 references

Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization

T0 review · 6 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper establishes that frontier LLMs score only about 50 out of 100 when summarizing long legal cases, with omission of key facts, especially settlement details, far more common than hallucination, and that a checklist-based evaluation

desk verdict A solid evaluation framework and a new long-context legal summarization benchmark, held back by an unvalidated judge-transfer assumption and an unsupported medical claim—worth serious peer review. read the letter →

arxiv 2601.04424 v3 pith:MPRB2MFI submitted 2026-01-07 cs.CL

Yao Dou , Benjamin Mamut , Wei Xu This is my paper
classification cs.CL
keywords long-contextLLMevaluationlegalsummarizationchecklist-basedmulti-valueextractionLLM-as-judgeagenticfactualcoverageomissionvshallucination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that current frontier LLMs are far from reliable at summarizing multi-document legal cases of 100K+ tokens, and that the right way to see this is fine-grained checklist evaluation rather than a single aggregate score. To show this, it builds Gavel-Ref, which extracts up to 26 checklist items as multi-value lists, scores residual facts and writing style, and meta-validates the LLM judge against 160 hours of human annotations. On 100 cases from 32K to 512K tokens, even the best model, Gemini 2.5 Pro, reaches about 51/100, with omissions (especially of settlement terms and related cases) far more common than hallucinations. The paper also introduces Gavel-Agent, a six-tool agent that pulls checklist facts directly out of case documents, using 36-59% fewer tokens than end-to-end or chunk-by-chunk baselines while staying competitive. If right, these results imply that today's long-context models cannot yet be trusted for legal summarization, and that document-grounded checklist agents are a promising efficient evaluation route.

What carries the argument

Gavel-Ref, a three-part reference-based evaluation framework. Its core unit is a 26-item legal checklist (filing date, parties, remedies, decrees, settlements, monitors, etc.) where each item is extracted as a list of values, each paired with verbatim supporting text; single-value items are scored by a four-way containment relation and multi-value items by element-wise F1, with non-applicable items excluded. Two further components capture residual facts (content outside the checklist) and writing-style similarity across five dimensions, combined as a weighted score. Gavel-Agent is a companion scaffold that gives an LLM six tools—list documents, read a window, regex search, get/append/update

What would settle it

Take a random sample of, say, 20 of the 100 cases, including all five length bins, and have a fresh team of annotators extract checklists and compare model summaries using the same instructions. If the LLM-judge-based Schecklist disagrees with the human-judge-based score by more than a pre-specified margin (e.g., 10 points on average), or if GPT-oss 20B's self-evaluation is systematically inflated relative to other models, the benchmark's validity and rankings fail.

Watch

Extended reading notes

Core claim

The central claim is that modern LLMs, despite having 1M-token context windows, score only about 50 of a possible 100 on a composite Gavel-Ref score when summarizing legal cases, and that their failures are patterned. They capture simple single-value items like filing dates, but largely miss rare multi-value items such as settlement terms and related cases, and their performance declines as case length grows even for models that claim 1M-token support. The paper argues this granular view is only possible because Gavel-Ref evaluates each summary by extracting 26 checklist items as lists of values with verbatim supporting text, matching residual facts, and rating writing-style similarity again

Load-bearing premise

The whole benchmark rests on the assumption that two open-source LLMs, GPT-oss 20B and Gemma3 27B, chosen as judges by human agreement on 40 long summaries, keep judging faithfully when applied to all 100 cases (including 256K–512K ones) and to all 12 model summaries—including the case where GPT-oss judges its own outputs.

Editorial extensions

If this is right

  • If valid, current frontier models are not reliable for legal summarization at scale; a score of ~50/100 means roughly half of key facts are missed or wrong.
  • Omission being the dominant error type suggests evaluation and improvement efforts should focus on recall — extracting all applicable values — rather than hallucination control alone.
  • The 36–59% token savings of Gavel-Agent with only ~7% score drop suggests agentic document navigation is a practical evaluation and feedback route for long-context domains.
  • Since all models degrade on 256K–512K inputs even with 1M context, claims of unlimited context usefulness need re-examination for this task.
  • The paper's extension to the medical domain indicates the checklist-based approach may transfer to other expert summarization areas.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the judge-transfer assumption holds, the same checklist machinery could be inverted: generate summaries conditionally on checklists extracted directly from documents, a direction the paper only leaves as future work.
  • The finding that models underspecify multi-value items 76.5% of the time suggests a testable extension: recall-oriented training signals derived from checklist extraction could improve summarization quality more than style matching.
  • The paper's meta-evaluation was on 40 summaries averaging 1,130 words; applying Gavel-Ref to 512K-token cases extends beyond that validated range, so per-length-bin human agreement would be a natural next check.
  • The residual-fact component might be reducible to a checklist of 'extra' facts, but the paper keeps it open-ended; a cost-geometry argument could be made that the 26-item checklist carries most of the signal, making style and residual components optional for ranking.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper presents Gavel, a two-part framework for evaluating LLM-generated multi-document legal case summaries. Gavel-Ref is a reference-based evaluation combining a 26-item multi-value checklist, residual-fact extraction, and a five-dimension writing-style similarity rating; Gavel-Agent is a tool-using agent scaffold that extracts the same checklist items directly from case documents. The authors collect human annotations for meta-evaluation, benchmark 12 LLMs on 100 legal cases at 32K--512K tokens, report that even the best model scores only about 50 on the composite S_Gavel-Ref, that omissions (especially of rare/multi-value items such as settlements) are more common than hallucinations, and that performance degrades with case length. They further claim that Gavel-Agent with Qwen3 matches end-to-end GPT-4.1 extraction within 7% while using 36--59% fewer tokens, and the abstract states that Gavel-Agent generalizes to the medical domain with 77% fewer tokens.

Significance. If the evaluation is valid, the paper makes a useful contribution to long-context summarization evaluation: the multi-value checklist with supporting evidence is a clear improvement over the binary comparison in ExpertLongBench, the human annotation effort (5,442 item-level judgments) is substantial, and the Gavel-Agent scaffold is a practical step toward source-grounded, reference-free evaluation. The paper also ships code and data, which supports reproducibility. However, several load-bearing claims are currently under-supported: the abstract's medical generalization claim appears nowhere in the body, the judge models are validated on human references but then applied to a different distribution of model outputs without re-validation, one judge is also a benchmarked model, and headline scores have no confidence intervals. These issues affect the trustworthiness of the main quantitative conclusions, though they appear fixable within the manuscript's scope.

major comments (6)
  1. [Abstract] The abstract states that Gavel-Agent "also generalizes to the medical domain, performing the best with at least 77% fewer tokens." No medical experiments, medical dataset, or medical evaluation appears anywhere in the paper; Section 4 uses only legal cases from the Civil Rights Litigation Clearinghouse. This unsupported empirical claim must either be removed or substantiated with actual medical-domain experiments and results.
  2. [§2.3, Table 1; §3, Eq. (2)] Judge reliability is established only on human reference summaries (40 summaries averaging 1,130 words), 150 comparison pairs, and 25 style pairs. In Section 3, the same judges (GPT-oss 20B for extraction/style, Gemma3 27B for comparison) are applied to all 100 cases and all 12 models, including 256K--512K bins where model summaries are 500--800 words versus the human ~1,200 words (Fig. 6) and structurally different, e.g., GPT-5's list-like output (Fig. 9). No per-bin or per-model judge agreement is reported, and no confidence intervals are given for S_Gavel-Ref. Because Eq. (2) is a weighted combination of checklist/residual and style scores, any judge bias on the actual evaluation distribution propagates directly into the headline ~50 scores and the omission/hallucination analysis. I would like to see validation on a stratified sample of model summaries, per-bin agreement, or at least
  3. [§2.3 and §3.1] GPT-oss 20B is selected in §2.3 as the checklist extractor and style rater, and is simultaneously one of the 12 evaluated models in §3, reported as the best open-source model at 45.9. This is a self-evaluation confound: the judge may systematically favor or disfavor its own output style compared with other models. The paper does not discuss this. Please either use a judge that is not among the evaluated models, or provide a leave-one-out analysis showing that the ranking of GPT-oss 20B relative to other models is stable when a different extraction/rater model is used.
  4. [§3, Figure 2; §4.3, Figure 5] The main benchmark results are point estimates from 20 cases per length bin with no confidence intervals or significance tests. Claims such as "Gemini 2.5 Pro leads," "GPT-5 is the weakest among the proprietary models," and the "7% drop" between Gavel-Agent and end-to-end extraction are comparative statements that require error bars or paired statistical tests; the differences among top models may be within noise. Please report intervals at least for the headline scores and for the §4 efficiency/performance comparison.
  5. [Appendix D; Eq. (2)] Writing-style ratings show low human inter-annotator agreement: Krippendorff's α averages 0.32, and the best LLM-human Cohen's κ in Table 1 is 0.157. This component enters the final score with 10% weight and supports the specific claim that Claude and Gemini are the most human-like while GPT-5 is lowest. Given the low reliability, I recommend either re-analyzing style with per-aspect agreement, reporting the sensitivity of the top-model ordering to the α=0.9 weighting, or treating style-related findings as descriptive rather than as a validated evaluation component.
  6. [§3.1, §3, dataset construction] The length-degradation analysis is confounded by two factors: (i) 14 of 20 cases in the 512K bin are pre-2025, whereas only 3 of 20 in the 32K bin are pre-2025; (ii) inputs are truncated by removing tokens from the end of each document for models with smaller context windows. The vintage imbalance could affect even 1M-context models through pretraining contamination, and truncation mechanically disadvantages smaller-context models. Please report results on the 83-case 2025-only subset, and separate truncated from non-truncated evaluation, to support the claim that all models degrade as case length increases.
minor comments (6)
  1. [Abstract and §2.3] The human annotation effort is reported as "150 hours" in §2.3, "160 hours" in the abstract, and the cost of $3,000 at $18/hour implies ~167 hours. Please reconcile.
  2. [Table 1] Qwen3 30B-A3B has a negative Cohen's κ (-0.011) for style rating. This should be discussed; as reported it suggests agreement worse than chance on a small sample.
  3. [Limitations] The limitations section mentions "GPT-5.2" and "Claude 4.5 Pro," but neither appears in the model list or implementation details. Please clarify or correct.
  4. [Figure 4] The caption says "Top-5 LLMs" but does not identify which five models are included. Please name them in the caption or text.
  5. [§4.2 and Figure 5] The 36% token reduction compares Gavel-Agent (Qwen3 30B-A3B) to end-to-end extraction with GPT-4.1, confounding model and method. The same-model comparison against chunk-by-chunk is stronger and should be emphasized; the GPT-4.1 comparison should be framed as an absolute efficiency benchmark, not a controlled method comparison.
  6. [Throughout] The naming "GPT-oss 20B" and "GPT-oss 120B" may be confused with OpenAI's GPT-oss series; please add a brief clarification of the exact model identifiers used, especially since vLLM serving and API versions are reported for other models.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Gavel-Ref is grounded in human annotations and Gavel-Agent claims are measured, not derived from the evaluator by construction.

full rationale

No circular step meets the required standard. Gavel-Ref computes checklist, residual-fact, and writing-style scores via explicit formulas (Eqs. 1–2) that compare model summaries against human-annotated references collected independently in §2.3 and Appendix D. The gold standard is human annotation, not the model outputs being benchmarked, so the framework is not self-definitional. The meta-evaluation selects GPT-oss 20B and Gemma3 27B based on human agreement; applying those judges to all 100 cases is a transfer/validity assumption, not a by-construction reduction—no score is defined in terms of the quantity it is used to predict. Gavel-Agent's token-savings and S_checklist comparisons are measured against fixed human-summary-derived checklists and do not reduce to fitted parameters. The only self-citation (Thresh, Heineman et al. 2023) is for the annotation interface and is not load-bearing. The use of GPT-oss 20B both as evaluator and as one of the 12 evaluated models is a legitimate confound (possible self-preference bias), but it does not make any benchmark score equal to its own input by construction; without an equation showing such a reduction, it is outside the circularity definition used here. External-validity concerns (judge reliability on 256K–512K inputs and on model-generated summaries) are correctness risks, not circularity.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper's contributions rest primarily on human-annotated meta-evaluation and on the prior ExpertLongBench checklist rather than on new fitted parameters. One hand-set scalar, α=0.9, controls the final S_Gavel-Ref score; the larger burden is the assumption that LLM judges validated on 40 summaries remain reliable across all models and lengths.

free parameters (1)
  • α (content/style balance) = 0.9
    Equation (2) sets S_Gavel-Ref = α·(content) + (1−α)·S_style with α = 0.9 throughout; chosen by hand, not fitted to data, but it directly sets the final score scale.
assumptions (5)
  • domain assumption Human summaries from the Civil Rights Litigation Clearinghouse are reliable expert references for legal cases.
    Used throughout Section 3 as the gold standard; no independent verification of these summaries is provided.
  • domain assumption The 26-item checklist inherited from ExpertLongBench is the right ontology for what matters in legal case summaries.
    The checklist is adapted from Ruan et al. (2025) and not re-derived or validated against an external legal standard.
  • ad hoc to paper LLM-based extraction/comparison validated on 40 long summaries transfers to all 100 cases and all 12 models.
    Section 2.3 meta-evaluates on a small sample; Section 3 applies the same evaluators everywhere without per-bin or per-model re-validation.
  • ad hoc to paper Truncating over-length cases by removing tokens from the end of each document still measures long-context summarization fairly.
    Applied in Section 3 for models with shorter context windows; the paper treats this as a standard proxy rather than testing its effect.
  • ad hoc to paper Writing-style similarity ratings are meaningful despite low inter-annotator agreement.
    Section D reports Krippendorff α = 0.32 across style aspects, yet style scores are used in the main S_Gavel-Ref and in model comparisons.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization." pith.science (2026). https://pith.science/paper/MPRB2MFI

@misc{pith2026260104424,
  author       = {Pith},
  title        = {Pith review of: Gavel: Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MPRB2MFI}},
  note         = {Machine review of arXiv:2601.04424}
}
read the original abstract

Large language models (LLMs) now support contexts of up to 1M tokens, but their strengths and weaknesses on complex long-context tasks remain unclear. To study this, we focus on multi-document legal case summarization, where a single case often spans many documents exceeding 100K tokens. We systematically evaluate 12 frontier LLMs with Gavel, which consists of Gavel-Ref, a reference-based evaluation framework with checklist, residual-fact, and writing-style evaluations, and Gavel-Agent, a reference-free agent for evaluating factual coverage directly from source documents. Our results show that current models are more prone to omitting key information than hallucinating. They all perform well on simple checklist items, such as filing date, but struggle with rare and complex items, such as settlements. Performance also declines as case length increases. To meta-evaluate Gavel, we collect 160 hours of human annotations. Gavel-Agent reduces token usage by at least 36% compared to end-to-end and chunk-by-chunk methods while achieving competitive performance. Gavel-Agent also generalizes to the medical domain, performing the best with at least 77% fewer tokens.

Figures

Figures reproduced from arXiv: 2601.04424 by the authors.

Figure 1
Figure 1. Example of evaluating a Gemini 2.5 Pro summary with [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Benchmarking results of 12 LLMs on long-context legal summarization with our [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Gemini 2.5 Pro performance breakdown: top/bottom 5 checklist items by matching score and most [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (25 more)
Figure 4
Figure 4. Figure 4: Top-5 LLMs’ performance across checklist [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Schecklist versus total token usage for different methods extracting from case documents. method’s extracted checklist against the human￾created checklist from the summary. Results [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Summary length and overall evaluation for [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Performance breakdown for the top-3 models in checklist evaluation (Gemini 2.5 Pro, Gemini 2.5 Flash, [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Schecklist versus total token, input token, and output token usage for different methods extracting from case documents [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Example summaries from GPT-5, Gemini 2.5 Pro, and a human reference for a case in the 32K bin. This [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Checklist item-level performance for each LLM in the checklist evaluation. The metric is the matching [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Checklist item-level performance for each LLM in the checklist evaluation. The metric is the matching [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Checklist item-level performance for each LLM in the checklist evaluation. The metric is the matching [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Screenshot of a visualization for one case, comparing checklists extracted directly from case documents [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Screenshot of a visualization for one case, comparing checklists extracted directly from case documents [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Screenshot of a visualization for one case, comparing checklists extracted directly from case documents [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Screenshot of a visualization for one case, comparing checklists extracted directly from case documents [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Screenshot of a visualization for one case, comparing checklists extracted directly from case documents [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18: Screenshot of a visualization for one case, comparing checklists extracted directly from case documents [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 19
Figure 19. Figure 19: Screenshot of a visualization for one case, comparing checklists extracted directly from case documents [PITH_FULL_IMAGE:figures/full_fig_p026_19.png]
Figure 20
Figure 20. Figure 20: Screenshot of a visualization for one case, comparing checklists extracted directly from case documents [PITH_FULL_IMAGE:figures/full_fig_p027_20.png]
Figure 21
Figure 21. Figure 21: Screenshot of a visualization for one case, comparing checklists extracted directly from case documents [PITH_FULL_IMAGE:figures/full_fig_p028_21.png]
Figure 22
Figure 22. Figure 22: Screenshot of a visualization for one case, comparing checklists extracted directly from case documents [PITH_FULL_IMAGE:figures/full_fig_p029_22.png]
Figure 23
Figure 23. Figure 23: Checklist item–level performance and statistics for end-to-end checklist extraction from full case [PITH_FULL_IMAGE:figures/full_fig_p030_23.png]
Figure 24
Figure 24. Figure 24: Checklist item–level performance and statistics for [PITH_FULL_IMAGE:figures/full_fig_p031_24.png]
Figure 25
Figure 25. Figure 25: Checklist item–level performance and statistics for chunk-by-chunk checklist extraction from full case [PITH_FULL_IMAGE:figures/full_fig_p032_25.png]
Figure 26
Figure 26. Figure 26: Screenshot of the annotation interface for checklist extraction from summaries. Annotators can add, [PITH_FULL_IMAGE:figures/full_fig_p033_26.png]
Figure 27
Figure 27. Figure 27: Screenshot of the annotation interface for checklist comparison. Annotators match items between two [PITH_FULL_IMAGE:figures/full_fig_p034_27.png]
Figure 28
Figure 28. Figure 28: Screenshot of the annotation interface for rating writing style similarity. Annotators compare two [PITH_FULL_IMAGE:figures/full_fig_p035_28.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 4 linked inside Pith

  1. [1]

    Filing Date: The date when the lawsuit was first initiated with the court

  2. [2]

    The city

    Parties: Description of each plaintiff and defendant involved, including relevant po- sitions or offices held. Use specific terms (e.g., “The city”, “The parents”) rather than generic terms (e.g., “The defendant”, “The plaintiffs”)

  3. [3]

    InFindings of the Association for Compu- tational Linguistics: ACL 2024, Bangkok, Thailand

    FENICE: Factuality evaluation of summariza- tion based on natural language inference and claim extraction. InFindings of the Association for Compu- tational Linguistics: ACL 2024, Bangkok, Thailand. Association for Computational Linguistics. Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, and Doug Downey. 2022. Multi- lexsum: Real-worl...

    arXiv 2024

  4. [4]

    Type of Counsel: The type(s) of counsel representing each side. Use brief category labels (e.g., private counsel, public inter- est nonprofit, government counsel, pro se) and include specific organizations (if appli- cable) in parentheses (e.g., Public interest nonprofit (ACLU)). B. Legal Foundation

  5. [5]

    § 1983, Title II ADA, FTCA) or judge-made vehicles (e.g., Bivens)

    Cause of Action: The legal vehicle(s) used to bring the claims (the “how” of suing), such as statutes that create a private/en- forcement right of action (e.g., 42 U.S.C. § 1983, Title II ADA, FTCA) or judge-made vehicles (e.g., Bivens)

    1983

  6. [6]

    Statutory/Constitutional Basis: The sub- stantive rights and sources of law allegedly violated (the ’what’ was violated), such as specific constitutional provisions/clauses (e.g., Fourteenth Amendment—Equal Pro- tection, First Amendment—Freedom of Association, Eighth Amendment) and statutory rights (e.g., ADA Title II, Rehab Act § 504)

  7. [7]

    Class Action or Individual Plaintiffs: Whether the case involves class action plaintiffs or individual plaintiffs with de- scriptions

  8. [8]

    Do not include Supreme Court Justices

    Judge Name: The first and last name of the judge(s) involved in the case. Do not include Supreme Court Justices. D. Related Cases

Show all 49 references
  1. [9]

    Consolidated Cases: Cases that were com- bined with this case for joint proceedings

  2. [10]

    Related Cases: Other cases referenced or connected to this case, listed by case code number. E. Filings and Proceedings

  3. [11]

    Include both sides if the defendant seeks relief

    Remedy Sought: What each party asks the court to grant, not what the court ordered or what the parties settled. Include both sides if the defendant seeks relief. C. Judge Information

  4. [12]

    Court Rulings: Judicial decisions on im- portant filings such as motions to dismiss, summary judgment, preliminary injunc- tions, class certification, and attorneys’ fees (excluding amended complaints and statements of interest)

  5. [13]

    2020 WL 4218003

    Reported Opinions: Citations of reported opinions using shortened Bluebook format (e.g., “2020 WL 4218003”), without case name, court, or date unless from a different case

    2020

  6. [14]

    Trials: Information about trial proceedings including scheduling, outcomes, and re- lated motions or rulings

  7. [15]

    Important Filings: Significant motions filed, including temporary restraining or- ders, preliminary injunctions, motions to dismiss, and motions for summary judg- ment

  8. [16]

    This in- cludes consent decrees and stipulated judg- ments/injunctions because they are entered as court orders

    Significant Terms: The substantive obli- gations ordered by the court. This in- cludes consent decrees and stipulated judg- ments/injunctions because they are entered as court orders

  9. [17]

    Decrees include injunctions, consent decrees, or stipulated judgments/injunctions

    Decree Dates: All decree-related dates such as entry date, modification/amend- ment dates (of the order), suspension/stay dates, partial termination dates, and full termination/vacatur dates. Decrees include injunctions, consent decrees, or stipulated judgments/injunctions

  10. [18]

    Duration: The duration of all decrees obli- gations (each as a separate entry). A ‘de- cree’ is any formal order or judgment is- sued by a court such as an injunction, con- sent decree, or stipulated judgment/injunc- tion, as opposed to a negotiated agreement between parties. ...

  11. [19]

    Appeals: Whether appeals were filed, which parties appealed, to which court, and the outcomes. F. Decrees

  12. [20]

    Settlement Date: All settlement-related dates (each as a separate entry) such as exe- cution/signing date(s), court approval date (if approved but not entered as an order), amendment dates, enforcement/retention dates without incorporation (e.g., court re- tains jurisdiction o...

  13. [21]

    A ’settlement’ is any negotiated agreement between parties that resolves a dispute, as opposed to a formal order or judgment is- sued by a court

    Duration: The duration of all settlements obligations (each as a separate entry). A ’settlement’ is any negotiated agreement between parties that resolves a dispute, as opposed to a formal order or judgment is- sued by a court

  14. [22]

    Answer Yes if the court explicitly retains jurisdiction to enforce the settlement without incorporating it into an order/judgment (e.g., Kokkonen retention)

    Court Enforcement: Whether the settle- ment (not entered as an order/judgment) is court-enforced. Answer Yes if the court explicitly retains jurisdiction to enforce the settlement without incorporating it into an order/judgment (e.g., Kokkonen retention). Answer No if it’s a p...

  15. [23]

    A settle- ment may be court-approved or enforced, but as long as it is not entered as an order, it is a settlement

    Settlement Terms: The substantive obli- gations the parties agree to in a settlement that is not entered as a court order. A settle- ment may be court-approved or enforced, but as long as it is not entered as an order, it is a settlement

  16. [24]

    Monitor Name: Name of any court- appointed monitor or special master

  17. [25]

    Monitor Reports: Monitor’s findings re- garding defendant compliance with court orders, including which terms are being met. I. Context

  18. [26]

    two-agree

    Factual Basis: The underlying facts and evidence supporting the legal claims, in- cluding: (i) details of relevant events (what, when, where, who), (ii) supporting ev- idence (physical, documentary, testimo- nial), and (iii) background context. C Writing Style Similarity Evalu...

    2023

  19. [27]

    Enforcement Disputes: The disputes about enforcing a settlement (a negoti- ated agreement not entered as a court order)—e.g., motions to enforce/contempt or requests invoking retained jurisdiction— each as a separate value with date, movant, issue, and outcome (or pending). H....

  20. [31]

    **extracted** and **evidence** is always a list, even if they hold a single object

  21. [32]

    Copy the **evidence** exactly as it appears in the case summary-no rewriting

  22. [33]

    reasoning

    If the case summary contains no relevant information, output the **extracted** as an empty list:,→ ``` {{ "reasoning": "<brief analysis>", "extracted": [] }} ``` Figure 29 Prompt for Comparing Single-Value Checklist Item You are given two pieces of legal information (A and B) ...

  23. [34]

    A contains B

    **"A contains B"** - A includes all the information in B, plus additional information,→

  24. [35]

    B contains A

    **"B contains A"** - B includes all the information in A, plus additional information,→

  25. [36]

    A equals B

    **"A equals B"** - A and B convey the same information (semantically equivalent),→

  26. [37]

    A and B are different

    **"A and B are different"** - A and B contain different or conflicting information,→ # Output Format Structure your response as follows: **Reasoning:** Provide your detailed analysis of how the two pieces of information relate to each other,→ **Final Answer:** State one of the...

  27. [38]

    Items that appear in **both A and B** (i.e., semantically equivalent),

  28. [39]

    Items that appear **only in A**,

  29. [40]

    common": [ {{

    Items that appear **only in B**. # Information to Compare ## List A: {information_A} ## List B: {information_B} # Output Format Structure your response as follows: **Reasoning:** Provide your detailed analysis of how the two lists relate to each other. Explain any mappings bet...

  30. [41]

    **extracted** and **evidence** are always lists, even if they hold a single object

  31. [42]

    Copy the **text** in evidence objects exactly as it appears in the case documents-no rewriting or paraphrasing.,→

  32. [43]

    Always include **source_document** and **location** for each piece of evidence

  33. [44]

    reasoning

    If the case documents contain no relevant information, output the **extracted** as an empty list:,→ ``` { "reasoning": "<brief analysis>", "extracted": [] } ```

  34. [45]

    Extract information from all relevant documents-do not stop after finding information in just one document.,→

  35. [46]

    Each distinct piece of information should be a separate item in the **extracted** list

  36. [47]

    beginning of document

    If you cannot determine the specific page number or section, you may use descriptive locations like "beginning of document", "middle section", or "near the end".,→ Figure 36 Prompt for Chunk-by-Chunk Extracting Checklist Items from Case Documents You are assisting a lawyer in ...

  37. [48]

    **Issue one function call**, or

  38. [49]

    decision

    **Stop** if all applicable checklist items are fully extracted and any non-applicable items are marked.,→ - When stopping, return **only** this JSON (no extra text): ```json { "decision": "stop", "reason": "<brief justification>" } Figure 40

  39. [2018]

    InProceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797–1807, Brussels, Bel- gium

    Don’t give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. InProceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1797–1807, Brussels, Bel- gium. Association for Computational L...

    2018 arXiv

  40. [2024]

    In The Twelfth International Conference on Learning Representations

    BooookScore: A systematic exploration of book-length summarization in the era of LLMs. In The Twelfth International Conference on Learning Representations. Howard Chen, Ramakanth Pasunuru, Jason Weston, and Asli Celikyilmaz. 2023a. Walking down the mem- ory maze: Beyond contex...

    2025 arXiv

  41. [2025]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    Qwen3 technical report.arXiv preprint arXiv:2505.09388. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information...

    2024 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.