Pith. sign in

REVIEW 4 major objections 6 minor 53 references

Inject or Navigate? Token-Efficient Retrieval for LLM Analysis of Transactional Legal Documents

T0 review · 4 major / 6 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Structured retrieval can replace full-corpus injection for transactional legal QA without losing judged answer quality, at a fraction of the tokens and often lower dollar cost.

desk verdict Solid engineering paper with honest uncertainty: two retrieval modes nearly match full injection on a small legal multi-doc set, plus a clean caching-crossover rule. read the letter →

arxiv 2607.05764 v1 pith:RCG47T2T submitted 2026-07-07 cs.CL cs.IR

Mahmoud Hany , Mourad ElSheraey , Mahmoud Said , Peter Naoum This is my paper · ORCID
classification cs.CLcs.IR
keywords legaldocumentQAretrieval-augmentedgenerationtokenefficiencypromptcachingstructure-awarechunkingcontractanalysisLLMnavigationembeddingretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The simplest way to answer questions over a set of related legal contracts is to put the whole corpus in the model’s context every time. That maximises recall, but the token bill and long-context degradation grow with the corpus, not the question. This paper shows what it took to replace that baseline with two structured alternatives over the same structure-aware chunks: embedding retrieval (NAVEMBED) and LLM navigation over a compact boolean-flagged index (NAVINDEX). On a 20-question benchmark with verified ground-truth answers, a position-controlled, reference-anchored judge found semantic retrieval with reranking tied with injection on 16 of 18 document-bound questions while attending to 17.3× fewer input tokens, and NAVINDEX tied on all 18 at 1.61× smaller total token footprint, a roughly 56× smaller answering context, and 25% lower recorded dollar cost. The authors also give a closed-form rule: once prompt caching is in play, cached full injection is cheaper in dollars only while the corpus stays below about ten times the retrieval payload. The practical message is that quality can be held as the corpus grows, provided the index carries the structural signals legal answers actually depend on.

What carries the argument

The NAVINDEX dual-file format: a compact navigation surface (boolean semantic flags, cross-reference and defined-term graphs, hard cap of ten nodes) that an LLM scans, plus a deterministic fetch of verbatim provisions for a short answering call. Together with the caching-crossover equations that separate token footprint from dollar cost under prompt-cache pricing.

What would settle it

A forced-decision (ties disallowed) re-judge of NAVINDEX Config 6 on the full 20-question scored set, or ground-truth node labels showing whether the two injection wins in the rerank run were retrieval misses rather than reasoning failures.

Watch

Extended reading notes

Core claim

On a scored legal-QA set with verified references, two structured retrieval modes over structure-aware chunking were judged interchangeable with full-corpus injection on the large majority of document-bound questions (NAVEMBED semantic+rerank 16/18 ties; NAVINDEX token-optimised 18/18 ties) while cutting attended input tokens by 17.3× and 1.61× respectively, with NAVINDEX also delivering a ~56× smaller answering context and 25% lower dollar cost; cached injection remains cheaper in dollars only below a closed-form corpus-to-payload threshold of roughly 10×.

Load-bearing premise

That a same-family LLM judge, even with forward/reverse position control and a verified reference answer, certifies true answer interchangeability—and that twenty answer-aware questions over six deliberately window-sized documents are enough to support the quality claims.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper compares full-corpus injection (INJECT) against two structured retrieval modes—embedding retrieval with optional reranking (NAVEMBED) and LLM navigation over a compact boolean-flagged index (NAVINDEX)—for QA over coordinated transactional legal documents. On a 20-question set with verified ground-truth answers (18 document-bound + 2 out-of-scope controls), a position-bias-controlled, reference-anchored pairwise LLM judge finds semantic+rerank NAVEMBED tied with INJECT on 16/18 document-bound questions at 17.3× fewer input tokens, and token-optimised NAVINDEX tied on all 18 at 1.61× lower total token footprint, ~56× smaller answering context, and 25% lower dollar cost. The authors also derive a closed-form caching-crossover rule: warm-cached INJECT is cheaper in dollars only while corpus size stays below roughly ten times the retrieval payload (Eqs. 1–3), with a NAVINDEX variant. Uncertainty is quantified via Clopper–Pearson intervals and a dedicated threats section.

Significance. If the quality claims hold under stronger validation, the work is practically significant for legal-document LLM systems: it shows that structure-aware retrieval can approach full-injection answer quality at a fraction of the attended tokens, and it cleanly separates token footprint from dollar cost under prompt caching—an operational distinction many RAG papers blur. Strengths include treating INJECT as an explicit upper-bound baseline, releasing prompts and the scored question set, reconciling per-question ledgers to aggregates, reporting out-of-scope controls, and stating a falsifiable crossover rule from a transparent price card. The NAVINDEX design (typed flags, cross-reference edges, hard selection cap) is a concrete, transferable engineering contribution for clause-structured instruments. The evaluation is small and judge-dependent, so significance is conditional on the interchangeability claims surviving forced-decision and/or human checks.

major comments (4)
  1. Section 3.2 and Section 8: The headline quality claims (RUN-012 16/18 document-bound ties; Config 6 18/18) rest entirely on a same-provider LLM judge (claude-opus scoring claude-sonnet) under a reference-anchored forward/reverse protocol. Position control and key-fact criteria help, but residual family-level coherence can still inflate ties. The only forced-decision (ties-disallowed) probe—on a disjoint 15-question Phase II pool—yielded a 10–5 INJECT lean (exact binomial p=0.30, CI [0.38,0.88]), and the manual audit found 2/15 verdicts likely wrong (Table 5). The paper itself calls a forced-decision run on Config 6 “the single most important follow-up.” Without that result (or a cross-provider / multi-annotator human audit of the 20 scored pairs) reported in the manuscript, the abstract’s interchangeability language overstates what the current protocol certifies. This is load-bearing for
  2. Section 4.2, Section 6, and Table 5: Scale and uncertainty. All scored claims come from one answering model, one judge, one run per configuration, n=18 document-bound questions over six deliberately window-sized documents. Clopper–Pearson intervals are correctly wide (e.g., 16/18 → [0.65, 0.99]; 18/18 → [0.81, 1.00]), but the abstract and conclusion state the point estimates without carrying that uncertainty into the prose claims. A single additional INJECT win would materially change the narrative. The evaluation should either expand (more questions, a second model family, or DocNavBench open-search results) or systematically temper the abstract/conclusion to match Table 5.
  3. Section 4.2 and Section 8 (“No ground-truth node labels”): Retrieval recall and precision are not measured. The two RUN-012 INJECT wins are diagnosed as “likely retrieval failure (relevant nodes ranked outside the top-10),” but this is conjectural without gold node labels. Because the paper’s contribution is retrieval strategy (chunking held fixed), the absence of any retrieval metric means answer-level ties cannot be attributed to successful retrieval versus lucky reasoning over incomplete context. Adding even a partial gold-node set for the six scored documents—or a must_contain substring rubric as proposed in Section 9—would make the quality claims diagnostic rather than only comparative.
  4. Table 3 note ‡ and Section 5.3: NAVINDEX Config 6 changes serialisation, question set, and harness together relative to Configs 1–5. The within-run pairing against INJECT is valid, but the claim that Config 6 is “the highest single-run tie rate in either comparison” and the descriptor-ladder narrative are confounded by the set change. The manuscript should either re-run Config 6’s serialisation on the Phase II pool (or Configs 1–5 on the scored set) or clearly demote cross-configuration quality comparisons and rest the NAVINDEX claim solely on the paired Config-6 vs INJECT result.
minor comments (6)
  1. Section 3.1 / Appendix C: The reviewing lawyer notes that scored questions are answer-aware (written knowing the document contains the answer), unlike DocNavBench’s issue-driven register. Flag this more prominently in the abstract or introduction so readers do not over-generalise to open legal search.
  2. Section 6: Explicitly state that RUN-012 and Config 6 were never judged head-to-head; Table 4’s side-by-side layout invites an invalid quality ranking. A one-sentence box or caption warning would help.
  3. Figure 2: The left panel omits NAVINDEX with a parenthetical; a short caption note on why (index size is document-dependent) would reduce confusion.
  4. Eqs. (1)–(3): State units and that output tokens are excluded by empirical parity (±8% median); a sensitivity line if output diverges would strengthen the cost model.
  5. Appendix A Table 6: Phase I dollar cells are correctly omitted; a footnote that Phase II corpus size (~214.5K) differs from the scored 163K prefix would prevent misreading cross-phase dollar ratios.
  6. Typos / consistency: abstract uses “17.3x” and “29.9x” without the × symbol used in the body; “navembed”/“NAVEMBED” casing varies; arXiv date line says “7 Jul 2026” which may be a placeholder.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical systems comparison with algebraically derived cost crossover from stated prices; quality claims are measured outcomes, not predictions forced by inputs.

full rationale

This paper is a controlled empirical comparison of full-corpus injection against two structured retrieval modes (NAVEMBED, NAVINDEX) on a 20-question legal-QA benchmark with verified ground-truth answers. The headline quality claims (RUN-012 16/18 document-bound ties at 17.3× fewer tokens; Config-6 18/18 ties at 1.61× smaller footprint and 25% lower dollar cost) are measured pairwise-judge outcomes, not theoretical predictions derived from fitted parameters. The closed-form caching-crossover rule (Eqs. 1–3, Section 7) follows by direct algebra from the stated Anthropic price card (p_in = $3/MTok, p_r = $0.30/MTok ⇒ p_in/p_r = 10) and measured payload sizes; no free parameter is fitted to the quality results and then re-presented as a prediction. Design choices (ten-node hard cap, boolean flag set, structure-aware chunking held fixed) are engineering parameters validated by ablation, not self-definitional constructions of the reported tie rates. There is no load-bearing self-citation chain, no uniqueness theorem imported from the authors’ prior work, and no renaming of a known empirical pattern as a first-principles derivation. The evaluation is self-contained against its own inject baseline and external price schedule; residual threats (same-provider judge, small n, no node-level recall labels) are validity concerns, not circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The central quality and cost claims rest on standard LLM/RAG machinery plus a small set of design choices (node cap, flag vocabulary, price card, judge protocol) and the untested assumption that structure-aware proprietary chunking is held fixed without harming the comparison. No new physical entities; free parameters are engineering knobs and commercial prices.

free parameters (4)
  • hard node-selection cap t=10
    Chosen after Configs 1–3; directly bounds S3 cost and is load-bearing for the NAVINDEX efficiency claim.
  • retrieve-40 then rerank-keep-10
    Pipeline shape for RUN-012; determines the 9.4K mean payload used in the crossover calculation.
  • Anthropic price card (pin=$3, pr=$0.30, pw=$3.75 per MTok)
    Sets the factor pin/pr=10 that produces the ‘corpus < 10R’ rule; prices are external but the numerical threshold is price-card-dependent.
  • boolean flag vocabulary (isDef, hm, hd, hpct, hll, hc, hpty)
    Hand-designed semantic pre-filters; quality of NAVINDEX navigation depends on this choice.
assumptions (5)
  • domain assumption Full-corpus injection maximises retrieval recall and is therefore the correct upper-bound baseline when the corpus fits the context window.
    Stated in Introduction and Section 3.1; frames every paired comparison.
  • domain assumption A position-bias-controlled, reference-anchored LLM judge (same provider family) yields verdicts that certify answer-level equivalence for the purpose of the quality claim.
    Section 3.2 and 8; the paper quantifies residual error via audit and forced-decision probe.
  • ad hoc to paper Structure-aware proprietary chunking can be held fixed so that measured differences are attributable only to retrieval strategy.
    Section 1 and 3; no ablation against fixed-size chunking is reported.
  • domain assumption Output length does not differ systematically between inject and navigate, so output tokens can be omitted from the cost model.
    Section 7; median output within ±8% in these runs.
  • standard math Standard dense-retrieval, cross-encoder, BM25, and prompt-caching mechanisms behave as in the cited literature.
    Related Work and method sections rely on established IR/LLM components.
invented entities (2)
  • NAVINDEX dual-file format (*.index.json + *.full.json) with typed boolean flags, crossReferencedIds, and hard selection cap
    purpose: Enable LLM navigation over legal structure without embedding similarity and without shipping full provision text in the scan step.
    Core engineering contribution of Comparison 2; independent evidence is the Config-6 tie result on the scored set, not an external physical measurement.
  • caching-crossover rule C < 10R (and the NAVINDEX variant C < I + 10S) independent evidence
    purpose: Separate token-footprint reduction from dollar cost under prompt caching.
    Closed-form from the stated price card; falsifiable by changing provider prices or measuring different payloads.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Inject or Navigate? Token-Efficient Retrieval for LLM Analysis of Transactional Legal Documents." pith.science (2026). https://pith.science/paper/RCG47T2T

@misc{pith2026260705764,
  author       = {Pith},
  title        = {Pith review of: Inject or Navigate? Token-Efficient Retrieval for LLM Analysis of Transactional Legal Documents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RCG47T2T}},
  note         = {Machine review of arXiv:2607.05764}
}
read the original abstract

Answering questions over a set of transactional legal documents is most simply done by injecting the whole corpus into the LLM's context window on every query. That baseline maximises retrieval recall, but its token footprint scales with the corpus rather than the question, and long-context degradation scales with it. We report what it took to replace full-corpus injection in a legal-document analysis system, comparing it against two structured retrieval modes over our proprietary structure-aware chunking: embedding retrieval (NAVEMBED) and LLM navigation over a compact structured index (NAVINDEX). On a 20-question benchmark with verified ground-truth answers, a position-bias-controlled, reference-anchored pairwise judge scored semantic retrieval with reranking tied with injection on 16 of 18 document-bound questions (injection preferred on 2) while attending to 17.3x fewer input tokens (a general-text-embedding (GTE) configuration reaches 29.9x at a lower tie rate); both modes were judged tied on the 2 out-of-scope controls. NAVINDEX was judged tied on all 18 at a 1.61x smaller total token footprint, a ~56x smaller answering context, and 25% lower dollar cost. We derive a closed-form caching-crossover rule: cached injection is cheaper in dollars only while the corpus stays below roughly ten times the retrieval payload. Scope and uncertainty are quantified in Section 8.

Figures

Figures reproduced from arXiv: 2607.05764 by the authors.

Figure 1
Figure 1. The three pipeline modes. Left: inject: full corpus in context on every query. Centre: navembed: co￾sine retrieval selects the top-k nodes at query time. Right: navindex: three-step structured navigation over a compact boolean-flagged index, no embedding similarity required. 4 INJECT vs. NAVEMBED 4.1 Approach navembed pre-indexes each document into a tree of nodes. Every node carries the section’s verbatim text plus… view at source ↗
Figure 2
Figure 2. Cumulative dollar cost versus repeated queries; navembed at the measured RUN-012 payload (9.4K tokens/query, full input price). Left: a single ∼35K-token document (navindex omitted; its cost depends on that document’s index size). Right: the measured 163K six-document corpus; navindex at its Config 6 structure (cached 98.6K index re-read per query plus ∼2.9K provisions at full input price; Eq. 3). Phase III settled … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 53 canonical work pages

  1. [1]

    Introducing the Model Context Pro- tocol

    Anthropic. Introducing the Model Context Pro- tocol. Anthropic, November 2024.https://www. anthropic.com/news/model-context-protocol

    work page 2024

  2. [2]

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang. Lost in the Middle: How Language Models Use Long Contexts. TACL, 2024

    work page 2024

  3. [3]

    Hendrycks, C

    D. Hendrycks, C. Burns, A. Chen, S. Ball. CUAD: An Expert-Annotated NLP Dataset for Legal Con- tract Review. InNeurIPS Datasets and Bench- marks, 2021

    work page 2021

  4. [4]

    Koreeda, C

    Y. Koreeda, C. D. Manning. ContractNLI: A Dataset for Document-Level Natural Language In- ference for Contracts. InFindings of EMNLP, 2021

    work page 2021

  5. [5]

    Reconstructing Context: Evaluating Advanced Chunking Strategies for Retrieval-Augmented Generation

    C. Merola, J. Singh. Reconstructing Context: Eval- uating Advanced Chunking Strategies for Retrieval- Augmented Generation.arXiv:2504.19754, 2025

    work page Pith review arXiv 2025

  6. [6]

    Taiwo, M

    S. Taiwo, M. A. Yusoff. Evaluating Chunking Strate- gies for Retrieval-Augmented Generation in Oil and Gas Enterprise Documents.arXiv:2603.24556, 2026

    work page arXiv 2026

  7. [7]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V.Karpukhin, N.Goyal, etal. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In NeurIPS, 2020

    work page 2020

  8. [8]

    Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, et al. Retrieval-Augmented Generation for Large Lan- guage Models: A Survey.arXiv:2312.10997, 2023

    work page Pith review arXiv 2023

Show all 53 references
  1. [9]

    Karpukhin, B

    V. Karpukhin, B. Oğuz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, W. Yih. Dense Passage Re- trieval for Open-Domain Question Answering. In EMNLP, 2020

    2020

  2. [10]

    Reimers, I

    N. Reimers, I. Gurevych. Sentence-BERT: Sen- tence Embeddings using Siamese BERT-Networks. InEMNLP, 2019

    2019

  3. [11]

    L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, F. Wei. Text Em- beddings by Weakly-Supervised Contrastive Pre- training.arXiv:2212.03533, 2022. 10

    2022 arXiv

  4. [12]

    Z. Li, X. Zhang, Y. Zhang, D. Long, P. Xie, M. Zhang. Towards General Text Embed- dings with Multi-stage Contrastive Learning. arXiv:2308.03281, 2023

    2023 arXiv

  5. [13]

    S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, D. Lian, J.-Y. Nie. C-Pack: Packaged Resources to Advance General Chinese Embedding.arXiv:2309.07597, 2023

    2023 arXiv

  6. [14]

    K. Song, X. Tan, T. Qin, J. Lu, T.-Y. Liu. MP- Net: Masked and Permuted Pre-training for Lan- guage Understanding. InNeurIPS, 2020

    2020

  7. [15]

    Robertson, H

    S. Robertson, H. Zaragoza. The Probabilistic Rele- vance Framework: BM25 and Beyond.Foundations and Trends in Information Retrieval, 3(4), 2009

    2009

  8. [16]

    G. V. Cormack, C. L. A. Clarke, S. Buettcher. Re- ciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. InSIGIR, 2009

    2009

  9. [17]

    Nogueira, K

    R. Nogueira, K. Cho. Passage Re-ranking with BERT.arXiv:1901.04085, 2019

    1901 arXiv

  10. [18]

    Prompt Caching with Claude

    Anthropic. Prompt Caching with Claude. An- thropic Documentation, 2024.https://docs. anthropic.com/en/docs/build-with-claude/ prompt-caching[Accessed: June 2026; content subject to change as a living document]

    2024

  11. [19]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. InNeurIPS, 2023

    2023

  12. [20]

    P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, et al. Large Language Models are not Fair Evaluators.arXiv:2305.17926, 2023

    2023 arXiv

  13. [21]

    Tuggener, P

    D. Tuggener, P. von Däniken, T. Peetz, M. Cieliebak. LEDGAR: A Large-Scale Multi- label Corpus for Text Classification of Legal Provisions in Contracts. InLREC, 2020

    2020

  14. [22]

    Sarthi, S

    P. Sarthi, S. Abdullah, A. Tuli, S. Khanna, A. Goldie, C. D. Manning. RAPTOR: Recursive AbstractiveProcessingforTree-OrganizedRetrieval. InICLR, 2024

    2024

  15. [23]

    Zhang, Y

    M. Zhang, Y. Tang, et al. (VectifyAI). PageIndex: Document Index for Vectorless, Reasoning-based RAG. 2025.https://github.com/VectifyAI/ PageIndex

    2025

  16. [24]

    C. J. Clopper, E. S. Pearson. The Use of Confidence or Fiducial Limits Illustrated in the Case of the Bi- nomial.Biometrika, 26(4):404–413, 1934

    1934

  17. [25]

    W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, M. Zhou. MiniLM: Deep Self-Attention Distilla- tion for Task-Agnostic Compression of Pre-Trained Transformers. InNeurIPS, 2020

    2020

  18. [26]

    Anthropic. Pricing. Anthropic, 2026.https: //www.anthropic.com/pricing[Accessed: June 2026; prices subject to change]

    2026

  19. [27]

    Red.” (reduction) is theinject/navembedinput-token ratio. “Inj. w

    A. Panickssery, S. R. Bowman, S. Feng. LLM Evalu- ators Recognize and Favor Their Own Generations. InNeurIPS, 2024. 11 A Run History and Lessons The twelvenavembedconfigurations were run in three phases as the evaluation harness matured. Each run is an internally controlled pa...

    2024

  20. [29]

    For each sub-question, identify which document(s) contain relevant text

  21. [30]

    an amendment that modifies a clause of the original)

    Resolve cross-references between documents (e.g. an amendment that modifies a clause of the original)

  22. [31]

    - A more specific provision usually overrides a more general one

    Reconcile precedence: - A later-dated amendment, side letter, restatement, or supplement OVERRIDES the original to the extent of any inconsistency. - A more specific provision usually overrides a more general one. - When in doubt, state which document you treated as controllin...

  23. [32]

    [doc_id] Clause X

    Compose a single integrated answer. # Answer format - Lead with the direct answer (1-2 sentences). Then a short supporting analysis. - Cite every load-bearing fact as: "[doc_id] Clause X". - Quote short, decisive phrases verbatim where wording matters. - Preserve exact numeric...

  24. [33]

    Decompose the question: sub-questions, concepts, thresholds, conditions, parties

  25. [34]

    Scan documentTags.Topic across all documents to identify relevant documents

  26. [35]

    Discard any node where isTocEntry is true

  27. [36]

    The answer often lives in more than one document

    Treat EVERY document as in scope until you have a structural reason to rule it out. The answer often lives in more than one document

  28. [37]

    Use boolean flags as fast filters before reading summaries: - defined term in question -> isDefinition: true nodes across all docs - amount or fee -> hasMoney: true nodes - date or deadline -> hasDate: true nodes - rate or percentage -> hasPercentage: true nodes - liability ca...

  29. [38]

    Confirm each candidate via title, snippet, summary, keywords

  30. [39]

    When an amendment addresses the same subject as the original, include BOTH

  31. [40]

    [doc=..., clause_ref=..., node_id=...]

    Follow crossReferencedIds only when the node supplies a definition or condition the selected clause explicitly depends on. # Hard rules - Return AT MOST 10 (doc_id, node_id) pairs. - doc_id MUST exactly match a <index doc_id="..."> value in the block below. - Do not invent doc...

  32. [41]

    Decompose the question into sub-questions, conditions, thresholds, parties

  33. [42]

    For each sub-question, identify which document(s) bear on it

  34. [43]

    - A more specific provision usually overrides a more general one

    Resolve precedence: - A later-in-time amendment overrides the original to the extent of inconsistency. - A more specific provision usually overrides a more general one

  35. [44]

    Flag pointers outside it

    Resolve cross-references within the extract. Flag pointers outside it

  36. [45]

    [doc_id] Clause X

    Compose a single integrated answer. # Answer format - Lead with the direct answer (1-2 sentences). Then a brief supporting analysis. - Cite every load-bearing fact as: "[doc_id] Clause X". - Quote short, decisive phrases verbatim where wording matters. - Preserve exact numeric...

  37. [46]

    Key facts -- does the answer include the specific facts stated in the reference (exact numbers, dates, party names, amounts, percentages, clause citations)? Missing or wrong key facts are the primary basis for a loss

  38. [47]

    No hallucination -- does the answer avoid asserting facts that contradict the reference or that are not supported by the reference?

  39. [48]

    Completeness -- if the reference has multiple parts, does the answer address all of them?

  40. [49]

    not found

    No-answer correctness -- if the reference says the answer is not in the documents, an answer that correctly says "not found" is correct; an answer that fabricates a plausible-sounding fact is wrong. # Verdict rules - Return "A" if Answer A is closer to the reference. - Return ...

  41. [50]

    Factual correctness -- does the answer state the right facts? An answer with one wrong fact loses to an answer with all right facts, even if the right-fact answer is shorter

  42. [51]

    Completeness -- does it cover everything the question asks? A multi-part question requires every part addressed

  43. [52]

    Specificity -- does it cite the controlling clause? does it preserve exact numeric values, currencies, dates, parties?

  44. [53]

    Honesty about uncertainty -- an answer that correctly flags missing context beats one that hallucinates a confident wrong answer

  45. [54]

    A" if A is clearly better on the criteria above. - Return

    Clarity -- is it unambiguous and easy to follow? (least important tiebreaker) # Verdict rules - Return "A" if A is clearly better on the criteria above. - Return "B" if B is clearly better. - Return "tie" if both are equally good, equally bad, or differ only in style. - Do not...

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.