pith. sign in

arxiv: 2606.21676 · v1 · pith:RMISAMSGnew · submitted 2026-06-19 · 💻 cs.IR

CRAwLeR -- Cross-Reference Aware Legal Retrieval

Maciej Jalocha , William Michelsen This is my paper

Pith reviewed 2026-06-26 12:35 UTC · model grok-4.3

classification 💻 cs.IR
keywords context-aware retrievallegal documentscross-referenceschunk retrievalconstruct validityinformation retrieval benchmarkDanish legal datasetPolish legal dataset
0
0 comments X

The pith

A pipeline builds legal retrieval benchmarks where queries require cross-reference context, exposing that current models top out at 55-59% recall.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes a focused benchmark for context-aware chunk retrieval in legal documents by operationalizing the specific phenomenon of cross-references. Existing benchmarks rarely confirm that their queries actually need surrounding context, which makes performance numbers hard to interpret. The authors create a pipeline that detects cross-references, generates demanding queries via LLM, and filters them with adversarial checks plus manual validation showing roughly 80% genuine cases. They release Danish and Polish datasets plus a contextualization baseline, demonstrating that the remaining performance gap traces to the contextualizing LLM rather than the retriever itself.

Core claim

CRAwLeR operationalizes cross-reference-aware context utilization for chunk retrieval in legal documents through a pipeline that detects cross-references, identifies query candidates, links target chunks to relevant context, generates context-demanding queries with an LLM, and filters via adversarial non-contextual baseline and assurance prompts. The resulting CRAwLeR-DK and CRAwLeR-PL datasets achieve approximately 80% validity on manual review, yet best Recall@10 reaches only 55% and 59% respectively. Ablation shows the gap stems from the contextualising LLM, not the retriever, and labelled context chunks routinely outrank the target even when it is retrieved.

What carries the argument

The CRAwLeR pipeline that detects legal cross-references, generates LLM queries, and applies adversarial non-contextual filtering plus assurance prompts to ensure queries require the labelled context.

If this is right

  • The benchmarks remain unsolved at 55-59% Recall@10 despite a strong contextualization baseline.
  • Performance gaps are attributable to the contextualising LLM rather than the underlying retriever.
  • Even when the target chunk is retrieved in the top ten, labelled context chunks routinely rank higher.
  • Failures follow systematic and named patterns that can be inspected for targeted fixes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same cross-reference detection and adversarial filtering steps could be reused to create analogous benchmarks in other document domains that rely on internal references.
  • Future work could test whether replacing the contextualising LLM with a more capable model closes the observed gap without changes to the retriever.
  • The named failure patterns provide a concrete starting point for improving query generation or context handling in legal retrieval systems.

Load-bearing premise

The filtered LLM-generated queries genuinely require the labelled context chunks rather than being answerable from the target alone.

What would settle it

A fresh manual review of the released queries finding substantially below 80% validity, or a retrieval system that reaches high recall without using the provided context chunks.

Figures

Figures reproduced from arXiv: 2606.21676 by Maciej Jalocha, William Michelsen.

Figure 1
Figure 1. Figure 1: Visual definition and explanation of a possible context-aware chunk retrieval task item [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: shows chunking strategy for the CRAwLeR-PL dataset. A chunk is meant to cap￾ture an individual piece of information. In practice it was split by newlines. Art. 37s. 1. Head of the organizational unit of the State Fire Service referred to in Article 37l point 2, in relation to firefighters delegated to perform tasks in civilian institutions: 1) is a disciplinary superior and supervises compliance with offic… view at source ↗
Figure 3
Figure 3. Figure 3: shows how chunking was performed on the CRAwLeR-DK dataset. The splits capture "pieces" (stykke / stk. These are parts of legal sec￾tion. Legal sections being denoted by the symbol §). SECTION 12. A public housing association’s articles of association must contain provisions on: 1) Name and domicile of the housing association. 2) Capital relations of the housing association. Pcs. 2. Articles of association… view at source ↗
Figure 4
Figure 4. Figure 4: Visualisation of the definitions of implicit context, context chunks, explicit references, and target chunk, [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Overview of the CRAwLeR dataset creation pipeline. Legal documents are chunked and cross-references [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The query asks “who” performs the examina [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: An example of a well-formed query targeting [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 7
Figure 7. Figure 7: A partial context selection failure caused by [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 10
Figure 10. Figure 10: A query generation failure for CRAwLeR￾DK. The target chunk merely cross-references §119 as one of the changes permitted during liquidation. The substantive conditions live entirely in the context chunk. Original Danish version in the Appendix 24. Query (barnetslov_201_stk1): Does the state pay the costs of the free coun￾seling and guidance provided to children and young people with reduced physical or me… view at source ↗
Figure 11
Figure 11. Figure 11: An example of a well-formed query target [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Example of a contextualisation failure (near [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Manual analysis ratios for CRAwLeR-PL with 95% exact CIs. targeted and context-dependent. BGE-M3 retrieves the target chunk in the top 10 for 13 of 24 queries; BM25 does so for 13 of 24. We inspected the 8 queries missed by both re￾trievers. In 3 of 8, the contextualisation describes the wrong topic, often reusing a generic prefix shared across sibling chunks ( [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: A contextualisation failure (wrong-topic pre [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: A contextualisation failure (underspecific [PITH_FULL_IMAGE:figures/full_fig_p021_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Manual analysis ratio for CRAwLeR-DK with 95% exact CIs [PITH_FULL_IMAGE:figures/full_fig_p021_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Token distances from the target chunk to the [PITH_FULL_IMAGE:figures/full_fig_p022_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Token distances from the target chunk to [PITH_FULL_IMAGE:figures/full_fig_p022_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Original Polish version of Figure [PITH_FULL_IMAGE:figures/full_fig_p035_19.png] view at source ↗
Figure 21
Figure 21. Figure 21: Original Polish version of Figure [PITH_FULL_IMAGE:figures/full_fig_p036_21.png] view at source ↗
Figure 20
Figure 20. Figure 20: Original Polish version of Figure [PITH_FULL_IMAGE:figures/full_fig_p036_20.png] view at source ↗
Figure 22
Figure 22. Figure 22: Original Polish version of Figure [PITH_FULL_IMAGE:figures/full_fig_p036_22.png] view at source ↗
Figure 24
Figure 24. Figure 24: Original Danish version of Figure [PITH_FULL_IMAGE:figures/full_fig_p037_24.png] view at source ↗
Figure 23
Figure 23. Figure 23: Original Danish version of Figure [PITH_FULL_IMAGE:figures/full_fig_p037_23.png] view at source ↗
Figure 25
Figure 25. Figure 25: Original Danish version of Figure [PITH_FULL_IMAGE:figures/full_fig_p037_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Original Danish version of Figure [PITH_FULL_IMAGE:figures/full_fig_p038_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Original Danish version of Figure [PITH_FULL_IMAGE:figures/full_fig_p038_27.png] view at source ↗
Figure 29
Figure 29. Figure 29: Token distributions of chunks for the Danish [PITH_FULL_IMAGE:figures/full_fig_p039_29.png] view at source ↗
Figure 28
Figure 28. Figure 28: Token distributions of chunks for the Polish [PITH_FULL_IMAGE:figures/full_fig_p039_28.png] view at source ↗
Figure 31
Figure 31. Figure 31: Document Cosine Similarity Heatmap for the CRAwLeR-DK dataset. Computed us￾ing paraphrase-multilingual-MiniLM-L12-v2 sen￾tence transformer (Reimers and Gurevych, 2019) H Code and Artifacts The repository is in the abstract’s footnote. Inside the repository there is also a link to datasets [PITH_FULL_IMAGE:figures/full_fig_p040_31.png] view at source ↗
read the original abstract

Existing benchmarks for context-aware chunk retrieval rely heavily on repurposed task items and rarely demonstrate that their queries genuinely require context, making score interpretation difficult. We focus on a specific kind of context dependence, legal cross-references, and introduce CRAwLeR, an operationalization of a narrow, well-defined phenomenon: cross-reference-aware context utilization for chunk retrieval in legal documents. Our pipeline detects legal cross-references, identifies query candidates, links target chunks to their relevant context, generates context-demanding queries with an LLM, and filters them through both an adversarial non-contextual baseline and an assurance prompt. We release CRAwLeR-DK and CRAwLeR-PL, Danish and Polish datasets built with this pipeline, alongside a strong Anthropic-style contextualization baseline. Manual analysis finds that approximately 80% of randomly sampled queries genuinely target the labelled target chunk and require context, with failures following systematic and named patterns. The benchmarks are hard but not solved: best Recall@10 reaches 55% on CRAwLeR-DK and 59% on CRAwLeR-PL. Ablation and failure analysis attribute the remaining gap to the contextualising LLM, not the retriever. Even when the target is retrieved in the top ten, labelled context chunks routinely outrank it. We are the first dataset for context-aware chunk retrieval to carefully consider construct validity and inspect our results in the light of such a narrow, well-defined phenomenon.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The manuscript introduces CRAwLeR, a pipeline for constructing datasets that operationalize context-aware chunk retrieval for legal cross-references. It detects cross-references, generates LLM queries intended to require context, applies adversarial filtering and assurance prompts, and releases CRAwLeR-DK and CRAwLeR-PL (Danish and Polish). Manual analysis is reported to find ~80% validity; benchmarks show Recall@10 of 55% (DK) and 59% (PL), with ablations attributing the gap to the contextualizing LLM rather than the retriever. The work positions itself as the first to carefully address construct validity for this narrow phenomenon.

Significance. If the manual validation holds, the paper contributes new datasets and empirical measurements on a well-defined legal IR task, including concrete Recall@10 figures, a strong baseline, and ablations that isolate the contextualizer as the bottleneck. The emphasis on construct validity and systematic failure patterns is a positive step beyond repurposed benchmarks.

major comments (1)
  1. [Abstract] Abstract: the central claim that the queries 'genuinely require context' (and thus that the datasets demonstrate careful construct validity) rests on the ~80% manual validity rate, yet no sample size, annotation criteria for 'requiring context', blinding procedures, or inter-annotator agreement are provided. Without these, the reliability of the 80% figure cannot be assessed and systematic biases in the validation cannot be ruled out.
minor comments (1)
  1. The abstract mentions 'failures following systematic and named patterns' but does not name them; expanding this in the main text would aid interpretability of the failure analysis.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive comment on the transparency of our manual validation. We address it point by point below and will revise the manuscript to incorporate the requested details.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim that the queries 'genuinely require context' (and thus that the datasets demonstrate careful construct validity) rests on the ~80% manual validity rate, yet no sample size, annotation criteria for 'requiring context', blinding procedures, or inter-annotator agreement are provided. Without these, the reliability of the 80% figure cannot be assessed and systematic biases in the validation cannot be ruled out.

    Authors: We agree that the manuscript does not currently report the sample size, annotation criteria, blinding procedures, or inter-annotator agreement for the manual validation. The abstract summarizes the ~80% figure and notes systematic failure patterns, but the full methodological details are absent. We will add a dedicated subsection in the revised manuscript that specifies: the exact sample size drawn for manual review, the annotation guidelines used to determine whether a query 'genuinely requires context' (i.e., cannot be answered from the target chunk alone), whether annotators were blinded to query provenance, and any inter-annotator agreement metrics. This will allow readers to evaluate reliability and potential biases directly. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical dataset and benchmark construction

full rationale

The paper describes an empirical pipeline for generating legal cross-reference datasets (CRAwLeR-DK, CRAwLeR-PL) via LLM query generation followed by adversarial filtering and manual validation (~80% validity). No equations, fitted parameters renamed as predictions, self-citations as load-bearing premises, or derivations that reduce to inputs by construction appear in the abstract or described process. The central claims are the release of new datasets and retrieval performance measurements on them, which stand as independent empirical results rather than tautological outputs of prior self-referential steps.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The work rests on standard IR and LLM usage assumptions for data generation and filtering; no new free parameters, mathematical axioms, or invented entities are described in the abstract.

axioms (1)
  • domain assumption LLM-generated queries after filtering can be made to require specific cross-reference context
    The pipeline and assurance prompt step rely on this to produce valid test items.

pith-pipeline@v0.9.1-grok · 5785 in / 1250 out tokens · 31973 ms · 2026-06-26T12:35:35.897668+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 3 canonical work pages · 1 internal anchor

  1. [1]

    In Gim, Guojun Chen, Seung seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong

    Cost-effective synthetic data generation for post-training using QWICK. In Gim, Guojun Chen, Seung seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong. 2024. Prompt cache: Modular attention reuse for low-latency infer- ence.Preprint, arXiv:2311.04934. Omer Goldman, Alon Jacovi, Aviv Slobodkin, Aviya Maimon, Ido Dagan, and Reut Tsarfaty. 2025. Is it r...

    work page arXiv 2024

  2. [2]

    everyone wants to do the model work, not the data work

    “everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, New York, NY , USA. Association for Computing Machinery. Nicolas Sannier, Morayo Adedjouma, Mehrdad Sabet- zadeh, and Lionel Briand. 2016. Automated classi- fication of legal c...

    work page arXiv 2021

  3. [3]

    InFindings of the Associa- tion for Computational Linguistics: ACL 2025, pages 8063–8075, Vienna, Austria

    Document segmentation matters for retrieval- augmented generation. InFindings of the Associa- tion for Computational Linguistics: ACL 2025, pages 8063–8075, Vienna, Austria. Association for Compu- tational Linguistics. Junjie Wu, Jiangnan Li, Yuqing Li, Lemao Liu, Liyan Xu, Jiwei Li, Dit-Yan Yeung, Jie Zhou, and Mo Yu

    2025

  4. [4]

    SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehension

    Sitemb-v1.5: Improved context-aware dense retrieval for semantic association and long story com- prehension.Preprint, arXiv:2508.01959. Liyan Xu, Jiangnan Li, Mo Yu, and Jie Zhou. 2024a. Fine-grained modeling of narrative context: A co- herence perspective via retrospective questions. In Proceedings of the 62nd Annual Meeting of the As- sociation for Comp...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  5. [5]

    • The query should not target any statement that is made in the context chunks or neighboring chunks

    Targets the target chunk— The answer to the query is a piece of information established in the target chunk, that is dependent on some context chunks. • The query should not target any statement that is made in the context chunks or neighboring chunks

  6. [6]

    The target chunk will not be retrieved based on its own content alone

    However, the retriever requires context chunks- for the target chunk to be selected as positive, at least one of the context chunks is required. The target chunk will not be retrieved based on its own content alone. • This means that the query should contain information that is only found in the context chunks, and that is required to understand the targe...

  7. [7]

    Article 5 para- graph 2 point 3 letter a

    The query must not contain explicit references to the context chunks- the point is that if you are crafting a query, that contains the relationship between the target chunk and context chunks, refer to the substance of the context chunk, not simply place its identifier like "Article 5 para- graph 2 point 3 letter a". This totally misses the point

  8. [8]

    No multiple questions hidden in the query: create a query poiting to a SINGLE statement made in the target chunk

  9. [9]

    Use of the Target Implicit Context Chunks: They might be helpful to understand the target chunk

  10. [10]

    This is to ensure that the retriever is not just matching keywords, but actually understanding the content

    Avoid unnecessary lexical overlap: The query should not simply repeat the same words as the target chunk, but rather use different wording to express the same meaning. This is to ensure that the retriever is not just matching keywords, but actually understanding the content. However, ensure that the words that are crucial, especially domain-specific terms...

  11. [11]

    Multiple statement in the target chunk- If the target chunk contains multiple statements, the query should be about only one such statement, which is dependent on some context chunks

  12. [12]

    art. 45 ust. 2 pkt 1 ustawy

    Pinpoint citations may be abbreviated: A target chunk may cite a context chunk in full (e.g., "art. 45 ust. 2 pkt 1 ustawy") or in abbreviated form when the context is nearby (e.g., "ust. 2 pkt 1", or even just "pkt 1"). Context chunks are themselves prefixed with their own (sometimes shorter) identifier. All context chunks provided have been explicitly r...

  13. [16]

    ust. 2 pkt 1

    od 0,03 do 0,05 najni˙zszego uposa˙zenia - za ka˙zdy dzie´ n wykonywania zada´ n w strefie działa´ n wojennych w warunkach zwi ˛ azanych z bezpo´srednim udziałem w akcjach o charakterze bojowym, akcjach zapobiegania aktom terroryzmu lub ich skutkom albo pełnieniem słu˙zby patrolowej, ochronnej lub z udziałem w konwojach. No neighboring chunks. Good query:...

  14. [17]

    In Danish: Target chunk: § 10. Stk. 2. For ansatte omfattet af § 2, nr. 3, finder stk. 1, nr. 2, ikke anvendelse. Context chunks: § 2. Ydelser efter denne bekendtgørelse tilkommer ansatte, der er udsendt til tjeneste uden for landet: 3) for at sikre driften af en offentlig institution i dens tjenesteområde. § 10. Stk. 1. Særtillægget udgør: 4) 300 kr. pr....

  15. [18]

    Do ˙zołnierzy, o których mowa w § 2 pkt 2 lit

    In Polish: Target chunk: 2. Do ˙zołnierzy, o których mowa w § 2 pkt 2 lit. e, nie stosuje si˛ e ust. 1 pkt 2. Context chunks: § 2. Nale˙zno´sci pieni˛ e˙zne okre´slone przepisami niniejszego rozporz ˛ adzenia przyznaje si˛ e˙zołnierzom zawodowym:

  16. [19]

    skierowanym do pełnienia zawodowej słu˙zby wojskowej poza granicami pa´ nstwa: e) w celu zabezpieczenia funkcjonowania jednostki wojskowej u˙zytej zgodnie z przepisami ustawy z dnia 17 grudnia 1998 r. o zasadach u ˙zycia lub pobytu Sił Zbrojnych Rzeczypospolitej Polskiej poza granicami pa´ nstwa, w rejonie jej działania albo zapewnienia organizacji, funkc...

    1998

  17. [20]

    No neighboring chunks

    od 0,03 do 0,05 najni˙zszego uposa˙zenia - za ka˙zdy dzie´ n wykonywania zada´ n w strefie działa´ n wojennych w warunkach zwi ˛ azanych z bezpo´srednim udziałem w akcjach o charakterze bojowym, akcjach zapobiegania aktom terroryzmu lub ich skutkom albo pełnieniem słu˙zby patrolowej, ochronnej lub z udziałem w konwojach. No neighboring chunks. Good query:...

  18. [21]

    Minister

    podmiot wła´sciwy do przeprowadzania bada´ n, o których mowa w ust. 1, Implicit context (containing the noun “Minister”): Art. 124z. Minister wła ´sciwy do spraw wewn˛ etrznych okre´sli, w drodze roz- porz ˛ adzenia:. . . Figure 19: Original Polish version of Figure 6 Query (obligationtodefend_2223): Czy Rada Ministrów okre ´sla, które organy administracj...

  19. [22]

    Figure 20: Original Polish version of Figure 7 Generated context (financingeducation_688): Art

    Terenowe organy administracji rz ˛ adowej [...] mog ˛ a by´ c zobowi ˛ azane do odpłatnego wykonania zada´ n mobilizacyjnych na rzecz Sił Zbrojnych. Figure 20: Original Polish version of Figure 7 Generated context (financingeducation_688): Art. 66 okre ´sla, ˙ze w szkołach prowadzonych przez jednostki samorz ˛ adu terytorialnego zadania i kompetencje orga...

  20. [23]

    1, jest przekazywana w terminie od dnia 6 maja do dnia 15 pa´ zdziernika

    Dotacja celowa, o której mowa w ust. 1, jest przekazywana w terminie od dnia 6 maja do dnia 15 pa´ zdziernika. Utilized context chunk: Art. 62. 1. Na sfinansowanie kosztu zakupu podr˛ eczników, materiałów edukacyjnych lub materiałów ´ cwiczeniowych w zakre- sie, o którym mowa w art. 55 ust. 1, pub- liczne szkoły artystyczne realizuj ˛ ace kształce- nie og...

    2026