WildTrace: Benchmarking Natural Evidence Trails in Long-Context Reasoning

Dayiheng Liu; Fei Huang; Haobo Li; Huamin Qu; Jianhong Tu; Kashun Shum; Peng Liu; Rui Sheng; Xiaodong Deng; Zixin Chen

Frontier long-context models still fail to integrate evidence that long documents disperse by their own logic, not by planted needles.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 03:50 UTC pith:54UZWQE6

load-bearing objection Solid source-first long-context benchmark with real validity work; the 75% ceiling and geometry split look like genuine diagnostics, not leaderboard noise. the 3 major comments →

arxiv 2607.09328 v2 pith:54UZWQE6 submitted 2026-07-10 cs.CL cs.AI

WildTrace: Benchmarking Natural Evidence Trails in Long-Context Reasoning

Zixin Chen , Peng Liu , Haobo Li , Rui Sheng , Jianhong Tu , Xiaodong Deng , Fei Huang , Kashun Shum

show 2 more authors

Dayiheng Liu Huamin Qu

This is my paper

classification cs.CL cs.AI

keywords long-context reasoningnatural evidence trailsmulti-hop QAsource-internal evidenceevidence geometriesfull-document evaluationcounterfactual reasoningbenchmark construction

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Real analytical work over long documents means recovering clues the source itself scattered—operating conditions, design flaws, delayed motives—and preserving the causal, temporal, or comparative relation that makes them jointly warrant an answer. Existing long-context tests mostly plant needles, reverse-engineer hops, or pre-select passages, so high scores can reflect distributional artifacts rather than genuine source reasoning. WILDTRACE builds 481 tasks from 214 real technical reports and literary narratives by mining trails from the document first, then writing questions only after multi-stage checks for necessity, grounding, and contamination resistance. Under full-document, evidence-withheld evaluation, the strongest of 18 systems reaches 75.3 percent mean rubric credit, with clear weaknesses on counterfactual branching and causal attribution even when context windows are large enough. The paper argues this gap between access and natural evidence integration is now a defining challenge for long-context research.

Core claim

When questions must be answered from naturally occurring evidence trails inside raw long sources—trails defined by the document’s own causal, temporal, and narrative logic—frontier models leave substantial headroom: the best of 18 systems scores 75.3 percent, and performance drops most on geometries that demand branch management or explanatory attribution rather than simple constraint accumulation.

What carries the argument

Natural evidence trails: source-induced sets of dispersed spans and typed relations that jointly warrant an answer. They are organized into seven evidence geometries (forward chains, intersections, comparisons, temporal reconstructions, causal fan-in, abductive explanations, counterfactual branches) and produced by a source-first pipeline that graphs the document before writing any question, then gates every item through necessity, grounding, and contamination checks.

Load-bearing premise

That the multi-stage source-first mining and validation gates truly remove residual construction cues and shortcuts so that high scores require recovering the document’s own relational structure.

What would settle it

Release a held-out set of the same sources with human-written questions that independent auditors rate as needing the identical multi-span relations, then re-score the same models; if top systems suddenly approach full credit without architecture or training changes, the claimed necessity of natural trails is undermined.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Long-context progress must be measured by relation-preserving integration of source-dispersed evidence, not only by token capacity or needle retrieval.
Geometry-specific scores become capability signatures: systems can look similar overall yet differ sharply on counterfactual branches versus intersections.
Training and memory design should supervise intermediate structures (entity bindings, order, causal direction, factual-versus-alternative states) rather than final answers alone.
High-stakes analytical deployments over incident reports or long narratives remain unreliable until models close the remaining 25-point gap under evidence-withheld conditions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Architecture ablations that keep or drop compressed memory of order and branch identity could be scored directly against the geometry profile to link systems design to the observed failure modes.
The same source-first trail mining could generate training supervision for relation-preserving compression, turning the benchmark’s construction artifacts into intermediate targets rather than only evaluation labels.
Legal, medical, and scientific document families will likely expose different geometry difficulty orderings than the current technical-plus-literary mix.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid source-first long-context benchmark with real validity work; the 75% ceiling and geometry split look like genuine diagnostics, not leaderboard noise.

read the letter

This is the cleanest attempt I’ve seen to measure full-document reasoning over evidence the source itself disperses, rather than needles or reverse-engineered hops. What’s new is the package: mine trails from the document graph first, write questions only after a supported trail exists, then force a 13.7% survival rate through necessity ablations, no-document probes (38% rejected), leave-one-out single-clue rejection, and dual review. Seven geometries (chain, intersection, comparison, temporal, causal fan-in, abductive, counterfactual) give a usable diagnostic axis, and the 18-system panel under evidence-withheld prompts is the right protocol.

They did the hard parts well. Blinded geometry relabeling (κ ~0.88–0.93), expert–judge correlation (~0.85 on a stratified 60), evidence-visible judge spot-checks, route accounting, and a public locked release with score matrices make the 75.3% ceiling and the counterfactual/causal weakness credible. The running comparative example shows they know the difference between retrieving shared topic and recovering mechanism contrast. Citation map is honest about LongBench, RULER, MuSiQue, NarrativeQA, etc.; positioning is a boundary map, not a ranking.

Soft spots are real but secondary. Domain is still technical reports plus literary (English-heavy, small CJK slice). LLM-assisted extraction can leave residual cues even after gates—paper doesn’t claim purity, and residual bias doesn’t erase the unsaturated headroom. Free parameters (D4/D7 thresholds, tier floors) are disclosed with sensitivity; D8 is nearly non-binding. Closed endpoints drift; they log route caps rather than hide them. Multi-geometry tasks and harnessed systems are left for later, which is fine.

For anyone building or training long-context systems for analytical work, this is worth reading and citing. I’d send it to peer review without hesitation; the construction and evaluation are careful enough that referees can argue about scope, not whether the core measurement is fake.

Referee Report

3 major / 6 minor

Summary. WILDTRACE introduces a 481-task benchmark over 214 naturally occurring long documents (technical incident reports and literary narratives) for full-document, evidence-withheld multi-hop reasoning. Unlike needle, planted-fact, or reverse-engineered multi-hop suites, candidate trails are mined source-first from typed document graphs and promoted only after multi-stage validity gates (D1–D9), leave-one-out necessity, no-document contamination probes (38% rejection), and human/external review, yielding a 13.7% survival rate from 3,506 candidates. Seven primary evidence geometries (forward chain, intersection, comparison, temporal, causal, abductive, counterfactual) structure the relational demands. Evaluating 18 frontier systems with three non-contestant LLM judges, the strongest reaches 75.3% mean rubric credit, with systematic degradation by document scale and a clear geometry profile (intersection strongest; counterfactual weakest). The paper argues that the remaining gap is relation-preserving integration of source-internal evidence, not mere context access.

Significance. If the construction and evaluation hold, WILDTRACE cleanly isolates a capability that existing long-context and multi-hop benchmarks largely confound: recovering and composing evidence trails that the source itself induces, under evidence-withheld full-document prompts. The source-first pipeline, high rejection rate, leave-one-out necessity checks, geometry relabeling reliability (κ≈0.88–0.93), expert–judge correlation (r≈0.85 on a stratified 60-response sample), public release (CinderD/wildtrace), and frozen multi-judge score matrices are concrete strengths that make the unsaturated frontier result (75.3% ceiling) and geometry-specific weaknesses diagnostically useful rather than merely another hard leaderboard. The work is well positioned to guide architecture, memory, and harness research on relation-preserving long-context reasoning.

major comments (3)

Table 1 and Appendix A.7 / Table 16: The main score panel mixes models with very different route caps, and OOC items appear to receive no route credit (Figure 1 MiniMax example). The caption and main text should state explicitly whether Score(m) is the mean over all 481 tasks (OOC=0) or only over Am, and should report both the eligible-n and an OOC-aware denominator so that low scores for limited-window systems are not misread as pure reasoning failures. This is load-bearing for interpreting the 18-system ranking and the claim of substantial headroom under full-document conditions.
Section 3.3 and Figure 5B / Table 17: The geometry profile is a central diagnostic claim (21-point spread; counterfactual hardest), but many geometry×tier cells have only 8–12 items and Table 17 reports top-six means without uncertainty. Please add task-bootstrap or source-cluster CIs (as in Table 20) for the geometry means in Figure 5B, and temper any interaction language that Table 17 cannot support at current n. The ordering can remain; the precision of the claimed separation needs to match the sample floor.
Abstract and §2.3: The abstract states that the seven geometries draw on Pearl’s causal hierarchy, but the operational taxonomy (Figure 3) is primarily a multi-hop relational typology (chains, intersections, comparisons, temporal order, fan-in, abduction, branches). Causal and counterfactual items map partially onto Pearl’s ladder; the others do not. Either supply an explicit mapping of each geometry to associational / interventional / counterfactual structure, or soften the Pearl framing so the contribution rests on the source-internal geometries as defined and validated (κ study in Appendix A.2).

minor comments (6)

Table 2 is a useful boundary map; a short paragraph in A.1 stating which prior suites come closest on each column (e.g., NovelQA, MuSiQue, LongBench v2) would help readers place the conjunction claim without over-reading the P/part./– symbols.
Figure 1 case card is excellent for intuition; ensure the main PDF keeps C1–C5 offsets and PASS/FAIL rows legible at print scale, or move the full card to the appendix and keep a compact schematic in the main text.
Appendix A.5 Table 10: Qwen-family judge bias is small and well reported; still note in the main evaluation section that one judge shares a family with several contestants, with the remove-Qwen sensitivity (max shift 1.8 points) as the reassurance.
Limitations §5 correctly flags domain and multi-geometry coverage; a one-sentence note on whether technical-report vs. literary difficulty is confounded with length/tier balance would help readers of Table 18.
Typos / polish: “Eval/Q10” appears in A.5 without definition in the main text; expand once. Ensure consistent spelling of model names (e.g., GPT-5.x series) across Table 1 and appendices.
Public release path CinderD/wildtrace is cited; confirm the frozen score matrix and D1–D9 audit ledger are version-pinned in the camera-ready so the 75.3% figure remains reproducible.

Circularity Check

0 steps flagged

No circular derivation: WILDTRACE is an empirical benchmark whose performance claims are measured, not forced by fitted inputs or self-definition.

full rationale

The paper’s load-bearing claims are empirical measurements on a locked 481-task release (strongest system 75.3% mean rubric credit; geometry-specific weaknesses under evidence-withheld full-document evaluation), not first-principles predictions derived from parameters fitted to the same targets. Source-first trail induction, D1–D9 gates, leave-one-out necessity, no-document probes, and dual human/external review define item validity; they do not redefine model scores as equal to construction inputs. Geometry labels organize diagnostic slices drawn from Pearl and multi-hop typologies; reporting lower credit on counterfactual/causal items is an observed outcome, not a ratio or quantity fixed by the geometry definition. Non-contestant multi-judge scoring is validated against expert consensus (r≈0.85) and judge-family sensitivity checks; Qwen-family presence among both systems and one judge is disclosed and does not make the leaderboard equal to a fit by construction. No uniqueness theorem, ansatz, or prior result by the same authors is invoked as an external mathematical fact that forces the central claim. The evaluation is self-contained against external frontier endpoints on a fixed release, so circularity score is 0.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 3 invented entities

As an empirical benchmark paper the load-bearing commitments are operational thresholds chosen for release promotion, domain assumptions imported from causal and multi-hop literature, and the newly defined objects (natural evidence trail, seven geometries, D1–D9 gates). No physical constants or free-form theoretical parameters appear; the free parameters are the numerical cut-offs that decide which candidates enter the locked set.

free parameters (5)

D4 no-document rejection threshold = 0.35
Candidates fail if any strong closed-book probe scores ≥0.35 rubric credit; sensitivity table shows the operating point is stable but still hand-chosen.
D7 single-clue sufficiency threshold = 0.60
Any ±2000-character window scoring ≥0.60 on D5 core facts is rejected; final leave-one-out also rejects unique full-answer clues.
D6 dispersion gates = spread≥0.10, gap≥0.05
Normalized clue spread ≥0.10 and minimum relative gap ≥0.05 required before expensive validation.
Geometry-by-tier sample floors = 8–12 per cell
L0/L6 ≥12 items per geometry, L1–L5 ≥8; used to declare the release powered.
Half-octave context tier boundaries = half-octave strata
L0–L7 cut-points (≤128K … >1M estimated tokens) chosen for analysis balance rather than derived from theory.

axioms (4)

domain assumption Pearl’s causal hierarchy and prior multi-hop typologies supply an adequate taxonomy for source-internal analytical reading.
Invoked to define the seven geometries (Section 2.3, Figure 3); not re-derived.
domain assumption Leave-one-out and single-clue ablations operationalize multi-hop necessity for natural documents.
Core of D2/D7 gates (Appendix A.5); assumes the ablation faithfully detects shortcut solvability.
ad hoc to paper Criterion-level rubrics scored by non-contestant LLM judges (with source verification front-loaded) are a valid proxy for expert analytical completeness.
Supported by a 60-response expert check (r≈0.85) but remains a methodological choice of the benchmark.
domain assumption Lesser-known literary narratives plus public technical incident reports are sufficiently free of memorization and representative for the claimed evaluation boundary.
Source selection premise (Section 2.3, Limitations); Chinese literature is only an initial slice.

invented entities (3)

natural evidence trail independent evidence
purpose: Defines the core evaluation object: source-induced dispersed clues plus typed relations that jointly warrant an answer.
Central construct of the paper; independent evidence is the locked release itself and the necessity audits.
seven source-internal evidence geometries independent evidence
purpose: Diagnostic taxonomy (forward, intersection, comparative, temporal, causal, abductive, counterfactual) that labels the primary answer-critical dependency.
New primary labels with blinded relabeling reliability; secondary relations are recorded but not scored.
source-first construction pipeline with D1–D9 validity gates independent evidence
purpose: Operational procedure that mines trails before writing questions and enforces grounding, necessity, contamination resistance and answerability.
The promotion funnel and 13.7% survival rate are the paper’s methodological contribution.

pith-pipeline@v1.1.0-grok45 · 29550 in / 3138 out tokens · 57218 ms · 2026-07-13T03:50:57.573243+00:00 · methodology

0 comments

read the original abstract

Answering complex questions over long documents frequently requires integrating evidence that the source itself disperses naturally across distant passages. In an incident report, the operating condition, design flaw, and missed safety check that jointly explain a disaster may appear dozens of sections apart; in a novel, a character's true motive may surface only through scenes far removed from the moment it becomes relevant. This source-internal evidence integration is central to real-world long-document analysis, yet existing benchmarks largely sidestep it. Needle probes, planted facts, and reverse-engineered multi-hop chains embed evidence that may differ from the host text in distribution, placement, or register, making it unclear whether strong performance reflects genuine source reasoning or distributional artifacts. We introduce WILDTRACE, a benchmark of 481 tasks over 214 naturally occurring long-form sources such as technical incident reports and lesser-known literary narratives, where all evidence trails arise from the document's own causal, temporal, and narrative logic. Drawing on Pearl's causal hierarchy and prior multi-hop reasoning typologies, we define seven source-internal evidence geometries that characterize the distinct relational demands of analytical reading in long documents. A source-first construction pipeline mines candidate trails from document structure before writing questions; each item then undergoes multi-stage validation covering clue necessity, answer groundedness, rubric fidelity, contamination resistance and answerability. As models are increasingly entrusted with real-world high-stakes analytical tasks, this gap between accessing information and reasoning over naturally dispersed evidence emerges as a defining challenge for the next stage of long-context research.

Figures

Figures reproduced from arXiv: 2607.09328 by Dayiheng Liu, Fei Huang, Haobo Li, Huamin Qu, Jianhong Tu, Kashun Shum, Peng Liu, Rui Sheng, Xiaodong Deng, Zixin Chen.