{"id":"6f618788-f147-4840-8543-71606b7bdb9f","arxiv_id":"2607.22610","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Recursively attributing each conversation turn to earlier turns recovers multi-hop provenance paths (up to ~90% source recall) where single-pass attribution collapses to under 20%.","lead":"This paper introduces a method and benchmark for tracing which earlier turns in a long conversation influenced a later AI answer, by recursively applying attribution turn-by-turn instead of once over the whole context. The authors report that recursive tracing recovers multi-hop sources far more often than flat attribution, but the benchmark is not yet released and uses AI-generated annotations with limited human review.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold graphs are produced under a protocol that hardcodes the same tool-result→invocation→source rule Tokengeist implements, so the headline gap may measure protocol agreement rather than causal provenance.","rationale":"The paper's internal experiments are well-controlled: flat vs recursive differ only in recursion depth, backends are held fixed, ablations saturate where expected, and the depth-saturation pattern tracks gold depth distributions. The recursive framework is plausible and likely does help recover multi-hop structure. However, the central quantitative claim rests on a gold standard that is model-annotated under a protocol encoding Tokengeist's own structural rule, with only a 2.6% single-author spot-check. The paper acknowledges this in Limitations, so it is not a hidden flaw, but it is still load-bearing: the 20%→90% source-recall figure is the headline, and if the gold graph is shaped by the protocol's strict chain rule, the gap partly measures how well the method implements the protocol. Because the issue could be settled by human annotation and released artifacts, CONDITIONAL is appropriate; I see no basis for REJECT (no internal inconsistency or misconduct) and no basis for full acceptance until the gold standard's independence is established.","tokens_in":25568,"tokens_out":7450,"duration_ms":67875,"concrete_test":"Human gold standard study: have 3 independent annotators, blinded to the annotation protocol and to each other, label ≥300 stratified targets (≥8% of the benchmark) using only the causal-necessity definition from the prompt ('without which the target would be wrong, incomplete, or different'), with no stated rule about tool-result chains and no 'should have attended to' language. Measure inter-annotator agreement (e.g., Krippendorff's alpha). Re-run Table 2 (flat vs TG+Attention/TG+AT2) against this human gold. If the source-recall gap narrows materially or Tokengeist's edge F1 drops by more than ~10 points on tool-heavy targets, the headline is largely protocol-driven. A cheap complementary test: re-run Tokengeist with the deterministic structural tool-result→invocation edge replaced by soft attribution; if source recall collapses, the hardcoded shortcut is a primary driver.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline claim—flat attribution under 20% source recall vs Tokengeist 90%—presupposes that MTCABench's gold graphs are valid ground truth for causal dependency. That presupposition is undercut by the annotation protocol itself. Appendix C.2 (Listing 2) instructs GPT-5.4 that 'The correct structure is a strict chain: tool_result --> invocation --> [arg sources]. Never connect a tool result directly to arg sources' and provides explicit CORRECT/WRONG examples. Tokengeist hard-codes exactly this structure in Section 4.1: tool-result turns receive a deterministic structural edge (score 1.0) to the nearest preceding invocation, and recursion resumes from that invocation. Thus, on 96%+ of targets with tool-call paths (Table 1), the gold edges align with what Tokengeist constructs by fiat. The flat baseline cannot use the structural shortcut, so a large part of the 20%→90% source-recall gap is a handshake between benchmark definition and method, not discovered multi-hop structure. Human validation is thin: 100/3,845 targets (2.6%), one author, 11% minor issues, 2% wrong before relabeling (Appendix C.5). The paper's own Limitations section explicitly concedes 'evaluation therefore measures agreement with this model-assisted annotation standard rather than with a human-only gold standard.' If the gold standard is protocol-derived, the numbers measure protocol adherence, not true causal provenance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces multi-turn context attribution (MTCA), a task of recovering, for a target span in an assistant turn, the full dependency DAG over prior turns. It proposes Tokengeist, a recursive framework that applies a single-turn attribution backend at each frontier turn, follows a deterministic structural edge from tool-result turns to their invocations, and terminates at exogenous user/system turns. The authors construct MTCABench from ConFETTI and TauBench with GPT-5.4-annotated gold provenance graphs, and report that recursive Tokengeist greatly outperforms flat attribution on edge F1 and source recall across four open-weight models and three backends. The central empirical claim is that flat attribution achieves under 20% source recall while Tokengeist reaches about 90%.","tokens_in":25970,"tokens_out":4950,"duration_ms":55670,"significance":"If the empirical claims hold, the paper makes a useful contribution: it identifies a real phenomenon (single-pass attribution missing multi-hop provenance), offers a simple method-agnostic recursive procedure with a clean comparison design (flat is literally d_max=1 in the same tracer), and includes ablations supporting the default hyperparameters. The task definition and benchmark, once released with human-validated gold graphs, could be valuable to the community. However, the current evaluation is not yet convincing as external evidence: the gold annotation protocol hardcodes the same tool-result-to-invocation chain that Tokengeist implements by fiat, the gold standard is produced by a single model with a 2.6% single-author spot-check, and the headline 'under 20%' claim is contradicted by the paper's own tables for some backends. The core idea is defensible, but the quantitative claims need substantial re-grounding.","major_comments":[{"comment":"The evaluation has a circularity problem that bears directly on the headline 20%-to-90% source-recall gap. The annotation protocol (Listing 2) instructs GPT-5.4 that 'The correct structure is a strict chain: tool_result --> invocation --> [arg sources]' and forbids connecting a tool result directly to argument sources. Tokengeist, in §4.1, hardcodes exactly this rule as a deterministic structural edge (score 1.0) from a tool-result turn to the nearest preceding invocation. Since Table 1 shows 96.1%/96.2% of targets in ConFETTI/TauBench have tool-call paths, a large part of the recovered structure is guaranteed by construction rather than discovered by attribution. The flat baselines do not use this shortcut, so the reported gap partly measures protocol adherence, not genuine provenance recovery. The Limitations section itself concedes that 'our evaluation therefore measures agreement wit","section":"§4.1, §5.2, Appendix C.2 (Listing 2), Table 1"},{"comment":"The abstract and conclusion claim that flat attribution methods 'achiev[e] under 20% source recall' and that Tokengeist raises it 'from under 20% to over 90%.' This is not supported by the paper's own tables. On TauBench, Flat AttnTrace achieves source recall of 28.3% for Llama-3.1-8B and Qwen2.5-7B (Table 2) and 22.7% for Phi-4-Mini (Table 3); on ConFETTI, Flat AttnTrace reaches 54.8% and 55.0%. Even restricting to attention and AT2 backends, the 'under 20%' statement holds only for TauBench, not ConFETTI (where flat source recall is 28-46%). The claim should be rephrased to reflect the actual range, e.g., 'as low as under 20% on the harder tool-call split,' or the claim should be dropped from the abstract. This is not a cosmetic issue, since the magnitude of the improvement is the paper's main takeaway.","section":"Abstract and §7 Conclusion vs Table 2 and Table 3"},{"comment":"The gold-standard validation is too thin to support the quantitative conclusions. Only 100 of 3,845 targets (2.6%) were reviewed, by a single author, using a subjective three-level scale; 11% had 'minor issues' and 2% were 'wrong' before relabeling. The paper states that independent multi-annotator review is planned prior to release, and the Limitations note that no inter-annotator agreement is currently available. Because the gold graphs are produced by GPT-5.4 alone, and because the protocol embeds the exact structural rule the method uses, the current numbers could shift substantially under independent human annotation. At minimum, the paper should either (a) perform and report multi-annotator agreement on a larger stratified sample, or (b) explicitly limit all quantitative claims to 'agreement with the model-assisted annotation standard' throughout, including the abstract and conclus","section":"§5.2 and Appendix C.5"}],"minor_comments":[{"comment":"The table header says 'ConFETTI(n=495)' while Section 5 and Table 1 report 688 ConFETTI targets. The discrepancy is explained only in Appendix A.4 (30 held-out conversations for alpha selection). Please state this in the main text near Table 2.","section":"Table 2"},{"comment":"The annotation pipeline uses GPT-5.4 to select targets and then GPT-5.4 to annotate provenance. It would be useful to report the agreement of target selection against an alternative selector or at least to note the potential selection bias.","section":"§5.2"},{"comment":"The definition of 'leaf source' as an exogenous node excludes tool-result turns, but tool-result turns are not exogenous per Section 3. Please clarify whether tool results ever count as leaf sources, since the structural edge makes them non-leaf by construction.","section":"§6.2"},{"comment":"The attention extraction re-projects queries and keys from cached hidden states. Please report whether the resulting attention weights match the true trained attention (e.g., due to numerical precision or rotary-embedding reapplication) on a small validation set.","section":"Appendix A.3"},{"comment":"The name 'MTCABench' is written as 'MTCABench' in the title and 'MTCABENCH' elsewhere; unify.","section":"General"},{"comment":"The prompt says a dependency must be 'causally necessary' but then also instructs a strict structural chain. These two criteria can conflict (a tool result may be causally necessary without the invocation being causally necessary). It would be helpful to note how the annotator resolves such cases.","section":"Appendix C.2, Listing 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is from a strong group and the proposed framework is simple and plausible, but the evaluation currently rests on a model-annotated gold standard that encodes the method's own structural assumption, and the headline numbers are not consistent across the tables. I believe the central idea can be salvaged by adding a neutral human-annotated sample, ablating the structural edge, and recalibrating the abstract claims. I did not find evidence of intentional misreporting; the discrepancies appear to stem from over-generalizing from the TauBench split and from relying on the model-assisted annotation protocol as ground truth."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this paper's headline claim—flat attribution under 20% source recall, Tokengeist at 90%—is real only relative to a gold standard that the authors' own method helps construct. The annotation protocol (Appendix C.2) tells GPT-5.4 that tool results must connect strictly to invocations, and Tokengeist hardcodes exactly that edge with score 1.0. So on 96% of targets, the recursive method is being rewarded for following the annotation rule, not for discovering structure. That doesn't make the paper worthless, but it means the numbers measure protocol adherence more than causal provenance.\n\nWhat is genuinely new: they formalize multi-turn context attribution as a task, propose recursive DAG construction over turns, and build a benchmark with gold provenance graphs. That is a useful framing, and the empirical work is substantial: four models, three attribution backends, consistent gains from recursion, plus ablations over depth, branching, pruning. The paper is well-structured and, to its credit, the Limitations section concedes the core issue: evaluation measures agreement with the model-assisted annotation standard, not a human gold standard.\n\nSoft spots, in proportion: (1) the benchmark's ground truth is GPT-5.4-annotated, with only 100 of 3,845 targets spot-checked by one author (2.6%; 2% wrong pre-relabeling). That is thin. (2) The structural edge handshake inflates the gap; without a human-validated gold on a held-out set, we can't know how much. (3) Code and data are promised, not released; that's fine for a preprint but matters for a benchmark paper. (4) The abstract overstates universality—\"under 20%\" should read \"under our annotated standard.\" The AttnTrace weakness is acknowledged and not a problem.\n\nThe central hypothesis—single-pass attribution misses multi-hop dependencies—is plausible and likely true. But this paper does not yet prove it. I'd send it to review, with a strong request for the artifacts and either a human-annotated subset or a perturbed-context validation that breaks the handshake. For a reading group, it's a good case study in how annotation protocols can become self-fulfilling.","headline":"Useful task framing and solid engineering, but the headline numbers are partly a handshake between the annotation protocol and the method; wait for the released artifacts and human validation.","tokens_in":26383,"tokens_out":2298,"would_cite":false,"duration_ms":25082,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that recursive turn-by-turn tracing — not a single backward pass over the full context — recovers multi-hop provenance in agentic conversations, lifting source recall from under 20% to 90%.","keywords":["multi-turn context attribution","provenance graph","recursive attribution","agentic conversations","tool calling","provenance collapse","attribution benchmark","large language model interpretability"],"falsifier":"A human study with independent annotators, blind to the annotation protocol, constructing provenance graphs for a random sample of MTCABench targets and comparing them to the gold graphs would settle whether the reported source-recall and edge-F1 numbers reflect true causal provenance. If humans frequently do not reproduce the protocol's strict tool-result→invocation→argument-source chains and the paper's claim of over 90% source recall depends on matching that convention, the headline numbers would not transfer to human judgments.","tokens_in":25484,"feed_emoji":"🔁","tokens_out":7364,"duration_ms":67963,"temperature":0.7,"pith_summary":"The paper introduces multi-turn context attribution (MTCA): given a specific span in a model's response, recover not only which earlier turns directly contributed, but the whole chain of intermediate turns through which that information propagated. It argues that existing attribution methods, which score the entire context in a single pass, systematically miss this layered structure — a failure the paper names 'provenance collapse' — and proposes Tokengeist, which applies any base attribution method recursively, walking backward from the target turn through intermediate assistant and tool-result turns until it reaches original user inputs. On a new benchmark of 3,845 target spans across 665 agentic conversations, flat methods recover sources of under 20% of multi-hop dependencies, while Tokengeist recovers over 90%. If the benchmark's annotation standard is accepted, this would mean that verifying or debugging agentic systems requires recursive attribution rather than one-shot context scoring.","feed_headline":"Recursive tracing lifts source recall from under 20% to 90%","feed_subtitle":"A single backward pass misses how information moves across turns; recursion finds the true source.","key_machinery":"The central object is the provenance DAG: a weighted directed acyclic graph whose nodes are conversation turns and whose root is the target span. The machinery is a backward breadth-first traversal that, at each node, concatenates all preceding turns, runs a single-turn attribution scorer to assign each prior turn a relevance score, keeps the top-k (default 3), and dispatches by role: user/system turns terminate the branch; tool-result turns follow a deterministic structural edge to their invoking assistant turn; assistant turns are recursively attributed. A relative-threshold pruning step (alpha=0.85) trims edges based on the local score maximum, and a visited set keeps the graph acyclic wh","core_discovery":"The central claim is that attribution in multi-turn conversations is inherently multi-hop: the answer to a later turn often depends on an intermediate assistant turn that itself paraphrases, filters, or transforms an earlier tool result or user instruction. Tokengeist formalizes this as building a weighted directed acyclic graph over conversation turns by recursively applying a single-turn attribution scorer, with special handling for tool-result turns (which connect deterministically to their invoking call) and exogenous user/system turns (which act as leaves). Across four open-weight instruction-tuned models and three base scorers, the recursive framework raises source recall from under 20","pith_inferences":["If the central claim is right, then existing single-pass attribution results on long-context questions may understate what context actually drove the answer; a testable extension would be to re-run long-context benchmarks with recursive tracing and compare whether apparent 'lost in the middle' effects shrink when intermediate transformations are traced.","The annotation protocol's strict tool-result→invocation→argument-source convention is baked into both the gold graphs and the tracer, so the framework's effectiveness partly measures its match to that convention; a natural stress test would be conversations with asynchronous tool calls or shared mutable state, where the convention may break.","The recursion depth and branching factor needed in practice track the provenance-depth distribution of the conversation; one could predict that persistent-memory or multi-agent transcripts will require amortized graph updates rather than per-target backward BFS.","The relative-threshold pruning has a flat plateau over alpha in [0.65, 0.85], which suggests the pruning rule is not the bottleneck; a testable improvement would be to condition the threshold on dependency type, since multi-source targets are the hardest category."],"forward_implications":["Flat, single-pass attribution is systematically insufficient for multi-turn contexts: on MTCABench it recovers sources of under 20% of chained dependencies, while recursive tracing exceeds 90% source recall.","Recursion, not the choice of base scorer, is the dominant factor: the gains hold across all four models and three attribution backends evaluated, including the weakest one.","Reaching original source turns (rather than stopping at a nearby assistant summary) is the bar that makes provenance useful for verification, audit, and hallucination grounding, and is the bar the recursive method clears.","For hallucinated spans, recursive tracing markedly improves locating the turn that primed or propagated the erroneous content, helping identify where a wrong value entered the agentic trace.","The performance gap is concentrated in the cases MTCA targets: flat methods perform comparably on single-hop 'direct' dependencies but drop below 35% edge F1 on chained dependencies, while recursive tracing reaches 70%."],"fun_headline_variants":["Recursive tracing lifts source recall from 20% to 90%","Tokengeist: recursive tracing recovers multi-hop provenance","Attribution across turns: recursion hits 90% recall","Single-pass misses, recursion finds: 20% to 90%","Multi-turn attribution: recursive tracing reaches 90% recall"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the gold provenance graphs — produced by the annotation model under a strict protocol that encodes the exact structural rule Tokengeist assumes, and human-checked on only 2.6% of targets by a single reviewer — are a valid ground truth for what actually caused each response; if that standard does not match human judgments of causal provenance, the reported gains measure agreement with the annotation protocol rather than true attribution quality","fun_headline_variants_meta":{"raw":{"variants":["Recursive tracing lifts source recall from 20% to 90%","Tokengeist: recursive tracing recovers multi-hop provenance","Attribution across turns: recursion hits 90% recall","Single-pass misses, recursion finds: 20% to 90%","Multi-turn attribution: recursive tracing reaches 90% recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000586,"raw_usage":{"total_tokens":2589,"prompt_tokens":738,"completion_tokens":1851,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":1764}},"tokens_in":482,"tokens_out":1851,"duration_ms":12036,"temperature":1.0,"reasoning_tokens":1764,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T11:13:15.135260+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A human study with independent annotators, blind to the annotation protocol, constructing provenance graphs for a random sample of MTCABench targets and comparing them to the gold graphs would settle whether the reported source-recall and edge-F1 numbers reflect true causal provenance. If humans frequently do not reproduce the protocol's strict tool-result→invocation→argument-source chains and the paper's claim of over 90% source recall depends on matching that convention, the headline numbers would not transfer to human judgments.","supporting_citations":[],"review_version":1}