{"id":"0f891429-d38d-4045-814b-b68677216057","arxiv_id":"2605.19092","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Counterfactual likelihood tests detect indirect influence through public channels in private reasoning models, validated on a 7B role-channel model showing asymmetric A-to-B influence and complete pathway identification via graph-separation controls.","lead":"This paper introduces a counterfactual likelihood test that swaps length-matched private reasoning blocks while holding public tokens fixed to measure downstream influence shifts in AI models. Smart generalists and AI researchers should read it to better evaluate hidden information flow across private and public channels in reasoning systems.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Length matching of donor blocks may not fully control RoPE positional confounds when isolating NLL shifts for indirect influence.","rationale":"The reader's weakest assumption matches the load-bearing point for the counterfactual test itself. The graph-separation result is strong internal evidence, but the test's ability to attribute shifts specifically to public-channel influence hinges on the RoPE control being complete. Verifying the proposed length-mismatch check would directly test whether that assumption holds without requiring external data.","tokens_in":1762,"tokens_out":330,"duration_ms":28121,"concrete_test":"Recompute the downstream NLL shifts for the 13,734 directional contrasts using donor blocks whose lengths are deliberately mismatched by ±5 tokens while keeping content otherwise neutral; if the bit-identical result in the graph-separation control or the A-to-B vs B-to-A asymmetry changes by more than the reported noise floor, the length-matching control is insufficient.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that the graph-separation control identifies the public-channel pathway as the complete carrier depends on the counterfactual test cleanly measuring influence via public hidden states. This requires that replacing the upstream private block with a length-matched donor produces an NLL shift attributable only to content influence, not to RoPE rotations that depend on absolute token positions within the fixed public sequence. The abstract asserts length matching controls the confound, but without explicit verification that donor insertion preserves relative positional relationships or that attention patterns remain unaffected beyond length, residual positional leakage could inflate or mask the measured A-to-B asymmetry across the 13,734 contrasts.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces a counterfactual likelihood test for measuring indirect influence between private reasoning channels in LLMs. It replaces an upstream private block with a length-matched donor block, holds the public token sequence and downstream target fixed, and measures the resulting shift in negative log-likelihood on the target. Validation on a 7B role-channel model shows textual probes (n-gram overlap, canary reproduction) are unreliable, while the likelihood test separates unmasked and masked conditions and detects asymmetric influence (persistent A-to-B via public hidden states, near-zero B-to-A). A graph-separation control blocking private-to-public edges yields bit-identical natural and counterfactual scores across all 13,734 contrasts, identifying the public-channel pathway as the complete carrier under the role-visibility mask. The asymmetry replicates across three checkpoints, five seeds, and the full set of directional contrasts.","tokens_in":1900,"tokens_out":586,"duration_ms":43640,"significance":"If the central claims hold, the work supplies a practical, falsifiable method for auditing information flow across private-public boundaries in reasoning models and demonstrates that private-channel evaluation must separately report direct and indirect influence. Notable strengths include the bit-identical graph-separation control across 13k contrasts and the multi-checkpoint, multi-seed replication, both of which support reproducibility.","major_comments":[{"comment":"§4 (Counterfactual Likelihood Test and Length Matching): the claim that length matching of donor blocks controls RoPE positional confounds lacks an explicit verification (e.g., ablation confirming preserved relative positional relationships or unchanged attention patterns beyond token length). This is load-bearing for attributing observed NLL shifts and the reported A-to-B asymmetry solely to content influence rather than residual positional leakage.","section":"§4 (Counterfactual Likelihood Test and Length Matching)"},{"comment":"§5 (Graph-Separation Control): while bit-identical scores across 13,734 evaluations are reported, the manuscript does not test whether the result remains stable under modest changes to the role-visibility mask; dependence on the specific mask implementation weakens the generality of the 'complete carrier' conclusion.","section":"§5 (Graph-Separation Control)"}],"minor_comments":[{"comment":"Abstract: the phrase 'hardened masked validation' is used without a forward reference or brief definition; add a parenthetical pointer to the relevant subsection.","section":"Abstract"},{"comment":"Notation: ensure 'natural' and 'counterfactual' scores are defined once at first use and used consistently thereafter to avoid reader ambiguity.","section":"Notation"}],"recommendation":"major_revision","confidential_remarks":"The replication scale and control design are strong; the manuscript would benefit from an explicit data-availability statement to allow independent verification of the 13k contrasts."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their careful reading and constructive comments on our manuscript. We address each of the major comments below and outline the revisions we plan to make.","responses":[{"response":"We appreciate the referee highlighting the need for explicit verification. Length matching ensures donor blocks occupy identical sequence positions to the originals, preserving the positional indices applied to the fixed public token sequence under RoPE. To provide the requested confirmation, we will add an ablation in the revised manuscript that compares attention head patterns and NLL shifts between length-matched and non-matched donor blocks. This will isolate content-driven effects from any residual positional contributions and directly support attribution of the A-to-B asymmetry.","revision_made":"yes","referee_comment":"[§4 (Counterfactual Likelihood Test and Length Matching)] §4 (Counterfactual Likelihood Test and Length Matching): the claim that length matching of donor blocks controls RoPE positional confounds lacks an explicit verification (e.g., ablation confirming preserved relative positional relationships or unchanged attention patterns beyond token length). This is load-bearing for attributing observed NLL shifts and the reported A-to-B asymmetry solely to content influence rather than residual positional leakage."},{"response":"We agree that the 'complete carrier' conclusion is scoped to the specific role-visibility mask implemented in the model. The bit-identical natural and counterfactual scores across all 13,734 contrasts rigorously demonstrate that, under this mask, the public-channel pathway accounts for the entire measured signal with no residual private-to-public leakage. We did not vary the mask, as the experiment was designed to validate the counterfactual test within the model's fixed architecture. In revision we will expand the discussion of the mask definition, explicitly state the scope of the claim, and note the desirability of mask-variation tests as future work.","revision_made":"partial","referee_comment":"[§5 (Graph-Separation Control)] §5 (Graph-Separation Control): while bit-identical scores across 13,734 evaluations are reported, the manuscript does not test whether the result remains stable under modest changes to the role-visibility mask; dependence on the specific mask implementation weakens the generality of the 'complete carrier' conclusion."}],"tokens_in":1495,"tokens_out":467,"duration_ms":71925,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"This paper introduces a counterfactual likelihood test to measure indirect influence between private reasoning channels in models that separate computation. The core move is swapping an upstream private block for a length-matched donor while keeping the public tokens and downstream target the same, then tracking the shift in negative log likelihood. On a 7B role-channel model it separates unmasked and masked conditions where raw overlap and canary checks do not. The graph-separation control that blocks private-to-public edges produces bit-identical natural and counterfactual scores across all 13,734 cases, which is the strongest piece of evidence they present. It also shows clear directional asymmetry, with A-to-B influence persisting through public hidden states while reverse influence stays near zero, and they replicate across checkpoints and seeds. That combination of replacement, NLL shift, and explicit separation validation is what is new here and where the work is most useful. The length-matching step is offered as a control for RoPE positional confounds, but that assumption is not obviously airtight. Swapping a donor block into a fixed public sequence could still shift absolute or relative positions enough to affect attention patterns and downstream likelihoods beyond pure content influence. Without more explicit checks on attention or position encodings in the results, some of the measured asymmetry might carry residual positional leakage. The work is aimed at people auditing information flow or privacy boundaries in separated reasoning systems. Anyone testing channel isolation for safety or evaluation purposes would get practical value from trying the probe. I would send it for peer review so the methods section and any additional positional diagnostics get proper scrutiny.","headline":"The counterfactual replacement test plus graph control gives a cleaner read on indirect influence than text probes, though the RoPE length-matching step still looks like the weakest link.","tokens_in":2374,"tokens_out":387,"would_cite":true,"duration_ms":54887,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[{"relation":"unclear","rs_module":"IndisputableMonolith/Cost/FunctionalEquation.lean","rs_theorem":"washburn_uniqueness_aczel","paper_passage":"replaces an upstream private block with a donor block, holds the public token sequence and downstream target fixed, and measures the downstream target’s negative-log-likelihood shift"},{"relation":"unclear","rs_module":"IndisputableMonolith/Foundation/AlexanderDuality.lean","rs_theorem":"alexander_duality_circle_linking","paper_passage":"graph-separation control that blocks private-to-public carrier edges produces bit-identical natural and counterfactual scores"}],"headline":"Counterfactual NLL intervention on role-channel graphs; no RS cost or distinction structure","alignment":"orthogonal","rationale":"The paper's core machinery is a behavioral intervention (replace private block X with length-matched donor X', hold public tokens P and target T fixed, score ΔNLL) plus a graph-cut control that blocks private-to-public carrier edges. This is a standard causal-tracing / mediation technique in LLM evaluation. RS framework derives J-cost, φ-ladder, 8-tick periodicity, and spacetime constants from a single distinction (reality_from_one_distinction, AbsoluteFloorClosure, Cost.FunctionalEquation). No shared primitives, no cost functional, no ratio symmetry, no ladder spacing appear in the paper. Domain is cs.LG model auditing; RS has no theorems about attention masks or NLL shifts.","tokens_in":46653,"confidence":"high","tokens_out":349,"duration_ms":13375,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Graph separation controls produce bit-identical scores, showing public channels fully carry indirect private influence under role masks.","keywords":["counterfactual likelihood","indirect influence","private reasoning channels","graph separation","role-visibility mask","negative log likelihood","public channel pathway","influence measurement"],"falsifier":"Observing different natural and counterfactual scores after blocking private-to-public carrier edges in the graph-separation control would show that the public-channel pathway does not fully account for the measured signal.","tokens_in":2649,"feed_emoji":"🧪","tokens_out":693,"duration_ms":66058,"temperature":0.7,"pith_summary":"The paper develops a counterfactual likelihood test that replaces an upstream private block with a length-matched donor block, keeps the public token sequence and downstream target fixed, and measures the resulting negative-log-likelihood shift. This isolates indirect influence through public channels while controlling for positional confounds. On a 7B role-channel model, textual probes prove unreliable for detecting leakage, but the likelihood method cleanly separates masked and unmasked conditions and reveals asymmetric influence that persists from A to B through public hidden states but not in reverse. Validation across three checkpoints, five seeds, and 13,734 directional contrasts replicates the pattern. A graph-separation control that blocks private-to-public carrier edges yields identical natural and counterfactual scores in every case, establishing the public pathway as the complete carrier of the measured signal.","feed_headline":"Graph separation shows public channels carry all indirect private influence","feed_subtitle":"Bit-identical scores in 13734 tests confirm the public pathway is the complete carrier under role masks","key_machinery":"Counterfactual likelihood test that replaces an upstream private reasoning block with a length-matched donor block, holds the public token sequence and downstream target fixed, and measures the negative-log-likelihood shift on the target.","core_discovery":"Under the implemented role-visibility mask, the tested public-channel pathway is the complete carrier of the measured counterfactual signal. This is shown by a graph-separation control that blocks private-to-public carrier edges and produces bit-identical natural and counterfactual negative-log-likelihood scores across all 13,734 valid directional contrasts. The test replaces upstream private blocks with length-matched donor blocks to isolate influence on downstream targets while holding public sequences fixed.","pith_inferences":["The test could be applied to audit information flow in other role-based or multi-agent systems that separate private and public computation.","Directional asymmetries may affect coordination in masked multi-role setups and warrant checks in larger models.","Focusing evaluation on public outputs alone may suffice once the carrier pathway is verified."],"forward_implications":["Private-channel evaluation should report direct and indirect influence separately.","Counterfactual likelihood probes provide a practical default for measuring influence boundaries where textual methods fail.","Influence is asymmetric: A-to-B persists through public-speech hidden states while reverse B-to-A influence is near zero.","The public-channel pathway accounts for the complete counterfactual signal under the role-visibility mask.","The asymmetry and pathway identification replicate across multiple checkpoints and seeds."],"fun_headline_variants":["Graph separation confirms public channels as sole carriers of indirect influence","13734 tests show bit-identical scores proving public pathway completeness","Counterfactual likelihood isolates indirect influence via public channels","Length matching eliminates RoPE confounds in counterfactual influence tests"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Length matching of donor blocks sufficiently controls for RoPE positional encoding confounds when measuring the negative-log-likelihood shift on the downstream target.","fun_headline_variants_meta":{"raw":{"variants":["Graph separation confirms public channels as sole carriers of indirect influence","13734 tests show bit-identical scores proving public pathway completeness","Counterfactual likelihood isolates indirect influence via public channels","Length matching eliminates RoPE confounds in counterfactual influence tests"]},"model":"grok-4.3","cost_usd":0.011615,"raw_usage":{"total_tokens":5026,"prompt_tokens":706,"num_sources_used":0,"completion_tokens":64,"cost_in_usd_ticks":116153000,"prompt_tokens_details":{"text_tokens":706,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":4256,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":706,"tokens_out":64,"duration_ms":68799,"temperature":1.0,"reasoning_tokens":4256,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-20T12:59:14.225147+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Observing different natural and counterfactual scores after blocking private-to-public carrier edges in the graph-separation control would show that the public-channel pathway does not fully account for the measured signal.","supporting_citations":[],"review_version":1}