{"id":"f352dc1c-e72a-4d3b-99c2-3f86384d2502","arxiv_id":"2608.13385","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The Selection-Realization Hypothesis holds that static task vectors suffice when demonstration-induced activation changes are largely shared across queries; query-conditioned, multi-site, or routing interventions are warranted only when the changes are query-predictable, dispersed, or not…","lead":"This paper proposes an empirical theory for when demonstrations in multimodal in-context learning can be compressed into a static internal 'task vector' rather than a more expensive query-dependent intervention. It finds that the fraction of demonstration-induced activation change shared across queries predicts how much a static vector recovers, and that query-specific, distributed, or non-additive structures call for more expressive interventions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (3) assumes a query-independent basis; the paper never tests it, and the gating-attribute split in the controlled task could falsify it.","rationale":"The reader's weakest_assumption identifies Eq. (3) as the load-bearing premise, and I agree. The central claim is that intervention complexity can be predicted from explicit-M-ICL diagnostics, but all four diagnostics inherit the query-independent-basis assumption. H1's reconstruction and recovery results are necessary evidence but not sufficient: they show that a global low-rank basis exists, not that the basis is the correct causal decomposition. The controlled task's gating attribute creates a natural per-branch test because the two branches (g=0 and g=1) are designed to have different query-dependence structures; if per-branch bases fit better, the global basis is an oversimplification and the diagnostics would be miscalibrated in a way that directly affects the frozen selector. The proposed test is concrete, uses the paper's own data and splits, and would settle whether the assumption holds or fails. No secondary issue—such as the derangement counterfactual or missing confidence intervals—is as consequential, because those are addressable without altering the central theory. The verdict should remain CONDITIONAL, as the concern is real but not yet demonstrated; the test would determine whether it actually lands.","tokens_in":11997,"tokens_out":10741,"duration_ms":105137,"concrete_test":"Using the controlled task family and the same 20 episodes per α as in §5.1, group all held-out queries by the gating attribute g(x) in Eq. (7). Fit separate low-rank bases B_k^0 and B_k^1 on the estimation traces from each group, and also fit the global basis of Eq. (3) on all estimation traces. Compare held-out reconstruction R^2 and behavioral recovery (Eq. 10) for the three bases on the held-out queries. If the per-group bases improve reconstruction or recovery over the global basis by more than 0.05 in either metric, the query-independent basis assumption of Eq. (3) is violated. If they do not, the concern is settled and the diagnostics are adequate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every diagnostic that drives the selector—sharedness (Eq. 6), coefficient predictability, support dispersion κ (Eq. 12), and the fit–recovery gap G_add (Eq. 8)—is computed from a decomposition that assumes a query-independent basis B_k(D) in Eq. (3). If the true internal computation uses query-dependent basis elements or nonlinear interactions, then the residual after the low-rank fit is not noise but structured signal, and all four diagnostics are miscalibrated. The paper's support for Eq. (3) is H1 (Table 1), which shows that a global low-rank basis reconstructs held-out changes. That is necessary but not sufficient: a high-R^2 global approximation can still miss a small, behaviorally critical query-dependent subspace, and reconstruction R^2 does not guarantee that the basis supports causal recovery. The controlled task family provides a direct way to test query-independence: queries are split by the gating attribute g(x) into a constant-rule branch (g=0) and a query-selected-rule branch (g=1). If separate per-branch bases reconstruct or recover substantially better than the global basis, then Eq. (3) is violated and the shared component v_s conflates branch-specific structure with true query-wise selection. Because the selector's thresholds are frozen on controlled diagnostics and then applied to natural tasks, any such miscalibration would propagate into method choices and the claimed 0.3-point regret.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the Selection–Realization Hypothesis as an empirical account of when demonstrations in multimodal in-context learning can be compressed into a static task vector and when more expressive interventions are needed. It defines a decomposition of demonstration-induced activation changes into a query-independent basis with query-dependent coefficients (Eq. 3) and derives four diagnostics: sharedness, coefficient predictability, support dispersion, and the additive fit–recovery gap. Using a controlled multimodal task family where a gating attribute selects between a constant rule and a query-selected rule, the authors test hypotheses H1–H4 and report that the diagnostics predict static versus conditional selection and local versus multi-site versus routing realization. A family selector calibrated only on controlled tasks is then frozen and evaluated on natural VQA benchmarks, where it stays within 0.21–0.37 percentage points of the post-hoc best method at a relative cost of 0.58.","tokens_in":12342,"tokens_out":6100,"duration_ms":65383,"significance":"If the central decomposition assumption holds, the paper provides a valuable empirical theory and a practical, cost-aware method-selection procedure. The strengths are genuine: controlled episodes vary query dependence while holding task primitives fixed, intervention families are matched in rank, norm, and supervision, counterfactuals are designed to isolate the demonstrated mapping, cluster-bootstrap confidence intervals are used for correlations, and the decision rule is frozen before natural-task evaluation. The hypotheses are explicit and falsifiable, and the natural-task validation with a pre-specified selector is a strong design. However, the entire diagnostic machinery rests on the untested query-independence of the basis in Eq. (3), and part of the H2 claim is constructional rather than empirical. The absence of confidence intervals on the main recovery tables and the lack of released code or data further limit verification.","major_comments":[{"comment":"The decomposition T_D(x) ≈ Σ_k c_k(x,D) B_k(D) with a query-independent basis B_k(D) is load-bearing for all four diagnostics: sharedness (Eq. 6), coefficient predictability, support dispersion κ (Eq. 12), and the additive gap G_add (Eq. 8). The only support offered is H1/Table 1, which shows that a global low-rank basis reconstructs held-out changes and supports behavioral recovery. Reconstruction R² is necessary but not sufficient: a global approximation can have high R² while missing a small, behaviorally critical query-dependent subspace, and reconstruction quality does not guarantee that the recovered basis supports causal intervention. The paper should test query-independence directly using the gating split in §4.1: at α=1, fit separate per-branch bases for g(x)=0 and g(x)=1 and compare reconstruction and recovery against the global basis. If per-branch bases recover substantially better, Eq. (3) is violated and the sharedness, predictability, dispersion, and gap diagnostics are miscalibrated.","section":"§2, Eq. (3); §4.1; H1/Table 1"},{"comment":"The relation between sharedness and static-vector sufficiency is partly true by construction at the representation level. v_s(D) is defined as the query mean of Δz_s, the static intervention inserts exactly v_s, and Shareds(D) is the fraction of demonstration-induced energy contained in v_s. Thus high sharedness mechanically implies high representational overlap between the static intervention and the average demonstration-induced change. The empirical content reduces to whether that representational overlap transfers to behavioral recovery, as measured in Tables 2–3 and Figure 3. The text should state this explicitly and separate the constructional part from the transfer part; for example, it could test whether recovery tracks sharedness after conditioning on the representation-level alignment, or compare episodes matched on Shareds but differing in behavioral transfer.","section":"§3, H2; Eq. (5)–(6)"},{"comment":"The main quantitative claims that carry the theory—static recovery declining with α, multi-site gain of 0.04–0.09 over local interventions, routing gain of 0.04–0.06 on GQA/OK-VQA/CVQA, and the selector regret of 0.29 points in Table 4—are reported as point estimates without confidence intervals. Figure 3 and Figure 4 provide cluster-bootstrap intervals only for the correlation analyses, not for the recovery comparisons that motivate the family selector. The matched-capacity comparisons are the key evidence for H3 and H4, so confidence intervals on these recovery differences are necessary to judge whether the observed gains are reliable.","section":"Tables 2 and 3; §5.3"}],"minor_comments":[{"comment":"The rank criterion (90% variance, cap 32) and the sufficiency threshold ρ=0.65 are free parameters; please add a sensitivity analysis over these choices to show that the controlled-to-natural transfer does not depend on their exact values.","section":"§4.2"},{"comment":"The admissible set C(D) and the predicted-recovery mapping from diagnostics d(D) are described qualitatively; specify exactly how diagnostic thresholds and the recovery estimator produce C(D) so that the selector is fully reproducible.","section":"Eq. (9), §4.4"},{"comment":"The random-subspace entry '23.2 / −0.0125.8' appears to be a formatting error and should be corrected.","section":"Table 2, α=0.25 row"},{"comment":"The sentence 'compactness does not imply that one direction is shared by all queries' seems to contain a typo; 'direction' should likely be 'intervention' or 'basis direction'.","section":"§5.3, H1 discussion"},{"comment":"No code, data, or paired traces are released; given that the diagnostics require precise extraction and intervention pipelines, releasing the code is important for reproducibility and for independent verification of the decomposition assumption.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The decomposition assumption in Eq. (3) is the crux of the paper. The experimental framework is careful and the controlled-task design is a real contribution, but the lack of a direct test of query-independence, the partly constructional nature of the sharedness result, and missing confidence intervals on the main recovery tables make the current version premature. I would support acceptance after a revision that adds the per-branch basis test, clarifies the constructional versus empirical content of H2, and reports CIs on the recovery comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I largely agree with the conditional read, and I'd go a bit further on the soundness score. Here's the key thing you should know: the paper is not a theory in the mathematical sense, but it's a genuinely well-designed empirical study that gives practitioners a recipe for choosing the cheapest sufficient implicit ICL intervention. The controlled-task design is the highlight — they vary query dependence while holding task primitives and prompt format fixed, use derangement counterfactuals to isolate the demonstrated mapping, match capacity across intervention families, and freeze the selector's thresholds on controlled data before touching natural benchmarks. That's the right way to test a hypothesis like this.\n\nThe two-axis separation (selection vs. realization) is a useful synthesis, and the four diagnostics are new enough to be citeable. The H2 relation is partly definitional: the shared component is defined as the mean change, and sharedness is its energy fraction, so high sharedness trivially means the static vector captures most of the representation change. What's non-trivial — and what the paper does show — is that the representational overlap transfers to behavioral recovery. That's the empirical finding worth keeping.\n\nThe stress-test note about Eq. (3) is on the mark. The paper assumes a query-independent basis and never directly tests it. Reconstruction R^2 from a global basis is necessary but not sufficient; a small, behaviorally critical query-dependent subspace could be missed. The controlled task has a gating attribute that makes a clean falsification test possible — compare per-branch bases (g=0 vs g=1) against the global basis. The paper doesn't run it. I'd want that added, plus confidence intervals in the main tables and code/data release. The missing artifacts are a real problem for a paper whose practical value depends on reproducing the diagnostic pipeline.\n\nThe citation pattern is fine; the author's own prior baselines are cited appropriately. No red flags there.\n\nBottom line: the central argument holds up. The diagnostics predict natural-task performance reasonably well, with modest regret and a 42% cost reduction over routing. The flaws are addressable — they don't undermine the core contribution. This deserves a serious referee, and I'd expect major revision rather than rejection.","headline":"A solid empirical framework for choosing the cheapest sufficient implicit ICL intervention; the core relationship holds up, but the query-independent basis assumption needs a direct test.","tokens_in":12819,"tokens_out":3445,"would_cite":true,"duration_ms":34849,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that whether a static task vector suffices for implicit multimodal in-context learning is determined by the structure of the demonstration-induced computation, and that four measurable diagnostics predict when…","keywords":["implicit multimodal in-context learning","task vectors","Selection–Realization Hypothesis","activation interventions","representation engineering","query-conditioned interventions","attention routing","cost-aware method selection"],"falsifier":"On a task deliberately built so that each query's correct rule lives in a disjoint internal subspace, estimate held-out reconstruction $R^2$ of the fixed low-rank basis; if reconstruction stays high yet both static and conditional interventions fail while a per-query refit basis succeeds, the decomposition in Eq. (3) is falsified.","tokens_in":11786,"feed_emoji":"🧠","tokens_out":6760,"duration_ms":59437,"temperature":0.7,"pith_summary":"The paper asks when a static task vector—a fixed shift in a model's internal state that encodes the demonstrations—is enough to reproduce in-context learning, and when the intervention must instead depend on the current query or act at several sites. It proposes the Selection–Realization Hypothesis: demonstrations induce a compact family of internal changes, the query selects a member of that family, and the model's computation determines how the selection can be implemented. Using controlled multimodal episodes where query dependence is varied without changing task primitives, the paper shows that the fraction of demonstration-induced change shared across queries predicts static-vector success, while coefficient predictability, support dispersion, and an additive fit–recovery gap predict when conditional, multi-site, or routing interventions are useful. A selector calibrated only on controlled tasks, using these diagnostics, picks interventions on natural VQA benchmarks within about 0.3 points of the post-hoc best method while reducing deployment cost. The contribution is an empirical theory that makes intervention complexity a measurable quantity rather than a post-hoc justification.","feed_headline":"Task-vector success predicted by one number: sharedness","feed_subtitle":"Four diagnostics pick the minimal intervention within 0.3 points of the best method, at lower cost.","key_machinery":"The carrying object is the demonstration-induced transformation $T_D(x) \\approx \\sum_{k=1}^r c_k(x,D)B_k(D)$: a low-rank basis of internal changes, with query-dependent coefficients. Four diagnostics derived from this decomposition drive the argument: sharedness $S(D)$, the fraction of change energy that is query-invariant; the held-out coefficient-predictor $R^2$, which tests whether query-specific coefficients can be predicted from the zero-shot query; support dispersion $\\kappa$, the normalized entropy of site-level recovery that distinguishes local from distributed causal support; and the additive fit–recovery gap $G_{\\text{add}} = \\text{Fit}_{\\text{add}} - \\text{Recovery}_{\\text{add}}$, which flags when additive shifts reproduce representations but fail behavior. Each is measured on calibration traces and mapped to a minimal intervention family without seeing test labels.","core_discovery":"The central claim is that whether a static task vector suffices is determined by the structure of the computation that explicit multimodal in-context learning induces, not by task difficulty or architecture. For a fixed demonstration set, the paper measures the change in internal activations relative to matched counterfactuals, $\\Delta z_s(x;D)$, and decomposes it into a shared component $v_s(D)$ and a low-rank, query-dependent residual $U_s(D)c_s(x,D)$. When the shared component carries most of the energy (high sharedness $S(D)$), a static vector recovers the behavioral effect of explicit demonstrations. When the residual varies predictably with the query, conditional interventions are needed; when causal support is dispersed across sites, multi-site additive interventions help; and when additive shifts reconstruct the representations but not the behavior, attention routing helps. These four relationships are verified on controlled episodes with varying conditionality $\\alpha$, then used with frozen thresholds to select minimal interventions on VQAv2, GQA, OK-VQA, and CVQA within about 0.3 points of the best post-hoc method.","pith_inferences":["The same selection–realization split likely applies to unimodal ICL and to prompt-routing architectures, so these diagnostics could index a reusable intervention library across LLMs rather than only multimodal models.","Equation (3) suggests a testable taxonomy: tasks whose computation is basis-stable (query-independent basis) versus tasks that require query-dependent basis elements; the latter would break the diagnostics and may need nonlinear or memory-augmented interventions.","The derangement counterfactuals assume shuffling output symbols isolates the demonstrated mapping; if marginal label statistics also drive behavior, sharedness would be overestimated. Comparing derangements with per-example shuffles and full rerandomizations would reveal such contamination.","Because diagnostics are cheap relative to re-encoding demonstrations, the selector could be run online and switch intervention families as the demonstration set changes, adapting cost to task drift."],"forward_implications":["A practitioner can choose among static, conditional, multi-site, and routing interventions using only calibration traces, without running test evaluation.","Static task vectors are not universally limited: at $\\alpha=0$ they recover 0.92 of the normalized behavioral gain, and the drop to 0.33 at $\\alpha=1$ tracks the sharedness diagnostic rather than task difficulty.","Routing is a targeted remedy, not a default: it gives little gain on controlled tasks and VQAv2, but 0.04–0.06 on GQA, OK-VQA, and CVQA, and this gain is predicted by the additive fit–recovery gap.","The theory-selected intervention lands within 0.29 mean regret of the post-hoc best method at relative cost 0.58, cutting cost by 19% relative to always using M2IV and 42% relative to always using ICR.","Compactness holds across settings, with effective rank far below model width, so compressing demonstrations into reusable interventions is broadly feasible."],"supporting_citations":[{"why":"Establishes that in-context learning creates task vectors, the conceptual foundation for the static intervention family.","marker":"[8]"},{"why":"Introduces multimodal task vectors from selected attention heads, the canonical static baseline compared here.","marker":"[9]"},{"why":"Provides I2CL, the general context-vector compression baseline for implicit ICL.","marker":"[18]"},{"why":"Supplies LIVE, a multi-site learned vector baseline against which site-dispersion claims are tested.","marker":"[24]"},{"why":"Provides M2IV, the multi-site MHA/MLP intervention used as a principal comparison and cost reference.","marker":"[17]"},{"why":"Introduces MimIC's query-conditioned shifts, the conditional-selection baseline for H2.","marker":"[12]"},{"why":"Supplies ICR's attention-routing intervention, the routing baseline for H4.","marker":"[15]"},{"why":"Establishes function vectors in LLMs, supporting the claimed generality of compact interventions.","marker":"[30]"}],"fun_headline_variants":["Sharedness predicts when task vectors suffice","One number picks the simplest intervention","Task vectors work if demonstrations share structure","Query-dependent structure demands smarter interventions","Sharedness score guides cost-aware method choice"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that the internal effect of the demonstrations can be written as a small set of reusable change patterns whose per-query weights vary; if the patterns themselves must change with the query, or if interactions between patterns are nonlinear, the four diagnostics will be miscalibrated.","fun_headline_variants_meta":{"raw":{"variants":["Sharedness predicts when task vectors suffice","One number picks the simplest intervention","Task vectors work if demonstrations share structure","Query-dependent structure demands smarter interventions","Sharedness score guides cost-aware method choice"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1467,"prompt_tokens":993,"completion_tokens":474,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":414}},"tokens_in":609,"tokens_out":474,"duration_ms":5779,"temperature":1.0,"reasoning_tokens":414,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:19:04.234596+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a task deliberately built so that each query's correct rule lives in a disjoint internal subspace, estimate held-out reconstruction $R^2$ of the fixed low-rank basis; if reconstruction stays high yet both static and conditional interventions fail while a per-query refit basis succeeds, the decomposition in Eq. (3) is falsified.","supporting_citations":[],"review_version":1}