{"id":"2a5e0899-9c66-47c1-af17-41e7a6483bdb","arxiv_id":"2510.03839","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"M-FISHER combines an anytime-valid martingale shift detector with Fisher/natural-gradient prompt updates for streaming test-time adaptation of CLIP, with modest empirical gains and largely standard theory.","lead":"M-FISHER is a sequential test-time adaptation method that triggers CLIP prompt updates with a martingale shift detector and shapes the updates with a Fisher-information preconditioner. It is a coherent engineering integration, but most of the theoretical machinery is a restatement of classical e-process and natural-gradient results, and the reported gains are modest.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Adaptation feedback invalidates the anytime-valid guarantee: after any detection (including a false alarm), prompt updates change the score distribution, so the calibrated e-process is no longer a supermartingale under the null.","rationale":"The reader's weakest assumption is calibration-set representativeness and bootstrap validity. That is real: Proposition A.1's bootstrap correction is not proven to be a valid 1−α upper bound, and using the same sample to estimate μhat and the empirical MGF creates a bias. However, even if the bootstrap were fixed, the anytime-valid guarantee would still not apply to the full algorithm, because every detection triggers a prompt update that changes the score distribution. The false-alarm theorem conditions on a fixed null distribution; Algorithm 1 violates that condition after the first update. Table 1's false-alarm numbers are for the full closed-loop method, so the paper claims a guarantee it does not prove. The proposed empirical test would settle whether the feedback loop in practice breaks the bound. If it does, the central claim fails; if it does not, the authors still need a proof. This moves the verdict from CONDITIONAL to REJECT in the current form.","tokens_in":11131,"tokens_out":11081,"duration_ms":103519,"concrete_test":"Run Algorithm 1 on a pure no-shift stream (all test inputs drawn from the same distribution as the calibration set), with τ = 100, λ and η as in §5.2, over T = 10^4 samples and ≥100 seeds. Count the fraction of runs with at least one detection (false alarm) for M-FISHER and compare with the same detector with the adaptation step disabled. If the closed-loop false-alarm rate exceeds α + 1/τ or is materially above the martingale-only rate, the anytime-valid bound fails because prompt updates alter the null score distribution.","verdict_should_be":"REJECT","load_bearing_attack":"The most load-bearing gap is the closed-loop interaction between detection and adaptation. Section 4.1/3.2 proves that M_t is a supermartingale when scores S_t are drawn from a fixed null distribution P0 and ψbar(λ) upper-bounds the log-MGF. However, Algorithm 1 updates the prompt P via Eq. 7 every time M_t ≥ τ. After the first such event — even if it is a false alarm — the score distribution for all subsequent S_t is different because P has changed. The increments exp(λ(S_i − μhat) − ψbar(λ)) are therefore no longer conditionally mean ≤ 1 under the null; the supermartingale condition and Ville's inequality no longer apply to the full closed-loop process. The paper never states that the bound holds only before the first adaptation, and Table 1 reports false-alarm rates for the complete M-FISHER algorithm, so the advertised 'anytime-valid' false-alarm control is not established for the method as deployed. This also compounds the calibration-set issue: ψbar is estimated once under the initial model and is not the correct MGF after P changes.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes M-FISHER, a sequential test-time adaptation method for vision-language models that couples martingale-based shift detection with Fisher-preconditioned prompt updates. The detection component forms an exponential process from non-conformity scores (Eq. 3/4), declares a shift when the process exceeds a threshold τ, and claims time-uniform false-alarm control via Ville's inequality (Eq. 10). The adaptation component updates prompt parameters with a damped diagonal Fisher preconditioner (Eq. 6–7), argued to be a natural-gradient step with local KL-optimality and reparameterization invariance (Theorem A.1). A detection-delay bound of order log(τ)/Γ is claimed under sustained shifts (Eq. 11–12). Experiments on ImageNet-C, ImageNet-R, and Office-Home report accuracy, calibration error, detection delay, and false-alarm rate, with ablations separating the contribution of the martingale trigger and the Fisher preconditioner.","tokens_in":11478,"tokens_out":5726,"duration_ms":138396,"significance":"If the theoretical claims held as stated, M-FISHER would be a useful template for principled TTA: the use of e-processes/martingales for anytime-valid detection is well motivated, the information-geometric perspective on prompt updates is standard, and the paper explicitly releases code and includes ablations. These are genuine strengths. However, the central anytime-valid guarantee is not established for the closed-loop algorithm as deployed, because prompt adaptation after a detection changes the score distribution and invalidates the supermartingale argument. The bootstrap finite-sample correction is asserted rather than proved, and the detection-delay claim relies on classical results whose hypotheses are not verified for the exponential e-process with plug-in estimates and post-adaptation score dynamics. These are load-bearing gaps, not presentation issues.","major_comments":[{"comment":"The supermartingale property and Ville bound (Eq. 10) hold only while the scores S_t are drawn from the fixed null distribution used to calibrate μhat and ψbar. In Algorithm 1, whenever M_t ≥ τ, the prompt P is updated via Eq. (7). After that first update — including after a false alarm — the score distribution changes, so the increments exp(λ(S_i−μhat)−ψbar(λ)) are no longer conditionally mean ≤ 1 under the null. The time-uniform guarantee therefore does not apply to the full M-FISHER process as deployed and as evaluated in Table 1. The paper should either state that the guarantee holds only up to the first adaptation, or prove a closed-loop supermartingale property under the adaptation rule, or present Table 1 as a validation of the detection module with adaptation disabled.","section":"§4.1 and Algorithm 1, Eq. (10)"},{"comment":"The 'finite-sample correction' is not proven. The proposition assumes P(ψ(λ) ≤ ψbar(λ)) ≥ 1−α and derives the unconditional bound α + 1/τ, but the main text does not prove that the empirical (1−α)-quantile of the bootstrap MGF provides such coverage, let alone uniformly over λ and over the stream. Bootstrap quantiles are first-order asymptotically valid under regularity conditions, not exact finite-sample confidence bounds. Moreover, μhat is also estimated on the same calibration set and appears inside the exponent, and the bootstrap procedure described in §3.2 does not account for this extra estimation step. Without a proof or a precise asymptotic statement, the advertised finite-sample false-alarm control is not established.","section":"§3.2 and Proposition A.1"},{"comment":"The detection-delay bound is asserted, not derived for the actual detector. Lorden's classical result applies to likelihood-ratio procedures with known pre-change and post-change distributions. Eq. (11) is invoked as 'classical', but the extension to exponential e-processes with plug-in μhat and ψbar, unknown P1, and a stopping rule based on M_t is not proved in the main text. Theorem A.2 gives an asymptotic IID statement under a fixed prompt, but Algorithm 1 changes the prompt upon detection, so E_{P1}[(T(τ)−t*)+] ≲ log(τ)/Γ is not a property of the closed-loop method. If the paper intends this as an analysis of the detection module in isolation, that limitation should be explicitly stated.","section":"§4.2, Eqs. (11)–(12)"},{"comment":"Theorem A.1 is stated and proved for the exact Fisher Information Matrix F_P, but Algorithm 1 and Eq. (6) use a damped diagonal estimator. The steepest-descent interpretation and reparameterization invariance in Theorem A.1 do not generally survive diagonalization or damping. The paper should either justify that the approximate update retains these properties under the stated assumptions, or weaken the 'geometrically stable' claim to hold only for the exact natural-gradient update.","section":"§4.3 and Theorem A.1"}],"minor_comments":[{"comment":"There are several missing references and placeholders: 'Ville's inequality [?]' after Eq. (10), '?Liang et al., 2020' in the introduction, 'Figure??' before Figure 1, and 'see detail A.1' with no clear pointer. These should be cleaned up.","section":"Throughout"},{"comment":"The notation is inconsistent: Eq. (5) writes conditioning on C, but the text says 'conditional on the calibration event {ψ(λ) ≤ ψbar(λ)}'. Proposition A.1 conditions on the event. Please make the conditioning object explicit.","section":"§3.2, Eq. (5)"},{"comment":"Algorithm 1 line 1 says 'Compute μS and ψ(λ)', while the corrected process in §3.2 uses ψbar(λ). Line 5 says 'Update M_t via Eq. 3', but Eq. (3) uses ψ, not ψbar. The algorithm and the main-text equations should be aligned.","section":"Algorithm 1"},{"comment":"DomainNet is listed in §5.1 and mentioned in the conclusion, but no DomainNet results appear in Table 2. Either report the results or remove the claim of evaluation on DomainNet.","section":"§5.1 and Table 2"},{"comment":"The detection statistics are averaged over only three random shift orderings. The paper itself notes the need for more runs; please provide per-shift-type breakdowns and confidence intervals or additional seeds before making strong comparative claims.","section":"§6.1, Table 1"},{"comment":"There are typos such as 'evlaute', 'perfroamnce', and 'benchmarking' in §5.1, and 'more' fragment in the reader's summary is not the issue; in the manuscript, please run a careful proofread.","section":"Typos"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid kernel — standard martingale and information-geometry tools are applied to a relevant problem, and the empirical study, while small, is suggestive. The decisive issue is that the headline anytime-valid false-alarm guarantee is not established for the full closed-loop algorithm because adaptation changes the score law after the first detection. This is a load-bearing gap that requires either a re-scoping of the claims or substantial new analysis. The bootstrap correction is another place where the paper asserts rather than proves a finite-sample property. I recommend major revision: the authors should clearly separate the detection-module guarantees from the full-system claims and provide rigorous statements appropriate to what is actually proven."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper couples a martingale/exponential e-process change detector (on non-conformity scores) with Fisher-preconditioned prompt updates for CLIP-style test-time adaptation. The integration is new at the system level and the math they invoke is mostly correctly applied. But the central advertised guarantee – anytime-valid false-alarm control – is proven only for the score process before any adaptation. Once a trigger fires, even a false one, the prompt changes and the null distribution for future scores is no longer the calibrated one. The paper doesn't state this, and Table 1 reports false alarms for the full closed-loop algorithm, so the claim as deployed isn't established.\n\nWhat's good: the use of non-conformity scores plus an exponential supermartingale is a sensible way to decide when to adapt; Ville's inequality and Lorden's bound are the right tools, and their statements are fine in the open-loop setting. They also include a limitations section (A.2) that concedes the calibration issue and the diagonal-Fisher approximation, which is more honest than most. The empirical gains (1–3% accuracy) are modest but consistent across benchmarks.\n\nSoft spots, in rough order of severity:\n\n1. The closed-loop problem above is the big one. There is no argument that the supermartingale property holds after any update. A false alarm triggers an update and changes the score distribution, so the next increments are no longer conditionally mean ≤ 1. The 'unconditional α + 1/τ' bound is only unconditional over the calibration randomness, not over adaptation. This needs a fix – maybe resetting the martingale after each adaptation, which would give a per-window guarantee instead of anytime-valid over the whole stream, or some rigorously bounded drift.\n\n2. The bootstrap correction for ψ is asserted, not proven. A bootstrap (1−α) quantile does not generally guarantee P(ψ ≤ ψbar) ≥ 1−α without strong assumptions. An empirical-Bernstein bound would be a straightforward alternative and would actually deliver the guarantee.\n\n3. The natural-gradient optimality is proved for the full Fisher, but the implementation uses a damped diagonal approximation; that's a reasonable practical choice, but then the KL-optimality statement is approximate. They cite Kunstner et al., so they know the caveat, and the limitations section says 'may not capture full curvature.'\n\n4. Empirically, three random orderings with standard deviations only for detection stats, no accuracy error bars, and no accessible code (the link is anonymous) makes the results hard to verify.\n\n5. The delay bound is presented as O(log τ/Γ), but the derivation is an asymptotic heuristic for general exponential e-processes. They do include a remark that it's heuristic under dependent data, which is fine.\n\nWho it's for: people working on sequential TTA or on bridging change detection and model adaptation. The open-loop guarantee is a nice building block; the closed-loop issue is a good discussion topic. It deserves a serious referee – I would send it to review, not desk-reject, but with the expectation of heavy revision. If I were the editor, I'd ask for a proof or explicit caveat for the adapted process, a proper concentration bound for ψ, and release of the code.\n\nWould I cite it? Probably not until the closed-loop gap is addressed; currently the main theoretical claim isn't valid for the deployed algorithm.","headline":"Reasonable system paper, but the anytime-valid guarantee is open-loop only – adaptation breaks it.","tokens_in":11918,"tokens_out":3362,"would_cite":false,"duration_ms":28481,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"M-FISHER couples martingale shift detection with Fisher-preconditioned prompt updates to deliver anytime-valid false-alarm control and locally optimal adaptation for streaming vision-language models.","keywords":["sequential test-time adaptation","martingale detection","e-process","Ville's inequality","Fisher information","natural gradient","prompt tuning","vision-language models"],"falsifier":"Run the detector on a stream where the calibration set is drawn from a distribution that differs only slightly from the test stream's null distribution (e.g., a different domain or time period). If the empirical false-alarm rate consistently exceeds α + 1/τ, or if the bootstrap quantile fails to cover the true ψ(λ) in a simulation study, the central validity claim is refuted. A second decisive test: measure expected detection delay on a sustained shift and check whether it scales as log(τ)/Γ across multiple thresholds τ; if the scaling breaks or the constant is far off for non-IID streams, the","tokens_in":10994,"feed_emoji":"🎯","tokens_out":2250,"duration_ms":21322,"temperature":0.7,"pith_summary":"This paper proposes M-FISHER, a sequential test-time adaptation framework that answers two coupled questions: when to adapt and how to adapt. For detection, it builds an exponential martingale from non-conformity scores, uses a bootstrap-corrected log-moment generating function, and applies Ville's inequality to guarantee a time-uniform false-alarm probability of at most α + 1/τ at any stopping time. For adaptation, it updates prompt embeddings via Fisher-preconditioned natural gradients, which the paper shows are locally KL-optimal and invariant to reparameterization. If correct, M-FISHER provides finite-sample, anytime-valid shift detection with a detection delay that scales as log(τ)/Γ, where Γ is the post-shift information growth rate, while keeping prompt updates stable and calibrated. A sympathetic reader would care because this unifies statistical sequential testing with geometric adaptation, potentially making test-time adaptation safer and more principled in safety-critical streaming applications.","feed_headline":"Martingale trigger plus Fisher updates gives anytime-valid adaptation","feed_subtitle":"A new framework decides when to adapt and how, with provable false-alarm control and geometrically stable prompt updates for streaming visio","key_machinery":"The central objects are (i) an exponential supermartingale (e-process) built from non-conformity scores S_t = KL(p_θ(y|x_t) ‖ Uniform) + α·Mahalanobis distance, with a bootstrap-corrected log-MGF ψ̄(λ) that restores the supermartingale property under finite calibration data; and (ii) the Fisher Information Matrix F_P of the prompt parameters, which preconditions gradient updates to implement natural gradient descent in the Fisher–Rao metric. Ville's inequality converts the supermartingale property into a time-uniform false-alarm bound valid at arbitrary stopping times, while the Fisher geometry ensures that each update is locally KL-optimal and stable under reparameterization. The combinatio","core_discovery":"The paper's central claim is that sequential distribution shift detection and prompt adaptation can be unified in a theoretically grounded way. Concretely, the corrected process M_t = ∏ exp(λ(S_i − μ̂) − ψ̄(λ)) is shown to be a supermartingale conditional on a high-confidence bootstrap upper bound on the log-moment generating function, so Ville's inequality yields P(sup_t M_t ≥ τ) ≤ α + 1/τ unconditionally. Under sustained shift, the expected detection delay is bounded by log(τ)/Γ, where Γ = sup_λ {λ(E_{P1}[S] − μ_S) − ψ(λ)} is the post-shift exponential growth rate, linking detection speed to distributional divergence. For adaptation, a Fisher-preconditioned step P ← P − η(F_P + γI)^{-1}∇L","pith_inferences":["The unconditional bound α + 1/τ hinges on the bootstrap quantile being a genuine 1−α upper confidence bound on ψ(λ); this is asserted but not proven, so the practical guarantee could degrade if the bootstrap undercovers or the calibration set is not representative of the null distribution.","The Γ-based delay bound is asymptotic and assumes IID or well-mixing score processes; for dependent or adversarial streams, the log(τ)/Γ scaling may only hold approximately, and empirical delay could be much larger—so the bound should be treated as a design heuristic outside those regimes.","Because the non-conformity score mixes confidence (KL from uniform) with feature-space distance, Γ implicitly depends on both predictive uncertainty and feature drift; this suggests the method could be extended to detect label shift or subpopulation shifts by redefining the score, though the paper does not test that.","The reported empirical gains are modest (1–3% accuracy, 0.5-sample delay improvement), but the theoretical guarantees—if valid—justify using the method in settings where false alarms are costly, even when average accuracy improvements are small."],"forward_implications":["If the false-alarm bound α + 1/τ holds, practitioners can set τ to target a desired false-alarm rate and trust the detector at any stopping time, enabling safe automation in streaming decision systems.","The detection-delay scaling log(τ)/Γ quantifies how shift magnitude (via Γ) trades off against detection speed, giving a principled way to set thresholds based on the expected cost of delay versus false alarms.","Fisher-preconditioned updates mean that after a detected shift, prompt adaptation makes the locally smallest KL step that improves the detection loss, reducing the risk of overfitting to transient noise compared to unconstrained gradient updates.","The anytime-valid nature of the detector means the same guarantee holds even if the stream length is unknown or unbounded, which is natural for online deployment.","The ablation results, if reproducible, imply that both the martingale trigger and Fisher preconditioning contribute complementary gains, suggesting that trigger-then-adapt is a better design than continuous or fixed-interval adaptation."],"fun_headline_variants":["Martingale-driven Fisher prompting for sequential shift adaptation","Anytime-valid detection and Fisher-prompt adaptation for streams","Exponential martingale triggers, Fisher updates adapt streaming prompts","Time-uniform shift detection with natural-gradient prompt updates","M-FISHER: martingale detection, geometric adaptation for streaming data"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the held-out calibration set accurately represents the null (no-shift) score distribution for the entire stream, so that the bootstrap-derived high-confidence bound on ψ(λ) is a genuine upper bound; if the calibration set is non-representative or the bootstrap undercovers, the claimed false-alarm guarantee α + 1/τ can fail.","fun_headline_variants_meta":{"raw":{"variants":["Martingale-driven Fisher prompting for sequential shift adaptation","Anytime-valid detection and Fisher-prompt adaptation for streams","Exponential martingale triggers, Fisher updates adapt streaming prompts","Time-uniform shift detection with natural-gradient prompt updates","M-FISHER: martingale detection, geometric adaptation for streaming data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1161,"prompt_tokens":728,"completion_tokens":433,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":365}},"tokens_in":472,"tokens_out":433,"duration_ms":4185,"temperature":1.0,"reasoning_tokens":365,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T11:34:12.532998+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the detector on a stream where the calibration set is drawn from a distribution that differs only slightly from the test stream's null distribution (e.g., a different domain or time period). If the empirical false-alarm rate consistently exceeds α + 1/τ, or if the bootstrap quantile fails to cover the true ψ(λ) in a simulation study, the central validity claim is refuted. A second decisive test: measure expected detection delay on a sustained shift and check whether it scales as log(τ)/Γ across multiple thresholds τ; if the scaling breaks or the constant is far off for non-IID streams, the","supporting_citations":[],"review_version":1}