{"id":"64f9b7fe-03e1-4e99-8e26-d47a430685a0","arxiv_id":"2506.01718","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Path scaling and hyperparameter tuning can dramatically lower Type 2 errors in signature MMD two-sample tests without inflating Type 1 errors, though the scaling factors are chosen per example.","lead":"This paper studies how to make a statistical test for whether two collections of time series come from the same process more reliable, using a kernel-based comparison called the signature maximum mean discrepancy (sig-MMD). It shows that rescaling the paths, together with other tuning choices, can sharply reduce the chance of wrongly saying two processes are the same, and it gives practical examples and code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline power gains rest on scaling factors chosen by looking at Type 2 error on the same data; without a prespecified selection rule, the reported reductions are in-sample optimism.","rationale":"I read the paper's central claim as an empirical prescription: with appropriate path scaling and optional preprocessing, sig-MMD can have dramatically lower Type 2 error while keeping Type 1 error near its nominal level. The level-contribution analysis in Section 2.5 and the accompanying figures is a genuine and useful diagnostic, and the public code is a real asset for reproducibility. The reader-identified math issues—the reversed rejection inequality in Section 3 and the normalization in Section 3.3—are concerning, but they appear to be presentation slips rather than the main threat: Equation (7) and the empirical protocol are consistent with the correct 'reject if statistic > quantile' rule, so I do not treat them as the load-bearing problem. The load-bearing weakness is that every headline power number is produced by selecting the scaling factor after looking at the Type 2 error curve on the same data. In a controlled simulation the labels are known, so this is oracle selection; in Section 5, the calibration step assumes the alternative on unlabelled data, which is the same circularity. Without a prespecified or principled rule for choosing θ, the claim that scaling 'reduces' Type 2 error is not yet established as a reproducible operating characteristic. The proposed two-stage test would settle this by separating selection from evaluation, and would also provide the missing Type 1 error estimate for the Section 5 pipeline. I therefore agree with the reader's conditional verdict: the contribution is credible as a proof of concept, but the evidence as presented overstates the universality of the prescriptions.","tokens_in":20002,"tokens_out":6083,"duration_ms":61287,"concrete_test":"Re-run the Section 4.1 Brownian and Section 4.2 GARCH experiments with a two-stage protocol. Stage 1: on one independently simulated dataset, choose the scaling factor θ by a fixed rule that uses only Stage-1 data—e.g., minimize the bootstrapped Type 2 error over a grid of θ, or use a moment-matching heuristic based on estimated variances (for Brownian) and autocorrelations (for GARCH). Stage 2: freeze θ and estimate Type 2 error and Type 1 error on a fresh independent dataset generated with the same parameters. Compare the Stage-2 Type 2 error at the selected θ to the reported 2.0% and 0.0%. If it is close to the unscaled 72.6% and 90.2%, or varies widely across repeated Stage-1/Stage-2 splits, the reported gain is a selection artefact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that path scaling reduces Type 2 error from 72.6% to 2.0% (Brownian) and from 90.2% to 0.0% (GARCH)—depends on the scaling factor being chosen before the test. In Section 4, the authors state that they 'tested various scaling factors' and then report the value that yields the dramatic improvement (3, 5.5, 2, 0.8, 5). This is oracle selection: the scaling is chosen by looking at the Type 2 error curve on the same simulated data that is later used to report power. A Type 2 error rate is an operating characteristic of a fixed test procedure; if the procedure includes 'inspect the Type 2 error curve and pick the best θ', then the reported numbers are in-sample fits, not a property of the sig-MMD test. The paper does not provide a legitimate selection rule, such as a moment-based heuristic, cross-validation on separate data, or a diagnostic that works without alternative labels. Section 5 attempts an 80/20 split, but the calibration stage computes Type 2 error under an assumed alternative on the 80% calibration data, which uses exactly the labels the test is supposed to infer; the final decision on the 20% holdout is a single realization with no Type 1 error estimate and no repeated-selection evaluation. Thus the headline reductions are not shown to be reproducible under a valid protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the signature Maximum Mean Discrepancy (sig-MMD) as a test statistic for two-sample tests on path-valued data. It introduces the phi-weighted signature kernel and the corresponding phi-MMD, decomposes the MMD into signature-level contributions, and argues that path scaling reweights higher-order signature levels, thereby reducing Type 2 errors. Through simulations with scaled Brownian motions, GARCH processes, and mixture models, the authors report large power improvements from scaling (e.g., Type 2 error falling from 72.6% to 2.0% for Brownian motion and from 90.2% to 0.0% for GARCH), while Type 1 error remains near the 5% significance level. The final section applies the pipeline to sector return data and concludes that the distributions are different. Public code is provided.","tokens_in":20227,"tokens_out":7420,"duration_ms":78792,"significance":"If the reported power gains were reproducible under a fixed protocol, the paper would be practically useful: it gives accessible numerical evidence that the signature kernel's level weighting strongly affects test power, and it ships public code and transparent level-decomposition diagnostics. The checks showing Type 1 error stability across scaling factors are a useful robustness observation. However, the central empirical claim is not yet established because the scaling factors are selected post hoc on the same data used to evaluate power, and the Section 5 calibration uses labels that the test is supposed to infer. The technical errors in Section 3 also need correction before the results can be relied upon.","major_comments":[{"comment":"The rejection rule is stated backwards: the null hypothesis is 'rejected with significance alpha if d-hat^2 < c-hat_{1-alpha}'. Since the MMD is nonnegative and large values indicate disagreement between distributions, the correct rule is rejection when d-hat^2 > c-hat_{1-alpha}. As written, the procedure rejects H0 when the statistic is small, which is inconsistent with all power calculations in Section 4 and with the Type 2 error formula in Eq. (7). This must be corrected, and the numerical code should be checked against the corrected rule.","section":"Section 3, first paragraph after fixing distributions"},{"comment":"The asymptotic null distribution is mis-normalized: the paper states that (1/N) d-hat^2_{kSig,2} converges to sum_i lambda_i (z_i^2 - 2), but the standard result in the cited references is that N times the unbiased MMD statistic converges to that weighted chi-square sum. With the displayed 1/N factor, the right-hand side would not be a valid asymptotic threshold. Please correct the normalization and confirm that no numerical result in Section 4 relies on this formula.","section":"Section 3.3, asymptotic null distribution"},{"comment":"The headline power improvements are obtained with scaling factors (3, 5.5, 2, 0.8, 5) that were selected by inspecting Type 2 error curves on the same simulated data used to report the power. The paper states that 'various scaling factors' were tested and then reports the value that gives the dramatic improvement; this is oracle selection. A Type 2 error rate is an operating characteristic of a fixed test procedure, so the reductions from 72.6% to 2.0% and from 90.2% to 0.0% are not yet shown to be achievable under a legitimate protocol. Please either prespecify a selection rule (e.g., a moment-based heuristic, a separate validation set, or an aggregated test over a grid) and evaluate power after selection, or explicitly reframe the reported numbers as illustrative best-case effects of scaling.","section":"Sections 4.1-4.3, scaling factor selection"},{"comment":"The calibration procedure computes the probability of a Type 2 error on the 80% calibration split, but in an uncontrolled environment the true hypothesis is unknown; computing a Type 2 error requires assuming an alternative, which uses exactly the labels the test is supposed to infer. The final permutation test on the 20% holdout is a single decision with no repeated-selection analysis and no Type 1 error estimate. Consequently, the real-data conclusion that the two sector distributions are not equal is not supported as a demonstration of a valid testing procedure. Please provide a protocol that does not use alternative labels for calibration, or clarify that Section 5 is an illustration whose statistical validity is not claimed.","section":"Section 5, uncontrolled environment calibration"},{"comment":"The displayed definition of the level-m contribution reads Gamma^phi_m(PX,PY) = phi(m)[Lambda_m(PX,PY) - 2 Lambda_m(PX,PY) + Lambda_m(PX,PY)], with identical arguments in all three terms. It should be phi(m)[Lambda_m(PX,PX) - 2 Lambda_m(PX,PY) + Lambda_m(PY,PY)] to match the MMD expansion in Eq. (6) and the estimators used in the numerical sections. As written, the object is identically zero and cannot underlie the level-contribution plots in Figs. 2, 4, 8, 10, 13, and 14.","section":"Section 2.5, definition of Gamma^phi_m"}],"minor_comments":[{"comment":"The text refers to 'the high probability 85.5%', but the preceding sentence reports 85.8% for the unbiased estimator; please make the numbers consistent.","section":"Section 4.1, paragraph after Fig. 1"},{"comment":"The notation dF_H0 and dF_H1 is ambiguous: if F denotes a cumulative distribution function, write F(c-hat); if dF denotes a density, Eq. (7) should be an integral rather than a point evaluation. Please clarify the notation.","section":"Section 3, Eq. (7) and surrounding text"},{"comment":"The statement that the level-K term of the expected signature is 'associated with the K-th moment' is imprecise: signature level K contains iterated integrals such as integral integral dX^i dX^j, which mix moments of different orders and include quadratic variation terms. Since this heuristic is used to justify the scaling mechanism, please replace it with a precise statement in terms of expected iterated integrals and the fact that scaling a path by theta multiplies the level-m signature term by theta^m.","section":"Section 2.5, moment interpretation"},{"comment":"The number of bootstrap or permutation replications used in the calibration and final decision is not stated; please report these values for reproducibility.","section":"Section 5, Figs. 18 and 19"},{"comment":"The set XSeq is used before it is defined; please define it at first use.","section":"Section 2.5, Definition 2.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads more as an experimental/how-to study than a theoretical contribution. I would ask for a revision that either provides a legitimate scaling-selection protocol or clearly downgrades the headline claims to illustrative best-case results. The reversed inequality and the normalization error in Section 3 should be corrected before further consideration; if the code implements the correct inequality, the authors should state this explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: the level-contribution decomposition is a genuinely helpful way to see why sig-MMD can be blind to higher-moment differences, and the paper shows empirically that scaling the paths can fix that. But the dramatic Type 2 error reductions are computed with scaling factors chosen by looking at Type 2 curves on the same data, so they are not the operating characteristic of a prespecified test.\n\nThe new piece is the decomposition of the phi-MMD into level-Γ_m terms. Making the factorial decay of signature terms visible and showing how low levels dominate the MMD explains the power failure when distributions differ in variance or higher moments. The explanation of path scaling as reweighting by θ^m (with proper attribution to Cass-Lyons-Xu) is clear, and the Brownian, GARCH, and mixture examples are easy to follow. The code is public. That part is solid.\n\nThe soft spots are real. Section 3 states the null is rejected when the MMD is smaller than the critical value—the inequality is reversed. The asymptotic null distribution is written with 1/N instead of N, the opposite of the correct convergence. Both look like typographical slips, but they are in the core statistical section and must be fixed.\n\nMore substantively, the headline numbers (72.6%→2.0%, 90.2%→0.0%) come from trying several scalings and reporting the best. The authors are candid that they 'tested various scaling factors,' but that turns the reported Type 2 error into a post-selected minimum, not the error of a fixed procedure. The Type 1 error plots are reassuring but do not remove the selection problem. The Section 5 'uncontrolled' example tries to address this with an 80/20 split, yet the calibration stage computes Type 2 error under an assumed alternative on the calibration data—requiring the labels the test is supposed to infer. The final permutation test on the holdout is a single realization, so the real-data conclusion is much weaker than the simulations suggest.\n\nThe central insight—that scaling reweights signature levels to target moments where distributions differ—is sound and worth having. The paper overstates the universality of its prescriptions, but it is a useful practical exploration. I would send it to peer review, expecting the authors to fix the statistical statements and either provide a principled scaling-selection rule or explicitly reframe the results as gains under oracle-like selection.\n\nFor readers working with signature kernels or MMD in low-sample time-series settings, this is a worthwhile read and a good starting point for discussion. Deserves a serious referee.","headline":"Useful level-contribution analysis and a real practical insight, but the headline power gains rely on post hoc scaling choices and the hypothesis-testing section has fixable but real errors.","tokens_in":20786,"tokens_out":4004,"would_cite":false,"duration_ms":44924,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G10","62M10"],"pacs":[],"model":"deepseek-v4-flash","headline":"Path scaling turns a failing signature two-sample test into a reliable one.","keywords":["signature kernel","maximum mean discrepancy","two-sample testing","stochastic processes","path signature","Type 2 error","kernel mean embedding","GARCH"],"falsifier":"Run the Section 4.3 mixture-model pair—two processes with nearly equal first and second moments but different third moment—at batch size 128 and compute the per-level contributions under increasing scaling; if no finite scaling makes the third-level term separate from noise, then scaling reweights levels but cannot by itself make high-moment differences detectable in low-data regimes, which the paper's own 94.6% Type 2 error already hints at.","tokens_in":19753,"feed_emoji":"📊","tokens_out":6672,"duration_ms":65458,"temperature":0.7,"pith_summary":"The paper tries to establish that the signature maximum mean discrepancy (sig-MMD) can serve as a practical two-sample test for stochastic processes, and that its main failure mode—accepting different processes as the same—is largely fixable. The fix is to scale the input paths before computing the signature kernel, because scaling is equivalent to reweighting the higher-order signature levels where distributional differences actually show up. On scaled Brownian motions with different volatilities, path scaling by 3 drops the Type 2 error from 72.6% to 2.0%; on GARCH models, scaling by 5.5 drops it from 90.2% to 0.0%, while Type 1 error stays near the 5% significance level. This matters because path-valued two-sample tests are useful in finance and time-series modelling, where limited data makes false agreement costly.","feed_headline":"Scaling paths cuts signature test misses from 73% to 2%","feed_subtitle":"Reweighting high signature levels catches Brownian and GARCH differences while keeping false alarms near the 5% level.","key_machinery":"The load-bearing identity is the scalar-multiplication equivalence for signature kernels: $k_{\\mathrm{Sig}}(\\theta x, y) = k_{\\mathrm{Sig}}(x, \\theta y) = k^{\\varphi}_{\\mathrm{Sig}}(x, y)$ with $\\varphi(m)=\\theta^m$, which makes path scaling identical to weighting level-$m$ signature inner products by $\\theta^m$. The paper also decomposes the $\\varphi$-MMD into per-level contributions $\\Gamma^{\\varphi}_m$, so one can see which signature level carries the separation between the null and alternative distributions; the factorial decay estimate $\\|\\Phi_{\\mathrm{Sig},m}(x)\\| \\le \\|x\\|_p^m / m!$ explains why, without reweighting, lower levels that do not separate the distributions dominate the statistic.","core_discovery":"The central claim is that the default sig-MMD two-sample test is systematically underpowered when two stochastic processes differ mainly in higher-order moments, because signature terms decay factorially and low-level terms dominate the statistic. Scaling both collections of paths by a common factor $\\sqrt{\\theta}$ is not a neutral preprocessing step: by [9, Corollary 2.10] it is exactly the $\\varphi$-signature kernel with weight $\\varphi(m)=\\theta^m$ on level-$m$ signature terms, so it reweights the statistic toward the levels that separate the distributions. With this reweighting, the test detects differences between scaled Brownian motions and between GARCH models at batch size 128 with Type 2 errors near zero and Type 1 errors near the nominal 5%, and it detects distributional differences at much smaller batch sizes than the unscaled test.","pith_inferences":["Because scaling is equivalent to changing the weight of each signature level, the same data expressed in different units can lead to different test conclusions; practitioners should treat scaling as a tunable hyperparameter, not an arbitrary normalization.","The reweighting idea is not tied to the signature kernel: any graded feature map whose terms decay across levels could be reweighted analogously to emphasize the levels where two distributions differ.","When paths are lifted through an RBF kernel, the scaling must be applied to the lifted path rather than the raw path, or the level-reweighting effect is lost.","The mixture-model example suggests a diagnostic use of the level decomposition: checking whether any finite scaling makes the separating level dominate tells you in advance whether the test can succeed at a given batch size."],"forward_implications":["For scaled Brownian motions with volatilities 0.2 and 0.3 and batch size 128, scaling by 3 lowers Type 2 error from 72.6% to 2.0% with the biased estimator and from 85.8% to 6.6% with the unbiased estimator.","For GARCH models with distinct parameters, scaling by 5.5 lowers Type 2 error from 90.2% to 0.0% at batch size 128.","Type 1 error remains centered near the 5% significance level across scaling factors and batch sizes, for both biased and unbiased estimators.","Scaling lets the test separate distributions at much smaller batch sizes, which matters when data are scarce or computation is expensive.","The per-level decomposition shows which signature level is responsible for separation, guiding the choice of scaling and preprocessing before the test is run."],"supporting_citations":[{"why":"Defines the kernel MMD and the two-sample testing framework that the paper extends to path space.","marker":"[21]"},{"why":"Establishes that signature-based kernels are characteristic and universal, making the sig-MMD a metric on distributions of processes.","marker":"[13]"},{"why":"Provides the PDE-based signature kernel computation and the factorial-decay estimate for signature terms.","marker":"[40]"},{"why":"Defines the signature kernel on path space and the kernel mean embedding used to formulate the sig-MMD.","marker":"[41]"},{"why":"Introduces general signature kernels and proves the scalar-multiplication identity linking path scaling to $\\varphi$-weighting.","marker":"[9]"},{"why":"Proves the weighted signature kernel results, including the injectivity that makes the $\\varphi$-MMD a metric.","marker":"[10]"},{"why":"Supplies the U-statistic asymptotic null and alternative distributions used for the two-sample test.","marker":"[20]"},{"why":"Provides the gamma approximation to the null distribution of the biased MMD used in the experiments.","marker":"[30]"}],"fun_headline_variants":["Scaling path data reweights signature terms, slashing missed detections","Signature MMD test gains power via common scaling of paths","Scaling both path sets reweights signature kernel to catch differences","Common path scaling rescues signature MMD from Type 2 errors","Path scaling turns signature MMD into a practical two-sample test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The power gains assume the user can find a good scaling factor, RBF bandwidth, and preprocessing pipeline without knowing the true relationship between the samples; in the paper every successful scaling (3, 5.5, 2, 0.8, 5) was selected by inspecting Type 2 error curves, and the Section 5 calibration estimates Type 2 error under an assumed alternative on unlabelled data.","fun_headline_variants_meta":{"raw":{"variants":["Scaling path data reweights signature terms, slashing missed detections","Signature MMD test gains power via common scaling of paths","Scaling both path sets reweights signature kernel to catch differences","Common path scaling rescues signature MMD from Type 2 errors","Path scaling turns signature MMD into a practical two-sample test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000901,"raw_usage":{"total_tokens":3866,"prompt_tokens":918,"completion_tokens":2948,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":2860}},"tokens_in":534,"tokens_out":2948,"duration_ms":22895,"temperature":1.0,"reasoning_tokens":2860,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:35:07.784608+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Section 4.3 mixture-model pair—two processes with nearly equal first and second moments but different third moment—at batch size 128 and compute the per-level contributions under increasing scaling; if no finite scaling makes the third-level term separate from noise, then scaling reweights levels but cannot by itself make high-moment differences detectable in low-data regimes, which the paper's own 94.6% Type 2 error already hints at.","supporting_citations":[{"cited_title":"Gretton, K","cited_arxiv_id":null,"evidence_quote":"Defines the kernel MMD and the two-sample testing framework that the paper extends to path space."},{"cited_title":"Chevyrev and H","cited_arxiv_id":null,"evidence_quote":"Establishes that signature-based kernels are characteristic and universal, making the sig-MMD a metric on distributions of processes."},{"cited_title":"Salvi, T","cited_arxiv_id":null,"evidence_quote":"Provides the PDE-based signature kernel computation and the factorial-decay estimate for signature terms."},{"cited_title":"Salvi, M","cited_arxiv_id":null,"evidence_quote":"Defines the signature kernel on path space and the kernel mean embedding used to formulate the sig-MMD."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces general signature kernels and proves the scalar-multiplication identity linking path scaling to $\\varphi$-weighting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Proves the weighted signature kernel results, including the injectivity that makes the $\\varphi$-MMD a metric."},{"cited_title":"Gretton, K","cited_arxiv_id":null,"evidence_quote":"Supplies the U-statistic asymptotic null and alternative distributions used for the two-sample test."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the gamma approximation to the null distribution of the biased MMD used in the experiments."}],"review_version":1}