{"id":"158a3289-84cc-4bc0-8046-a007225ea264","arxiv_id":"2507.11528","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"FlexCAST preserves the design of a scientific analysis as a reusable functional, enabling reinterpretation with changed input data and parameters, demonstrated on a machine-learning anomaly detection analysis.","lead":"FlexCAST is a new framework for reusing particle physics analyses: instead of preserving the exact analysis code, it preserves the design itself so both the input data and the analysis settings can be changed. A demonstration on LHC-like data shows the approach can retrain machine-learning classifiers and scan different signal regions while reporting automated validity checks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FlexCAST's central validity guarantee is unvalidated: all case-study tests pass or are yellow, with no negative controls showing the tests flag invalid reinterpretations, so the claim that changed analyses remain scientifically meaningful rests on asserted rather than demonstrated test sensitivity.","rationale":"The reader's weakest assumption identifies the task-based tests and their heuristic, non-exhaustive nature as the main weakness. My stress-test sharpens this into a concrete, load-bearing concern: the validity principle is central to FlexCAST's contribution, yet the paper provides no evidence that the tests are capable of failing on invalid configurations. All reported results are green or yellow, and the authors explicitly acknowledge that the tests are not exhaustive and use heuristic thresholds. Without negative controls, passing tests cannot be interpreted as evidence that a reinterpreted analysis is scientifically valid. This directly threatens the claim that FlexCAST preserves the analysis design such that new results remain meaningful. The concern is addressable by a negative-control benchmark, so it does not invalidate the framework; it strengthens the condition under which the paper should be accepted. Therefore the reader's CONDITIONAL verdict remains appropriate, and no verdict adjustment is needed.","tokens_in":13307,"tokens_out":6472,"duration_ms":85604,"concrete_test":"Run the FlexCAST demonstrator against a set of deliberately invalid configurations and require the corresponding tests to turn red: (1) shuffle the SR/SB labels before ClassifierTraining, (2) fit the sideband with a constant background instead of the exponential, and (3) inject a second signal peak inside the sideband region. If any of these configurations is reported as green or yellow, the validity layer does not detect known invalid analyses and the paper's core validity claim is unsupported; if all turn red, the framework's tests are sensitive as claimed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that preserving the analysis functional F(d,p) and validating each task lets users change the input data and parametrization while obtaining scientifically meaningful results. Section 2 makes validity a core principle: 'Validity ... ensures that the results produced by a function f remain valid and meaningful.' In the implementation, validity rests entirely on the task-based tests of Section 3.2 (TestClassifierBias, TestHistogramSpread, TestHistogramStats, TestFit). The case study reports only green and yellow statuses (Fig. 5b); no test is ever red. The paper explicitly states in Section 4 that the tests are heuristic and 'not necessarily exhaustive.' What is missing is any demonstration that the tests are sensitive to invalid reinterpretations. For example, a classifier trained with shuffled labels, a fit using the wrong background model, or a signal injected into the sideband would each produce a scientifically invalid analysis; there is no evidence that any of the four test tasks would turn red. If the tests cannot fail on such configurations, then green/yellow statuses convey no validity guarantee, and the central claim that FlexCAST preserves the design so that new results remain meaningful collapses to an unverified assertion. This is the load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"FlexCAST is a framework for the flexible reinterpretation of scientific data analyses. The paper introduces a formal distinction between an analysis function f(d), which maps input data to results, and an analysis functional F(d,p), which produces the analysis function from data and parameters. RECAST preserves f, whereas FlexCAST preserves F, allowing the entire input data and parametrization to change during reinterpretation. The implementation uses a DAG of modular, parametrized tasks; each production task can be accompanied by a test task that yields a green/yellow/red status. The framework is demonstrated on a CWoLa anomaly detection analysis with LHC Olympics data, scanning multiple signal windows across four datasets and reporting test statuses. The paper claims that this approach enables valid and robust reinterpretation of data-driven ML analyses.","tokens_in":13625,"tokens_out":6434,"duration_ms":72729,"significance":"FlexCAST addresses a real gap in analysis reinterpretation: standard frameworks such as RECAST assume a fixed analysis implementation, which is inadequate for data-driven methods whose behavior changes with input data. The formalization of analyses as functionals, and the emphasis on preserving the design rather than the frozen implementation, is a valuable conceptual contribution. The authors provide a complete, publicly available demonstrator with automated tests and CI/CD, which is a strength. However, the central validity claim is not yet underpinned by evidence that the task-based tests can detect invalid reinterpretations; all reported statuses are green or yellow, and the tests are acknowledged as heuristic. Thus the paper's significance will fully materialize only after the validation machinery is shown to be meaningful.","major_comments":[{"comment":"The central claim that FlexCAST ensures scientifically meaningful results rests on the task-based tests, but the paper does not demonstrate that these tests are sensitive to invalid reinterpretations: Fig. 5b shows only green and yellow statuses, and the tests are explicitly heuristic and \"not necessarily exhaustive,\" so a classifier trained on shuffled labels or a fit with an incorrect background model might pass all checks; the authors should add negative control experiments to prove that the tests can turn red on known-invalid configurations, or qualify the validity guarantee as conditional on test sensitivity.","section":"Section 2 (Validity) and Section 4"},{"comment":"The claimed advantage of FlexCAST over RECAST is asserted rather than measured: the paper contrasts preserving the analysis function with preserving the analysis functional, but no baseline comparison (e.g., applying the original fixed classifier and fixed signal window to the same blackbox datasets) is provided; adding such a comparison would substantiate the claim that retraining and re-optimization are necessary for sensitivity in anomaly detection reinterpretation.","section":"Section 1 and Section 4"},{"comment":"The description of the p-value calculation for TestHistogramSpread is incomplete: the sum of z-scores over classifiers and bins is used to derive a p-value, but the null distribution (e.g., a normal approximation with variance equal to the number of terms, or a chi-square with a stated number of degrees of freedom) and the treatment of correlations between bins are not specified, which is necessary for reproducing and interpreting the test status.","section":"Section 3.2 (TestHistogramSpread)"}],"minor_comments":[{"comment":"The text contains a typo: \"withing\" should be \"within.\"","section":"Section 3.2"},{"comment":"The code listings display broken character spacing (e.g., \"P r o d u c t i o n T a s k\"); the formatting should be cleaned up.","section":"Code listings 1 and 2"},{"comment":"Figure 5b has no caption; a caption should be added to explain the bar-chart contents.","section":"Figure 5b"},{"comment":"In the dataset description, \"600.000\" should be written as \"600,000\" for English-language consistency.","section":"Section 4"},{"comment":"The displayed formula \"Flexibility = Modularity + V alidity + Robustness\" has a stray space; it should be typeset correctly.","section":"Section 2"},{"comment":"The phrase \"the signal injected in blackbox 1 is too small to be detected\" is slightly awkward; consider rephrasing to \"the signal in blackbox 1 is too small to be detected.\"","section":"Section 4"},{"comment":"Reference [24] lacks complete publication metadata (year, journal or arXiv identifier); please complete it.","section":"Reference [24]"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely of interest to the JHEP readership as a methods contribution. The main weakness is the lack of validation of the testing mechanism; I strongly recommend that the authors include negative controls or a sensitivity analysis of their test tasks in the revision. The open-source demonstrator is a valuable asset and should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FlexCAST is a genuine, well-scoped contribution. Nachman and Noll reframe analysis preservation from freezing the function f to preserving the functional F(d,p) that produces the function, and they ship a fully automated, public demonstrator for a CWoLa anomaly-detection analysis on LHC-like data. This is a real conceptual extension beyond RECAST, and the implementation is reproducible: Law-based DAG, Docker container, GitLab CI/CD, single-command execution, and public code. The paper does not oversell the mathematics, and the writing is clear.\n\nCredit where it is earned: the function-to-functional shift is new and correctly motivated by data-driven ML methods that must be retrained when the input data changes. The task-based testing idea, with status reports tied to DAG tasks, is a practical contribution to analysis preservation practice. The case study demonstrating retraining and sliding-window scanning is the first reinterpretable AD demonstration I know of, and the authors are appropriately modest about its scope.\n\nThe soft spot is real and the stress-test note lands on it. The validity guarantee is only as strong as the tests, and the paper provides no negative controls showing that any of the four tests would turn red on a deliberately invalid configuration—shuffled labels, wrong background model, or signal injected into the sideband. As it stands, a green status means 'passed these heuristic checks,' not 'scientifically meaningful,' and the paper admits the tests are not exhaustive. I do not think this is fatal, because the framework is designed to let analysts define their own tests and the paper's central conceptual claim—that preserving the design functional enables flexible reuse—does not depend on these specific thresholds. But a referee should ask for at least one or two negative-control experiments, and ideally a quantitative comparison against a static RECAST-style rerun, since the claimed advantage in scientific return is asserted rather than measured.\n\nThe paper is for people working on analysis preservation, reinterpretation, and ML-driven searches at the LHC. It deserves a serious referee, not a desk reject; the issues are addressable and the supporting code is real.\n\nRecommendation: engage with it, send it to review.","headline":"Genuine conceptual step beyond RECAST with a real public demonstrator, but the validity tests are heuristic and never shown to catch an invalid analysis, so a referee should push for negative controls.","tokens_in":14042,"tokens_out":2616,"would_cite":false,"duration_ms":30140,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FlexCAST preserves the design of an analysis so it can be retrained and reparametrized on new data.","keywords":["analysis reinterpretation","RECAST","anomaly detection","Classification Without Labels","workflow preservation","data-driven machine learning","validity testing","LHC-like data"],"falsifier":"A concrete test: run a FlexCAST-preserved analysis on a new dataset with an injected signal whose mass or width lies outside the original design range, and compare the reported significance and test statuses against a manual re-analysis by a physicist. The central claim would be falsified if a configuration passes all green/yellow tests yet yields a significance that disagrees with the true injected signal, or if the flexibly retrained pipeline misses a signal that a human re-optimization of the same analysis finds.","tokens_in":13113,"feed_emoji":"⚛️","tokens_out":12328,"duration_ms":116599,"temperature":0.7,"pith_summary":"The paper introduces FlexCAST, a framework for scientific data analysis reinterpretation that extends existing approaches like RECAST. RECAST preserves an analysis as a fixed function applied to new signal models; FlexCAST preserves the logic of how the analysis was designed, encoded as a functional that takes data and parameters and returns a ready-to-run analysis. This allows changes to the entire input dataset and to the analysis parametrization—retraining classifiers, adjusting signal windows, and re-optimizing selections—which is essential for data-driven machine-learning analyses such as anomaly detection. The framework rests on three principles: modularity (a DAG of parametrized tasks), validity (automated per-task tests with status reports), and robustness (designing tasks to remain valid over a wide range of data and parameters). A case study on LHC-like data shows a weakly-supervised anomaly detection analysis being flexibly rerun on new datasets, with significance of an injected signal improved from 3σ to almost 8σ.","feed_headline":"FlexCAST makes data-driven analyses fully reinterpretable","feed_subtitle":"Preserving the design rather than the code lets analyses be retrained and reparametrized on new data.","key_machinery":"The key object is the analysis functional $F(d,p)$, which maps input data and parameters to an analysis function $f$; the analysis is decomposed into a directed acyclic graph of modular, parametrizable tasks, each with its own production, testing, and supervision logic. Task-based testing assigns green/yellow/red statuses from statistical and heuristic checks, and robustness measures are embedded in tasks (e.g., iterative fitting, classifier ensembles) to maximize the region of data/parameter space where the tests pass. This machinery carries the argument by turning 'preserve the analysis' into 'preserve the process that builds the analysis,' and by making each rebuilt analysis self-certifying through automated tests.","core_discovery":"The central claim is that preserving the analysis functional $F(d,p)$—the design process that produces an analysis function $f=F(d,p)$—rather than the fixed function $f$, makes an analysis fully reinterpretable. During reinterpretation, new data $d_R$ and new parameters $p_R$ produce a new function $f_R=F(d_R,p_R)$, applied to the changed data to give a new result $r_R=f_R(d_R)$. The paper argues that this enables changes to the entire input data and the complete parametrization, which the RECAST-style preservation of a fixed function cannot do, and demonstrates it with a Classification Without Labels (CWoLa) anomaly detection analysis whose classifiers are retrained and signal windows rescanned when the input dataset is changed. The authors state that a user implements not only the analysis strategy but also the logic behind how they arrived at a particular analysis strategy, including the steps taken to optimize it.","pith_inferences":["A quantitative benchmark against a RECAST-style fixed-function rerun on the same reinterpretation tasks would directly test the paper's central design advantage; the paper argues for it by principle but does not measure it.","The heuristic nature of the demonstrator's validity tests (e.g., 1% and 10% loss-difference thresholds) invites a stress test that injects a pathological dataset designed to pass all tests yet produce a wrong physics conclusion, which would reveal where the validity guarantee needs strengthening.","The functional-preservation idea is not limited to high-energy physics: any data-dependent analysis pipeline where the optimal preprocessing or model depends on the data could be preserved as a functional and flexibly reinterpreted, provided the validity tests are made domain-specific.","Fine-tuning a base model rather than retraining from scratch, mentioned as future work, would reduce the computational overhead of FlexCAST and could make it the default mode for large analyses; this is an extension of the framework's robustness principle rather than something demonstrated here."],"forward_implications":["If FlexCAST's central claim holds, data-driven analyses whose classifiers must be retrained on the data can be reinterpreted for entirely new datasets and signal hypotheses, not just new signal models on fixed selections.","Automated validity tests with documented status reports give analysts and reviewers a chain of validation for each reconfigured analysis, supporting publication of new results produced by a changed pipeline.","The modular DAG structure means that changing a parameter reruns only the affected tasks, so flexible reinterpretation can be computationally selective even though it is heavier than a fixed-function rerun.","FlexCAST's definition of validity—per-task automated tests on actual runtime data—extends RECAST's implicit assumption that a fixed analysis needs no revalidation, shifting the burden to explicit, checkable tests for every reconfigured analysis."],"supporting_citations":[{"why":"Defines the RECAST reinterpretation approach that FlexCAST generalizes; preserving the analysis function $f$ is the baseline against which FlexCAST's functional $F(d,p)$ is contrasted.","marker":"[1, 2, 3]"},{"why":"Introduces Classification Without Labels, the weakly supervised anomaly detection method used in the case study; its data-dependent classifier training motivates preserving the analysis design.","marker":"[21]"},{"why":"Provides the LHC Olympics R&D dataset and official data files used to build the four benchmark datasets for the demonstration.","marker":"[22, 23]"},{"why":"Describes the Law/Luigi workflow library that implements the DAG-based modular task structure underpinning FlexCAST's modularity principle.","marker":"[12, 13]"},{"why":"Supplies the example of optimizing a signal window or training a neural network as steps inside the analysis functional, illustrating what preserving the design means.","marker":"[11]"},{"why":"Contributes the ML testing practices (data schemas, distributions, model checks) that the validity principle adapts into task-based tests.","marker":"[16]"}],"fun_headline_variants":["FlexCAST: preserve design, not code, for analysis reuse","Retrain and reparametrize with FlexCAST's flexible design","FlexCAST turns fixed analyses into reusable designs","Change data, retrain model: FlexCAST makes it possible","Design preserved, analyses reinterpreted: FlexCAST"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that breaking an analysis into a DAG of modular, parametrized tasks fully preserves the logic of the analysis design, so that retraining classifiers and changing parameters yields scientifically valid results, and that the automated tests—heuristic and acknowledged as non-exhaustive—are enough to certify that validity.","fun_headline_variants_meta":{"raw":{"variants":["FlexCAST: preserve design, not code, for analysis reuse","Retrain and reparametrize with FlexCAST's flexible design","FlexCAST turns fixed analyses into reusable designs","Change data, retrain model: FlexCAST makes it possible","Design preserved, analyses reinterpreted: FlexCAST"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1388,"prompt_tokens":949,"completion_tokens":439,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":359}},"tokens_in":565,"tokens_out":439,"duration_ms":5314,"temperature":1.0,"reasoning_tokens":359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:06:28.160749+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: run a FlexCAST-preserved analysis on a new dataset with an injected signal whose mass or width lies outside the original design range, and compare the reported significance and test statuses against a manual re-analysis by a physicist. The central claim would be falsified if a configuration passes all green/yellow tests yet yields a significance that disagrees with the true injected signal, or if the flexibly retrained pipeline misses a signal that a human re-optimization of the same analysis finds.","supporting_citations":[{"cited_title":"The ML test score: A rubric for ML production readiness and technical debt reduction","cited_arxiv_id":null,"evidence_quote":"Contributes the ML testing practices (data schemas, distributions, model checks) that the validity principle adapts into task-based tests."}],"review_version":1}