{"id":"3d84cac2-2da7-43eb-96db-516ad0f879d4","arxiv_id":"2608.05207","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CRAFTER corrects frozen time-series forecasters by mining residual features with an MCTS search and an LLM, gating candidates by validation error, and fitting a small corrector; gains concentrate on weak backbones with residual headroom.","lead":"The paper introduces CRAFTER, a pipeline that finds interpretable features of a frozen forecasting model's errors and uses them to correct the forecasts with a small extra model. Across six datasets and six frozen backbones, the authors report that this correction approach beats dedicated feature engineering tools on average and helps most when the original model leaves large structured errors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The aggregate headline may rest on unstable validation-to-test transfer: split variance dominates, and the gate is only a validation-time safeguard, not a test guarantee.","rationale":"The reader's weakest assumption is exactly that validation-grounded gate and corrector selection generalize out-of-sample. My reading agrees and sharpens it: the paper's own variance decomposition (seed-std 0.49, split-std ~3.2) shows that the averaged result is dominated by temporal drift, and the paper itself flags BIZITOBS/Chronos as a regression cell. The central claim is empirical and aggregate, so a per-split instability would undermine the abstract headline even if the averaged numbers are as reported. The external baselines are not run natively, which is disclosed and motivated as an attribution device; that is a secondary concern about the phrase 'surpasses every dedicated system' but not the deepest risk. Code and data are not shipped, which supports the reader's CONDITIONAL verdict rather than ACCEPT. The proposed per-split and Wilcoxon check is direct, cheap, and would settle whether the validation gate transfers. This preserves the CONDITIONAL verdict; no change is needed.","tokens_in":29819,"tokens_out":5332,"duration_ms":63185,"concrete_test":"Recompute the main grid (Table S12) with each of the three rolling splits reported as a separate column for CRAFTER, CRAFTER-dagger, and the best external baseline, and run the paired one-sided Wilcoxon test within each split. If CRAFTER's advantage over the best external does not hold in at least two of the three splits, or reverses on the latest split (where distribution shift is largest), then the abstract claim is a mean artifact of split averaging rather than a stable property.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that features and corrector selected on validation blocks continue to help on later test windows. The paper explicitly disclaims this in Sec. 3.4: the NONE option and gate are 'a validation-time safeguard, not a test guarantee,' and App. D reports median split-std ~3.2 versus seed-std 0.49 for CRAFTER, i.e., split/temporal variance dominates. Since the headline averages three rolling splits, a single favorable window can drive the aggregate. Per-cell regressions do occur: BIZITOBS/Chronos goes 24.6 (RAW) to 31.4 (CRAFTER), and several FAVORITA and M5 cells are at or below RAW. The fixed gate threshold tau=0.05 is a low absolute correlation on validation noise, and with many candidates spurious correlations can pass; the corrector's NONE option only floors validation risk, not test risk. Additionally, Table S11 shows only a 60% win rate versus the best external at K=5, so 'surpasses every dedicated feature-engineering system at every feature budget' is supported only as a mean over cells, not per cell. The empirical headline is therefore not yet established under rolling-origin distribution shift.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies corrective feature discovery for frozen black-box forecasters: rather than fine-tuning the backbone, CRAFTER mines interpretable features of the backbone's residual and fits a lightweight post-hoc corrector. Two candidate generators are used, a compositional Monte-Carlo tree search over raw channels and an LLM that proposes named feature combinations, flags, and code. A source-blind validation-grounded gate accepts or rejects candidates based on Spearman correlation with the unexplained validation residual, and a validation-selected corrector (identity, additive GBDT, or multiplicative GBDT) produces the final forecast. The method is evaluated on six public datasets and six frozen backbone families under a rolling-origin three-split multi-seed protocol, with three external feature-engineering systems run inside the same harness. The paper claims that CRAFTER surpasses every dedicated feature-engineering system at every feature budget, roughly doubles the corrector-only improvement, and that gains concentrate where residual headroom is large while being neutral or harmful in saturated cells.","tokens_in":30009,"tokens_out":4931,"duration_ms":52890,"significance":"If the headline results hold, the paper makes a useful contribution: a source-agnostic, validation-grounded pipeline for residual correction, coupled with an explicit characterization of when correction helps, would be valuable for deploying frozen forecasters. The strengths of the manuscript include the careful rolling-origin protocol, the use of one shared harness for all feature generators, the honest reporting of negative and saturated cells, detailed appendices (feature grammar, prompts, per-cell tables), and the explicit acknowledgment that the validation gate is not a test guarantee. The central claim is plausible and the empirical evidence is substantial, but the abstract's 'surpasses every system at every feature budget' statement goes beyond what the per-cell results support, and the validation-to-test transfer risk is not quantified to the standard the headline requires.","major_comments":[{"comment":"The claim that CRAFTER 'surpasses every dedicated feature-engineering system at every feature budget' is supported only as a mean over cells, not as a per-cell statement. Table S11 shows that at K=5 CRAFTER beats the best external system in only 60% of the 30 cells, and at K=10/20/50 the win rate is 73%, with mean differences of -0.020 to -0.039 wMAPE. Per-cell regressions exist in Table S12, e.g. BIZITOBS/Chronos goes from RAW 24.6 to CRAFTER 31.4 (GPT-5.2), and several FAVORITA and M5 cells are at or above RAW. The abstract and Section 5.1 should state that the advantage is an aggregate mean over cells and that negative cells occur; as written, the headline overclaims.","section":"Abstract, §5.1, Table S11"},{"comment":"The load-bearing assumption that validation-selected features and correctors transfer to later test windows is explicitly disclaimed in §3.4: the gate is 'a validation-time safeguard, not a test guarantee.' This is not merely a cosmetic caveat. App. D reports that split variance dominates for CRAFTER (median split-std ~3.2 versus seed-std 0.49), so with only three rolling splits the reported means can be driven by favorable windows. In addition, the fixed threshold tau=0.05 in §3.3 is a low absolute correlation on validation residual, and the paper does not report the number of candidates tested per cell or any multiple-testing control, leaving the rate of spuriously accepted features unquantified. The empirical headline would be much more convincing with per-split, per-cell ranges and an analysis of gate false discoveries under validation-to-test shift.","section":"§3.4, §3.3, App. D"}],"minor_comments":[{"comment":"The label '30 GPT-5.2 cells' is inconsistent with the 6×6=36-cell main grid; please clarify whether the Chronos-2 single-seed rows are excluded from the budget sweep and why.","section":"Table S11"},{"comment":"In Figure 2a the legend appears to list 'CRAFTER' twice without clearly distinguishing the dagger variants; please fix the legend rendering.","section":"Figure 2"},{"comment":"App. D states that seed-std is quantified on the DeepSeek grid, while Table S13's caption says the reproduction is single-seed; please clarify the seeding actually used for the DeepSeek runs.","section":"App. D, Table S13"},{"comment":"The phrase 'roughly doubling the improvement achieved by the corrector alone' should define what 'the corrector alone' refers to (COV? a corrector with no accepted features?) at first use, since the corrector set includes an identity option.","section":"Abstract"},{"comment":"The statement in §4 that the comparison is 'fair by construction' is too strong: the external systems are run inside the CRAFTER harness with their native selection mechanisms replaced by the shared gate, so what is measured is each generator under CRAFTER's selection rule, not the external system as originally designed. App. E acknowledges this, but the main-text phrasing should be softened.","section":"§4, App. E"}],"recommendation":"major_revision","confidential_remarks":"The paper is well executed and the appendices are unusually detailed, but the abstract and Section 5.1 overstate the per-cell strength of the results. I would like to see the aggregate claims explicitly qualified and the validation-to-test transfer risk quantified before publication. The use of future-dated references is consistent with the arXiv date and does not affect my assessment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"X, here's my take on 2608.05207.\n\nThe real contribution is the setup: a source-blind validation gate that lets you attribute forecast gains to the feature source alone, plus a two-generator pipeline (compositional search + LLM) for corrective features on frozen backbones. The paper does that evaluation honestly: six datasets, six frozen backbones, rolling-origin splits, three seeds, same harness for all methods, and negative cells reported. The core empirical finding — correction helps when the backbone leaves exploitable residual structure, is neutral or harmful when the series is saturated — is supported by the per-cell grid and the scatter in Fig S1. The transferability replay (Sec 5.5) is a nice piece of evidence, and the cross-LLM robustness check is the right control.\n\nThe soft spots are concentrated in the abstract. \"Surpasses every dedicated feature-engineering system at every feature budget\" is an aggregate statement, not a per-cell one. The paper's own Table S11 gives a 60% win rate against the best external at K=5, and there are cells where CRAFTER makes things worse (BIZITOBS/Chronos: 24.6 to 31.4; several FAVORITA and M5 rows at or below RAW). The authors know this: App D reports median split-std ~3.2 versus seed-std 0.49, so the between-split temporal variance dominates and the three-split average is fragile. The gate threshold tau=0.05 is a low bar, and the paper explicitly says the NONE option is a validation-time safeguard, not a test guarantee. None of this kills the work, but it means the headline claim is not established as a per-cell property, and no code or data is shipped, so the adaptation of TSFRESH/CAAFE/LLM-FE cannot be checked.\n\nWho should read this: anyone working on post-hoc correction of frozen forecasters or LLM-based feature engineering. The instrument idea and the headroom characterization are worth carrying forward. I would not cite the abstract's blanket claim; I would cite it for the regime result and the evaluation protocol.\n\nMy recommendation: send to peer review. The paper deserves serious refereeing, and the referee reports should push the authors to qualify the abstract, report per-cell win rates as prominently as the aggregates, and release the code and adapted baseline definitions.","headline":"Careful empirical study with an oversold headline: the headroom characterization holds, the blanket 'surpasses every system at every budget' claim does not.","tokens_in":30607,"tokens_out":3163,"would_cite":true,"duration_ms":32884,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that on six public datasets and six frozen backbones, its residual-mining agent CRAFTER beats every dedicated feature-engineering system at every feature budget, roughly doubling the lift of the corrector alone and…","keywords":["corrective feature discovery","residual correction","frozen forecaster","large language model","feature engineering","time series forecasting","source-blind gate","validation headroom"],"falsifier":"Re-run the exact harness on a seventh public dataset with a frozen backbone and an abrupt regime shift between the validation window and the test window: if CRAFTER's test wMAPE is worse than the uncorrected backbone on a substantial share of cells, or if at any feature budget K an external generator beats it, the empirical headline fails. Concretely, find one (dataset, backbone, K) cell where CRAFTER loses to the best external system under the shared gate.","tokens_in":29588,"feed_emoji":"🤖","tokens_out":4965,"duration_ms":42843,"temperature":0.7,"pith_summary":"The paper studies corrective feature discovery: instead of repairing a frozen pretrained forecaster by fine-tuning, it mines interpretable features of the forecaster's residual—the structured ways it fails—and feeds them to a lightweight post-hoc corrector. The central claim is that a source-blind pipeline, which lets a compositional search and a large language model propose candidate features and admits them through one validation-grounded gate, outperforms dedicated feature-engineering systems on all six public datasets and six frozen backbones at every feature budget. The gain roughly doubles what the corrector alone achieves and cuts the weakest backbones' error by up to 27%, because the accepted features name mechanisms the backbone misses rather than re-describing the data. A reader should care because it offers a cheap, interpretable way to correct deployed forecasters without retraining, and a controlled way to attribute forecast improvements to the feature source alone.","feed_headline":"Residual mining doubles forecast-correction gains on six benchmarks","feed_subtitle":"A source-blind gate lets an LLM and a search mine the backbone's residual, lifting weak forecasts by up to 27%.","key_machinery":"The load-bearing mechanism is the source-blind acceptance gate. A candidate feature is accepted only if its Spearman correlation with the still-unexplained validation residual exceeds a fixed threshold τ=0.05, if it is not collinear (|ρ|≤0.75) with an already-accepted feature, and if per-family and per-round budgets hold. Because the same test applies to every candidate regardless of which generator produced it, the gate makes the two generators commensurable and also turns the pipeline into an instrument: external feature-engineering systems run through the identical gate, corrector, and budget, so forecast differences are attributable to the feature source alone. A validation-selected corrector (identity, additive, or multiplicative gradient-boosted trees) then applies the survivors or leaves the forecast unchanged.","core_discovery":"On the paper's own terms, the discovery is that a frozen forecaster's residual contains exploitable, model-specific structure that can be mined as interpretable features, and that mining it with two heterogeneous generators—a compositional Monte-Carlo tree search over raw channels and an LLM proposing named combinations, flags, and code—through a single source-blind gate produces consistent gains. Across the 6×6 grid of datasets and backbones, CRAFTER surpasses every dedicated feature-engineering system at every feature budget, roughly doubling the improvement of the corrector alone and reducing the error of the weakest backbones by up to 27%. The gain is governed by residual headroom: it is large on weak backbones with informative forecast covariates, where LLM-named features are the differentiator, and neutral or harmful once a strong backbone saturates the series.","pith_inferences":["The headroom rule implies a practical diagnostic: measuring the correlation between a frozen backbone's validation residual and available covariates could predict in advance whether correction will pay, without running the full pipeline.","The gate's blindness to source suggests it could be repurposed as an audit tool for LLM-proposed features in other domains, separating whether a gain comes from feature semantics or from the proposer's priors.","The finding that LLM-code features are almost always pruned hints that executable-code proposals are a weak lever in this setting; a testable extension would be to restrict proposers to named combinations and flags to reduce variance.","A natural stress test the paper does not run is a dataset where the residual structure shifts abruptly between validation and test windows, directly probing the weakest assumption about validation-time safeguards."],"forward_implications":["If the headroom condition holds generally, the deployment rule is to apply corrective feature discovery only when validation residuals remain structured; a larger feature budget amplifies but does not create gains.","The source-blind pipeline doubles as an evaluation instrument, so any future feature generator can be tested under identical conditions and its contribution attributed to the feature source alone.","Gains persist when the corrected backbone is fine-tuned rather than used zero-shot, so corrective features act as a deployment-time layer that survives backbone adaptation.","Features learned on one backtest window transfer to later windows with mild decay, supporting a lightweight feature-memory that amortizes search across time.","Weak backbones benefit most, suggesting corrective features as a cheap alternative to per-series fine-tuning in large-scale frozen-model deployments."],"supporting_citations":[{"why":"Supplies the TSFRESH statistical feature bank baseline that CRAFTER must outperform under the shared gate and corrector.","marker":"(Christ et al., 2018)"},{"why":"Supplies the CAAFE baseline, an LLM-based feature-code generator used as one of the three dedicated feature-engineering systems.","marker":"(Hollmann et al., 2023)"},{"why":"Supplies the LLM-FE baseline, an LLM-guided evolutionary search that serves as the third external feature-engineering comparison.","marker":"(Abhyankar et al., 2025)"},{"why":"Provides the Chronos frozen backbone used across the 6×6 dataset-backbone grid.","marker":"(Ansari et al., 2024)"},{"why":"Provides the Timer frozen backbone, one of the six pretrained forecasters whose residual CRAFTER corrects.","marker":"(Liu et al., 2024)"},{"why":"Provides the Moirai frozen backbone, another of the six forecasters in the evaluation grid.","marker":"(Woo et al., 2024)"},{"why":"Establishes the classic residual-modeling idea of predicting a fixed forecaster's errors with a second learner, which motivates the corrective feature setting.","marker":"(Zhang, 2003)"}],"fun_headline_variants":["Mining frozen forecaster residuals doubles correction gains","LLM and search mine model residuals, cutting weak errors 27%","Source-blind feature gate doubles forecast fix gains","Residual headroom: CRAFTER lifts weak backbones by 27%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that features accepted because they correlate with the validation residual, and correctors chosen by validation wMAPE, keep helping on later, out-of-sample test windows; the paper itself flags that the validation gate is a safeguard, not a test guarantee.","fun_headline_variants_meta":{"raw":{"variants":["Mining frozen forecaster residuals doubles correction gains","LLM and search mine model residuals, cutting weak errors 27%","Source-blind feature gate doubles forecast fix gains","Residual headroom: CRAFTER lifts weak backbones by 27%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1530,"prompt_tokens":973,"completion_tokens":557,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":487}},"tokens_in":589,"tokens_out":557,"duration_ms":6166,"temperature":1.0,"reasoning_tokens":487,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T18:14:17.596774+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the exact harness on a seventh public dataset with a frozen backbone and an abrupt regime shift between the validation window and the test window: if CRAFTER's test wMAPE is worse than the uncorrected backbone on a substantial share of cells, or if at any feature budget K an external generator beats it, the empirical headline fails. Concretely, find one (dataset, backbone, K) cell where CRAFTER loses to the best external system under the shared gate.","supporting_citations":[{"cited_title":", journal=","cited_arxiv_id":null,"evidence_quote":"Supplies the TSFRESH statistical feature bank baseline that CRAFTER must outperform under the shared gate and corrector."},{"cited_title":"Peter , journal=","cited_arxiv_id":null,"evidence_quote":"Establishes the classic residual-modeling idea of predicting a fixed forecaster's errors with a second learner, which motivates the corrective feature setting."}],"review_version":1}