{"id":"a5fd886d-ce46-4d10-a791-d63fd7b22793","arxiv_id":"2601.15503","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"On 30 Maine lakes, ridge regression with MICE-imputed covariates can stay within 5% of full-history forecasting accuracy using roughly 64 recent samples and one predictor per lake.","lead":"This paper tests how little recent data and how few measurements are needed to forecast lake water clarity (Secchi depth) within 5% of a full-history model, using 30 data-rich Maine lakes. It reports that about 64 recent samples and a single predictor (often dissolved oxygen status) typically suffice in this dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-block leakage in imputation and configuration selection likely inflates the minimal-resource rule; nested evaluation is required.","rationale":"The reader's weakest assumption is exactly the leakage-unsafe protocol: MICE fit on the full covariate matrix and minimal n/k selected on test-block errors. My review of the full text confirms this is the most load-bearing issue. The central claim (64 samples, 1 predictor) is the abstract's headline contribution and the practical recommendation; if the evaluation leaks test information, the numbers are optimistic and the rule may not transfer. The paper itself lists 'a leakage-safe, nested train–validation–test evaluation in which imputation is fit on the training fold only' as future work (Section V), which is an explicit admission that the current pipeline is not fully valid. I do not find a stronger concern: the methodological framing (joint feasibility) is coherent, the negative R2 issue is secondary given the nMAE metric, and the MDI-for-ridge mismatch is a known limitation acknowledged in the Discussion. The concrete test I propose would settle whether the leakage materially changes the minimal configurations. If it does not, the rule stands; if it does, the paper should be revised to report leakage-safe numbers. Since the reader already reached CONDITIONAL, my recommendation is UNCHANGED — the verdict remains CONDITIONAL pending this test. I agree with the reader's identification of the weakest assumption; I see no additional load-bearing flaw that would move the verdict to REJECT or UNVERDICTED.","tokens_in":14068,"tokens_out":2047,"duration_ms":24679,"concrete_test":"Implement a nested leakage-safe evaluation for each of the 30 lakes: split the pre-test period into training (earliest 80%) and validation (most recent 20%); fit MICE on the training fold only, apply the fitted imputer to validation and test covariates; train ridge on the training fold, evaluate the reference error on validation; choose minimal (n,k) on validation using a 5% tolerance relative to that validation reference; finally evaluate the chosen configuration on the held-out test block. Compare the resulting median hat-n and hat-k, and the distribution of feasible pairs, with the paper's reported median n=64, median k=1. If the median shifts upward (e.g., hat-n > 64 or hat-k > 1) or if a majority of lakes no longer meet the 5% target on the true test block, the claimed rule is an artifact of leakage. Also re-run feature ranking (MDI and forward selection) using only training-fold da","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim — that ~64 recent samples and one predictor per lake suffice for 5% of full-history accuracy — rests on a protocol that leaks test information in two places. First, MICE is fit on the full covariate matrix, including the held-out test block and future records, so imputed feature values in the training window can encode test-block structure. Second, the minimal n and k are selected using nMAE on the same test block that defines the 5% target; the feasibility function f(n,k) is evaluated and the lexicographically smallest feasible pair is chosen on data that also sets the tolerance. This is a form of adaptive selection/optimism: the configuration is selected to pass on the very errors used to judge it. The paper itself acknowledges in Future Works (e) that a 'leakage-safe, nested train–validation–test evaluation in which imputation is fit on the training fold only' is still to be done. Until such an evaluation is run, the '64 samples, 1 predictor' rule cannot be considered a valid out-of-sample result. The concern is not about consensus or style; it is a correctness risk internal to the evaluation protocol. The manuscript is transparent about this limitation, but the headline numeric claims are nevertheless unsupported as general guidance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies Secchi Disk Depth (SDD) forecasting for 30 data-rich Maine lakes with irregular, missing multivariate time series. Missingness is handled by MICE imputation, and among six candidate forecasters ridge regression is selected as best. The authors define a normalized MAE (nMAE), then quantify a minimal sample size n* (mean approximately 176 under a recent-history backward protocol), a minimal feature set (four features), and introduce a joint feasibility function f(n,k) that selects, per lake, the lexicographically smallest pair (n,k) staying within 5% of the full-history, full-feature reference nMAE. The headline result is a median minimally sufficient configuration of n=64 recent samples and k=1 predictor, with OXIC dominant. The paper is transparent about its limitations and lists a leakage-safe nested evaluation as future work, but the current protocol fits MICE on the full covariate matrix (including the test block) and selects n,k using test-block errors, so the empirical claims are not out-of-sample validated.","tokens_in":1448,"tokens_out":1862,"duration_ms":70095,"significance":"If the results survive a leakage-free evaluation, the proposed joint feasibility rule would be a practically useful tool for lake-monitoring design: it unifies sample-size and feature-selection decisions under a single, interpretable accuracy target, and it addresses a real gap in the water-quality forecasting literature. The paper also contributes a scale-normalized nMAE metric that is appropriate for cross-lake comparison, and it openly acknowledges several of its own weaknesses. However, the central numeric claims (especially n=64 and k=1) currently rest on an evaluation protocol that can leak test information into both imputation and configuration selection, so the contribution is not yet demonstrated.","major_comments":[{"comment":"The evaluation is not leakage-safe. MICE is fit on the full covariate matrix Z_l, which includes the held-out test block and future records, so imputed covariate values in the training window can encode test-period structure. In addition, n* and k* are defined by thresholding nMAE computed on the very same test block used to set the 5% tolerance (Sec. II-D and II-F), so the minimal configuration is selected on the test set. The paper's Future Works (e) explicitly concedes that a leakage-safe nested evaluation is still needed. Until that is run, the headline numbers (n*~176, median n=64, k=1) cannot be treated as valid out-of-sample estimates.","section":"II-B and II-F; Future Works (e)"},{"comment":"The feature-selection loop also selects on the test set. The MDI importance ranking is computed and then, for each k, the ridge forecaster is evaluated by nMAE on the test block T_test to pick k*_l. This is a second channel of test-set adaptation beyond the imputation leakage. Moreover, the paper does not state whether MDI comes from a RandomForest or a Ridge model, and no evidence is given that MDI importances transfer to ridge regression. The authors should either derive the ranking from ridge coefficients or perform a nested selection where the ranking and k are chosen on validation data only.","section":"II-E"},{"comment":"The minimally sufficient configuration depends on the lexicographic tie-break, which minimizes n first and only then k. This is an arbitrary design choice that can substantially change the reported n and k; for example, a Pareto-minimal or cost-weighted rule might choose a slightly larger n to achieve k=1. The paper provides no sensitivity analysis for the tolerance (5%) or the tie-break, and lists such an analysis only as future work (d). Without it, the headline 64 samples and 1 predictor is not robust.","section":"II-F"},{"comment":"The manuscript uses two different minimal quantities without clearly distinguishing them: n*~176 is the mean minimal sample count for the full-feature model, while n=64 is the median joint n when k is also minimized. The abstract highlights the latter but the former appears in the results. Moreover, no confidence intervals or per-lake distributions are provided for n and k, and 10 of 30 lakes revert to the full-data/full-feature configuration. Such high variability and a small convenience sample of 30 record-rich lakes do not support a strong general per-lake rule.","section":"III-a,d"}],"minor_comments":[{"comment":"Clarify that n*~176 is the mean minimal count for the full-feature model, while n=64 is the median joint result; otherwise readers may conflate the two.","section":"Abstract / III"},{"comment":"The y-axis label 'percentage of lakes' should state the denominator is the 20 lakes with k=1, not all 30 lakes.","section":"Fig. 3"},{"comment":"Specify which model produces the MDI importances. The text mentions 'Reference Ridge model' and then 'MDI importances,' which are associated with tree-based models; this needs disambiguation.","section":"II-E"},{"comment":"The nMAE formula is not numbered and the notation 'nMAE_final' is not used consistently; use a single notation for the full-history reference.","section":"II-D"},{"comment":"There are minor typos such as 'T est' and inconsistent capitalization of 'Secchi Disk Depth' vs 'Secchi disk depth'; please polish the text.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its main weakness, but that weakness is load-bearing: the empirical minimal-resource rule is derived from a protocol with test leakage in both imputation and configuration selection. I do not think this is rejectable if the authors rerun the experiments with a nested, leakage-safe evaluation and report sensitivity to the tie-break and tolerance; however, the current version's headline numeric claims are not supported. The 30-lake convenience sample and lack of code/data also limit reproducibility. I would encourage the editor to request the nested evaluation as a condition of revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a clearly written applied ML paper with an honest limitations section, but its headline numbers (median 64 recent samples, 1 predictor) are not out-of-sample valid. The paper itself flags the leakage in Future Work (e), so it is not a hidden flaw. It means the practical rule should be read as a fitted summary of these 30 lakes, not a general guideline.\n\nWhat is genuinely new and good: the joint feasibility framing—simultaneously selecting recent-history length and predictor count under a fixed accuracy tolerance—is a reasonable extension over prior work that varies these in isolation. The nMAE metric for cross-lake comparability is sensible, the exclusion of chlorophyll to avoid target leakage shows care, and the per-lake results in Table III are transparent. The writing is clear, and the limitations are acknowledged without overclaiming: MDI bias, tolerance/tie-break dependence, and the restriction to 30 data-rich lakes.\n\nSoft spots, in proportion: the central numeric claims are not leakage-safe. MICE is fit on the full covariate matrix, including the held-out test block, so imputed training values can encode test-period structure. More importantly, the minimal n and k are selected using nMAE on the same test block that defines the 5% tolerance. This is adaptive selection: the configuration is chosen to pass on the very errors used to judge it, which inflates the optimism of the '64 samples, 1 predictor' result. There is also no code or data, no confidence intervals on the learning curves, and all methods report negative R², which suggests the forecasting task is hard and the 5% tolerance may be close to noise. The MDI importance ranking is computed with a tree-based model for a ridge forecaster, which is a methodological mismatch, though it only affects feature ordering, not the core evaluation.\n\nAre these fatal? I do not think so, because the paper is honest about its preliminary nature and the leakage-safe nested evaluation is explicitly left as future work. What it means is that the contribution is a case study and a methodological suggestion, not a validated sampling rule. The 64-and-1 numbers are best regarded as descriptive statistics of these 30 lakes under this protocol, not as a transferable finding.\n\nWho this is for: researchers in environmental ML who want a concrete demonstration of how learning curves and feature selection can be unified, and anyone looking for a clean teaching example of evaluation leakage. It deserves a serious referee: the real dataset, clear prose, and acknowledged limitations make it worth the effort of a major revision that adds a leak-safe evaluation and ideally releases code and data.\n\nRecommendation: send it to peer review with the expectation of major revision, not desk rejection.","headline":"Honest applied ML case study whose headline '64 samples, 1 predictor' rule is an in-sample fitted summary, not a validated out-of-sample result, because imputation and configuration selection leak test information.","tokens_in":14830,"tokens_out":2183,"would_cite":false,"duration_ms":25691,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A joint feasibility rule shows that lake water clarity forecasts can match full-history accuracy within 5% using about 64 recent samples and a single predictor per lake.","keywords":["lake water quality","Secchi disk depth","time-series forecasting","missing data imputation","MICE","ridge regression","sample size determination","feature selection"],"falsifier":"Re-run the same backward protocol but fit MICE only on the training fold and apply it to the test block, and select (n,k) using only validation data; if the median minimal configuration moves substantially above 64 samples or 1 predictor, the paper's central rule is an artifact of leakage.","tokens_in":13877,"feed_emoji":"🏞️","tokens_out":1885,"duration_ms":22133,"temperature":0.7,"pith_summary":"The paper asks how little data and how few measurements a lake water-quality forecaster needs while staying within 5% of the accuracy of a model trained on full history and all available features. Using three decades of volunteer-collected Secchi disk depth records from 30 Maine lakes, the authors impute missing covariates with MICE, pick ridge regression as the best of six forecasters, and then search over training-window length and feature count together. They report that a typical lake needs only about 64 of its most recent observations and one predictor—most often the oxic status variable—to meet the target. If this holds, monitoring programs could dramatically cut sampling effort and still produce reliable clarity forecasts.","feed_headline":"64 samples, one sensor match full lake forecasts","feed_subtitle":"A joint feasibility rule finds that a short recent history and a single predictor capture 95% of full-data accuracy across 30 Maine lakes.","key_machinery":"The central object is the joint feasibility function f(n,k), which formalizes 'good enough' as staying within 5% of a lake-specific reference nMAE achieved by ridge regression on all pre-test data and all features. The procedure combines MICE for missing covariates, a backward recent-history training protocol that predicts a held-out last-five-years test block, MDI-based feature ranking followed by greedy forward selection, and a lexicographic rule that prioritizes minimal n then minimal k. This yields a parameter-free, per-lake minimally sufficient configuration and an actionable aggregate target for sampling effort and measurement priorities.","core_discovery":"The paper introduces a joint feasibility function f(n,k) that is 1 when a ridge forecaster trained on the most recent n pre-test observations with the top-k ranked predictors achieves test nMAE within a 5% tolerance of the full-data, full-feature reference. Taking the lexicographically smallest feasible (n,k) per lake, the authors find a median minimal configuration of 64 recent samples and 1 predictor, with OXIC selected in 90% of the 20 lakes where a single predictor sufficed. Averaged over lakes, the minimal sample count alone is about 176, and a four-feature subset (TPEC, OXIC, SCHMIDT, CONDUCT) matches the thirteen-feature baseline within tolerance.","pith_inferences":["The 64-samples/1-predictor result plausibly reflects diminishing returns in learning curves: after a few seasonal cycles, additional history adds little signal for ridge regression, so the finding may hold qualitatively even for other lakes, though the exact numbers likely shift.","A natural testable extension would be to apply the same joint feasibility procedure to other targets (e.g., chlorophyll) and other imputation methods to see whether the minimal configuration is robust or an artifact of MICE's assumptions.","The paper's own flagged future work — a leakage-safe nested evaluation — suggests the central numbers should be treated as best-case estimates until the imputation and model-selection pipeline is re-validated without test-block information."],"forward_implications":["Volunteer monitoring programs could focus effort on roughly 60 recent observations per lake and a single priority measurement (e.g., oxic status) rather than maintaining full-history, thirteen-feature records.","The 5% tolerance can be tuned: a looser tolerance would shrink the required sample count and feature set, while a stricter tolerance would grow them, giving program managers a direct accuracy-versus-effort dial.","The per-lake feasibility output (hat-n, hat-k) translates directly into operational targets, so agencies can reallocate staff time toward instrumentation or targeted lab campaigns.","If the result generalizes beyond the 30 data-rich lakes, the same joint rule could guide monitoring design for data-poor lakes where remote sensing is unreliable."],"fun_headline_variants":["64 lake samples, 1 predictor hit 95% accuracy","Ridge regression: 64 samples, single sensor match","Lake forecasting shrinks to 64 samples, one variable","Joint feasibility rule: 64 recent samples, 1 predictor"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The evaluation protocol is leakage-free: the MICE imputer is fit on the full covariate matrix including test-block and future records, and the minimal n and k are selected using test-block errors that also define the 5% target, so if test information seeps into training the reported '64 samples, 1 predictor' rule may not hold out of sample.","fun_headline_variants_meta":{"raw":{"variants":["64 lake samples, 1 predictor hit 95% accuracy","Ridge regression: 64 samples, single sensor match","Lake forecasting shrinks to 64 samples, one variable","Joint feasibility rule: 64 recent samples, 1 predictor"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1187,"prompt_tokens":824,"completion_tokens":363,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":294}},"tokens_in":568,"tokens_out":363,"duration_ms":5123,"temperature":1.0,"reasoning_tokens":294,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T08:49:45.323351+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same backward protocol but fit MICE only on the training fold and apply it to the test block, and select (n,k) using only validation data; if the median minimal configuration moves substantially above 64 samples or 1 predictor, the paper's central rule is an artifact of leakage.","supporting_citations":[],"review_version":1}