{"id":"8dfe5614-9f10-4259-8bf6-3712be9c32af","arxiv_id":"2607.25367","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A leak-free stacking protocol and mask-conditioned synthetic generation improve sand-boil segmentation to 0.707 IoU, but stacking underperforms the best single model and synthetic gains come only from label-fidelity filtering.","lead":"This paper tests whether cross-validated stacking and synthetic data improve sand-boil segmentation without hidden data leakage, and reports that stacking does not beat the best single model. It also introduces a leak-free protocol for mixing synthetic images into training and a mask-conditioned generator that supplies labels at zero annotation cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Architecture selection appears to use the held-out test set, so the headline 0.707/0.681 numbers are not yet shown to be unbiased.","rationale":"The reader's formal weakest assumption concerns pseudo-label circularity in the synthetic pool. That is a real risk, but it bears mainly on the filtered synthetic-pool gain (0.718 vs 0.707), which the authors themselves label suggestive and whose spread is as wide as its mean. The test-set-informed architecture selection is more load-bearing because it directly threatens the headline 0.707-vs-0.608 improvement, the stack-vs-single negative result, and the paper's 'unbiased by construction' claim for the protocol. The reader does mention test-set-informed selection in the rationale, so there is partial agreement, but it is not the formally identified weakest assumption. The proposed re-run is feasible with the released artifacts and would settle whether the central numbers survive a clean selection firewall. I therefore recommend keeping the verdict CONDITIONAL: the methodology is promising and the negative ensemble result may well be real, but the quantitative claims need to be re-derived without touching the test set during model selection.","tokens_in":33542,"tokens_out":9505,"duration_ms":102381,"concrete_test":"Re-run the architecture search of Section III-D using only an internal validation split (e.g., a fixed 15% of the 199 training images, or five-fold CV within the training set) to select the family champions and the scSE gate. Freeze all architectural choices before computing any number on the 46-image test set, then evaluate the final champion and the stack on the test set exactly once. If the resulting test IoUs fall materially below 0.707/0.681, or if the stack-vs-single ordering changes, the reported numbers include test-set selection bias.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central empirical claims rest on architecture selection performed on the same 46-image held-out test set. Section V-A states that the architecture search keeps 'the best performer in each family' and that 'Table I reports the resulting family champions on the held-out test set'; Section III-D says eleven networks were searched and only the best in each family was retained. The test set is therefore used to choose the base learners and the scSE attention variant (Table VII), and then the same test set is used to report the final 0.707 IoU and the stack-vs-single comparison (0.681 vs 0.694). Algorithm 1's claim that 'the held-out test set is first touched at line 6' is thus false for the overall pipeline: model selection touched it earlier. With only 46 test images, selecting among 11 architectures plus several attention variants on test IoU can inflate the reported numbers and can change which member is 'best' or whether stacking closes the gap. The Cross-dataset DeepCrack check is helpful, but it does not repair the sand-boil selection leakage. If Table I's 'single 85/15 split' was actually an internal validation split, this concern would be weakened, but the text as written identifies the held-out test set as the selection ground truth.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a sand-boil segmentation framework built around a 'leak-free' five-fold cross-validated stacking protocol with per-architecture temperature calibration, a per-fold filter that excludes synthetic descendants of held-out parents, a synthetic hard-negative phase, and a mask-conditioned synthesis route (MaskCN). On 199 training / 46 test images, the Updated SandBoilNet reaches 0.707 IoU versus 0.608 for the published baseline; the calibrated stack reaches 0.681 IoU versus 0.694 for the best fold-averaged member; a label-fidelity-filtered synthetic pool lifts the champion to 0.718 versus a 0.707 control. The paper reports that stacking does not improve on the best single model and that the hard-negative phase does not help, framing these as honest negative results. The central methodological claims are (1) that the protocol closes two named leakage paths and therefore gives unbiased ensemble measurements, and (2) that the updated backbone improves on the published baseline.","tokens_in":33870,"tokens_out":5779,"duration_ms":57065,"significance":"If the protocol is valid, the paper makes a useful contribution to scarce-data defect segmentation: a carefully specified out-of-fold stacking pipeline, a simple parent-tagging filter for synthetic data, per-architecture calibration, extensive meta-learner ablations, bootstrap uncertainty intervals, a cross-dataset check on DeepCrack, and a reproducible artifact with code, checkpoints, and manifests. The reporting of negative results is candid and the sensitivity analysis is more thorough than typical for this area. However, the central validation is undermined by the fact that architecture selection is performed on the held-out test split, which contradicts the 'leak-free/unbiased by construction' claim and calls the headline numbers into question. The synthetic-label circularity is a further load-bearing concern for the synthetic-augmentation contribution.","major_comments":[{"comment":"The family champions are selected on the held-out test set: Section V-A states that the search keeps 'the best performer in each family' and that 'Table I reports the resulting family champions on the held-out test set'; Table I's caption identifies a single 85/15 split, and Table VII uses the same kind of split to choose the scSE attention variant. The same 46-image test set is later used to report the champion's 0.707 IoU and the stack-vs-single comparison (0.681 vs 0.694). Algorithm 1's assertion that 'the held-out test set is first touched at line 6' is therefore false for the overall pipeline: model selection touched it earlier. On 46 test images, selecting among 11 architectures plus attention variants on test IoU can inflate the reported numbers and can change which member is 'best' or whether stacking closes the gap. Please state clearly whether the 85/15 split is an internal val","section":"Section V-A, Table I; Algorithm 1"},{"comment":"The synthetic positive masks are generated by thresholding the public SandBoilNet probability map through Convex_Hull_Annotator, and the label-fidelity filter admits a synthetic pair only if a held-out segmenter reaches IoU >= 0.70 against that same pseudo-label. The filtered-pool gain (0.718 vs 0.707, Section V-G) may therefore reflect reinforcement of the baseline's biases rather than genuinely new signal. This is load-bearing for the synthetic-augmentation contribution. Please provide independent validation of a sample of synthetic masks (manual review, or agreement with a segmenter trained without SandBoilNet-derived labels), and report the rejection rate of the fidelity filter. Re-deriving labels with 'the strongest segmenter available here' does not remove the concern if that segmenter inherits the same training labels.","section":"Section III-B; Section V-G"}],"minor_comments":[{"comment":"The 'best single' model differs between Table I (Updated SandBoilNet, 0.707) and Table II (SegFormer, 0.694) because one is a single-split comparison and the other uses fold-averaged predictions. This is stated in captions but should be flagged prominently in the main text to avoid confusion.","section":"Tables I and II"},{"comment":"The label-fidelity admission criterion ('all five cross-validation folds of a held-out segmenter reach IoU 0.70 against the image's own mask') is ambiguous: which segmenter, and what does 'all five folds' mean for a held-out segmenter? Please clarify.","section":"Section III-B"},{"comment":"The IoU values shown in Figure 5 come from Table I, which is the architecture-search score on the test split. The figure should be labeled as such so readers do not mistake these for unbiased estimates.","section":"Figure 5"},{"comment":"The three-seed filtered-pool gain of 0.012 ± 0.012 is appropriately described as suggestive, but reporting the individual per-seed values would strengthen this transparency.","section":"Section V-G"}],"recommendation":"major_revision","confidential_remarks":"The architecture-selection-on-test issue is the primary gate. If the authors can demonstrate that the 85/15 split in Table I and Table VII is an internal validation split rather than the 46-image test set, the concern is substantially weakened. The synthetic-label circularity is secondary but still needs independent validation for the synthetic-augmentation claim. I do not see grounds for rejection if these experimental-validation issues can be fixed, because the negative stacking result and the leak-free protocol are otherwise carefully executed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: a careful, honest empirical study of stacking and synthetic augmentation for a scarce-data defect, where the main findings are negative — stacking doesn't beat the best member, hard-negative mining hurts — and those negative results are the most useful part. The paper deserves a serious referee, but the authors need to fix a serious ambiguity about what split the architecture search used.\n\nThe real contributions: a clean per-fold filter that prevents synthetic descendants of held-out parents from leaking into training folds — simple and right; a per-pixel cross-validated stacking protocol with per-architecture temperature calibration, fitted entirely on out-of-fold predictions; and a genuinely honest account of what does and doesn't help in this regime. The reproducibility discipline is unusually good: seeded folds, source_image_id manifests, released code and checkpoints.\n\nThe biggest soft spot is the architecture selection. Section V-A says Table I reports family champions 'on the held-out test set', but the table caption says 'on a single 85/15 split'. If it was the real test set, then the headline 0.707 and the stack-vs-single comparison are selected on the test set, and Algorithm 1's claim that the test set is only touched at line 6 is false for the overall pipeline. If it was an internal split, the text is misleading. Either way, this needs to be stated unambiguously. With 46 test images, selecting among 11 architectures plus attention variants can easily inflate the winner.\n\nSecond soft spot: the synthetic ground-truth masks come from thresholding the public SandBoilNet probability map, and the fidelity filter checks agreement with those same pseudo-labels. So the observed synthetic gain (0.718 vs 0.707) may partly reflect the model learning SandBoilNet's biases. MaskCN avoids this, but the main augmented pool doesn't. Worth stating as a limitation; the paper doesn't really.\n\nThe negative results are the paper's main value, and they look solid: eight meta-learners, three CV designs, bootstrap intervals. The 0.681 vs 0.694 gap is within the bootstrap noise [0.637, 0.745], and the authors say so — that's the right level of honesty.\n\nWho's this for: anyone working on scarce-data segmentation, synthetic augmentation, or stacking. It won't reorganize the field, but it's a useful counterweight to the usual 'ensembling helps' narrative.\n\nRecommendation: send to peer review. The reviewer should push on the split question and ask for a version where architecture selection is done on training folds only, or clearly shown to use an internal split. The paper is close to being a solid negative-result study.","headline":"A careful, honest negative-result study of stacking and synthetic augmentation for a scarce-data segmentation task — worth refereeing, but the architecture-selection split must be clarified before the headline numbers can be trusted.","tokens_in":34359,"tokens_out":4015,"would_cite":true,"duration_ms":37901,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Under a leak-free cross-validated stacking protocol that closes both the tuning-leak and synthetic-descendant leak, a five-backbone sand-boil ensemble (IoU 0.681) does not beat the best single model (0.694), while the modernized single netw","keywords":["sand boil segmentation","earthen levee inspection","cross-validated stacking","temperature calibration","synthetic data leakage","parent-image filter","label fidelity filtering","mask-conditioned synthesis"],"falsifier":"Re-train the champion on a synthetic pool whose labels are generated by an independent segmenter or human-verified masks, using the same diffusion source and filter logic; if the IoU gain over the 0.707 control disappears or reverses, the filter was selecting for agreement with the baseline's own errors, not for label fidelity. Concretely, compare the current filtered-pool result against a pool filtered on human-verified IoU rather than the held-out segmenter's agreement with pseudo-labels.","tokens_in":33443,"feed_emoji":"🌊","tokens_out":6002,"duration_ms":54942,"temperature":0.7,"pith_summary":"Sand boils are early visible signs of hidden erosion inside earthen levees, and deep networks are taught to spot them in inspection photos — but annotated examples are rare, and two common workarounds quietly inflate reported accuracy: tuning ensemble weights on the images that will later be scored, and training on synthetic images that descend from the very photographs held out for testing. This paper builds a segmentation pipeline that closes both loopholes: every synthetic image points to its real parent, a per-fold filter blocks descendants of held-out parents, and five backbones are combined only through out-of-fold predictions, calibrated by one temperature scalar per architecture. On a held-out test set, the modernized single model reaches IoU 0.707 versus 0.608 for the published baseline re-evaluated on the same split. Under the leak-free protocol, however, the calibrated five-model stack reaches only 0.681 against 0.694 for the strongest member — the ensemble does not help, and the paper attributes this to a mean pairwise error correlation of 0.894 among members. The point a sympathetic reader takes away is that honest measurement, not the score itself, is the contribution: it separates transfers that help scarce-data defect segmentation (a modern encoder, label-fidelity-filtered synthetic data) from those that do not (stacking, hard-negative mining, higher resolution).","feed_headline":"Stacking loses to its best member under leak-free evaluation","feed_subtitle":"A corrected protocol for sand-boil segmentation shows which tricks really help scarce-data models.","key_machinery":"The central object is the leak-free cross-validated stacking protocol: each of five encoder–decoder backbones is trained once per fold, predicts only on its held-out fold, and the resulting out-of-fold cube is recalibrated by one temperature scalar per architecture and combined by a per-pixel logistic-regression meta-learner fitted exclusively on those out-of-fold predictions. The companion mechanism is the parent-image filter: each synthetic record carries a source_image_id pointer to its real parent, and a per-fold set operation excludes any synthetic descendant whose parent sits in that fold's validation split, closing the synthetic-descendant leakage path in O(1). A second, complementary","core_discovery":"This paper claims that when both common leakage paths are closed, the measured value of ensembling in a scarce-data defect-segmentation regime is negative: the calibrated stack (0.681 IoU) falls short of the best fold-averaged single model (0.694 IoU), a result reproduced by eight meta-learner families and explained by a mean pairwise member error correlation of 0.894. It further claims that the modernized SandBoilNet (ConvNeXt-S encoder with scSE skip attention) reaches 0.707 IoU against 0.608 for the published original re-evaluated on the same held-out split, and that filtering the synthetic augmentation pool for label fidelity lifts the champion to 0.718. The author's own account is that","pith_inferences":["A natural extension is to apply the same parent-image filter and out-of-fold discipline to other synthetic-augmentation regimes in medical or infrastructure imaging, where diffusion-generated descendants of real patient or structure images are becoming common; the source_image_id field is cheap to add at generation time.","The error-correlation diagnosis suggests a testable recipe: decorrelate by pairing each backbone with a distinct pretrained encoder, and check whether the stacking shortfall (−0.013 IoU) shrinks further than the −0.008 observed when only two encoders were swapped.","Since the label-fidelity filter leans on the same baseline pseudo-labels that seed the synthetic masks, an editorial caution is warranted: the observed synthetic-data gain may partly reinforce the baseline's biases; re-validating with human-checked masks or a second independent annotator would disentangle genuine signal from inherited error.","MaskCN's zero-annotation labelling, if it generalises, would decouple dataset size from annotation budget for any segmentation task with a well-defined mask shape, potentially shifting the scarce-data bottleneck from labelling to mask design."],"forward_implications":["If the leak-free protocol is correct, any reported gain from weighted-averaging or naive synthetic augmentation in this domain must be re-read as partly an artifact of tuning on the scored split or of parent leakage.","Stacking's measured failure is attributed to member correlation (mean pairwise error correlation 0.894; five backbones share only three encoders), so future ensemble gains will require genuinely independent feature extractors rather than more decoders on the same encoders.","Label-fidelity-filtered synthetic data lifts the champion from 0.707 to 0.718 IoU, while unfiltered pools do not; the action is in filtering, not in the generator configuration, since sweeping guidance scale over four values produced no difference.","The negative hard-negative result (IoU −0.010, false-positive rate up) suggests that confusion-score-weighted synthetic negatives require a closed loop that re-scores against the model being trained, not the frozen baseline.","The modernized encoder (ConvNeXt-S + scSE) transfers to a second scarce-data defect dataset, indicating the architecture ranking is not an artifact of this particular split."],"fun_headline_variants":["Stacking fails to beat its best member in leak-free test","Leak-free eval: stack loses to single model in sand-boil detection","Sand-boil segmentation: stacking underperforms best single model","Per-architecture calibration can't rescue stacking in scarce-data test","Clean protocol shows ensemble loses to individual model in sand-boil task"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The synthetic training masks are produced by running the published SandBoilNet probability map through a convex-hull cleaning pipeline, and the label-fidelity admission filter checks a held-out segmenter's agreement with those same auto-generated labels — if the baseline's systematic errors are passed on to its successor, the observed synthetic-data gain (0.718 vs 0.707) may reflect reinforcement of those biases rather than genuinely new, correct signal.","fun_headline_variants_meta":{"raw":{"variants":["Stacking fails to beat its best member in leak-free test","Leak-free eval: stack loses to single model in sand-boil detection","Sand-boil segmentation: stacking underperforms best single model","Per-architecture calibration can't rescue stacking in scarce-data test","Clean protocol shows ensemble loses to individual model in sand-boil task"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1117,"prompt_tokens":862,"completion_tokens":255,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":179}},"tokens_in":606,"tokens_out":255,"duration_ms":3151,"temperature":1.0,"reasoning_tokens":179,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T02:37:26.878585+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-train the champion on a synthetic pool whose labels are generated by an independent segmenter or human-verified masks, using the same diffusion source and filter logic; if the IoU gain over the 0.707 control disappears or reverses, the filter was selecting for agreement with the baseline's own errors, not for label fidelity. Concretely, compare the current filtered-pool result against a pool filtered on human-verified IoU rather than the held-out segmenter's agreement with pseudo-labels.","supporting_citations":[],"review_version":1}