{"id":"d6bdb539-e4ed-403d-a9bb-fea90816b9e3","arxiv_id":"2507.04788","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A two-stage optimization pipeline that alternates label loss with a KL divergence between feature maps of masked and unmasked inputs improves accuracy and robustness in small-data classification.","lead":"This paper introduces a two-stage training pipeline that alternates label-based updates with a feature-alignment step, using simple explanation masks to guide models toward the 'right reasons' in small, imbalanced, or spurious datasets. The authors report consistent accuracy and robustness gains on one real and two synthetic image datasets, and argue that explanation-guided training can reduce sample complexity.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central \"two-stage\" contribution is never tested against the closest prior baseline: saliency guided training (Ismail et al., 2021), which the paper itself cites for its KL loss, is absent from all experiments, so the claimed gains cannot yet be attributed to the two-stage design.","rationale":"The reader's weakest_assumption is explanation-mask sufficiency and distinctiveness. I read that as an external-validity limitation: within the paper's three datasets the masks are hand-selected to be sufficient, so the assumption is not internally violated. The more load-bearing gap is internal: the paper's novelty is claimed to be the two-stage optimization, but the closest single-stage KL-based method, which the authors themselves cite for the loss design, is never compared. The joint-vs-two-stage ablation in Table 3 is useful but does not settle this, because it is a comparison against the authors' own loss rather than against the established prior method. A single added experimental arm would resolve whether the two-stage mechanism, rather than the explanation signal and KL alignment itself, is responsible for the observed gains. Since the reader already conditions acceptance on stronger verification and explicitly notes the missing baseline, my read leaves the verdict unchanged.","tokens_in":15224,"tokens_out":10094,"duration_ms":133234,"concrete_test":"Add saliency guided training (Ismail et al., 2021) as an experimental arm, using the same CNN architectures, optimizers, dataset sizes, early stopping, balanced/imbalanced and spurious variants, with its lambda tuned on a held-out split; report 30-seed mean and standard deviation for clean, alternative-rule, and spurious test sets. If it matches or beats Ours on accuracy or convergence, the two-stage contribution is not established; if Ours still wins, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the two-stage alternating pipeline (Algorithm 1) is what yields the empirical gains. This is argued in Section 2.2 by claiming that joint explanation losses fail because the two gradients race, and Section 2.3 adopts a KL feature-map loss explicitly \"similar to the saliency guided training loss (Ismail et al., 2021)\". Yet saliency guided training is never run. The only regularization baseline is Grad Reg (Ross et al., 2017). The Table 3 ablation compares two-stage with joint training of the authors' own feature loss, but not with the established joint method whose loss is the closest prior art. Without that arm, the observed superiority of Ours could be due to the explanation signal and KL loss rather than to the two-stage optimization. This is compounded by Appendix B, where eta1/eta2 are selected from empirical results and eta2 is raised specifically in the imbalance/spurious settings, without any held-out validation protocol, so the comparison may be tuned toward the proposed method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage training algorithm for image classifiers that uses simple explanation masks as an additional supervision signal. In stage one, all parameters are updated with a cross-entropy label loss; in stage two, only a newly inserted linear mapping layer is updated to minimize the KL divergence between the softmax-normalized feature map of the full input and the feature map of the explanation-masked input. The authors claim that this alternating schedule, unlike joint explanation-loss training, avoids gradient conflicts and leads to faster convergence, higher test accuracy, better consistency, robustness to spurious features, and reduced sample complexity in small, imbalanced, or spurious-data regimes. Experiments on two synthetic geometric datasets and a two-class CUB bird subset compare against vanilla training, gradient regularization, and focal loss, and include ablations of the mapping layer and the two-stage schedule.","tokens_in":15447,"tokens_out":7264,"duration_ms":76866,"significance":"If the claims hold, the paper offers a practical and low-cost way to inject human reasoning into deep learning under data scarcity. The evaluation is more thorough than typical for this line of work: it reports 30-trial statistics, includes a sparse-explanation setting, tests on alternative rule-sharing datasets, and stress-tests with injected spurious features. The two-stage optimization with a dedicated mapping layer is a plausible mechanism for avoiding the gradient conflicts that afflict joint explanation losses. However, the significance is limited by the narrow evaluation (the Bird dataset is the only real data, with hand-drawn beak masks) and by concerns about hyperparameter selection and the absence of the closest prior baseline. These issues do not invalidate the core idea but make the current evidence for the central claims incomplete.","major_comments":[{"comment":"The learning rates η1 and η2 are selected as \"the best combination\" from {0.1, 0.01, 0.001} based on empirical results, and the text further states that η2 is increased from 0.001 to 0.01 specifically when training under class imbalance or spurious correlations. No held-out validation set or cross-validation procedure is described for this selection, so the reported test-accuracy gains may reflect test-set tuning. Because the main claims are comparative, the authors should specify how η1 and η2 are chosen (e.g., validation split) and report the sensitivity of the results to these hyperparameters.","section":"Appendix B (hyperparameter selection)"},{"comment":"The paper cites Ismail et al. (2021) as the source of the KL feature-misalignment loss but never runs their saliency guided training as a baseline. The closest evidence against joint training is the \"JOINT OPT\" ablation in Table 3, which uses the same loss and mapping layer but trains with a joint objective. The authors should either explicitly run and report the Ismail et al. baseline, or clearly state that the JOINT OPT condition is an instantiation of that prior method with the additional mapping layer. Without this clarification, the reader cannot attribute the observed benefits to the two-stage schedule rather than to the KL alignment objective itself.","section":"Section 3 and Table 3 (baseline comparison)"},{"comment":"The method rests on the assumption that the feature map of the masked input x' = x ⊗ e(x) is \"distinctive enough for the classifier,\" which is asserted without direct evidence. The explanation masks in the experiments are hand-crafted to highlight exactly the discriminative region (vertex, arc, or beak), so they are informative by construction. In realistic annotation settings, masks may be coarse, misaligned, or partially occluded. A brief ablation varying mask size, adding mask noise, or shifting the mask would strengthen the paper by showing the method's sensitivity to mask quality. This is load-bearing because the KL loss in Eq. (3) directly pulls the full-input feature map toward the masked-input feature map.","section":"Section 2.3 (assumption on masked feature maps)"}],"minor_comments":[{"comment":"The pentagon and triangle-vs-circle test sets are constructed from the same cues as the explanation masks used in training. Please acknowledge in the text that these tests verify adherence to the specific provided cue, rather than giving evidence that the model has learned a general, semantically meaningful \"right reason\" beyond that cue.","section":"Section 3.3 (alternative test sets)"},{"comment":"The formatting of Table 3 is difficult to parse: many cells contain multiple numbers (e.g., \"0.704±0.051 0.704±0.051 0.704±0.051 0.567±0.089 0.632±0.099\"), and it is unclear which columns correspond to the three reported time points (20 epochs, 40/80 epochs, convergence). Please restructure the table so that each row and column is unambiguously labeled.","section":"Table 3 (readability)"},{"comment":"The sentence \"We empirically evaluated the model with three learning rates ... We then pick the best combination\" is immediately followed by \"Note that we did not tune these hyper-parameters to a great depth.\" This is contradictory and should be rephrased, since selecting the best among three values is still a form of tuning.","section":"Appendix B (contradictory statements)"},{"comment":"The cost analysis claims that explanations are more cost-efficient than collecting more data, but the arithmetic is not shown. Table 4 indicates roughly a 2-3x reduction in dataset size to reach a given accuracy, while explanations are stated to cost about 3x a label. Please make the cost comparison explicit and account for the variance in Table 4, since the conclusion is not obvious from the quoted numbers.","section":"Section D (cost analysis)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop submission with a promising idea, and the two-stage ablation is a useful control, but the hyperparameter-selection protocol and the missing explicit comparison to saliency guided training need to be addressed. I would be willing to see a revised version if these issues are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the training scheme—alternating a label step with a feature-alignment step on masked inputs, through a learned linear mapping layer—is genuinely different from the joint-loss regularization in Ross et al., Rieger et al., and Ismail et al. Second, the experiments are better than the typical small-data interpretability paper: 30 seeds, alternative test sets that reward the intended rule, spurious-feature injection, and ablations of the two new components. That is real work and it mostly supports the claim that explanations help in small-data regimes.\n\nThe strongest evidence is the alternative-rule test sets. Instead of eyeballing saliency maps, the authors construct pentagon and triangle-vs-circle sets that share the label rule, and their models transfer better. The sparse-explanation experiment, where only minority-class points have masks, is also a nice practical touch.\n\nNow the soft spots, in severity order. The biggest is that the paper never runs the closest prior method: saliency guided training (Ismail et al., 2021). The KL loss in Section 2.3 is explicitly \"similar to\" that work, and Section 2.2's argument against joint losses is used to motivate the two-stage design. Without Ismail et al. in the plots, the gains attributed to two-stage alternation could actually be due to the KL feature alignment itself or to the mapping layer. The ablation in Table 3 only varies the authors' own components; it cannot separate the two-stage idea from the specific loss. Second, the hyperparameters eta_1 and eta_2 are selected from empirical results (Appendix B) with no held-out protocol. The authors even say eta_2 needed to be raised for imbalanced/spurious settings \"because models need to update their focus more\"—that is post hoc fitting, and it weakens the claim that the method is robust. Third, no code or data is released, and the architecture details are compressed; reproduction would be harder than it needs to be. Fourth, the table formatting in the appendix is a mess (repeated numbers, misaligned columns), which is minor but doesn't inspire confidence.\n\nNone of these are fatal. The central idea is plausible and the evidence, while incomplete, is not circular in a damaging way—the alternative test sets are constructed from the same rules as the masks, which is a mild self-consistency issue, not a logical circle. The paper should go to peer review, but only if the authors add the Ismail baseline, fix the hyperparameter story, and release code. As it stands, I would not cite it as an established result yet.","headline":"Two-stage explanation training is a real variant, but without the Ismail baseline and with tuned eta hyperparameters, the headline gains are not yet established.","tokens_in":15970,"tokens_out":2516,"would_cite":false,"duration_ms":25122,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two-stage training with masks beats labels alone on small, skewed, spurious datasets","keywords":["learning from explanations","explanation-guided training","two-stage optimization","feature-map alignment","small data regime","spurious correlations","class imbalance","sample complexity"],"falsifier":"Train the pipeline with deliberately corrupted explanations, such as masks shifted to cover only background or only a spurious square, on the same synthetic datasets; if test accuracy and reason-alignment remain at the levels achieved with correct masks, the claimed mechanism is not driving the gains, while a collapse to label-only levels would confirm that the explanations are doing the work.","tokens_in":1482,"feed_emoji":"🎯","tokens_out":2335,"duration_ms":61310,"temperature":0.7,"pith_summary":"The paper argues that labels alone are insufficient for training reliable models on small datasets, because a model can fit any arbitrary rule that agrees with the labels, including spurious ones. It proposes supplying a binary feature mask, or explanation, for each training label and alternating between a standard label-loss update and a feature-alignment update that pulls the model's internal feature map toward the features of the masked input. The claim is that this two-stage cycle makes models learn the intended reasons, yielding higher test accuracy, faster convergence, lower variance, and greater robustness to spurious correlations than label-only training. The authors demonstrate this on two synthetic geometric datasets and a real bird-species dataset, including settings with severe class imbalance and injected spurious cues.","feed_headline":"Two-stage training with explanation masks beats labels alone","feed_subtitle":"Small, imbalanced, or spurious datasets gain accuracy, speed, and stability when models align to masks.","key_machinery":"The mechanism is a two-stage optimization loop over a CNN decomposed into a feature extractor $f$, a linear mapping layer $m$, and a classifier $c$. Stage one updates all weights with the cross-entropy label loss. Stage two masks the input with the explanation to form $x' = x \\otimes e(x)$, computes the softmax-normalized feature maps $\\text{softmax}(f(x'))$ and $\\text{softmax}(m(f(x)))$, and updates only $m$ to minimize their KL divergence. The mapping layer is trained to filter out irrelevant information and amplify the explanatory region, and updating only this layer avoids the conflicting-gradient race condition that the paper identifies in prior joint-loss methods.","core_discovery":"Standard empirical-risk-minimizing training on small datasets can fit any rule consistent with labels, including spurious ones. The paper's central claim is that supplying a binary mask over input features that explains each label, and alternating between a standard label-loss update and a KL-divergence update that aligns the feature map of the full input (passed through a learned mapping layer) with the feature map of the masked input, yields models that learn the intended reasons. As a result, on small, class-imbalanced, and spuriously correlated datasets, this two-stage pipeline reaches higher test accuracy faster, with lower variance and higher pairwise agreement between models trained on the same data, than label-only training, focal loss, or gradient-regularization baselines.","pith_inferences":["The approach would likely transfer to knowledge distillation, where a teacher provides both labels and explanation masks to train a student model; the paper mentions this as a natural extension.","Because the second stage only updates the mapping layer, the method could be combined with pretrained feature extractors or used as a lightweight fine-tuning procedure, though the paper does not test this directly.","The alternative test sets used to probe whether models learn the intended reasons are constructed from the same explanation-derived rules, so a stronger evaluation would use independently collected human annotations or counterfactual variations not derived from the masks.","If real-world explanation masks are noisy or coarse, the KL alignment may need confidence weighting or a robust loss; the paper assumes masks are informative and sufficient, which may not hold in practice."],"forward_implications":["Models trained with explanation masks reach a given accuracy with substantially fewer training examples, with the reported cost analysis indicating at least a 3x and sometimes a 10x reduction in sample size.","When the training set is severely class-imbalanced (9:1), the approach outperforms label-only training and focal loss, even when explanations are provided only for the minority class.","Models trained with explanations show much higher pairwise prediction agreement on the same training data, indicating that they converge to the same decision function rather than arbitrary label-consistent rules.","When spurious features are injected into training data, explanation-guided models maintain higher accuracy on clean test sets, indicating they rely more on the intended reasons.","The paper's comparison suggests that prior gradient-regularization approaches that penalize wrong reasons rarely outperform label-only training, whereas the proposed two-stage feature-alignment approach does."],"supporting_citations":[{"why":"Provides the gradient-regularization baseline that adds an explanation-misalignment penalty to the label loss; the paper compares against it and shows it rarely beats label-only training.","marker":"(Ross et al., 2017)"},{"why":"Another penalty-based explanation-training method that the paper cites as evidence that joint-loss approaches often fail to improve over vanilla models.","marker":"(Rieger et al., 2020)"},{"why":"Supplies the CUB-200-2011 dataset from which the Bird dataset (Indigo Buntings vs Blue Grosbeaks) is constructed.","marker":"(Wah et al., 2011)"},{"why":"Shows that many saliency-map explanations fail sanity checks, motivating the paper's use of feature-map alignment rather than attribution-based penalties.","marker":"(Adebayo et al., 2018)"},{"why":"Provides the saliency-guided training loss using KL divergence between normalized feature maps, which the paper adapts for its feature misalignment loss.","marker":"(Ismail et al., 2021)"}],"fun_headline_variants":["Explanations beat labels for small data training","Feature masks guide models to learn real reasons","Two-stage training with masks improves small-data models","Teach models with masks instead of just labels","Masks make small data training more accurate and stable"],"cache_read_input_tokens":18176,"weakest_assumption_plain":"The explanation masks must be informative enough to fully explain the label, and the masked input's feature map must be distinctive enough for the classifier; if a mask is incomplete, misaligned, or too coarse, the alignment step can steer the model toward partial or wrong reasons.","fun_headline_variants_meta":{"raw":{"variants":["Explanations beat labels for small data training","Feature masks guide models to learn real reasons","Two-stage training with masks improves small-data models","Teach models with masks instead of just labels","Masks make small data training more accurate and stable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000551,"raw_usage":{"total_tokens":2576,"prompt_tokens":839,"completion_tokens":1737,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":1667}},"tokens_in":455,"tokens_out":1737,"duration_ms":12845,"temperature":1.0,"reasoning_tokens":1667,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:39:19.407092+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the pipeline with deliberately corrupted explanations, such as masks shifted to cover only background or only a spurious square, on the same synthetic datasets; if test accuracy and reason-alignment remain at the levels achieved with correct masks, the claimed mechanism is not driving the gains, while a collapse to label-only levels would confirm that the explanations are doing the work.","supporting_citations":[{"cited_title":"The caltech-ucsd birds-200-2011 dataset","cited_arxiv_id":null,"evidence_quote":"Supplies the CUB-200-2011 dataset from which the Bird dataset (Indigo Buntings vs Blue Grosbeaks) is constructed."}],"review_version":1}