{"id":"ab35bedf-2c25-4d96-b9a5-c052db0ba5fa","arxiv_id":"2504.12185","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SALAD combines POS-tag-masked positives and LLM-generated counterfactual negatives in a contrastive loss to make fine-tuned language models more robust to spurious correlations.","lead":"The paper introduces SALAD, a training method that masks non-essential words identified by POS tagging and uses an LLM to create counterfactual sentences, then applies contrastive learning to reduce reliance on spurious correlations. It reports accuracy gains over several baselines on sentiment, sexism detection, and NLI, especially on out-of-distribution test sets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Positive samples may flip labels: the non-causal POS set in SALAD is never checked for label preservation, so the contrastive loss can pull contradictorily labeled sentences together.","rationale":"The reader's weakest assumption identifies the non-causal POS tag set as the key vulnerability; I agree partially and sharpen it into a specific, testable failure mode: label-flipping of positive samples. The paper's central claim is empirical, and the reported averages, ablations, and LLM data-quality metrics are internally consistent, so there is no ground for rejection. However, the method's novel component, structure-aware positive construction, is never checked for the property that makes it valid: that masking tokens from G preserves the label. The NLI case makes this acute because semantically decisive words can fall into the non-causal set, and the contrastive loss would then train against the classification objective. The threshold inconsistency between Section 4.3 and Appendix B.4 is a concrete documentation flaw that should be fixed, but it is secondary to the label-preservation check. The proposed test would settle whether the concern lands: if positive samples preserve labels, the concern is resolved; if they do not, the central mechanism is unsound and the reported gains likely come from other components such as LLM negatives or generic regularization. The verdict should remain conditional pending this check and the threshold clarification.","tokens_in":19845,"tokens_out":8896,"duration_ms":96260,"concrete_test":"Generate the positive set D_pos for CF-NLI exactly as in Section 3.1 with the reported G and k=1, then measure label agreement between each original sentence and its positive counterpart using gold labels, with a stricter sample of 200 checked by human annotation or by a RoBERTa-large model fine-tuned on a held-out split. If more than a small fraction (e.g., >2%) of positives change gold label, the contrastive objective is pulling differently labeled sentences together and the central mechanism is unsound. As a secondary check, recompute R_Tj with three seeds or a different backbone and report whether Table 10's causal sets remain stable; if they change, the method's core component is an artifact of the chosen standard fine-tuned model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the POS-tag accuracy-reduction criterion (Eq. 1, Section 3.1) identifies a non-causal tag set G whose members can be masked without changing the label. The paper never verifies label preservation for positive samples. This is not just a threshold typo: for CF-NLI, G is the complement of {VERB, NOUN, ADJ, ADV} (Table 10), so DET, PRON, PART, NUM, and similar tags are treated as non-causal. Words such as 'not', 'no', 'some', and 'all' can be label-determining in NLI; masking even k=1 such token (Table 9) can flip entailment to contradiction or vice versa. The contrastive loss then pulls a label-flipped sentence toward the anchor, directly conflicting with the classification loss. The threshold ambiguity (Section 4.3 states 0.1, Appendix B.4 states 1%) matters only because it shifts G; the deeper issue is that average accuracy reduction over an entire POS category does not guarantee per-sentence label invariance. Moreover, the referee for this decision is the standard fine-tuned model that SALAD aims to correct, so the 'causal' set can encode the model's spurious dependencies rather than semantic causality.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SALAD, a fine-tuning method that augments the standard cross-entropy loss with a triplet contrastive loss. Positive samples are generated by randomly masking a small number of tokens belonging to a POS-tag set that is deemed non-causal, where the set is derived from the average accuracy reduction of a standard fine-tuned model when each POS tag is removed (Eq. 1, Section 3.1). Negative samples are counterfactual sentences generated by an LLM that are prompted to flip the label by changing only causal words. The method is evaluated on sentiment classification, sexism detection, and natural language inference, with additional cross-domain and prompt-based few-shot experiments. The main claimed results are improved in-domain robustness, particularly on counterfactual test sets, and improved out-of-distribution and cross-domain accuracy relative to several data-augmentation and robust-learning baselines.","tokens_in":20145,"tokens_out":6990,"duration_ms":66322,"significance":"The empirical program is broad: three tasks, in-domain and out-of-distribution evaluation, cross-domain transfer, prompt-based few-shot evaluation, component ablations, and prompt-instruction analysis. If the results hold, SALAD would be a useful contribution to mitigating spurious correlations in fine-tuned PLMs. The paper also provides several useful analyses, including a comparison of LLM prompt templates for counterfactual generation and a diversity/overlap/BERTScore characterization of the generated data, which strengthen the empirical narrative. However, the central mechanism currently rests on an unverified assumption about label preservation under POS-based masking, and several reporting practices, including an inconsistent threshold, potential test-set tuning of hyperparameters, missing variance information, and incomplete cross-domain baselines, weaken the evidence. These issues are addressable within the scope of the manuscript.","major_comments":[{"comment":"The threshold used to define the non-causal tag set is inconsistent: Section 4.3 states that the threshold is 0.1 across all datasets, while Appendix B.4 and Figure 3 state that θ is estimated to be 1% and that non-causal tags are those with an average accuracy reduction of less than 1%. These thresholds differ by an order of magnitude and can change the tag set G materially. Please state the correct threshold, report the measured accuracy-reduction values for each POS tag, and reconstruct the tag sets with a justified, consistent threshold.","section":"§4.3 vs. §B.4, Fig. 3"},{"comment":"The hyperparameters k and λ are selected per dataset, and Section B.3 states that k=8 was chosen because it 'showed significant performance improvement for the CF-IMDB dataset, particularly on the out-of-distribution dataset (ODD)'. If k, λ, and the threshold are selected using the same ODD test sets whose accuracies are then reported in Tables 1-4, the reported gains are partly selection artifacts. The manuscript does not specify a validation protocol that keeps the reported test sets out of hyperparameter selection. Please either describe a protocol that uses only held-out validation data or report results with hyperparameters fixed a priori.","section":"§4.3, §B.3, Table 9"},{"comment":"All results are reported as averages over three random seeds without standard deviations or significance tests. Several claimed improvements are small relative to the strongest baselines, for example 93.05 vs. 91.49 overall on sentiment in Table 1, 91.61 vs. 89.92 on cross-domain in Table 4, and 88.31 vs. 89.47 on sexism in Table 2. Without variance information or significance testing, the reader cannot tell whether SALAD is actually better than the comparison methods. Please report per-seed results or standard deviations and, where possible, significance tests.","section":"Tables 1-4, 13; §4.3"},{"comment":"The cross-domain experiment omits several baselines that appear in the main single-domain tables, including AugGPT, Human-CAD, CORE-CAD, and DISCO. The statement that 'except in IMDB→SST2, all accuracy achieves the best performance' is therefore only relative to a subset of the considered baselines. Please add the missing baselines to Table 4, or explicitly restrict the claim to the compared methods.","section":"Table 4"},{"comment":"The discussion of the sexism task should be more careful: in Table 2, SALAD's overall accuracy is 88.31, which is below Human-CAD's 89.47, and on the counterfactual test set SALAD is 88.47 vs. Human-CAD's 91.80. The paper acknowledges this in passing, but the abstract's general claim that SALAD improves robustness and generalization should be qualified by this comparative result, especially since Human-CAD is not included in the cross-domain experiments.","section":"§5.1, Table 2"}],"minor_comments":[{"comment":"The symbol α is used both as the margin in the triplet loss in Eq. (3) and as the scaling factor for k in Sections 3.1 and 4.3; please rename one of them to avoid ambiguity.","section":"§3.3, Eq. (3)"},{"comment":"There is a typo in 'non-casual words' in the paragraph following Eq. (1); the intended word is 'non-causal'.","section":"§3.1"},{"comment":"The column header 'FindFood' should read 'FineFood' for consistency with the rest of the paper.","section":"Table 1"},{"comment":"Table 10 is headed 'Causal tag set G', but Section 3.1 defines G as the non-causal tag set; please align the notation in the table with the text.","section":"Table 10 vs. §3.1"},{"comment":"The axis labels in Figure 3 render with garbled characters, making the accuracy-reduction values unreadable; the figure should be regenerated with a standard font.","section":"Fig. 3"},{"comment":"The sentence 'Table 4 reports the performance of applying SALAD...' appears to refer to Figure 4 in the appendix rather than Table 4 in the main text; please correct the cross-reference.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and the experimental design is extensive, but the empirical claims currently outrun the evidence. I recommend a major revision rather than rejection because the method is plausible and the requested additions, such as label-preservation verification, a consistent and justified threshold, an honest validation protocol, and variance reporting, are feasible within the scope of the manuscript. If the authors can supply these, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: The paper's specific combination—POS-tag-masked positives plus LLM-generated counterfactual negatives under a triplet loss—is new relative to the cited baselines, and the reported accuracy gains are consistent across sentiment, sexism, NLI, and cross-domain settings. But the central assumption of the positive-sample branch, that masking tokens from the non-causal POS set never changes the label, is never verified and is likely false in NLI. That is the first thing a referee should press on.\n\nWhat works: The pipeline is concrete and mostly well-motivated. Using average accuracy reduction of a standard fine-tuned model to pick non-causal POS tags is a real mechanism, even if it inherits the reference model's biases. The LLM prompt that tells the model to change only causal words (Instruction 4) yields CAD with overlap and BERTScore close to human data, and the ablation in Table 5 shows the POS-tag positives add value beyond LLM negatives. The cross-domain results are directionally consistent. So the paper earns serious consideration.\n\nWhere it gets soft: The label-invariance issue is load-bearing. For CF-NLI, the non-causal set is the complement of {VERB, NOUN, ADJ, ADV}, so determiners and particles are masked. Words like 'some', 'all', 'no' can be label-determining in NLI. Since k=1 for CF-NLI, a single masked token can flip entailment to contradiction. The contrastive loss then pulls a label-flipped sentence toward the anchor, directly fighting the classification loss. The paper never checks label preservation. Average accuracy reduction over an entire POS category does not guarantee per-sentence label invariance. Also, the 'causal' set is computed from the same standard fine-tuned model SALAD aims to correct, so the set may encode the model's spurious dependencies rather than semantic causality.\n\nMinor but real issues: The threshold is 0.1 in Section 4.3 but 1% in Appendix B.4. Hyperparameters (k, lambda, alpha, prompt template) are tuned on the evaluation benchmarks, and no error bars are reported, so the improvements are partly a product of selection. Cross-domain Table 4 omits several baselines that appear in the main tables, making comparison incomplete. The sexism task shows SALAD below Human-CAD, so 'state-of-the-art' in the abstract is overstated. Code/data are not released, which limits reproducibility.\n\nWho this is for: Practitioners and researchers working on spurious-correlation robustness and counterfactual augmentation will find the method interesting, but they should treat the reported numbers as provisional.\n\nRecommendation: This deserves peer review, not desk rejection. A referee should ask for a label-preservation analysis on positive samples, a consistent threshold, error bars, hyperparameter selection on held-out data, and ideally code. With these fixes the method could be solid; without them, the central mechanism rests on an unverified assumption.","headline":"Unverified label preservation in POS-masked positives is the load-bearing flaw; the combination is new and the gains are consistent, but the numbers are provisional until that assumption is checked.","tokens_in":20639,"tokens_out":5068,"would_cite":false,"duration_ms":47359,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-16T12:35:23.672089+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}