{"id":"810148a2-c95c-46f2-ba03-f0ad4f4de169","arxiv_id":"2509.03725","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A metric learning approach selects few-shot fine-tuning samples that improve cross-target and cross-domain stance detection over random selection across six models.","lead":"This paper introduces a method called MLSD that picks a small set of training examples from a new target or domain to fine-tune a stance detection model, using metric learning to choose examples most similar to the source target. In tests on two datasets and six models, the selected examples consistently beat random selection for cross-target and cross-domain stance detection.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Statistical significance of MLSD's improvement over random selection is unverifiable: no per-seed/per-shot F1s or exact p-values, so the central 'p<0.05' claim does not yet have auditable support.","rationale":"The reader's CONDITIONAL verdict already captures the need for stronger statistical evidence, so my concern reinforces the conditions rather than changing the verdict. I deliberately chose not to make the reader's weakest_assumption—the correlation between source-likeness and downstream fine-tuning utility—the load-bearing issue. That assumption is an explanatory mechanism for why MLSD works, but the paper's central claim is an empirical comparison against random selection. If the end-to-end comparison is statistically sound, the claim stands even if the mechanism is only imperfectly understood. Conversely, if the paired t-test is invalid or the reported p-values cannot be reproduced, the central claim itself is unsupported, regardless of how plausible the selection mechanism sounds. The missing per-seed/per-shot results and unspecified pairing are thus the most direct threat to the paper's headline assertion.","tokens_in":11497,"tokens_out":10164,"duration_ms":117071,"concrete_test":"Require the authors to release the per-seed, per-shot macro-F1 values for every model and transfer direction, then independently recompute the paired t-test separately for n=5, 10, and 15 (paired by seed) and after a multiple-comparison correction for the eight transfer directions. If MLSD does not beat random selection at p<0.05 in each shot setting (or at least the majority after correction), the blanket p<0.05 claim in Tables 5–8, and with it the central improvement claim, must be withdrawn or narrowed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MLSD-selected few-shot samples improve cross-target/cross-domain stance detection over random selection, with p<0.05. The load-bearing problem is that the statistical support for this claim is unauditable as written. Section 4.5 says results are means over five fixed seeds, and Tables 5–8 report only one macro-F1 number per model/transfer, captioned as averaged over five seeds and 5/10/15-shot settings. Exact p-values, standard deviations, per-seed and per-shot F1s, and even the pairing structure of the claimed paired t-test are absent. It is therefore impossible to tell whether the p<0.05 statements in the table captions come from paired seed-level comparisons, from pooled 5/10/15-shot runs treated as independent, or from comparisons on already-aggregated cells; either of the latter would invalidate the test. Since the headline claim is an empirical superiority claim over random selection, the unvalidated source-likeness mechanism in Section 3.3 is not the most load-bearing assumption; the same end-to-end comparison, if statistically sound, would establish the claim even with an imperfect mechanism story. But without the underlying per-run data, the significance claim, and hence the central claim, is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MLSD, a metric-learning-based few-shot sample selection method for cross-target (CTSD) and cross-domain (CDSD) stance detection. It first trains a triplet-loss model on source-target vs noise-target pairs using BERT embeddings and hard negative mining, then uses the model's confidence scores to select the top-n destination training samples per stance class. Those samples are used to fine-tune a stance classifier previously trained on a source target. The method is evaluated on SemEval-2016 and WT-WT with six classifiers (BiCond, BiLSTM, CrossNet, TAN, TextCNN, RoBERTa), comparing MLSD selection to random selection and standard training. The paper reports consistent macro-F1 gains for MLSD over random selection, with captions claiming statistical significance (p < 0.05) via paired t-tests.","tokens_in":11806,"tokens_out":3993,"duration_ms":42651,"significance":"If the empirical claims hold, MLSD would be a simple, model-agnostic, and external-knowledge-free sample selection strategy that improves cross-target and cross-domain stance detection. The experimental scope is broad: two datasets, six classifiers, multiple transfer directions, and both same-domain and cross-domain settings. The authors provide a code link, which supports reproducibility. However, the central significance claim—statistically significant superiority over random selection—is not currently auditable from the manuscript, and the hyperparameter tuning protocol is underspecified. The contribution is potentially useful, but the evidence as presented does not yet meet the bar for a strong empirical claim.","major_comments":[{"comment":"The statistical support for the central claim is not auditable. Tables 5-8 report single macro-F1 numbers per model/transfer, captioned as 'averaged across five seeds and 5, 10, and 15-shot settings,' with the statement that MLSD's improvement over random selection is 'statistically significant, with p < 0.05, based on a paired t-test.' However, no exact p-values, standard deviations, confidence intervals, degrees of freedom, per-seed values, or per-shot (5/10/15) values are provided. It is unclear whether the paired t-test was performed on seed-level pairs, on 5/10/15-shot runs pooled as independent observations, or on already-aggregated table cells. Each choice has different statistical validity. Without the underlying per-run data or effect sizes, the headline significance claim and hence the central empirical contribution cannot be verified.","section":"Section 4.5 / Tables 5-8"},{"comment":"The hyperparameter tuning protocol is underspecified and could affect the validity of the comparison. The margin for triplet loss is said to be 'determined through a grid search to optimize performance' (Section 3.2), the number of triplets per anchor was chosen after 'experimenting with different numbers (5, 10, and 15)' (Section 3.1), and hard-negative top-k is fixed at 5. No validation split or selection procedure is described. If any of these choices were made using the destination test set (or on the same folds used for the final evaluation), the reported improvements over random selection would be inflated. The authors must specify how hyperparameters were chosen and, ideally, provide an ablation or nested validation.","section":"Section 3.1-3.2 / Section 4.5"}],"minor_comments":[{"comment":"Numerous typos and inconsistencies: 'decending' (Section 3.1), 'Hilary' vs 'Hillary', 'MSLD' for MLSD (Section 4.3), 'differnent' (Section 5.1), 'there approaches' (Section 2.1). Please proofread carefully.","section":"Throughout"},{"comment":"The FM→DT row for BiCond appears malformed: '34 28.35 40.71%' should likely be '34% 28.35% 40.71%' or similar. Formatting of percent signs is inconsistent.","section":"Table 7"},{"comment":"The notation 'arg N max i=1' is confusing. This should be written as a standard top-N selection procedure, e.g., choosing the N instances with the highest confidence scores within each class.","section":"Eq. (1), Section 3.3"},{"comment":"The reported 'performance increased by 11.72%' etc. should clarify whether these are absolute percentage-point improvements or relative percentage improvements. The current phrasing is ambiguous.","section":"Sections 5.2 and 5.4"},{"comment":"The sentence 'These results suggest then a Top N > 5 often provides better results' is not well supported by Table 9, where the trend is inconsistent across models and settings. Please revise or qualify.","section":"Section 5.3 / Table 9"},{"comment":"The RoBERTa citation (Liu, 2019) should conform to the standard citation format used in the paper; currently it appears incomplete.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the empirical idea is interesting, but the verifiability of the central significance claim is a serious gap. If the authors provide per-seed/per-shot data, exact p-values, and a clear hyperparameter tuning protocol, the contribution may be solid. I encourage the editor to request those details before further consideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on MLSD. The core idea is a new combination: train a triplet-loss model to separate source target from a noise target, then use its confidence scores to pick the most 'source-like' destination examples as few-shot fine-tuning samples. That's a plausible, model-agnostic way to adapt stance classifiers without external knowledge, and it's not in the papers they cite. The empirical pattern is consistent: across six architectures and two datasets, MLSD-selected samples beat random selection, and the cross-domain gains are large.\n\nCredit where due: the paper is honest about the key limitation—it still needs labeled destination data—and the authors ran a wide evaluation with multiple classic baselines as the underlying classifiers. They also ship code, which helps reproducibility.\n\nNow the soft spots. The main one is exactly what the reader flagged: the 'p < 0.05' claims are not auditable. They say results are means over five seeds and 5/10/15-shot settings, but report one number per cell, with no standard deviations, no exact p-values, and no description of the t-test pairing structure. Given that the headline is an empirical superiority claim, that's a load-bearing problem, not a minor one. The stress-test note is right on this.\n\nSecond, the selection mechanism in Section 3.3 is under-specified. The model outputs a binary source-vs-noise probability, but they say they select top-n per stance class. How do they get per-class confidence? That's never explained. The few-shot samples are labeled, so maybe they just group by true label, but the text doesn't say. This makes the method description partly incoherent.\n\nThird, the magnitude of the cross-domain gains (e.g., 30 to 70 F1) is surprising. It could be real, but with random selection sometimes performing below standard training, I'd want to see the per-run numbers and a sanity check on the fine-tuning protocol before believing it. There's also no comparison to existing selection baselines like Arakelyan et al.'s topic-guided sampling, which they cite. That weakens the claim that their selection strategy is the reason for the gains.\n\nThe paper is worth engaging with. The idea is clever, the evaluation is broad, and the limitations are acknowledged. But it needs a serious referee to demand full per-seed/per-shot results, a clear selection description, and at least one non-random baseline. I'd send it to review rather than desk reject, but I wouldn't want it accepted without those fixes.","headline":"A promising few-shot selection idea for cross-target/cross-domain stance detection, but the statistical evidence and method description are not yet solid enough to fully credit the gains.","tokens_in":12254,"tokens_out":3128,"would_cite":true,"duration_ms":31774,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A few-shot selection method that learns which destination-target examples look most like the source target consistently improves cross-target and cross-domain stance detection, beating random selection across six models.","keywords":["stance detection","cross-target stance detection","cross-domain stance detection","few-shot learning","metric learning","triplet loss","hard negative mining","sample selection"],"falsifier":"Take any held-out dataset pair, rank destination examples by MLSD source-likeness and by their actual leave-one-out effect on fine-tuning accuracy, then compare the two rankings and the fine-tuned performance of the top sets. If MLSD's top set fails to beat random selection while the influence-ranked set does, the source-likeness proxy is the thing that breaks.","tokens_in":11393,"feed_emoji":"🎯","tokens_out":7863,"duration_ms":77816,"temperature":0.7,"pith_summary":"This paper introduces MLSD, a few-shot sample-selection method for adapting stance detectors to new targets and domains. The key move is to train a triplet-loss similarity model that separates source-target tweets from an unrelated noise target, then use it to score every destination-target example by how confidently it looks like the source; the top examples per stance class become the few shots for fine-tuning. The paper reports that these selected samples consistently beat randomly selected samples of the same size in cross-target and cross-domain settings, across six stance-detection models and two datasets, with paired-t-test significance at p<0.05. The implication is that a cheap, model-agnostic selection step can substitute for external knowledge sources and prompt engineering when a stance classifier must transfer.","feed_headline":"Learned few-shot picks beat random in cross-domain stance","feed_subtitle":"A triplet-loss selector chooses destination examples that fine-tune six stance models better than random samples do.","key_machinery":"The load-bearing mechanism is a triplet-loss embedding trained to distinguish source from noise, followed by confidence-based top-N selection. Hard negative mining with Sentence-BERT embeddings makes the triplet model learn a boundary tight around the source target; the softmax confidence of that model on destination examples is then read as a source-likeness score. That score, not any task-specific stance signal, chooses the few shots per stance class.","core_discovery":"MLSD's central discovery is that source-likeness, learned using only source and noise labels plus unlabeled destination text, is an effective proxy for how useful a destination example is for fine-tuning. Concretely, the method first builds triplets from source anchors, source positives, and hard negatives mined from a noise target, and trains a BERT-embedding model with triplet loss. It then runs that model on the destination target and, for each stance class, keeps the N examples the classifier is most confident belong to the source class. Fine-tuning a source-trained stance model on those N examples—5, 10, or 15 per class—improves macro-F1 over random selection in every reported scenario,","pith_inferences":["If source-likeness is the selection signal, the method likely favors destination examples that resemble the source; a natural stress test is to apply it when the destination has a large subpopulation unlike the source and check whether that subpopulation's stance accuracy drops.","The same selection procedure could plausibly transfer to other classification tasks with target or domain shift, such as sentiment or hate-speech detection, where a source-versus-noise similarity model could rank unlabeled target examples for annotation.","The paper's observation that RoBERTa gains least suggests the value of MLSD is inversely related to how much the base model already generalizes; one could test this by measuring gain as a function of the source-trained model's zero-shot destination accuracy.","Confidence-based selection may concentrate on easy, source-like examples; combining source-likeness with diversity-aware tie-breaking or uncertainty sampling could yield a family of selection criteria worth benchmarking."],"forward_implications":["Any stance classifier—CNN, RNN, or transformer—can be adapted to a new target by fine-tuning on 5–15 MLSD-selected examples, so the method layers onto existing models without changing their architecture.","Because the similarity model is trained once per source/noise pair, the same scores can rank many destination targets, reducing the annotation needed for new targets to a handful of examples per class.","The largest gains occur where standard transfer is weakest (entertainment→healthcare and healthcare→entertainment), suggesting selection of representative examples matters most when domains are far apart.","Across six classifiers, the method outperforms random few-shot selection, which is the natural baseline for limited-label settings, and it also improves on standard training without any destination data.","Using only 0.0006%–0.03% of the destination training data as few shots, the approach offers a low-cost alternative to external knowledge graphs, Wikipedia, or LLM-based methods."],"supporting_citations":[{"why":"Supplies the SemEval-2016 Task 6 dataset and stance labels used for the cross-target and cross-domain evaluations.","marker":"Mohammad et al., 2016"},{"why":"Supplies the WT-WT dataset and documents the poor cross-domain baseline that MLSD is shown to improve.","marker":"Conforti et al., 2020"},{"why":"Supplies Sentence-BERT embeddings used to rank candidate negatives for hard negative mining in triplet construction.","marker":"Reimers and Gurevych, 2019"},{"why":"Supplies BERT embeddings from which triplets are encoded for the metric learning model.","marker":"Devlin et al., 2019"},{"why":"Introduces the BiCond model and the cross-target stance detection formulation that MLSD extends and is tested against.","marker":"Augenstein et al., 2016"},{"why":"Introduces CrossNet and the source/destination target conventions for cross-target stance detection used in the experiments.","marker":"Xu et al., 2018"},{"why":"Supplies the TAN model, one of the six stance classifiers evaluated with MLSD.","marker":"Du et al., 2017"},{"why":"Supplies the TextCNN model, one of the six stance classifiers evaluated with MLSD.","marker":"Kim, 2014"},{"why":"Supplies the RoBERTa model, one of the six stance classifiers evaluated with MLSD.","marker":"Liu, 2019"},{"why":"Supplies the BiLSTM architecture used as one of the six stance classifiers evaluated with MLSD.","marker":"Schuster and Paliwal, 1997"}],"fun_headline_variants":["Metric learning picks better few-shot examples for stance","Triplet-loss selection outperforms random in cross-domain stance","Source-likeness guides example choice, improving stance fine-tuning","MLSD trains on selected examples to beat random in stance tasks","Cross-domain stance gains from triplet-loss example selection"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"Everything rests on the premise that destination examples the similarity model most confidently labels as source are the same examples that most improve a fine-tuned stance classifier.","fun_headline_variants_meta":{"raw":{"variants":["Metric learning picks better few-shot examples for stance","Triplet-loss selection outperforms random in cross-domain stance","Source-likeness guides example choice, improving stance fine-tuning","MLSD trains on selected examples to beat random in stance tasks","Cross-domain stance gains from triplet-loss example selection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000439,"raw_usage":{"total_tokens":2006,"prompt_tokens":627,"completion_tokens":1379,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":371,"completion_tokens_details":{"reasoning_tokens":1300}},"tokens_in":371,"tokens_out":1379,"duration_ms":12566,"temperature":1.0,"reasoning_tokens":1300,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:42:39.237053+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any held-out dataset pair, rank destination examples by MLSD source-likeness and by their actual leave-one-out effect on fine-tuning accuracy, then compare the two rankings and the fine-tuned performance of the top sets. If MLSD's top set fails to beat random selection while the influence-ranked set does, the source-likeness proxy is the thing that breaks.","supporting_citations":[{"cited_title":"Stance Detection with Bidirectional Conditional Encoding","cited_arxiv_id":"1606.05464","evidence_quote":"Introduces the BiCond model and the cross-target stance detection formulation that MLSD extends and is tested against."}],"review_version":1}