{"id":"bb39e816-139c-4b45-b5ef-7d887bfcb471","arxiv_id":"2505.22076","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuning a Gemma model on a mix of seed, generated, and general instruction data (ArgInstruct) improves zero-shot performance on unseen computational argumentation tasks while preserving general NLP performance.","lead":"The authors build a specialized instruction-tuning recipe for computational argumentation (CA), crafting 105 seed tasks, generating 52k CA tasks, and training a Gemma model that beats general LLMs on zero-shot CA evaluations. The result suggests domain-specific instruction tuning can improve unseen tasks within a field without sacrificing general NLP ability.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Held-out CA task instructions are included in the seed pool used to generate the genCA training data, so 'unseen' task generalization is potentially inflated by instruction-level leakage.","rationale":"The central claim of the paper is that CA-specialized instruction fine-tuning significantly enhances an LLM on unseen CA tasks while preserving general capabilities. The reader's identification of leakage in the generation step is correct and is the most load-bearing concern because the 'unseen' evaluation is the linchpin of the claim. The paper explicitly states that all 105 seed tasks were used to generate the 52,445 additional tasks (Section 4.1), and 21 of these seed tasks are later designated as held-out test tasks (Section 5.1). The self-instruct process samples few-shot examples from the full seed pool T0, which includes these held-out tasks. Therefore, information about the held-out tasks' instruction phrasing and structure is baked into the genCA training data. The model does not see the exact held-out tasks, but it sees many generated tasks that are semantically and structurally close to them, which is exactly the kind of exposure that improves zero-shot instruction following. This is a classic test-set contamination issue, distinct from the intended cross-task generalization. It directly affects the headline result in Table 3 (best mean rank 2.0 on unseen tasks) and the comparison against other LLMs in Table 6. The proposed concrete test—regenerating genCA with only the 84 training seed tasks and re-evaluating—would settle the question. If the improvement persists, the method's claim holds; if it vanishes, the paper overstates its contribution. The paper otherwise has merit: the dataset, benchmark, and method are useful, and the general-task stability result is valuable. But the leakage issue cannot be ignored, hence a conditional verdict pending re-validation.","tokens_in":31930,"tokens_out":5201,"duration_ms":51548,"concrete_test":"Retrain the ArgInstruct model (Gemma-2-9B-General + seedCA + genCA) with genCA generated from a seed pool restricted to the 84 training tasks only, excluding the 21 held-out task instructions. Use the same generation hyperparameters (Meta-Llama-3-70B, filtering threshold 0.7, 52k tasks) and the same evaluation protocol. Compare the mean rank and F1 on the 21 held-out CA tasks against the reported values (rank 2.0, F1 .65). If performance degrades materially (e.g., rank worsens by more than 1.5 or F1 drops more than 0.05), the leakage is real and the 'unseen' claim is overstated; if performance is stable, the concern is refuted.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.1 states that 'Based on the 105 seed tasks, we generated 52,445 additional CA tasks.' The 105 seed tasks T0 include the 21 tasks from 9 datasets later reserved as held-out test tasks (Section 5.1). In the self-instruct generation loop (Section 3.1), each generation step randomly samples few-shot instructions from I0 ∪ I<i, i.e., the full seed pool including the held-out instructions. These generated instructions become the genCA training set. Consequently, the model is trained on instructions that are derived from and structurally similar to the held-out tasks, violating the 'unseen' premise of the zero-shot evaluation in Table 3. The diversity filter (ROUGE-L < 0.7) keeps generated instructions within a similarity band of all seed instructions, so the training distribution is explicitly shaped to be close to the held-out tasks without being identical. The reported improvement on unseen tasks (rank 2.0, F1 .65) could thus reflect the model having seen paraphrases or variants of the test tasks rather than true generalization. This directly undermines the central claim of significant enhancement on 'unseen' CA tasks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ArgInstruct, a method for specialized instruction fine-tuning in computational argumentation (CA). The authors manually craft 105 seed tasks from 30 argumentation corpora, use a self-instruct-style pipeline with Meta-Llama-3-70B to generate about 52k additional CA tasks, and fine-tune Gemma-2-9B on combinations of seed, generated, and general instruction data. They evaluate on held-out instances of training tasks, on 21 tasks reserved as 'unseen', on SuperNI, and against task-specific SOTA and several instruction-following LLMs. The central claim is that combining seedCA, genCA, and general data yields the best mean rank on unseen CA tasks (rank 2.0) while preserving general instruction-following performance.","tokens_in":32105,"tokens_out":6879,"duration_ms":70179,"significance":"If the unseen-task results are valid, the paper makes a useful contribution: a large CA instruction dataset, a 105-task benchmark, and evidence that domain-specialized instruction tuning can outperform general instruction-following models of comparable size. The release of code and data, the per-task results in the appendices, the manual quality evaluation of generated data, and the use of significance tests are clear strengths. However, the central generalization claim is conditional on the test tasks being excluded from the data-generation pool, and the current setup does not satisfy that condition.","major_comments":[{"comment":"The 'unseen' test tasks are not truly unseen during data generation. T0 contains all 105 seed tasks, including the 21 tasks from the nine datasets marked with '*' in Table 1 and reserved for testing in Section 5.1. Section 4.1 states that 'Based on the 105 seed tasks, we generated 52,445 additional CA tasks,' and Section 3.1 describes a self-instruct loop where each generation step samples few-shot instructions from I0 ∪ I<i, with I0 being all seed instructions. The diversity filter (ROUGE-L < 0.7) additionally keeps generated instructions within a similarity band of every existing seed instruction, including the held-out ones. Moreover, instance generation uses 'tasks from T0 representative of each task type,' so held-out tasks also serve as templates for the generated input-output instances. As a result, the genCA training data likely contains instructions and instance patterns that are paraphrases or near-variants of the held-out test tasks. The key result in Table 3(b) — that the full combination (seedCA+genCA+general) outperforms seedCA alone on unseen CA tasks (rank 2.0 vs. 5.7) — is therefore confounded: the improvement may reflect training on task-structure information derived from the test tasks rather than genuine zero-shot generalization. This issue is load-bearing for the abstract and Section 5.2. Please regenerate genCA using only the 84 training seed tasks (or otherwise exclude the held-out instructions from the generation, filtering, and instance-generation pools) and re-run Table 3 and all subsequent comparisons that depend on it.","section":"Section 4.1 and Section 5.1"},{"comment":"The SuperNI evaluation is under-specified, which weakens the second half of the paper's central claim that general instruction-following ability 'remains stable.' The manuscript does not state which subset of the 1600+ SuperNI tasks was used, how the aggregate F1 and ROUGE-L scores were computed, or how the mean rank in Table 4 was obtained. With only four rows and no per-task results, the reader cannot independently verify that specialization does not harm general performance. Please provide the SuperNI evaluation protocol, the list of tasks, the aggregation method, and per-task results, ideally following the standard SuperNI evaluation setup.","section":"Section 5.3 and Table 4"},{"comment":"The Wilcoxon signed-rank tests are reported only as footnote markers ('†' and '‡'), without the number of paired observations, the test direction, or effect sizes. Since the unseen-task evaluation has only 21 tasks, these details matter for interpreting the claim of 'significant improvement.' More importantly, given the contamination described in the first major comment, even a statistically significant difference between ArgInstruct and the seedCA-only baseline would not currently establish zero-shot generalization; it would only show that training on generated tasks derived from the held-out instructions improves scores on those same held-out tasks. Please report the test details and re-evaluate after fixing the leakage.","section":"Table 3"}],"minor_comments":[{"comment":"The task name 'Same Aebate Argument' should be corrected to 'Same Debate Argument.'","section":"Appendix E, Table 9"},{"comment":"The reference for Wang et al. (2023) contains a corrupted author string, 'Swaroop and/ Liu'; the correct authorship should be 'Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.'","section":"References"},{"comment":"The Limitations section contains the sentence 'we hope that our analysis in Section 4, which uses the entire test sets, gives readers an idea about the transferability,' but Section 4 contains dataset statistics rather than analyses using the entire test sets; the cross-reference should point to the relevant evaluation section or be reworded.","section":"Section 7 (Limitations)"},{"comment":"Figure 2 is very dense and difficult to interpret because of the repeated arrows and the many small labels; a simplified numbered diagram of the pipeline would improve readability.","section":"Figure 2"},{"comment":"The description of the 100-instance sampling says 'covering the full range for regression,' but it is not clear whether all 100 instances are used for every regression task or how tasks with fewer than 100 unique values are handled; please clarify the sampling procedure.","section":"Section 5.1"},{"comment":"Table 5 reports MAE, while Section 5.1 defines MASE as the regression metric; please align the metric names and definitions between the main evaluation and the SOTA comparison.","section":"Section 5.4 and Table 5"}],"recommendation":"major_revision","confidential_remarks":"The leakage concern is the primary reason for the major-revision recommendation. The paper would be a solid contribution if the authors regenerate the generated data from only the training seed tasks and the main results still hold. I also suggest asking the authors to clarify whether the released genCA dataset already contains instructions derived from the held-out tasks, since this affects all downstream users of the dataset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that it builds a genuinely useful resource for computational argumentation—105 seed tasks with hand-written instructions, a 52k synthetic CA task set, and a clean training setup—but the headline claim about generalizing to unseen CA tasks is weaker than it looks because the held-out test instructions were part of the seed pool used to generate the synthetic training data.\n\nWhat's actually new: they adapt self-instruct to a domain, combine domain tasks with general instruction data, and show the combination beats ablations on their benchmark. The paper ships code and data, and the ablations are well-designed. Their comparison to general LLMs, including GPT-4o-mini, is a fair sanity check. They also honestly report that the generated data is noisy (62.5% correct outputs) and that their model loses to task-specific SOTA on all six tasks it was tested on, though it lands close on three.\n\nThe soft spot is the leakage. Section 4.1 says they generated 52k tasks \"based on the 105 seed tasks,\" and the held-out 21 tasks from 9 datasets are part of that pool. In the self-instruct loop, the generator samples few-shot instructions from the full seed set, so the held-out task instructions (and their label schemes) are feeding into the generated instructions that the model is trained on. The ROUGE-L filter keeps generated instructions from being identical to any seed instruction, but it does not prevent them from being paraphrases or variants of the held-out tasks. So the rank 2.0 on \"unseen\" tasks is at least partly a measure of how well the model learned the structure of the test tasks, not pure zero-shot transfer. The authors should re-run with the held-out instructions excluded from the generation pool, or at minimum report results under that cleaner split.\n\nMinor points: the regression numbers are all poor across models, which the paper acknowledges; the 62.5% quality on generated outputs is a real caveat but not disqualifying; and the six-task SOTA comparison is a nice grounding even though ArgInstruct loses.\n\nOverall: this is a solid contribution from an empirical standpoint, and the leakage is fixable with a clean split. The benchmark and dataset alone are worth having. I'd send it to peer review, with the leakage as the main point to resolve. For my own work, I'd cite the benchmark but not the \"unseen generalization\" claim until it's re-validated.","headline":"Useful CA benchmark and dataset, but the 'unseen task' claim is inflated by leaking held-out instructions into the generation seed pool.","tokens_in":32650,"tokens_out":3520,"would_cite":true,"duration_ms":38429,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A domain-specific instruction fine-tuning recipe for computational argumentation makes an LLM stronger on unseen CA tasks without sacrificing its general instruction-following ability.","keywords":["computational argumentation","instruction fine-tuning","self-instruct","domain specialization","zero-shot generalization","argument mining","argument assessment","argument generation"],"falsifier":"Retrain the pipeline while sampling few-shot examples for instruction generation only from the 84 training seed tasks (excluding the 21 held-out tasks), then evaluate zero-shot on the held-out tasks; if the gains over the base model shrink or vanish, the reported unseen-task improvement is substantially caused by leakage rather than transferable specialization.","tokens_in":31719,"feed_emoji":"💬","tokens_out":3653,"duration_ms":38974,"temperature":0.7,"pith_summary":"The paper tries to establish that an LLM can be turned into a computational-argumentation specialist without losing its general instruction-following ability. It manually crafts instructions for 105 existing CA tasks, synthesizes 52,445 more CA tasks through a self-instruct loop, and fine-tunes Gemma-2-9B on CA tasks alone or mixed with general instruction data. On 21 held-out CA tasks, the full mixture of seed CA, generated CA, and general tasks achieves the best mean rank (2.0), and the final model beats several instruction-following LLMs in zero-shot evaluation (best F1 .65, mean rank 2.33). On the SuperNI general benchmark, performance remains stable, supporting the claim that specialization and generalization can coexist.","feed_headline":"Domain tuning lifts LLMs on unseen argumentation tasks","feed_subtitle":"ArgInstruct blends 52k generated CA tasks with general data and beats instruction-tuned rivals zero-shot.","key_machinery":"The key machinery is the adaptation of the self-instruct process to a single knowledge domain: an LLM generates new CA instructions using few-shot examples drawn from the seed task pool, filters them for CA relevance and novelty (ROUGE-L F1 threshold 0.7), then creates input-output instances per task type (classification, regression, generation). Mixing the resulting 52k CA tasks with general instruction tasks at fine-tuning time is what lets the model keep its generalization abilities while gaining CA specialization.","core_discovery":"Specialized instruction fine-tuning for computational argumentation significantly improves zero-shot generalization to unseen CA tasks while preserving general NLP performance. The authors construct a seed set of 105 CA tasks from 30 corpora, generate 52,445 additional CA tasks with a self-instruct process adapted to the domain, and fine-tune Gemma-2-9B on combinations of seed CA, generated CA, and general instruction data. The combination of all three sources ranks best on unseen CA tasks (mean rank 2.0), and the final ArgInstruct model outperforms comparable instruction-following models on unseen CA tasks in zero-shot evaluation (F1 .65, mean rank 2.33), while retaining its performance on the SuperNI benchmark.","pith_inferences":["If the leakage from sampling few-shot examples over the full seed pool including held-out tasks is real, the reported gains on 'unseen' tasks are an upper bound; retraining with a clean split would likely shrink the margin, though relative ordering of data mixtures may persist.","Regression-type CA tasks remain unsolved by all tested models (all MASE scores above the mean baseline), suggesting that domain-specialized instruction tuning may need task-type-specific output decoding or training objectives for such tasks.","A direct transfer test to another knowledge-intensive domain, such as education, would clarify whether the mechanism is domain-specialized instruction diversity or something specific to argumentation data; the paper leaves this to future work."],"forward_implications":["CA-specialized instruction tuning yields a single model that handles argument mining, assessment, and generation tasks in zero-shot mode, without task-specific fine-tuning.","Mixing general instruction data with domain data prevents catastrophic forgetting of general NLP abilities, as measured on SuperNI.","The method is a general recipe: any domain with a collection of seed tasks and datasets could be turned into a specialized instruction-following model.","For tasks with large amounts of task-specific training data, dedicated fine-tuned models still outperform the specialist, so the approach targets broad-coverage settings rather than per-task peak performance."],"supporting_citations":[{"why":"Supplies the self-instruct generation loop, filtering criteria, and task-type taxonomy that the paper adapts to generate 52k CA tasks.","marker":"Wang et al. (2023)"},{"why":"Provides the 52k-instance scale target and the general instruction dataset used for the general fine-tuning component.","marker":"Taori et al. (2023)"},{"why":"Provides the SuperNI benchmark used to test whether general instruction-following ability is preserved after CA specialization.","marker":"Wang et al. (2022)"},{"why":"Provides the Gemma-2-9B base model that is fine-tuned into the CA-specialized variants.","marker":"Gemma Team et al. (2024)"},{"why":"Motivates the need for argumentation-specific knowledge in instruction-following LLMs, the premise behind CA-specialized tuning.","marker":"Wachsmuth et al. (2024)"},{"why":"Establishes the recent baseline of LLM performance on computational argumentation tasks that the paper aims to improve upon.","marker":"Chen et al. (2024a)"}],"fun_headline_variants":["ArgInstruct: 52k instructions sharpen LLM argumentation","Argumentation-specialized tuning lifts zero-shot performance","52k generated instructions improve unseen argumentation tasks","Domain-specific instruction tuning boosts LLM argumentation","LLM fine-tuned on argumentation outperforms general models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 21 tasks used to measure 'unseen' generalization are assumed to be entirely new to the model, but the instruction-generation step samples few-shot examples from the full seed task pool including those same 21 tasks, so the generated training data may encode the structure of the test tasks.","fun_headline_variants_meta":{"raw":{"variants":["ArgInstruct: 52k instructions sharpen LLM argumentation","Argumentation-specialized tuning lifts zero-shot performance","52k generated instructions improve unseen argumentation tasks","Domain-specific instruction tuning boosts LLM argumentation","LLM fine-tuned on argumentation outperforms general models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000839,"raw_usage":{"total_tokens":3620,"prompt_tokens":868,"completion_tokens":2752,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":2673}},"tokens_in":484,"tokens_out":2752,"duration_ms":22286,"temperature":1.0,"reasoning_tokens":2673,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:15:02.205189+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the pipeline while sampling few-shot examples for instruction generation only from the 84 training seed tasks (excluding the 21 held-out tasks), then evaluate zero-shot on the held-out tasks; if the gains over the base model shrink or vanish, the reported unseen-task improvement is substantially caused by leakage rather than transferable specialization.","supporting_citations":[],"review_version":1}