{"id":"8f757c27-4e06-4ee6-8c64-2b97d6a5d4cb","arxiv_id":"2412.11455","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A neural-network-guided iterative search over dataset combinations is claimed to improve multi-task LLM performance, but the paper's own figures and text contradict each other and no baselines or error bars are provided.","lead":"A neural network is trained to guess which combination of datasets will improve a large language model's performance on multiple tasks, and the guesses are tested in repeated rounds. The paper reports gains on several biomedical datasets, but its own figures contradict the text and no random-search baseline is provided, so the core claim is not established.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Internal contradictions in the reported F1 curves undermine the central empirical claim; the DDI figure shows large gains while the text states no better combinations were found.","rationale":"The reader's weakest assumption focused on the unvalidated neural-network surrogate. That is indeed a serious gap, because the efficiency and optimality claims collapse if the predictor cannot rank untested combinations. However, an even more load-bearing concern precedes it: the paper's own reported F1 numbers are internally inconsistent, which means the empirical foundation for the claim is missing. If the DDI figure shows a large improvement while the text says none exists, the reader cannot even determine what the framework actually did. The concrete test I propose would settle whether the inconsistency is an artifact of plotting or a genuine contradiction, and whether the claimed improvements survive replication. My analysis reinforces the reader's REJECT verdict rather than changing it, so the verdict remains unchanged; I only partially agree with the reader's identification of the weakest assumption because the unvalidated surrogate matters, but the internal measurement inconsistency is the more immediate blocker.","tokens_in":151,"tokens_out":2281,"duration_ms":29802,"concrete_test":"Re-run the DDI and BioRED experiments using the exact combination ordering implied by the framework loop, with at least three random seeds, and record the F1 after every iteration. Then check whether the resulting best-F1 trajectory matches Fig. 4(b) and whether DDI actually fails to improve despite the figure. If Fig. 4(b) does not reproduce, the central result is internally invalid; if it does reproduce, the text still contradicts the figure and the claimed improvement remains unsubstantiated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the neural-network-guided search finds dataset combinations that improve F1 over single-task baselines while using far fewer fine-tuning runs. This claim depends entirely on reliable measurements of F1 for each tested combination. Section 5.2 states that for DDI 'the framework cannot find better combinations and stops soon,' and Table 1 shows only 4 of 36 DDI combinations improving over the 71.10 baseline, with the best at 73.94. Yet Fig. 4(b) plots best F1 rising from about 71 to about 78 across roughly 18 iterations, implying substantial improvements that are never mentioned or tabled. Similar contradictions appear elsewhere: GENIA2013 is described as improving to 'around 58' F1 while Fig. 5(b) has a y-axis that ends at 55, and the BC4CHEMD curve in Fig. 5(e) shows F1 values above 100, which is impossible for a bounded metric. These are not minor typos; they affect the primary evidence for the framework's effectiveness. Additionally, no seeds, error bars, random-search baseline, or held-out validation of the surrogate are provided, so the reported gains cannot be separated from fine-tuning noise or checkpoint selection. If the measured F1 values are even partly inconsistent, the paper offers no trustworthy evidence that the framework identifies better combinations at all.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an iterative framework for selecting auxiliary training datasets in multi-task instruction tuning of LLMs. A neural-network regression surrogate is trained on evaluated dataset-combination F1 scores; after each fine-tuning run the surrogate is retrained, all 2^11 auxiliary combinations are enumerated at low cost, and the predicted best combination is tested until the surrogate repeats an already-tested combination. Experiments are reported on 12 biomedical datasets across NER, RE, EE, and TC with LLaMA-3-8B. The paper claims that the method finds F1-improving combinations with far fewer fine-tuning runs than brute-force enumeration, including for tasks where human intuition sees no MTL benefit.","tokens_in":19982,"tokens_out":6152,"duration_ms":51707,"significance":"If the claims were backed by reliable measurements, the framework would be a useful practical contribution to MTL dataset selection: it is model- and dataset-agnostic in principle, and surrogate-guided search has the potential to reduce computational cost substantially. The paper also deserves credit for evaluating on 12 datasets and for formulating the selection problem as an iterative, feedback-driven search. However, the current evidence is not trustworthy. The central empirical claim rests on F1 curves that contradict the corresponding tables and text, and the surrogate's ranking ability is never validated against held-out predictions or a random-search baseline. The scientific contribution is therefore not established.","major_comments":[{"comment":"The text states that for DDI 'the framework cannot find better combinations and stops soon,' and Table 1 shows only 4 of 36 DDI combinations improving over the 71.10 baseline, with the best at 73.94; nevertheless Fig. 4(b) plots best F1 rising from about 71 to about 78 over roughly 18 iterations. These are incompatible, and because Fig. 4 is the primary evidence for the RE task, the central claim is not supported for DDI.","section":"§5.2, Fig. 4(b), Table 1"},{"comment":"The text reports that GENIA2013 improved 'to around 58,' while the y-axis of Fig. 5(b) ends at 55 and the plotted values do not reach 58. Similarly, Fig. 5(e) for BC4CHEMD shows a y-axis extending past 100 and F1 values above 100, which is impossible for F1. These inconsistencies make the figures unusable as quantitative evidence for the framework's effectiveness.","section":"§5.3, Fig. 5(b), Fig. 5(e)"},{"comment":"The framework's stopping rule and efficiency claim depend on the surrogate's ability to predict untested combinations, but the paper provides no held-out evaluation of the surrogate, no correlation analysis, and no random-search or other model-based baseline. Without such a baseline, the claim that the method 'significantly improved efficiency' relative to brute force is not established; the loop could simply be performing noisy exploration.","section":"§3, §4.4, §5.3"},{"comment":"All results are reported without seeds, error bars, or repeated runs, and the best checkpoint is selected from checkpoints saved every 1000 steps. Given that many reported F1 differences are small (e.g., BioRED improvements of about 0.5–1 point in Table 1), fine-tuning noise could dominate the reported gains, and the paper should at minimum report variance or significance tests.","section":"§4.5, §5.1"},{"comment":"The text says that for NER and TC 'every combination we attempted resulted in a performance decline' and that the framework later found improvements for BC2GM, BC5CDR, and HealthAdvice; however, the appendix tables mark the baselines as the best among the data-preparation runs, and no tabled framework-iteration F1 values are provided for these three datasets. The reader therefore cannot verify the claimed 'unpromising tasks' improvements from the reported data.","section":"§5.3, Figs. 5(d), 5(f), 5(h), Tables 2–4"}],"minor_comments":[{"comment":"The sentence 'we enumerate 211 = 2048kinds of combinations' contains a typo: it should read '2^{11} = 2048,' and 'nerual' should be 'neural.'","section":"§5.2"},{"comment":"The prompt templates contain typos: 'You need to to accurately recognize' and 'sentece' should be corrected.","section":"Fig. 3"},{"comment":"The correspondence email address is listed as zhan1386@umn.edu for Rui Zhang, while the affiliations list Zaifu Zhan as the first author; this should be checked for accuracy.","section":"Title page"},{"comment":"The model name is written 'LLama3-8B'; the standard spelling is 'LLaMA-3-8B,' and the name should be used consistently.","section":"§4.2"},{"comment":"The appendix tables are dense and repeat checkmark patterns in a way that is hard to scan; a compact representation of the selected datasets, such as bit strings, would improve readability.","section":"Tables 1–4"}],"recommendation":"reject","confidential_remarks":"The internal contradictions in the reported F1 curves (DDI, GENIA2013, BC4CHEMD) are severe enough that I cannot treat any of the empirical results as reliable. Even if the figures were corrected, the missing random-search baseline and lack of surrogate validation would require substantial new experiments. I recommend rejection rather than major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe core idea here is sensible and worth a serious look: use a cheap neural-network surrogate trained on evaluated dataset combinations to rank the remaining combinations, test the top pick, and iterate until the surrogate only suggests already-tested points. That is a legitimate application of surrogate-based optimization to a real bottleneck in multi-task LLM training, and the authors did a lot of fine-tuning runs to explore it. The framework description is clear, and the limitations section is honest about the single-model, single-domain evaluation.\n\nThe problem is that the reported evidence does not hold together. The text says DDI \"cannot find better combinations and stops soon,\" and Table 1 shows only four of 36 combinations improving over the baseline, with a best of 73.94. Yet Figure 4(b) plots the best F1 rising from about 71 to about 78 over the iterations. That is not a typo; it is the primary evidence for the framework's effectiveness on that dataset. The same happens for GENIA2013, where the text claims improvement to around 58, but the figure's y-axis ends at 55, and for BC4CHEMD, where the plotted F1 exceeds 100. These contradictions are load-bearing.\n\nBeyond that, the efficiency claim (dozen iterations vs. 2,048 combinations) is never compared to random search or to existing task-selection methods like AUTOSEM or task-affinity grouping. The surrogate is not validated with held-out predictions, so we do not know if its rankings are any better than random. There are no seeds or error bars, so even the consistent numbers cannot be separated from fine-tuning noise. The stopping rule also depends on the surrogate's own belief that no better combination exists, which is circular without an external check.\n\nThe abstract's model-, dataset-, and domain-independence claim is contradicted by the actual evaluation on one model and one domain; the limitations paragraph acknowledges this, so the overclaim in the abstract is a writing problem as much as an evidence problem.\n\nFor whom is this useful? Someone working on dataset selection could take the framework idea as a starting point, but the paper as it stands does not give trustworthy evidence that the framework finds better combinations. It needs corrected figures, raw numbers, a random-search baseline, and some validation of the surrogate before the empirical claims can be assessed.\n\nI would not cite this for its results, though it might be worth a mention as a related approach. I would send it to a serious referee if the authors submit a revision, because the idea is worth engaging with and the inconsistencies are fixable. As is, it should not be published.\n\nBest,","headline":"A sensible surrogate-based search idea that is undermined by internally inconsistent reported numbers.","tokens_in":20566,"tokens_out":3124,"would_cite":false,"duration_ms":27848,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Paper claims that an iterative neural-network-guided search finds better multi-task dataset combinations than single-task baselines using only a handful of fine-tuning runs.","keywords":["multi-task learning","dataset combination selection","large language models","neural network surrogate","instruction tuning","biomedical NLP","F1-score optimization","iterative search"],"falsifier":"Using the paper's logged combination-score tables, train the same two-layer surrogate on a random half of the combinations, then measure how well its predicted F1 ranks the held-out half (e.g., Spearman correlation); if the ranking is not significantly better than random, or if a random-search loop given the same fine-tuning budget matches the surrogate-guided loop's best F1, the framework's search-efficiency claim would be refuted.","tokens_in":19514,"feed_emoji":"🧠","tokens_out":6697,"duration_ms":54542,"temperature":0.7,"pith_summary":"Multi-task learning for large language models should benefit from choosing which datasets to train on together, but the obvious way to choose—fine-tune on every subset and keep the best—costs exponentially many runs. This paper proposes a feedback loop in which a small neural network learns to predict a dataset combination's F1 score from which datasets are included, and each iteration fine-tunes the LLM only on the combination the network currently ranks best. Across 12 biomedical datasets covering named entity recognition, relation extraction, event extraction, and text classification, the loop finds combinations that beat single-task training in a dozen or so iterations, rather than trying the 2,048 possible auxiliary combinations per target. The central finding is that the surrogate-guided search can improve tasks that look unpromising from a human perspective, such as three NER/TC datasets where every hand-chosen combination had decreased performance. A sympathetic reader would take this as evidence that learned dataset-combination search is a viable route to unlocking more of multi-task learning's benefit.","feed_headline":"Neural net finds better data mixes for multi-task LLMs","feed_subtitle":"A feedback loop tests only a handful of combinations out of 2,048 possible, improving F1 where experts saw no hope.","key_machinery":"The central object is the surrogate network: a two-layer neural network with 12 binary inputs (one per dataset, set to 1 if the dataset is included) and a single regression output, the predicted F1 score. It is trained on the data table of tested combinations and their measured F1 scores, retrained after every iteration, and then used to enumerate all $2^{11} = 2048$ auxiliary combinations for the target dataset and pick the predicted best. This surrogate is what converts dataset selection from an exponential number of fine-tuning runs into a feedback loop: test, retrain, repredict, repeat, terminating when the predicted best has already been tested. The same mechanism can in principle wrap any LLM, any dataset pool, and any scoring metric, which is what carries the generality claim.","core_discovery":"The paper's claim is that a cheap two-layer neural network can act as a surrogate for expensive fine-tuning and reliably steer an iterative search toward high-F1 dataset combinations. The surrogate takes a 12-bit vector indicating which datasets are in a combination and outputs a predicted F1 score; after each real fine-tuning run, it is retrained on the growing table of observed combination-score pairs, then used to score all remaining combinations so the next run tests the top-ranked one. The loop stops when the network's current top pick is a combination already tested. On the RE-task data, 30 of 36 combinations improved BioRED while all tested combinations hurt GIT, yet the framework still found an improved combination for GIT; across the full experiment it found gains for three datasets (BC2GM, BC5CDR, HealthAdvice) where every human-screened combination had failed. The authors conclude that the framework is model-, dataset-, and domain-independent and greatly improves search efficiency over brute-force enumeration.","pith_inferences":["A natural extension would be to replace the final fine-tune-and-evaluate step with a cheaper proxy (smaller model, fewer steps, or few-shot inference) to lower cost further; the paper itself does not test this substitution.","The bit-vector encoding treats each dataset as an independent binary switch; adding pairwise interaction terms to the surrogate could capture synergy and may improve ranking accuracy for tasks where combinations matter.","The stopping rule stops when the surrogate predicts an already-tested combination, which conflates 'no better combination exists' with 'the surrogate's confidence is exhausted'; a more robust stop would require independent validation on held-out combinations.","The observation that weak baselines benefit most suggests a usable pre-screening heuristic—spend search budget on low-F1 tasks—that the paper reports as a pattern but does not elevate to a decision rule."],"forward_implications":["For a fixed target dataset, the framework reaches a near-best combination in roughly a dozen fine-tuning runs instead of trying all $2^{11}=2048$ auxiliary combinations.","Datasets with low single-task F1 (roughly below 60 on these biomedical tasks) tend to gain the most from multi-task training, while already-strong tasks gain little or nothing.","Useful dataset combinations can be invisible to human judgment: three datasets improved only after the surrogate proposed combinations that a human would not have tried.","Because the surrogate only sees binary dataset indicators and a score, the same loop transfers to other LLMs, dataset pools, and metrics without architectural changes.","The stopping rule means the search ends when the surrogate's predicted best candidate has already been evaluated, which is an automatic budget cap rather than a fixed number of runs."],"supporting_citations":[{"why":"Defines the brute-force 'train every subset' baseline that the proposed framework must beat to justify its efficiency claim.","marker":"(Standley et al., 2020)"},{"why":"Establishes that MTL gains depend on dataset characteristics, motivating a learned selector over hand-crafted rules.","marker":"(Bingel and Søgaard, 2017)"},{"why":"Prior automatic task-grouping method based on inter-task affinity; the paper contrasts its single-run gradient affinity with direct combination-score prediction.","marker":"(Fifty et al., 2021)"},{"why":"AUTOSEM's multi-armed bandit selects auxiliary tasks but only pairwise, the limitation the framework extends to full subsets.","marker":"(Guo et al., 2019)"},{"why":"Large-scale study showing weak correlation between probing and downstream performance, used to argue that exhaustive or learned combination search is needed.","marker":"(Pruksachatkun et al., 2020)"},{"why":"ADELIE shows multi-task instruction tuning improves biomedical LLMs, providing the application context for the 12 biomedical datasets.","marker":"(Qi et al., 2024)"}],"fun_headline_variants":["Surrogate network steers LLM data selection","Efficient search for LLM data mixes via neural net","Neural net predicts best dataset combos for multi-task LLMs","Surrogate network picks winning data combos for LLMs","Neural-guided search finds surprising LLM data mixes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that a small neural network trained on a limited set of observed combination-F1 pairs can rank untested combinations accurately enough to guide the search; if that ranking is poor, the loop degenerates to near-random exploration and the efficiency and optimality claims no longer follow.","fun_headline_variants_meta":{"raw":{"variants":["Surrogate network steers LLM data selection","Efficient search for LLM data mixes via neural net","Neural net predicts best dataset combos for multi-task LLMs","Surrogate network picks winning data combos for LLMs","Neural-guided search finds surprising LLM data mixes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000575,"raw_usage":{"total_tokens":2664,"prompt_tokens":848,"completion_tokens":1816,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":1736}},"tokens_in":464,"tokens_out":1816,"duration_ms":11803,"temperature":1.0,"reasoning_tokens":1736,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:54:46.201780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Using the paper's logged combination-score tables, train the same two-layer surrogate on a random half of the combinations, then measure how well its predicted F1 ranks the held-out half (e.g., Spearman correlation); if the ranking is not significantly better than random, or if a random-search loop given the same fine-tuning budget matches the surrogate-guided loop's best F1, the framework's search-efficiency claim would be refuted.","supporting_citations":[],"review_version":1}