{"id":"117c726b-d51d-413e-a2bf-6958de66acc7","arxiv_id":"2608.10579","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":11,"one_line_summary":"An agentic four-stage LLM cascade turns natural-language selection intent into curated instruction subsets and outperforms static baselines on most 7B-8B SFT settings.","lead":"DataMaster is an agentic pipeline that reads a natural-language data-selection instruction and uses four LLM-driven stages to pick a 10K training subset from a larger instruction pool. In experiments across math, medical, and code domains, it beats static selection baselines in most settings and sometimes beats training on the full pool.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Learning-rate selection and single-seed reporting may inflate DataMaster's headline win counts; a baselines-only LR choice with multi-seed reruns is required to support 16/18 and 12/18.","rationale":"I read the paper as making a central empirical claim: a 10K subset chosen from a natural-language instruction by DataMaster beats static baselines in 16 of 18 single-domain settings and beats full-pool training in 12 of 18 settings. What must be true for that claim to hold is that the comparison itself is fair and stable. The reader's weakest assumption concerns whether LLM-generated rubrics and weights are reliable proxies for downstream quality. That is a real gap, but it is secondary to the comparison protocol: even if the rubrics are not the cause, the method might still outperform; whereas if the shared learning rate is selected using DataMaster's own benchmark scores, or if single-seed noise reverses a few small margins, the headline empirical claim itself is unsupported. I therefore agree partially with the reader: the LR selection and variance issues were listed among the reader's risks, but the rubric-proxy issue was chosen as the weakest assumption. The paper deserves credit for disclosing the single-seed protocol and the lack of variance quantification, and for providing per-benchmark tables and a public code link, which makes the proposed recheck feasible rather than speculative. The reader's CONDITIONAL verdict remains appropriate, with the condition sharpened to include an LR-selection robustness check and multi-seed results.","tokens_in":939,"tokens_out":849,"duration_ms":80734,"concrete_test":"Re-run the 18 single-domain settings at all three learning rates with at least 5 seeds, but select each setting's shared LR using only the baseline methods' average or a held-out validation split that is never used for final reporting. Then recompute DataMaster's win counts against the six static baselines and against Full Pool in Table 1. If the counts fall materially below 16/18 or 12/18, or if the direction reverses under alternative seeds, the headline claim is not robust to the protocol; if the counts persist with consistent margins, the concern is resolved. The rerun should also report the chosen LR values per setting and the seed-level variance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the fairness and stability of the headline empirical comparison, not the internal rubric mechanism. Appendix I describes a protocol in which, for each setting, every method is run at all three retained learning rates, and the rate with the highest setting-level average across all methods is chosen; all methods are then reported at that shared rate on a single seed (42). The paper explicitly states that multi-seed variance is not quantified. Because DataMaster has the highest scores, it dominates the average that selects the shared LR, so the hyperparameter is partly chosen on DataMaster's own benchmark performance. The evaluation benchmarks thus serve both as the selection criterion for the comparison's learning rate and as the outcome measure, with no held-out validation. Several reported margins are small, for example Table 2 OpenHermes Code shows 41.11 for DataMaster versus 41.07 for Claude Code, and Table 1 contains multiple sub-1-point leads. The headline claims of 16/18 over static baselines and 12/18 over Full Pool are count-based, so even a few reversals would materially weaken the central claim. This concern is directly testable because the paper discloses the protocol, provides per-benchmark tables, and links public code.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DataMaster, an agentic instruction-data-selection system that interprets a natural-language selection instruction and composes a four-stage cascade of domain, characteristic, informativeness, and quality filters to curate a 10K subset from an instruction pool. Each stage is configured by the same frozen LLM (DeepSeek-V4-Flash), with tool-assisted pool inspection, target-model metrics, and an LLM-generated quality rubric. Across three 7B–8B target models and eight source pools, the reported experiments claim that DataMaster outperforms six static baselines in 16 of 18 single-domain settings, outperforms Claude Code in 14 of 18 settings, and beats full-pool training in 12 of 18 settings, with additional multi-domain gains.","tokens_in":29098,"tokens_out":5920,"duration_ms":56381,"significance":"If the empirical claims hold, this is a useful and timely contribution: it replaces per-task handcrafted selection heuristics with intent-conditioned, agentic orchestration, and it ships a transparent protocol, per-benchmark tables, prompt templates, and public code. The central idea is clearly novel relative to static selection metrics, and the low API cost (about US$4–5 per selection task) makes the system practical. The main obstacle is the fairness and stability of the headline comparison: the shared learning rate is selected using the reported benchmark scores themselves, and all results come from a single seed with no variance quantification. Because the headline claims are count-based and several margins are below one point, the comparison protocol must be repaired before the counts can be accepted. The same-LLM rubric-generation/judging coupling is a secondary internal-validity concern but is not a circular derivation, since the final numbers come from external benchmarks.","major_comments":[{"comment":"The shared learning rate is selected per setting by running every method at all three retained rates and picking the rate with the highest setting-level average across all methods. Because DataMaster has the highest scores in most settings, it disproportionately influences the average that selects the rate, so the evaluation benchmarks serve simultaneously as the learning-rate selection criterion and as the outcome measure, with no held-out split. The paper's central claims are count-based (16/18 over static baselines, 12/18 over Full Pool, 14/18 over Claude Code), and several margins are sub-1-point (e.g., Table 2, OpenHermes-2.5 Code: 41.11 vs. 41.07). This protocol can therefore materially inflate DataMaster's win count. Please rerun the comparison with multi-seed variance and with a learning-rate choice that is independent of the reported test scores, such as a per-model fixed LR or an LR selected on a development split, and report whether the win counts survive.","section":"Appendix I, 'Fine-tuning protocol' and 'Runs per setting'"},{"comment":"The paper explicitly states that 'multi-seed variance is not quantified' and that each setting is reported on a single seed (42). Given that the headline claims are counts of wins and losses and that multiple reported margins are well under one point, the absence of variance estimates leaves the 16/18 and 12/18 claims statistically unsupported. At a minimum, report 3–5 seeds for DataMaster and the strongest baselines in the settings with small margins, together with confidence intervals or per-seed ranges, and state whether the rank order is stable across seeds.","section":"Appendix I, 'Runs per setting'"},{"comment":"The same backbone LLM both generates the quality rubric (Eq. 13) and applies it as the judge (Eq. 14), and no analysis shows that these LLM-judge scores predict downstream fine-tuning quality. Because the final benchmark numbers are external, this is not a circular derivation, but it is an unvalidated internal proxy: if the judge has systematic taste biases, the selected subset is corrupted even though the reported accuracies remain independent. I ask for a sensitivity analysis, for example comparing against a fixed generic rubric, or a correlation of LLM-judge scores with per-sample downstream utility; if that is infeasible, the limitation should be stated explicitly and its potential impact on the quality-stage contribution discussed.","section":"Section 4, 'Stage 4: Quality Agent' (Eqs. 12–14); Appendix G.4"}],"minor_comments":[{"comment":"The full-text title reads 'LetDataMasterInterpret Your Intent' without spaces; this should be corrected to 'Let DataMaster Interpret Your Intent'.","section":"Title/Front matter"},{"comment":"The column layout of Table 1 is difficult to read because the column headers are not clearly separated from the row labels and the Full Pool column is interleaved with the selection methods; a clearer table design with explicit column headers would improve reproducibility.","section":"Table 1 and Appendix C.1"},{"comment":"Select2Reason is a reimplementation because the original implementation is not public; please release this reimplementation in the public repository and note the exact version used so that the baseline is fully reproducible.","section":"Appendix H, Select2Reason"},{"comment":"The related-work section cites agentic curation systems that optimize through downstream feedback, but no such system is included as a baseline; a direct comparison, or a clear statement of why those systems are out of scope, would strengthen the positioning of the feedback-free design.","section":"Section 2 and Appendix K"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well written, the protocol is unusually transparent, and the public code and per-benchmark tables are strong assets. The main obstacle is the learning-rate selection on the reported benchmarks and the single-seed reporting; multi-seed reruns with a held-out LR choice would likely resolve the issue. The Stage-4 judge/rubric coupling should also be addressed with a sensitivity check or an explicit limitation. If those points are addressed, the paper would be close to acceptable for publication in a serious journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DataMaster is a sensible engineering contribution: it wraps four known pieces—embedding clustering, rule-based filters, NLL/entropy/drift/MeanDiff scoring, and LLM-as-judge—into an intent-conditioned cascade that selects 10K samples from an existing SFT pool given a natural-language instruction. The writing is clear, the prompts are in the appendix, the code is linked, and the multi-domain results are consistent. If the gains hold, this is a practically useful tool: it beats full-pool training in 12/18 settings at roughly $5 per curation task.\n\nThe paper is honest about its protocol, which helps: Appendix I discloses the shared-LR selection and the single seed. That protocol is also the main problem. In each setting, the authors run all methods at three learning rates, average setting-level scores across methods, and pick the rate with the highest average. DataMaster dominates those averages, so its own benchmark scores help choose the learning rate that everyone is then compared at. That is a mild form of selection on the evaluation set: the benchmarks are both the criterion for picking a shared hyperparameter and the outcome measure. With several margins under half a point (Table 2 OpenHermes Code: 41.11 vs 41.07), the count-based claims of 16/18 and 12/18 could flip on re-runs. The paper states that multi-seed variance is not quantified; that is a real gap, not a nitpick. Also, the quality rubric and metric weights come from the same LLM that acts as judge, and there is no validation that those scores predict downstream accuracy. Author reimplementations of Select2Reason and other baselines add another reason to want multi-seed runs.\n\nNone of this is fatal. The central idea is not circular—selection itself never sees the benchmarks—and the ablation suggests each stage contributes. But the headline numbers need multi-seed runs with significance tests, and ideally an LR chosen from a validation split or from baselines-only averages. A held-out commit hash would help reproducibility.\n\nWho this is for: anyone working on data curation or fine-tuning pipelines. It is a reasonable baseline and a clear description of a new design. It deserves a serious referee, and I would accept after the experiments are tightened. Bring it to reading group—the LR selection protocol is a good cautionary example.","headline":"A useful, clearly-written agentic data-selection tool whose headline win counts are undercut by a benchmark-influenced LR choice and single-seed reporting.","tokens_in":29631,"tokens_out":2471,"would_cite":true,"duration_ms":23213,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that instruction-data selection should be configured dynamically from a user's natural-language intent and the candidate pool, and it reports that its four-stage agent pipeline beats static baselines in 16 of 18…","keywords":["instruction data selection","agentic pipeline","large language models","supervised fine-tuning","natural-language intent","data curation","LLM-as-judge","informativeness metrics"],"falsifier":"Take one pool and two contrasting user instructions, run DataMaster on each, and compute the overlap of the two selected 10K subsets; if the subsets are nearly identical, or if a single fixed high-quality subset outscores both on both target benchmarks, then intent-conditioned dynamic selection is not what drives the reported gains.","tokens_in":28602,"feed_emoji":"🎯","tokens_out":8900,"duration_ms":74815,"temperature":0.7,"pith_summary":"DataMaster is an agentic system that turns a natural-language instruction into a full instruction-data selection strategy. Its claim is that selection cannot be a static, one-size-fits-all metric because the right subset depends jointly on the user's goal and the heterogeneous composition of the pool. The system implements this with four cascaded agents that handle domain matching, characteristic filtering, informativeness scoring, and quality judging, all configured at run time from the instruction and a tool-assisted inspection of the data. In experiments on three 7B-8B models, 10K-sample subsets chosen this way beat static baselines in 16 of 18 single-domain settings and beat training on the full uncurated pool in 12 of 18, with per-task API costs below $5. If the claim holds, developers can replace manual heuristic design with plain-language curation requests.","feed_headline":"One prompt curates better data than static selectors in 16 of 18 tests","feed_subtitle":"DataMaster reads your natural-language request and builds a four-stage selection strategy, beating full-pool training in 12 of 18 settings","key_machinery":"The load-bearing machinery is a four-agent cascade, each stage reading the same user instruction and the data it receives. The Domain Agent embeds and clusters the pool, asks an LLM to label each cluster, and keeps only clusters matching the instruction; the Characteristic Agent inspects the filtered pool through a sandboxed data-analysis tool and emits deterministic filtering rules such as length bounds, refusal-phrase blacklists, and noise caps; the Informativeness Agent computes four target-model metrics (negative log-likelihood, token entropy, semantic drift, and MeanDiff, the magnitude of parameter change caused by one gradient step on a sample) and weights them according to the instruction; the Quality Agent synthesizes a task-specific rubric and applies it in batched listwise LLM judging to pick the final subset. The same frozen LLM configures all stages, while the target model is used only for the Stage-3 metrics, and the whole design keeps the selection objective conditioned on the user's request at every step instead of collapsing it into one proxy.","core_discovery":"The paper's central claim is that instruction-data selection should be configured dynamically from a user's natural-language intent and from the actual contents of the candidate pool, not by a fixed metric. It argues that the same pool should be curated differently for different goals because the optimal subset depends on the target task, the data distribution, and the target model together. The paper instantiates this as DataMaster, a four-agent cascade that composes domain matching, characteristic rule filtering, informativeness scoring against the target model, and LLM-judged quality at run time. Across six single-domain pools and two multi-domain pools, with three 7B-8B target models, the 10K subsets produced this way beat static selectors in 16 of 18 single-domain settings, beat the strongest static multi-dimensional baseline in 13 of 18, and beat full-pool training in 12 of 18 settings while using 2-41 times less data.","pith_inferences":["Because the selected subset is defined relative to an instruction, benchmark comparisons of selection methods may need to be reported per intent; a method's 'best subset' is not a property of the pool alone.","A direct test of the self-judging design would swap the stage-four LLM judge for an independent model or human scores; if gains persist, intent-conditioned rubric generation, not judge identity, is what matters.","The emitted rule policies and metric weights could be recorded as reusable curation recipes, letting users transfer a strategy across pools or audit what the agent actually optimized."],"forward_implications":["A developer can curate a task-specific fine-tuning set by writing a sentence in natural language, removing the need to hand-craft per-dataset heuristics.","A compact 10K subset selected for the stated intent can outperform training on the entire uncurated pool in most tested settings, so compute and data budgets can shrink without sacrificing benchmark scores.","Dynamic, agentic selection is a workable replacement for static metric-based selection in both single-domain and multi-domain pools, including target-domain extraction from mixed pools.","The first-stage domain filter can be reused as a preprocessing module for existing static selectors, improving them, which supports the value of separating domain selection from the rest of the pipeline."],"supporting_citations":[{"why":"Supplies MeanDiff, the per-sample parameter-change metric that the Informativeness Agent adapts from ResoFilter's resonance score.","marker":"Tu et al. 2025"},{"why":"Defines SuperFiltering, the single-dimensional informative-data baseline that DataMaster is compared against in single-domain tests.","marker":"Li et al. 2024a"},{"why":"Defines SelectIT, an uncertainty-aware informativeness baseline in the single-domain comparisons.","marker":"Liu et al. 2024a"},{"why":"Defines MIG, a quality-by-information-gain baseline that DataMaster must beat.","marker":"Chen et al. 2025"},{"why":"Defines Select2Reason, the difficulty-plus-reasoning-length baseline in the single-domain comparisons.","marker":"Yang et al. 2026"},{"why":"Defines DEFT, a multi-dimensional complexity-quality-knowledge baseline that is the strongest static competitor.","marker":"Dai, Liu, and Yu 2026"},{"why":"Defines DSIR, a target-guided importance-resampling baseline for the multi-domain comparisons.","marker":"Xie et al. 2023"},{"why":"Defines TSDS, an optimal-transport target-guided baseline for the multi-domain comparisons.","marker":"Liu, Karbasi, and Rekatsinas 2024"}],"fun_headline_variants":["DataMaster turns your intent into custom data selection","Your prompt writes the recipe: DataMaster curates data","Describe your goal, DataMaster picks the best data subset","Natural language to smarter data selection: DataMaster wins","DataMaster: one prompt beats fixed selectors in 16/18 tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM's own quality rubrics and metric weights are trustworthy proxies for what actually improves downstream fine-tuning; if that judge has systematic taste biases, the selected subset can be skewed even though the final benchmark scores are measured independently.","fun_headline_variants_meta":{"raw":{"variants":["DataMaster turns your intent into custom data selection","Your prompt writes the recipe: DataMaster curates data","Describe your goal, DataMaster picks the best data subset","Natural language to smarter data selection: DataMaster wins","DataMaster: one prompt beats fixed selectors in 16/18 tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1225,"prompt_tokens":902,"completion_tokens":323,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":241}},"tokens_in":518,"tokens_out":323,"duration_ms":3382,"temperature":1.0,"reasoning_tokens":241,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:31:02.414767+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one pool and two contrasting user instructions, run DataMaster on each, and compute the overlap of the two selected 10K subsets; if the subsets are nearly identical, or if a single fixed high-quality subset outscores both on both target benchmarks, then intent-conditioned dynamic selection is not what drives the reported gains.","supporting_citations":[{"cited_title":"Data Selection for Language Models via Importance Resampling , booktitle =","cited_arxiv_id":null,"evidence_quote":"Defines DSIR, a target-guided importance-resampling baseline for the multi-domain comparisons."}],"review_version":1}