{"id":"9a315a52-0be3-402d-b4b5-81ee37f1e2cc","arxiv_id":"2607.22769","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Reallocating fine-tuning data proportions according to fitted per-domain loss curves improved benchmark scores by 1.8–3.8 points on Qwen3-1.7B, in a single unseeded run; the planned second stage is not yet implemented.","lead":"This paper introduces a lightweight way to improve how large language models are fine-tuned: it monitors how fast the model learns each training domain (math, code, chat), fits simple curves to those learning speeds, and re-balances the data mixture. On a 1.7-billion-parameter model, the re-balanced mixture improved four benchmarks by 1.8–3.8 percentage points with the same data and compute.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's scaling-law fits are too poor (R²=0.18–0.64) to justify the Stage-1 reallocations; without a control reallocation, the reported benchmark gains cannot be attributed to the fitted parameters.","rationale":"The reader's weakest assumption and my own converge on Eq. (1) and Table 2. I looked for an even more fundamental flaw—such as the abstract/title overstating Stage 2, or the unexplained MATH-500 entry in Table 4—but those are presentation or data-consistency issues, not the load-bearing link for the central claim. The decisive link is from Eq. (1) fits to Eq. (2) scores to benchmark improvements. The reported R² values and implausible a_i values mean this link is currently unsupported. I credit the patch-based monitoring architecture as a credible engineering contribution, and the paper's Limitations explicitly say Stage 2 is future work, so I am not inclined to reject the paper outright. The fixes are standard: seeds/error bars, a permuted-control reallocation, and fit-diagnostic plots. If the control reallocation reproduces the gains, the paper's interpretation collapses; if not, the central claim survives. Hence CONDITIONAL remains the right verdict, and my read does not change it.","tokens_in":8125,"tokens_out":4577,"duration_ms":48426,"concrete_test":"Run three seed replicates for (i) original mixture, (ii) DomainPilot's Stage-1 mixture, and (iii) a control mixture with the same delta proportions (+25% termagent, +15% swe, −8% chat, etc.) assigned to random domains. If the control matches DomainPilot's average improvement on LiveCodeBench v5/BFCL v3 within error bars, the scaling-law signal is not responsible for the gains; if DomainPilot beats the control by >1.5 points on both, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the Stage-1 scaling-law parameters in Eq. (1) are informative enough to drive the reallocation decisions in Eq. (3). Table 2 reports R² values of 0.635 (math), 0.522 (termagent), 0.429 (swe), 0.386 (science), and 0.183 (chat). With single unseeded runs and the epoch-boundary loss discontinuities the paper itself notes in §1.1, these fits are indistinguishable from noise for the very domains that receive the largest boosts (termagent +25%, swe +15%). The fitted a_i values are also internally implausible: four domains have a_i ≤ 10⁻¹¹, contradicting the claimed interpretation as 'initial loss amplitude' and effectively making the model L_i(D) ≈ b_i. Since Eq. (2) multiplies b_i, 1/α_i, and L_curr−b_i, noisy α_i/b_i directly change the score and hence the mixture. No control reallocation is reported, so the benchmark gains on LiveCodeBench/BFCL could plausibly result from any increase in swe/termagent exposure rather than from the specific scaling-law signal.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"DomainPilot proposes a domain-level, loss-guided framework for data mixture optimization in LLM fine-tuning. It monitors per-domain token losses during SFT, fits per-domain scaling curves L_i(D)=a_i D^{-alpha_i}+b_i (Eq. 1), computes a multiplicative reallocation score (Eq. 2), and adjusts mixture proportions (Eq. 3). A second 'Mixing Law' stage is described in Eq. (4) but explicitly deferred to future work. On Qwen3-1.7B SFT, the Stage-1 reallocated mixture is reported to improve over the authors' reproduction of the original mixture by +2% on MMLU-Redux, +1.8% on AIME24, +3.8% on LiveCodeBench v5, and +3.6% on BFCL v3. The paper also contributes a patch-based architecture with ~30 lines of framework-specific adapter code for token-level domain loss monitoring.","tokens_in":8426,"tokens_out":4871,"duration_ms":55454,"significance":"If the reported gains survive scrutiny, the framework would be a genuinely lightweight alternative to expensive data selection and auxiliary-model mixture optimization. The proposed monitoring overhead (<1% throughput) and the thin adapter layer address a real industrial portability problem. The paper is honest in its limitations section, noting that only Stage 1 is validated and that the Mixing Law sweeps and Weight mechanism are not complete. However, the current evidence is single-run, the Stage-1 scaling-law fits are poor for the very domains that receive the largest reallocations, and no control reallocation is reported. Thus the central empirical claim is not yet convincingly supported, although it is plausible and testable.","major_comments":[{"comment":"The fitted scaling-law parameters are not informative enough to drive the Stage-1 reallocation. Table 2 reports R^2 values of 0.635 (math), 0.522 (termagent), 0.429 (swe), 0.386 (science), and 0.183 (chat); the termagent and swe domains receive the largest boosts (+25% and +15%), yet their fits are particularly weak. The fitted a_i values for four domains are ≤10^-11, making the model effectively L_i(D) ≈ b_i and contradicting the 'initial amplitude' interpretation. Because Eq. (2) multiplies b_i, 1/α_i, and (L_curr - b_i), noisy α_i and b_i directly change the score and thus the mixture. No confidence intervals, repeated runs, or residual diagnostics are provided. The paper needs to show that the scaling-law fit is statistically meaningful (e.g., via confidence bounds, more data points, or an alternative robust estimator) before the reallocation can be attributed to it.","section":"§3.2, Eq. (1), Table 2"},{"comment":"The benchmark gains cannot be attributed to the specific scaling-law signal without a control reallocation. The improved benchmarks are exactly the domains whose proportions were increased (swe → LiveCodeBench, termagent → BFCL). A control experiment that increases the same total data volume for a different set of domains, or applies a random/rule-based reallocation of comparable magnitude, is necessary to distinguish 'more data for a bottleneck domain helps' from 'the fitted parameters in Eqs. (1)–(3) identify the right bottleneck.' In addition, the original-mixture reproduction scores (69.8 MMLU-Redux) are far below the official Qwen3 report (73.9), so the comparison baseline itself is questionable. The paper also reports only single unseeded runs; SFT at this scale is noisy, and the reported differences are small. Multiple seeds and error bars are needed.","section":"§5.1, Table 3"},{"comment":"The paper's central claim describes a 'two-stage' framework, but Stage 2 is not implemented. Section 3.3 states that the Mixing Law refinement 'remains future work' and Section 5 says 'reported results reflect only Stage 1.' The abstract and contributions nevertheless describe a 'two-stage' pipeline and claim the optimizations are achieved by it. This overstates the validated contribution. Either Stage 2 must be implemented and evaluated, or all claims and the title should be revised to reflect that only the Scaling-Law stage is empirically demonstrated.","section":"§3.3, §5, Abstract"},{"comment":"The four-factor score in Eq. (2) is introduced without derivation or sensitivity analysis. The factors (difficulty, speed, headroom) are plausible but the particular functional form—products of ratios with arbitrary normalizations—is not justified. Since Eq. (3) normalizes the scores, only relative order matters, but no evidence is given that this order is robust to reasonable changes in the score construction (e.g., dropping one factor, changing the exponent on b_i, or using only α_i). Without such robustness checks, the method is a hand-designed heuristic rather than a principled scaling-law optimization.","section":"§3.2, Eq. (2)"}],"minor_comments":[{"comment":"The 'Action Δprop.' column reports percentages but no algorithm or pseudocode is given for how the fitted parameters and Eq. (2)–(3) produce exactly these proportions. The application of Eq. (3) should be reproducible from the table alone.","section":"§3.2, Table 2"},{"comment":"The section heading 'Mixing Law Guided Fine Optimization (Planned)' conflicts with the contribution list in §1.2, which presents two-stage optimization as a contribution. Please align these statements.","section":"§3.3"},{"comment":"The cost-benefit discussion says 'requires only lightweight sweep experiments (~10% of a full SFT run),' but Stage 2 sweeps are not performed in this paper. Clarify that this refers to the planned Stage 2 cost, not to the results reported here.","section":"§5.4, Table 5"},{"comment":"For AIME24, the table reports '48.3 / 13.4' etc., but the 'Optimization' column gives +1.8%, which appears to be computed on the first number (42.9−41.1). Please specify whether the reported improvement is on Pass@1 or Cons@64, and do the same for LiveCodeBench (Pass@1 vs Pass@5).","section":"§5.1, Table 3"},{"comment":"MATH-500 appears in Table 4 but is not defined or included in Table 3. Provide the evaluation setup for this benchmark and ensure all metrics are consistently reported.","section":"§5.3, Table 4"},{"comment":"The paper would benefit from a reproducibility statement: no code, configuration files, or seed values are provided. Given the dependence on fitted parameters, releasing the domain-loss trajectories and fitting scripts would materially strengthen the contribution.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper has a good engineering idea and a clear presentation of the proposed architecture, but the empirical validation is currently too weak for the claims made. The central issue is not the direction of the work but the lack of statistical grounding: poor fits, no control reallocation, no seeds, and a baseline that underperforms the official report. I would encourage the authors to run the obvious control experiments and report variance; these are feasible within the paper's scope and would determine whether the underlying scaling-law signal is actually doing the work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing: the token-level domain tagging and the MindSpeed patch are credible engineering, and the Limitations section is honest. The empirical core—fitted scaling-law parameters driving reallocation and producing the reported gains—is not supported by the evidence in the paper.\n\nWhat is new: token-level domain IDs maintained through knapsack packing and passed through Megatron/MindSpeed forward passes, with domain losses aggregated by DP All-Reduce. That is practical and the <1% overhead claim is plausible. The patch architecture is a real contribution. The Qwen3-1.7B evaluation is new.\n\nThe soft spots are where the claims are strongest. Table 2's fits are too weak to carry the reallocation: chat R²=0.183, science 0.386, termagent 0.522, swe 0.429, all from single unseeded runs. Fitted a_i values around 1e-11 to 1e-17 contradict the 'initial amplitude' reading; Eq. 1 is effectively L≈b_i for those domains. No control reallocation is reported, so the LiveCodeBench/BFCL gains could come from any increase in swe/termagent exposure. The original-mixture reproduction also underperforms the official report (69.8 vs 73.9 MMLU-Redux), undermining the baseline. The abstract/title promise a two-stage pipeline, but Section 3.3 states Stage 2 is planned and results are Stage 1 only. Table 4 introduces MATH-500=94.0 without explanation, and the claimed +10.1% relative gain over Qwen2.5-3B is arithmetically wrong (~+40%). These errors, taken together, make the results section unreliable.\n\nWho this is for: practitioners working on data mixture optimization who want a cheap monitoring hook in Megatron-class frameworks. The scaling-law framing should not be cited as evidence until the fits are redone with proper controls.\n\nRecommendation: I would send it to peer review rather than desk reject, because the engineering contribution is concrete and testable, but I would expect major revision: seeds/variance, a naive reallocation baseline, Stage 2 either implemented or removed from title/abstract, and cleanup of Table 4.","headline":"The monitoring patch is real; the scaling-law story and most benchmark numbers do not hold up.","tokens_in":8962,"tokens_out":3710,"would_cite":false,"duration_ms":39801,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DomainPilot claims that fitting per-domain loss curves during supervised fine-tuning and reallocating data by a scaling-law score lifts four benchmarks by up to 3.8 points at zero extra compute.","keywords":["data mixture optimization","domain-level loss","scaling law","supervised fine-tuning","token-level monitoring","loss-guided reallocation","language models","training data"],"falsifier":"Fit the same scaling law to per-domain loss trajectories obtained from a run where domain labels are randomly permuted; if the resulting parameters still produce benchmark gains when used for reallocation, the signal is not domain-specific. Alternatively, show that a reallocation based on random scores (matching the same total budget and perturbation magnitude) reproduces the reported gains, which would indicate the specific scaling-law parameters are not load-bearing.","tokens_in":7921,"feed_emoji":"📊","tokens_out":4749,"duration_ms":49518,"temperature":0.7,"pith_summary":"The paper introduces DomainPilot, a method that monitors per-domain training loss at token level, fits each domain's convergence to a power law, and uses the fitted parameters to rebalance the data mixture. On a 1.7B-parameter model during supervised fine-tuning, the reallocated mixture improves MMLU-Redux by 2 points, AIME24 by 1.8, LiveCodeBench v5 by 3.8, and BFCL v3 by 3.6 absolute percentage points over the original mixture, with no increase in data volume or training cost. The central claim is that domain-level learning signals provide a cheap, effective signal for mixture optimization, avoiding expensive data selection or auxiliary model training. If true, this makes data mixture optimization nearly free at scale and directly applicable to industrial training pipelines.","feed_headline":"Domain-loss scaling law lifts fine-tune scores up to 3.8 points","feed_subtitle":"A lightweight monitor fits each domain's learning curve and reshuffles training data, adding no compute yet improving four benchmarks.","key_machinery":"The central mechanism is the domain-level scaling-law fit and the reallocation score of Eq. (2). Token-level domain tags allow the training loop to aggregate loss per domain with negligible overhead; those trajectories are fit to a_i D^{-α_i} + b_i, and the score combines three factors—relative irreducible loss (b_i), inverse convergence speed (1/α_i), and remaining improvement headroom—to scale each domain's proportion via a multiplicative reweighting step. The patch-based architecture is the delivery vehicle, but the scaling-law score is what carries the argument: it converts raw loss curves into a principled prior for mixture adjustment.","core_discovery":"The paper's core discovery is that per-domain loss trajectories, aggregated from token-level domain tags, approximately follow a scaling law L_i(D) = a_i D^{-α_i} + b_i, and the fitted parameters—convergence loss, convergence speed, and remaining headroom—carry enough information to guide a coarse reallocation of training data across domains. Using a four-factor score derived from these parameters, DomainPilot increases the share of slow-converging domains (e.g., software engineering and agent tasks) and decreases a fast-converging domain (chat), yielding consistent gains on four benchmarks. The largest gains appear exactly in the domains flagged as under-resourced, which the authors interpr","pith_inferences":["The reported gains are measured against the authors' own reproduction of the original mixture, which underperforms the official reported baseline on at least one benchmark; the optimization may be partially recovering lost performance rather than creating net-new capability.","Given that some fitted scaling-law parameters have very low R² values (as low as 0.18 for one domain) and fitted amplitudes span dozens of orders of magnitude, the specific parameter values may be noise; the observed gains could come primarily from shifting data away from a saturated domain to under-trained ones, regardless of the exact power-law form.","A direct way to test whether the scaling-law signal is load-bearing would be to compare against a random reallocation with the same total token budget; if random shifts produce similar gains, the fitted α_i and b_i are not the driving factor."],"forward_implications":["Data mixture optimization can be performed online during training with negligible overhead, without extra data passes or auxiliary reference models.","The same domain-level loss monitoring could be applied to pretraining, where mixture optimization is currently dominated by heuristic ratios, if the scaling-law signal holds at that scale.","Domain-level loss signals may identify under-resourced domains more reliably than sample-level loss reweighting, which conflates noise, difficulty, and novelty.","Reallocating existing data can produce gains comparable to a substantial fraction of what would be achieved by doubling model parameters, at a tiny fraction of the cost."],"fun_headline_variants":["Domain loss scaling law reshapes fine-tune data, lifts scores up to 3.8 pts","Per-domain loss scaling guides data mix, boosts fine-tune by 3.8% on benchmarks","No-extra-compute domain loss scaling for data mix lifts fine-tune scores","Domain loss scaling tunes data mix without compute, gains up to +3.8","Scaling law for domain loss reshuffles data, boosting fine-tune up to 3.8"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire reallocation rests on the assumption that per-domain loss curves faithfully follow the power law L_i(D) = a_i D^{-α_i} + b_i, so that the fitted b_i and α_i values are reliable signals rather than artifacts of short, noisy training runs.","fun_headline_variants_meta":{"raw":{"variants":["Domain loss scaling law reshapes fine-tune data, lifts scores up to 3.8 pts","Per-domain loss scaling guides data mix, boosts fine-tune by 3.8% on benchmarks","No-extra-compute domain loss scaling for data mix lifts fine-tune scores","Domain loss scaling tunes data mix without compute, gains up to +3.8","Scaling law for domain loss reshuffles data, boosting fine-tune up to 3.8"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001305,"raw_usage":{"total_tokens":5221,"prompt_tokens":871,"completion_tokens":4350,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":4231}},"tokens_in":615,"tokens_out":4350,"duration_ms":31651,"temperature":1.0,"reasoning_tokens":4231,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T06:15:01.125495+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fit the same scaling law to per-domain loss trajectories obtained from a run where domain labels are randomly permuted; if the resulting parameters still produce benchmark gains when used for reallocation, the signal is not domain-specific. Alternatively, show that a reallocation based on random scores (matching the same total budget and perturbation magnitude) reproduces the reported gains, which would indicate the specific scaling-law parameters are not load-bearing.","supporting_citations":[],"review_version":1}