{"id":"0fb13175-d0cd-4802-a994-380c3a0821d0","arxiv_id":"2607.07023","paper_version":1,"verdict":"CONDITIONAL","confidence":"UNKNOWN","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"Online SFT data selection acts as an implicit preference model, shifting refusal rates, verbosity, and sycophancy in directions predictable from the selected data's attribute mixture.","lead":"This paper shows that choosing which training examples to keep during language-model fine-tuning silently shifts the model's safety and style behaviors, even without explicit alignment training. Anyone building or deploying LLMs should care because 'data-efficient' selection can introduce hidden refusals, verbosity, or sycophancy that look like deliberate alignment decisions.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The 'predictability' claim rests on post-hoc enrichment–drift correlation across only 4 selectors with no held-out validation, and the formal bridge (clustering assumption A4) is only partially verified without reported τ values.","rationale":"The reader correctly identifies A4 as a weak point, but I think the more load-bearing issue is one level up: the predictability claim itself lacks held-out validation. A4's partial verification is a symptom of this — the paper never rigorously tests whether enrichment ratios predict drift for selectors outside the 4 used to construct the framework. The formal theory (Proposition 1, Eq. 19) is sound as a first-order approximation; the gap is empirical validation of the specific enrichment-to-drift link.\n\nThat said, this concern does not move the verdict below CONDITIONAL. The existence claim (part 1) is well-supported and practically important regardless of whether the predictability claim fully holds. The paper's observation that data selection silently changes alignment-relevant behavior is valuable even if the direction is only sometimes predictable. The AAS results provide partial evidence that enrichment matters. The single-model limitation (Llama-3.1-8B with LoRA only, unreported full-FT control) is a secondary concern that the reader already noted.\n\nThe paper makes a genuine contribution by reframing data selection as an alignment intervention and providing a measurement protocol. The concern is that it slightly overclaims on predictability relative to the evidence, not that the core observation is wrong. CONDITIONAL remains appropriate: accept if the authors add held-out validation or soften the predictability claim to 'consistent with' rather than 'predictable from.'","tokens_in":19165,"tokens_out":4099,"duration_ms":129718,"concrete_test":"Construct 2–3 additional online selectors with deliberately different enrichment profiles (e.g., one that over-samples concise refusals, another that over-samples verbose compliant responses). Using only the Appendix B attribute schema, predict the drift direction for each from Eq. 19 before training. Then run ADA (train, evaluate on the 8-axis behavior suite) and compare predicted vs. observed signed drift vectors. If predicted directions match, the predictability claim is validated. If they don't, the claim weakens to post-hoc consistency. Additionally, report τ (within-group gradient concentration error) for all attribute groups in the existing runs to assess whether Eq. 19's bound is quantitatively meaningful.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two parts: (1) selectors matched on task accuracy diverge on behavioral axes, and (2) the direction of drift is predictable from the attribute mixture of the selected data. Part 1 is directly supported by Table 1 and is robust. Part 2 is the stronger, more novel claim and is load-bearing for the paper's contribution.\n\nThe formal bridge from enrichment to drift (Eq. 19, Appendix A) requires clustering assumption A4: within-group gradient concentration with bounded error τ. The paper states this is 'verified empirically in the mechanistic diagnostics' (§3.1), but §5.6 only describes diagnostics for 'a subset of runs' and never reports numerical values of τ across all attribute groups. Without quantifying τ, we cannot assess whether Eq. 19's bound is tight enough to be genuinely predictive rather than merely post-hoc consistent.\n\nThe empirical evidence for predictability is also limited: Table 2 shows enrichment ratios for the same 4 selectors (Loss, Quality, Diversity, AAS) whose behavioral drift is reported in Table 1. The consistency between enrichment and drift direction is suggestive, but with n=4 selectors and no held-out validation, 'predictable' overstates what is demonstrated. AAS's success (Tables 1, 4) provides partial support — if enrichment didn't matter, constraining it shouldn't help — but AAS is evaluated within the same framework and metrics, so it functions as a consistency check rather than independent validation. A reader cannot verify that enrichment ratios would correctly predict drift direction for a new selector not used in constructing the framework.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper argues that online data selection during SFT is not a neutral efficiency layer but an implicit alignment mechanism: different online selectors (loss-based, quality-based, diversity-based) induce directional behavioral drift along alignment-relevant axes (refusal rate, verbosity, sycophancy, jailbreak robustness) even when matched on task accuracy. The authors formalize selection as importance reweighting of the SFT objective (§3.1, Eqs. 4–7), derive a first-order drift bound (Proposition 1, Eq. 8), and connect it to data-mixture enrichment under a clustering assumption (Appendix A, Eq. 19). They introduce Alignment Drift Auditing (ADA), a controlled protocol for measuring behavioral drift under equal token budgets, and Alignment-Aware Selection (AAS), a diagnostic selector that constrains drift via an MMD penalty on attribute mixtures. Empirically, Table 1 shows that selectors within 0.8 points of task accuracy differ by >8 points in harmful-refusal rate; Table 2 shows enrichment ratios that track the observed drift directions; Tables 3–4 show budget trends and AAS ablations.","tokens_in":19374,"tokens_out":2239,"duration_ms":81390,"significance":"The conceptual reframing — that an online scorer occupies the role normally held by a reward model — is timely and practically important. The ADA protocol (matched budgets, paired seeds, eight-axis behavioral suite, enrichment diagnostics) is a genuine methodological contribution that could standardize how selection papers report side effects. The formal framework (selection as importance reweighting, first-order drift bound) is correct and provides useful vocabulary. The AAS selector is presented appropriately as a diagnostic rather than a production method. The paper ships falsifiable predictions (drift direction tracks enrichment) and a reproducible experimental grid (budget sweep, judge robustness in Appendix E, ablation in Table 4).","major_comments":[{"comment":"§5.6 and Appendix A: The clustering assumption A4 (within-group gradient concentration, bounded by τ) is the formal bridge between the abstract Proposition 1 and the empirical enrichment ratios (Eq. 19). The paper states this is 'verified empirically in the mechanistic diagnostics' (§3.1), but §5.6 only describes the diagnostics qualitatively for 'a subset of runs' and never reports numerical values of τ across attribute groups. Without quantifying τ, a reader cannot assess whether Eq. 19's bound is tight enough to be genuinely predictive or merely post-hoc consistent. The gradient alignment measure α(k)_π (Eq. 13) is defined but no actual values are reported in any table or figure. Reporting τ and α(k)_π for the attribute groups in Table 2 would substantially strengthen the mechanistic claim. This is load-bearing because the enrichment-to-drift link is the paper's stronger, more novel贡献","section":null},{"comment":"§5.3 and Table 2: The predictability claim ('the direction of the shift is predictable from the attribute mixture of the selected data') is supported by enrichment-drift consistency across only four selectors (Loss, Quality, Diversity, AAS). With n=4 and no held-out validation — e.g., predicting drift direction for a new selector from its enrichment profile alone — 'predictable' overstates what is demonstrated. AAS's success (Tables 1, 4) provides indirect support (if enrichment didn't matter, constraining it shouldn't help), but AAS is evaluated within the same framework and metrics, functioning as a consistency check rather than independent validation. Adding one or two additional selectors with a priori enrichment predictions, or softening the claim to 'consistent with,' would address this without changing the paper's scope.","section":null}],"minor_comments":[{"comment":"Table 1: The 'Avg. tokens' column header is ambiguous — it is unclear whether this refers to selected-data tokens, output tokens at evaluation, or training tokens per example. Clarifying would help reproducibility.","section":null},{"comment":"§4.1: The paper mentions a 'full-fine-tuning run to rule out an adapter artifact' but does not report its results in any table. A one-line summary (e.g., 'full FT confirms the same selector ordering on D_A') would suffice.","section":null},{"comment":"Eq. (9): The coverage term c(S) and its submodular properties are referenced but c(S) is never explicitly defined. The greedy rule in Eq. (10) references λΔc(x|S), but the reader must infer the form of c from context.","section":null},{"comment":"Figure 2 (right panel): The signed drift heatmap uses a color scale that is difficult to parse in grayscale; the values are readable but the visual encoding could be clearer.","section":null},{"comment":"§3.2: The claim that quality and diversity scorers 'subsume the utility-style online selectors' (citing [15, 16]) is stated without justification. A brief argument for why utility-based selectors are instances of these families would help.","section":null},{"comment":"Appendix B: The calibrated enrichment estimator (Eq. 20) is a nice detail, but the confusion rates ρ_fp, ρ_fn are reported only as a range (Cohen's κ 0.86–0.94). Reporting the actual confusion rates per attribute family would make the correction fully reproducible.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The reader's stress-test correctly identifies the two weakest points (A4 verification and n=4 predictability). Both are real but fixable within the manuscript's scope — the first by reporting τ and α values that the authors likely already computed, the second by adding a selector or softening language. The core finding (selection induces structured behavioral drift) is robust and does not depend on these gaps. The paper is a good fit for the journal's scope given the growing interest in post-training analysis."},"author_rebuttal":null,"desk_editor":{"model":"glm-5.2","letter":"The punchline: this paper shows that online data selectors matched on task accuracy can diverge by 8+ points on refusal rate and shift sycophancy and verbosity, all without any preference optimization. That is a real finding worth taking seriously. The formalization of selection as importance reweighting of the SFT pool is clean and correct, and the experimental design — same base model, optimizer, token budget, decoding settings, only the subset changes — is the right way to isolate the causal role of selection.","headline":"Solid core finding with fixable gaps","tokens_in":19947,"tokens_out":967,"would_cite":true,"duration_ms":62965,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Choosing training data is already alignment, not just efficiency","keywords":[],"falsifier":"If two selectors with identical task accuracy and identical enrichment ratios on all measured behavioral attributes still produce significantly different behavioral drift, the claimed link from data-mixture diagnostics to behavioral outcomes would break, since the enrichment gap would no longer predict drift direction.","tokens_in":19414,"feed_emoji":"⚖️","tokens_out":853,"duration_ms":120519,"temperature":0.7,"pith_summary":"This paper argues that when you filter which examples to train a language model on during supervised fine-tuning, the selection rule acts as an implicit reward model that shifts the model's behavior along axes normally associated with alignment: refusal rate, verbosity, sycophancy, truthfulness, and jailbreak robustness. The authors formalize online data selection as a reweighting of the SFT objective, where the selector's scores define a tilted training distribution that over-represents certain response styles or safety postures. They show empirically that selectors producing statistically indistinguishable task accuracy can diverge by more than 8 points in harmful-refusal rate and over 3 points in benign over-refusal, with the direction of each shift predictable from which behavioral attributes the selected data enriches. A loss-based selector enriches premise-agreement examples and increases sycophancy; a quality-based selector enriches long answers and refusal markers, raising both helpfulness and over-refusal simultaneously. The paper introduces Alignment Drift Auditing, a protocol that measures this drift under equal token budgets, and Alignment-Aware Selection, a selector that constrains the behavioral attribute mixture of the chosen data to reduce drift while preserving data efficiency.","feed_headline":"Choosing training data is already alignment, not just efficiency","feed_subtitle":"Selectors with identical task accuracy diverge sharply in refusal, verbosity, and sycophancy, with drift predictable from the data mixture.","key_machinery":"The central object is the reweighted SFT objective, where an online selector assigns weights w_π(i) to candidate examples, inducing a tilted training distribution q_π. The alignment drift is formalized through Proposition 1, which gives a first-order bound: the behavioral change between two selectors equals the Jacobian mapping parameter updates to behavior, applied to the difference in their weighted gradient means, plus second-order remainder. Under a clustering assumption that gradients concentrate within behavioral attribute groups, this bound reduces to a function of the enrichment gap, the difference in attribute-group mass between the selected subset and the full pool, weighted by how","core_discovery":"Two SFT runs with identical base model, optimizer, token budget, and task accuracy can produce models with sharply different refusal rates, verbosity, sycophancy, and jailbreak robustness, solely because the online data selector over-represents certain behavioral attributes in the training subset. The direction and magnitude of this drift is predictable from the enrichment ratios of the selected data, formalized through a first-order bound linking the attribute mixture shift to behavioral change via the gradient-behavior Jacobian.","pith_inferences":[],"forward_implications":["Any paper or system reporting online SFT data selection results should report behavioral drift metrics alongside task accuracy, since equal efficiency can mask divergent safety and style profiles.","Preference optimization stages in post-training pipelines are not starting from a neutral SFT base; the selected data has already moved the model toward or away from desired behavioral regions, so SFT selection and preference optimization should be co-designed.","The framing of 'data quality' as a neutral, behaviorally inert concept is insufficient; high-quality data carries a persona with specific verbosity, deference, and caution characteristics that become the model's default behavior.","Alignment-Aware Selection demonstrates that constraining the behavioral attribute mixture of selected data via a maximum mean discrepancy penalty can reduce drift by more than half relative to unconstrained selectors while preserving most task gains."],"fun_headline_variants":["Data selection during SFT shifts model behavior without preference optimization","Identical accuracy, divergent refusals from data selection alone","Online data selector functions as an implicit reward model","Same accuracy, different alignment: SFT data selectors steer behavior","Online data selection induces predictable behavioral drift in SFT"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The clustering assumption that training gradients concentrate within behavioral attribute groups, meaning examples sharing a style like verbosity or refusal format produce similar gradient directions. This bridges the abstract gradient-space theory to the empirical data-mixture diagnostics, but it is verified only on a subset of runs without full quantitative reporting across all attribute groups.","fun_headline_variants_meta":{"raw":{"variants":["Data selection during SFT shifts model behavior without preference optimization","Identical accuracy, divergent refusals from data selection alone","Online data selector functions as an implicit reward model","Same accuracy, different alignment: SFT data selectors steer behavior","Online data selection induces predictable behavioral drift in SFT","Task-identical models diverge in safety posture from data selection","SFT data selectors serve as implicit alignment mechanisms","What you select during fine-tuning changes model behavior, not just efficiency","Refusal rate and sycophancy shift predictably from data selection choices","Alignment happens during data selection, before any preference optimization"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":1547,"prompt_tokens":584,"completion_tokens":963,"prompt_tokens_details":null},"tokens_in":584,"tokens_out":963,"duration_ms":31196,"temperature":1.0,"reasoning_tokens":843,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T21:27:33.019742+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If two selectors with identical task accuracy and identical enrichment ratios on all measured behavioral attributes still produce significantly different behavioral drift, the claimed link from data-mixture diagnostics to behavioral outcomes would break, since the enrichment gap would no longer predict drift direction.","supporting_citations":[],"review_version":1}