{"id":"4020ea3c-88c4-413b-a2b5-2cd121d33534","arxiv_id":"2501.07886","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Under unreliable demonstrations and comparisons, DPO fails to improve over SFT, while iterative label refinement (ILR) of the SFT dataset does improve performance across math, coding, and safe instruction-following tasks.","lead":"This paper studies what happens to language model post-training when the human demonstrations and preference comparisons used to train the model are unreliable. It finds that DPO, a common preference optimization method, fails to improve over supervised fine-tuning in this setting, while a proposed method called iterative label refinement (ILR) does improve performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ILR's advantage over DPO in the LM-simulated experiments may be confounded by the task-specific 'sufficiently different' filter that ILR applies but DPO does not; an apples-to-apples DPO control is missing.","rationale":"The reader identifies the representativeness of the simulated unreliable supervisors as the weakest assumption. That is a legitimate external-validity concern, but the most load-bearing issue I see is internal to the comparison: in the LM-simulated experiments, ILR and DPO use different mechanisms for deciding which comparisons to query. ILR filters proposals by task-specific signals such as a different final answer or execution result, while DPO does not. This means the observed ILR-vs-DPO gap could be driven by better pair selection rather than by the conceptual difference between refining labels and optimizing a policy. If so, the paper's central claim, especially for math and coding where the filter is most task-specific, would not be established. The human study partially alleviates this concern because it applies the same confidence-based filtering to both methods, but it covers only Alpaca instruction following and only two rounds, so it cannot rescue the math/coding conclusions. Because a missing control is addressable by a straightforward experiment, the appropriate verdict remains conditional rather than accept or reject. I therefore keep the reader's CONDITIONAL verdict unchanged, while flagging a different and more pointed risk than the one in the reader's weakest-assumption statement.","tokens_in":21045,"tokens_out":5932,"duration_ms":61317,"concrete_test":"On GSM8K (2B->7B), rerun the DPO baseline with the same pair-construction filter used by ILR: for each prompt, compare the original unreliable demonstration with model-generated proposals, and collect unreliable preference feedback only for pairs whose final parsed answer differs; keep all other settings identical, including the 15% confidence subsample and the beta value. Report the 4-round accuracy and the gap to SFT+ILR. If DPO with this filter reaches or approaches the ILR accuracy around 0.385, the headline claim is largely a filtering artifact; if DPO remains near 0.36, the label-refinement mechanism is supported. A BIRD variant with execution-result filtering would test the generality of the conclusion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the LM-simulated experiments, ILR and DPO are not given the same comparison data. Section 5.1 and Appendix C state that ILR collects feedback only for model-generated proposals whose final answer differs (GSM8K), whose execution result differs (BIRD), or whose embedding distance is in the top 50% (SaferPaca). In contrast, DPO samples six completions, forms three arbitrary pairs, and keeps only the top 15% most confident unreliable judgments. Thus the headline comparison varies two things at once: the use of feedback (dataset refinement vs. direct policy optimization) and the curation of which pairs receive feedback. A task-specific difference filter may remove ambiguous or uninformative pairs, and this filtering could account for much of ILR's advantage. The human study in Section 6 is more apples-to-apples, but it covers only instruction following, not math or coding. Since the abstract's central claim about math, coding, and safe instruction-following rests on the LM-simulated results, the absence of a DPO baseline with the same difference filter is load-bearing. The paper's ablation 'Naive ILR' (Appendix E.2) shows that feedback matters, but it does not isolate the filter, and the sDPO/wsDPO variants in Appendix E.4 do not incorporate the same task-specific filtering, so they do not resolve the confound.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies LM post-training under unreliable human or model supervision. Using small LMs and time-constrained humans to generate noisy demonstrations and comparison feedback, it reports that SFT retains partial effectiveness but that DPO fails to improve beyond SFT when both demonstrations and comparisons are unreliable. The authors propose iterative label refinement (ILR), which uses comparison feedback to decide whether an unreliable demonstration in the SFT dataset should be replaced by a model-generated proposal, and then retrains the model via SFT on the refined data. They report that SFT+ILR outperforms SFT+DPO on GSM8K, BIRD, and SaferPaca under LM-simulated supervision and on an Alpaca instruction-following task with time-constrained human supervision, and interpret this as evidence that comparison feedback is better used for dataset refinement than for direct preference optimization when supervision is unreliable.","tokens_in":21338,"tokens_out":6457,"duration_ms":62227,"significance":"If the reported comparisons are sound, the paper offers a practically relevant and timely alternative to RLHF-style preference optimization in weak-supervision settings, with broad experiments across math, code, and safety and across model scales. The study's strengths include the use of external ground-truth evaluations (exact match, execution accuracy, GPT-4o win rate), the LM-simulated and human-studied supervision settings, the reported ablation of naive ILR, the comparison with robust DPO losses, and the stated release of code and data. However, the central LM-simulated comparison is currently confounded by a difference in comparison-pair selection between ILR and DPO, and the main figures lack uncertainty estimates. The headline conclusion should therefore be treated as conditional until these issues are addressed.","major_comments":[{"comment":"The headline LM-simulated comparison is confounded by different comparison-pair curation. ILR's step 4 only evaluates proposals that are 'sufficiently different' (different final answer on GSM8K, different execution result on BIRD, top-50% embedding distance on SaferPaca), whereas Appendix C says DPO samples six completions, forms three pairs, and keeps the top 15% most confident unreliable judgments, with no answer-level or execution-level difference filter. The comparison therefore varies two things at once: the use of feedback (dataset refinement vs. policy optimization) and the curation of which pairs receive feedback. The sDPO and wsDPO variants in Appendix E.4 do not incorporate the same task-specific filters, so they do not isolate the filter. Because the abstract's claims about math, coding, and safe instruction-following rest on these simulations, a DPO control trained on the same difference-filtered pairs, or an ILR variant without the filter, is needed before the central claim can be accepted. Section 6's human study is more apples-to-apples but only covers instruction following.","section":"Section 5.1 / Appendix C"},{"comment":"The main accuracy and win-rate comparisons in Figure 4 are presented as single curves with no error bars or multiple seeds, and the human-study comparison in Figure 6c likewise reports no uncertainty. The paper uses these curves for the strong statement that ILR consistently outperforms DPO; without variance information or at least a statement of the number of independent runs, the reader cannot assess whether the reported advantages are reliable or within run-to-run noise. Please report means and standard errors over independent seeds for the central comparisons, or explicitly state seeded replication counts.","section":"Section 4 / Figure 4"},{"comment":"The hyperparameters of the comparison are selected on performance: Appendix D.3 says that β and the confidence-percentile cutoff for DPO and α for ILR were chosen by testing first-round performance, and Appendix E.3 sets α=0.15 based on GSM8K results. The paper does not specify whether this tuning was done on a held-out split, and the main test results are then reported for the selected values. Please clarify the validation protocol and, if tuning was on test, add a held-out evaluation or a sensitivity analysis for the tuned hyperparameters.","section":"Appendix D.3 / Appendix E.3"}],"minor_comments":[{"comment":"There are several typos: the tested β set is given as {0.01, 0.1, 0.5, 0.1}, presumably with the last value intended to be 1; 'critiria' appears in Appendix D; 'separte' appears in the comparison-task instructions; and Figure 7's caption says 'lable refinement' instead of 'label refinement'.","section":"Appendix D.3 and general text"},{"comment":"The abstract and conclusion state that RLHF 'may no longer be the best use of human comparison feedback,' but the experiments cover DPO only, not PPO or reward-model-based RLHF. The Limitations paragraph acknowledges this for PPO, but the abstract's wording is broader than the evidence; a more cautious phrasing would better match the scope of the study.","section":"Abstract and Section 7"},{"comment":"In the human study, DPO variants with β values other than 0.1 are evaluated using preference comparisons collected from completions of the β=0.1 model. This is a reasonable offline-evaluation shortcut, but Section 6 should state that the β-sweep curves are not full iterative DPO runs with policy-specific rollouts.","section":"Section 6 / Appendix D.2"},{"comment":"The informal claim that 'as long as the annotator chooses better responses more than half the time, the overall accuracy of the SFT data will increase' would benefit from a precise statement of the conditions under which this holds, especially given the α-truncation of accepted proposals and the dependence on the proposal-generation procedure.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The confound identified in the LM-simulated comparison is real and load-bearing, but it is also fixable within the paper's scope by adding an apples-to-apples DPO control with the same difference filter. The breadth of the experiments, the external benchmarks, and the human study make this a worthwhile contribution, and I would not recommend rejection at this stage. The paper would also benefit from a more conservative title and abstract until the DPO control is provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core claim—that ILR beats DPO under weak supervision—is plausible and well-motivated, but the headline comparison in the LM-simulated experiments is not apples-to-apples, and that weakens the abstract's strong claim.\n\nWhat's new: the paper studies joint unreliability of demonstrations and comparisons, shows DPO fails to improve beyond SFT, and proposes ILR as a data-refinement alternative. That's a genuinely useful direction. The cross-labeling scheme is a nice practical touch, and the human study, though limited to instruction following, is a real attempt to validate the simulation. Code and data are released.\n\nSoft spots: the biggest issue is the difference filter. In the LM simulations, ILR only asks for comparisons when the model proposal differs from the original label (different final answer, different execution result, or top-half embedding distance), while DPO gets no such filter and instead keeps the top 15% most confident judgments. So you're varying two things at once: refinement vs. preference optimization, and pair curation. The DPO variants in Appendix E.4 don't adopt the same filter, and the 'Naive ILR' ablation doesn't isolate it. The human study is cleaner on this axis, but it only covers instruction following, not math or coding. Since the abstract's central claim about math and coding rests on the LM simulations, this needs a direct fix: run DPO with the same task-specific difference filter, or run ILR without it, and see if the ranking holds.\n\nOther, more minor issues: no error bars or multiple seeds reported; hyperparameters (alpha, beta, confidence thresholds) are selected on validation, which can inflate the apparent advantage; and the paper generalizes from DPO to 'RLHF' even though PPO may behave differently. The authors acknowledge the DPO-to-RLHF limitation, so that's not fatal.\n\nWho it's for: people working on scalable oversight and post-training under weak supervision. It'll likely spark useful discussion.\n\nRecommendation: send to peer review; it deserves serious refereeing. But the referees should focus on the confound, and the authors should be pushed to add the missing control before publication.","headline":"The paper asks a good question and proposes a sensible method, but the main comparison is confounded by a difference filter that only ILR gets; the result is promising, not proven.","tokens_in":21835,"tokens_out":2674,"would_cite":false,"duration_ms":24894,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Under unreliable supervision, refining training labels with comparison feedback beats optimizing preferences directly.","keywords":["iterative label refinement","direct preference optimization","unreliable supervision","weak-to-strong generalization","RLHF","post-training","overoptimization","language models"],"falsifier":"Run SFT+DPO and SFT+ILR on a task with known ground truth where annotators are real domain novices under time pressure on genuinely hard examples; if DPO matches or exceeds ILR at equal feedback budgets under those realistic error patterns, the central claim fails.","tokens_in":20832,"feed_emoji":"🏷️","tokens_out":4584,"duration_ms":47538,"temperature":0.7,"pith_summary":"The paper asks whether the standard post-training pipeline—supervised finetuning (SFT) on demonstrations followed by preference optimization—still works when the demonstrations and comparisons are unreliable, as they will be on tasks that are harder than humans. Simulating unreliable supervisors with small language models and time-pressured human workers, it finds that SFT retains some value but direct preference optimization (DPO) offers little or no improvement over SFT. The authors propose iterative label refinement (ILR), which uses comparison feedback to decide whether to replace a demonstration with a model-written alternative, then retrains the model by SFT. SFT+ILR outperforms SFT+DPO on mathematical reasoning, SQL code generation, and safe instruction-following. The central conclusion is that when supervision is weak, human comparison feedback is better spent improving the training data than continually optimizing the policy.","feed_headline":"When supervision is unreliable, label refinement beats DPO","feed_subtitle":"In math, coding, and safe instruction-following, iterative label refinement improves on SFT while DPO stalls.","key_machinery":"The central mechanism is comparison-gated dataset replacement with cross-labeling. The dataset is split in two, separate SFT models are trained on each half, and each model proposes new responses for prompts in the other half, so proposals come from prompts held out of that model's training. An unreliable annotator compares each proposal with the original demonstration, and the proposal replaces the demonstration only if preferred, with at most a fraction alpha of labels updated per round. The model is then retrained from scratch on the refined dataset, which allows large policy changes without the overoptimization risk that limits DPO.","core_discovery":"The paper claims that under unreliable demonstrations and unreliable comparison feedback, the canonical SFT-then-DPO pipeline fails to improve on SFT, while SFT followed by iterative label refinement does improve. In LM-simulated settings, models finetuned on unreliable demonstrations outperform the weak supervisor but remain far below models trained on ground truth, leaving a gap that DPO cannot close. The paper attributes DPO's failure to a regularization dilemma: unreliable comparisons cause overoptimization unless the KL penalty is large, but a large penalty prevents the substantial model updates needed to correct errors learned during SFT. ILR avoids this by modifying the SFT dataset instead of the policy: two models trained on disjoint halves of the data cross-generate replacement proposals, an unreliable comparator decides whether each proposal beats the current demonstration, and accepted proposals are capped at a fraction of the dataset before the model is retrained from scratch. Across GSM8K, BIRD, SaferPaca, and a time-constrained human study on Alpaca instruction-following, SFT+ILR improves over the initial SFT model more than SFT+DPO does and continues to improve over multiple rounds.","pith_inferences":["An untested implication is that ILR could be used as a front-end for preference optimization: after several rounds of label refinement, the SFT data is cleaner and DPO's overoptimization problem may be less severe, so a hybrid may inherit the strengths of both.","The comparison-gated replacement mechanism suggests that other noisy-label text-generation problems could benefit from data editing guided by preference judgments rather than robust loss functions, but the paper only tests three tasks.","A testable extension is to vary the correlation between supervisor errors and model-imitated errors; if errors are highly systematic, ILR's advantage over DPO may shrink, whereas random errors should be easier for both methods to handle."],"forward_implications":["If the central claim holds, RLHF-style preference optimization may be the wrong use of comparison feedback for tasks where human supervision is unreliable; improving the training labels is more robust.","ILR's gains grow with model scale: on GSM8K the improvement was larger when a 70B model was trained than when a 7B model was trained, suggesting it may remain useful for more capable future models.","Even unreliable comparison feedback can steadily raise the accuracy of SFT training data across rounds, and with stronger feedback the refined demonstrations can approach or exceed the quality of ground-truth demonstrations.","Combining ILR with scalable oversight techniques that make evaluation more reliable could further improve data quality, since the refinement step depends on the comparator's accuracy."],"supporting_citations":[{"why":"Supplies the weak-to-strong generalization result and the small-LM simulation design that motivate SFT's robustness to unreliable demonstrations.","marker":"Burns et al. (2023)"},{"why":"Establishes reward-model overoptimization, which the paper uses to explain DPO's failure under unreliable comparison feedback.","marker":"Gao et al. (2023)"},{"why":"Provides the DPO algorithm that is the main baseline and whose failure under unreliable supervision is the paper's starting point.","marker":"Rafailov et al. (2024)"},{"why":"Supports the use of smaller LMs as simulated unreliable supervisors for studying learning from human feedback.","marker":"Dubois et al. (2024)"},{"why":"Defines the standard RLHF pipeline that the paper's post-training setup compares against.","marker":"Ouyang et al. (2022)"}],"fun_headline_variants":["Label refinement beats DPO under weak supervision","When supervision is unreliable, refine labels, not preferences","DPO fails under weak supervision; label refinement wins","Fix the data, not the policy: label refinement outdoes DPO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The simulated unreliable supervisors, small LMs and time-pressured humans, are assumed to produce error patterns like those of real people supervising genuinely hard tasks; if real weak supervision has different systematic biases, the relative benefit of ILR over DPO could shift.","fun_headline_variants_meta":{"raw":{"variants":["Label refinement beats DPO under weak supervision","When supervision is unreliable, refine labels, not preferences","DPO fails under weak supervision; label refinement wins","Fix the data, not the policy: label refinement outdoes DPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000466,"raw_usage":{"total_tokens":2368,"prompt_tokens":1033,"completion_tokens":1335,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":1270}},"tokens_in":649,"tokens_out":1335,"duration_ms":10732,"temperature":1.0,"reasoning_tokens":1270,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:29:55.915451+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SFT+DPO and SFT+ILR on a task with known ground truth where annotators are real domain novices under time pressure on genuinely hard examples; if DPO matches or exceeds ILR at equal feedback budgets under those realistic error patterns, the central claim fails.","supporting_citations":[],"review_version":1}