{"id":"63c5d575-6a33-4197-bf7f-3e5100294b77","arxiv_id":"2411.14121","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuning Llama-3-8B on GPT-4-rewritten 'silly' versions of MMLU questions gives at most a 0.54% overall gain and no improvement over seed-only fine-tuning.","lead":"The authors tested whether rewriting MMLU training questions in the style of 'silly' questions from the Chinese website Ruozhiba makes a language model learn better. They found that the trick barely helps overall, and small gains on some tasks are offset by losses on others.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central negative result is only interpretable if GPT-4 rewrites preserve the original answer; the paper never checks this, so label corruption could fully explain the observed task-level effects.","rationale":"The paper's central empirical contribution is a null result: applying Ruozhiba-derived rules to MMLU via GPT-4 rewriting gives no significant overall improvement (66.28 vs 66.27, Section 4.2). For this null result to be meaningful, the augmented datasets must be valid instances of the rules applied to the original questions. The rewriter is instructed to preserve the reference answer, but there is no verification that it does; if rewrites change the correct choice, the training labels are wrong, and both the null result and the task-level variation are artifacts. This assumption is exactly the one the reader identified, and I agree it is the weakest link. The lack of error bars is a real issue for the abstract's 'significantly improve' language, but it is secondary: even with error bars, the experiment would not test the rules unless semantic preservation is confirmed. The proposed test—human or independent-model answering of a sample of rewritten questions with the original choices—directly settles whether labels were preserved. If agreement is high, the concern is retired; if not, the central claim needs to be restated as 'this specific GPT-4 rewriting pipeline did not help,' not 'Ruozhiba rules do not transfer.' The reader's CONDITIONAL verdict remains appropriate, so no adjustment is needed.","tokens_in":69389,"tokens_out":4751,"duration_ms":45283,"concrete_test":"Sample 100-200 rewritten questions per rule (800-1600 total) from the 13K D_rt sets. For each, present the rewritten question with the original choice options to human annotators (or to an independent, non-GPT-4 strong model as a proxy) who do not see the reference answer, and compare their selected choice with the original repi. Compute agreement per rule and overall. If agreement is materially below agreement on the unrewritten seed questions (e.g., <95%, or a statistically significant drop), the label-preservation assumption fails and the task-level results in Figure 6 cannot be attributed to the rules. Report per-rule agreement so that rules with higher corruption rates can be identified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The pipeline in Section 3.2 keeps the original response repi for every rewritten instruction insi,rt, relying on the prompt instruction that the rewritten question can still be accurately answered with repi. However, the rewriter is never checked. The Appendix D.1 prompts provide #Reference Choices# and #Reference Answer#, but they also forbid including choice content in the rewritten prompt, so GPT-4 may rephrase the question in a way that changes which option is correct. The only reported validation is the Judge-based scoring applied to the 4K mixing subset (Section 3.2), not to the 13K datasets used for the headline results. If even a few percent of rewrites alter the correct answer, the training set contains systematic label noise that varies by rule and by task. The task-level gains and drops (e.g., Counterintuitive Thinking +5.00% on Global Facts, Blurring the Conceptual Boundaries -6.14% on Econometrics, Figure 6) could then reflect corrupted labels rather than the effect of the extracted rules. This is not a hypothetical: the same prompts are used for all eight rules, and the paper itself concedes (Section 4.2, point 2) that GPT-4's rewrites are mostly stylistic, but no quantitative correctness check supports that claim. Without such a check, the central negative result (66.28 vs 66.27) is not a valid test of whether Ruozhiba-style augmentation improves SFT, because the augmented data may not preserve the seed answers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates why SFT data from the Chinese forum Ruozhiba ('silly questions') reportedly improves LLM fine-tuning. The authors use GPT-4 to extract eight stylistic/cognitive rules from Ruozhiba, then rewrite 13K MMLU training questions according to each rule while keeping the original choices and answer, fine-tune Llama-3-8B-Instruct with LoRA, and evaluate on MMLU. The main empirical finding is that no rule-generated dataset improves overall MMLU accuracy beyond the seed-only SFT baseline (best 66.28% vs 66.27%), though specific tasks show gains and drops up to about 5%. The paper also reports that the direction of task-level effects is largely consistent across rules, and that single-rule augmentation outperforms the tested mixing/filtering strategies.","tokens_in":69673,"tokens_out":3372,"duration_ms":33949,"significance":"If the results are valid, the paper delivers a useful, honest negative result: Ruozhiba-style rule-based rewriting does not transfer to MMLU SFT as a general gain, and its effects are concentrated in specific tasks. The work also contributes a transparent pipeline for rule extraction, eight well-documented rewriting rules, and a large set of task-level comparisons, which could be a reference point for future data augmentation studies. The significance is dampened, however, by the absence of any validation that the GPT-4 rewrites preserve the original correct answer, and by the lack of statistical support for the task-level claims; these issues are load-bearing because they determine whether the observed differences reflect the rules themselves or label noise and sampling variation.","major_comments":[{"comment":"","section":"Section 3.2 / Appendix D.1"},{"comment":"","section":"Section 4.2 / Figures 6 and 9"},{"comment":"","section":"Abstract / Introduction vs Section 4.2"},{"comment":"","section":"Section 4.2 / Figure 14"}],"minor_comments":[{"comment":"","section":"Section 3.2"},{"comment":"","section":"Abstract and Section 4.2"},{"comment":"","section":"References"},{"comment":"","section":"Figures 6 and 7"},{"comment":"","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The central negative result is potentially publishable, but the missing correctness-preservation check is a genuine validity threat for the task-level claims, and the statistical treatment is too weak for the 'significant' language used. The inconsistency between the 13K and 4K results on the abstract's headline examples is concerning and should be addressed head-on. I would not reject, because the required validation and repeated-seed experiments are feasible within the scope of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a careful, honest negative result about Ruozhiba-style data augmentation, but the experiment has a credibility gap that prevents the headline from being as clean as it looks. The central claim—that applying GPT-4-distilled 'silly question' rules to MMLU training data gives no overall gain over seed-only SFT (66.28 vs 66.27)—is coherent, and the paper is admirably upfront about its limitations. What's new is the scale: eight rules distilled from Ruozhiba via GPT-4, applied to a 13K MMLU sample, evaluated across all 57 tasks, plus filtering and mixing comparisons. That's a legitimate new application and a useful cautionary data point for people working on SFT data synthesis.\n\nThe soft spots are real, though. The biggest one: the pipeline keeps the original answer for every rewritten question, and there's no check that the rewrite preserves which choice is correct. The prompts include the reference answer but forbid including choice content, so GPT-4 can rephrase in ways that change the right answer. The only validation is the judge-based scoring on the 4K mixing subset, not the 13K datasets used for the headline. If any nontrivial fraction of rewrites corrupt the label, the task-level gains and drops (Counterintuitive +5% on Global Facts, Blurring -6.14% on Econometrics) reflect label noise, not rule effects. That makes the negative result not a clean test of Ruozhiba-style augmentation.\n\nSecond, the 'significantly improve' language in the abstract isn't supported: these are single-run point estimates with no confidence intervals, and 5% swings on ~100-example tasks are within noise. The consistency metric lacks a null baseline, so the 94.74% figure is hard to interpret. Third, no code or generated data are released, which hurts reproducibility.\n\nNone of this is fatal to the paper's existence. The authors explicitly acknowledge the rewrites are mostly stylistic and raise the LLM-as-judge bias explanation. As a descriptive, cautionary result, it's worth publishing after revision. But to make the headline credible, they need a label-preservation check, multiple seeds or confidence intervals, and the generated datasets. I'd send it to review, with the expectation of major revision on the validation front.","headline":"A well-motivated negative result about Ruozhiba-style augmentation, but an unverified label-preservation assumption and single-run statistics keep the headline from being clean.","tokens_in":70234,"tokens_out":4123,"would_cite":false,"duration_ms":36762,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper tests whether the Ruozhiba 'silly questions' style, distilled into eight rewriting rules and applied to MMLU, improves supervised fine-tuning, and finds the overall gain is negligible even though individual tasks move by several…","keywords":["supervised fine-tuning","data augmentation","Ruozhiba","MMLU","instruction rewriting","LLM-as-a-judge","task-level consistency","LoRA"],"falsifier":"Take a random sample of rewritten MMLU questions from each of the eight rules, have independent human annotators (or a strong non-GPT model) answer the original multiple-choice options against the rewritten stem, and measure how often the original reference answer remains correct; if a substantial fraction are corrupted, the observed task-level improvements and drops are confounded by label noise.","tokens_in":69200,"feed_emoji":"🤔","tokens_out":3957,"duration_ms":38529,"temperature":0.7,"pith_summary":"The paper asks why a Chinese website of deliberately 'silly' questions, Ruozhiba, seemed to improve LLM fine-tuning, and whether that benefit can be transferred to a standard benchmark. It uses GPT-4 to condense Ruozhiba's style into eight explanatory rules, then rewrites MMLU questions with those rules and fine-tunes Llama-3-8B-Instruct on the result. The headline result is negative overall: on the MMLU test set, the best rule-generated dataset reaches 66.28% versus 66.27% for plain seed fine-tuning. Gains and losses up to about five points appear scattered across tasks, and different rules affect the same task in the same direction more often than not. The authors suggest the original Ruozhiba advantage may have been inflated by LLM-as-judge self-preference rather than by the data itself.","feed_headline":"Silly-question training gains vanish on MMLU: 66.28 vs 66.27","feed_subtitle":"A 13k-sample retest of Ruozhiba-style rewrites across 57 tasks finds no overall boost, only scattered task-level wins and losses.","key_machinery":"The central machinery is the Ruozhiba-rule augmentation pipeline: eight named rules (Counterintuitive Thinking, Blurring the Conceptual Boundaries, Anthropomorphic Expressions, and others) extracted by GPT-4 from Ruozhiba data are injected into a WizardLM-style instruction-rewriting prompt that rewrites each MMLU question while keeping reference choices and answer unchanged, and the rewritten instructions are then used as additional SFT data for LoRA fine-tuning of Llama-3-8B-Instruct. The work this machinery does is to convert an anecdotal data-source effect into a controlled, task-level evaluation of stylistic rules.","core_discovery":"The central claim is that applying Ruozhiba-style augmentation rules to MMLU questions does not improve overall supervised fine-tuning performance beyond the seed data, and that the extracted rules act mainly as stylistic rewrites whose task-level effects are consistent regardless of which rule is used. The paper supports this with a 13,000-sample experiment: rule-generated datasets underperform plain seed fine-tuning on STEM subjects, offer small improvements on Humanities and Other subjects, and produce task-level movements such as a roughly 5% gain on Global Facts under Counterintuitive Thinking and a 6.14% drop on Econometrics under Blurring the Conceptual Boundaries. Across tasks, 94.74% showed at least 50% agreement in whether different rules helped or hurt, and 26.32% reached 100% agreement. The authors interpret this as evidence that the choice of generation rule is less critical than applying the right kind of data to the right task, and they point to LLM-as-judge self-enhancement bias as a possible reason the original Ruozhiba results were stronger.","pith_inferences":["I read the near-tie on the overall MMLU average as evidence that the original Ruozhiba benefit may be a property of the evaluation setup (GPT judging GPT) rather than of 'silly' phrasing itself; a direct test would rerun the original COIG-CQIA comparison with a non-GPT judge.","The high consistency across rules suggests task domain is the dominant factor, which could let future work predict per-task augmentation gains from properties of the seed questions before spending API budget.","Because no correctness check of rewritten questions is reported, some of the observed task-level drops could be label corruption rather than a genuine stylistic effect; adding round-trip verification would cleanly separate the two.","A testable extension is to apply the same eight rules to open-ended tasks, where stylistic variation may matter more than it does in multiple-choice question answering."],"forward_implications":["If the central claim is right, previously reported Ruozhiba-based gains do not automatically transfer to MMLU-style multiple-choice benchmarks.","Task-level benefits are real but narrow, so SFT dataset construction should be guided by per-task diagnosis rather than by a one-size-fits-all augmentation style.","The consistency result implies that the specific rule matters less than whether augmentation fits the task, which simplifies future dataset-design decisions.","Filtering and mixing strategies tested in the paper do not beat the best single-rule augmentation, suggesting that selection among generated variants is not where the main gains come from.","The proposed explanation, LLM-as-judge self-enhancement bias, means earlier comparisons that used GPT judges may overstate the value of any particular data source.","The load-bearing premise is that GPT-4's rewritten questions preserve the semantics and the correct answer of the original MMLU question."],"supporting_citations":[{"why":"Supplies the Ruozhiba dataset and the original claim that Ruozhiba data improves fine-tuning, which this paper re-examines.","marker":"[Bai et al., 2024]"},{"why":"Provides the LLM-as-a-judge bias mechanism the authors use to explain why the original Ruozhiba advantage may be inflated.","marker":"[Zheng et al., 2023]"},{"why":"Provides the WizardLM-style rewriting prompt that the paper adapts as the base prompt for instruction rewriting.","marker":"[Xu et al., 2024]"},{"why":"Identifies the Meta-Llama-3-8B-Instruct model used for all supervised fine-tuning experiments.","marker":"[Dubey et al., 2024]"},{"why":"Supplies the LoRA method used to fine-tune the model in the experiments.","marker":"[Hu et al., 2022]"}],"fun_headline_variants":["Silly-question SFT data yields no net MMLU gain","Ruozhiba rule rewrites: no overall LLM boost, task-specific wins","Silly-question fine-tuning: slight task gains, overall neutral","Silly-question SFT: +5% on some tasks, -6% on others, no net gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison assumes that GPT-4's rewritten questions still mean the same thing and still have the same correct answer as the original MMLU question, even though the rewrite is never checked for correctness.","fun_headline_variants_meta":{"raw":{"variants":["Silly-question SFT data yields no net MMLU gain","Ruozhiba rule rewrites: no overall LLM boost, task-specific wins","Silly-question fine-tuning: slight task gains, overall neutral","Silly-question SFT: +5% on some tasks, -6% on others, no net gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001239,"raw_usage":{"total_tokens":5141,"prompt_tokens":1058,"completion_tokens":4083,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":674,"completion_tokens_details":{"reasoning_tokens":3993}},"tokens_in":674,"tokens_out":4083,"duration_ms":29606,"temperature":1.0,"reasoning_tokens":3993,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:30:44.791163+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of rewritten MMLU questions from each of the eight rules, have independent human annotators (or a strong non-GPT model) answer the original multiple-choice options against the rewritten stem, and measure how often the original reference answer remains correct; if a substantial fraction are corrupted, the observed task-level improvements and drops are confounded by label noise.","supporting_citations":[],"review_version":1}