{"id":"99ce35eb-d57c-440e-a5a6-17a04be3e53c","arxiv_id":"2411.11933","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"METEOR combines weak-to-strong distillation, iterative GPT-4 feedback, and contrastive self-training to adapt 7B-8B LLMs to a domain, with gains measured only by GPT-4 as judge.","lead":"This paper proposes METEOR, a three-stage training recipe that uses GPT-4 to distill and refine domain data, then lets a smaller LLM self-train by contrasting beam-search and greedy outputs. The authors report large gains on Stack Overflow Q&A judged by GPT-4, but the judge is the same model that supplied the training signal.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All reported improvements are measured by GPT-4, which also generated the distilled training data and corrective feedback, so the evaluation may reward GPT-4 mimicry rather than domain expertise; no independent human or objective evaluation is provided.","rationale":"Good-faith reading: the paper proposes a plausible three-phase recipe and supports it with ablations comparing with/without guidelines and with/without iterative training. The strongest empirical claim, however, is that the full METEOR pipeline produces large improvements on five domain-quality criteria. That claim rests entirely on GPT-4-as-judge. Because GPT-4 is also the teacher that creates the training targets and the correctness feedback, the evaluation loop is closed: the model is trained to satisfy GPT-4 and then evaluated by GPT-4. This does not by itself prove the numbers are wrong, but it removes any independent check on the claim. The paper's own limitation section concedes that self-evolution gains are limited, and Table 6 even shows a decrease (LLaMA3 completeness: 47.9% vs. 52.1%) while Section 4.4 asserts improvement across all metrics. That inconsistency is localized, but it further undermines confidence that the GPT-4-based scores measure what the paper claims. My recommended verdict is unchanged from the reader: REJECT as submitted, with the path to acceptance requiring independent human or objective evaluation. The concrete test above is a minimal version of that requirement.","tokens_in":15254,"tokens_out":8766,"duration_ms":87704,"concrete_test":"Select 200 test-set questions; collect paired answers (base vs. METEOR) for both LLaMA3-8B-Chat and Qwen2-7B-Instruct. Have three domain-expert annotators (e.g., experienced ML/NLP/CV engineers) independently rate each pair using the same five criteria and prompt structure as Table 8, blinded to which answer is from the METEOR model. Compute per-criterion win rates and GPT-4-human agreement (e.g., Cohen's kappa). If human win rates are substantially lower than GPT-4's reported values, or agreement is near chance, then the headline improvements are not established and the central claim fails. If human win rates reproduce the pattern, the circularity concern is largely answered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 states that 'we use GPT-4 as a judge to evaluate both data quality and model performance.' This is load-bearing because the same model is the sole source of the training signal: in Section 2.1, GPT-4 produces the distilled answers from weak-model guidelines; in Section 2.2.1, GPT-4 judges each response as correct/incorrect and supplies the guiding sentences used for fine-tuning. The final comparison (Table 2 and Table 8) then asks GPT-4 to decide, on subjective five-criteria judgments, whether the METEOR-trained model's answers are better than the base model's. Since the trained model has been optimized to produce outputs that GPT-4 approves of and stylistically resemble GPT-4's own generations, the 60-82% win rates could reflect judge self-preference, length, formatting, or other surface cues rather than genuine domain expertise. The paper provides no human evaluation, no inter-annotator agreement, no objective metric (e.g., correctness against known Stack Overflow accepted answers), and no alternative judge. A secondary internal inconsistency - Section 4.4 claims self-evolution improves all metrics even though Table 6 shows LLaMA3 completeness at 47.9% with vs 52.1% without - reinforces that the GPT-4-based evaluation pipeline is not reliable evidence for the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes METEOR, a three-phase training pipeline for adapting general-purpose LLMs to specific domains. Phase one is weak-to-strong data distillation: the weak model writes answering guidelines and GPT-4 generates answers conditioned on those guidelines. Phase two is iterative training: the domain model generates chain-of-thought answers, GPT-4 judges whether they are correct and provides corrective feedback, and the model is fine-tuned on the corrected trajectories. Phase three is self-evolution: the model is trained with a contrastive objective so that beam-search outputs are preferred over greedy outputs. The method is evaluated on Stack Overflow data from ML, DL, NLP, and CV categories using two base models (LLaMA3-8B-Chat and Qwen2-7B-Instruct). All performance claims are based on GPT-4 pairwise judgments across accuracy, completeness, relevance, coherence, and reliability. The paper reports large improvements from the full pipeline and from each phase individually.","tokens_in":15546,"tokens_out":4396,"duration_ms":42135,"significance":"If the reported effects were supported by trustworthy evidence, METEOR would be a useful contribution: the weak-to-strong distillation recipe is a sensible idea, the three-phase organization is clear, and the release of code would aid reproducibility. However, the central empirical claim is currently under-supported. The evaluation loop is closed: GPT-4 generates the distilled answers, produces the corrective feedback, and then serves as the judge of the final model outputs. The reported win rates may reflect the trained models' improved agreement with GPT-4's stylistic preferences rather than improved domain competence. The manuscript also contains a direct internal inconsistency in the self-evolution evaluation, where the text claims improvement on all metrics but Table 6 shows a decrease on completeness for LLaMA3. Because the main evidence is circular and the self-evolution evidence is self-contradictory, the contributions cannot be assessed as stated.","major_comments":[{"comment":"GPT-4 is used as the judge for both data quality and model performance, but GPT-4 is also the source of the distilled data (Section 2.1) and of the corrective feedback and guiding sentences used in iterative training (Section 2.2.1). The final pairwise win rates in Tables 2, 5, and 6 may therefore reflect the trained models' increased agreement with GPT-4's stylistic preferences rather than real gains in domain expertise. This is load-bearing for the paper's main claim. The authors should provide external validation: human evaluation with inter-annotator agreement, comparison against objectively correct answers (e.g., accepted Stack Overflow answers), or at minimum an alternative judge with a bias analysis.","section":"Section 3 (Metrics) and Sections 2.1, 2.2.1"},{"comment":"The text states that both models \"showed improved performance across all metrics following self-evolution,\" but Table 6 reports LLaMA3 completeness at 47.9% with self-evolution versus 52.1% without, which is a decrease. This directly contradicts the claim that the self-evolution component improves all evaluated dimensions. The authors must correct the table or the text and explain why the completeness metric regressed.","section":"Section 4.4 and Table 6"},{"comment":"No baseline comparisons against existing domain-adaptation or self-evolution methods are provided. The only comparisons are METEOR-trained models versus their untrained starting checkpoints. To establish that the proposed pipeline is valuable, the paper needs comparisons against direct fine-tuning on GPT-4 distilled data, self-refine-style methods, and other self-evolution baselines on the same data and with the same evaluation. Without such baselines, the improvements cannot be attributed to the specific design of METEOR.","section":"Section 4.1 and Table 2"},{"comment":"The claim that weak-to-strong distillation is superior is supported solely by GPT-4 scores of data that GPT-4 itself generated in both with-guideline and without-guideline conditions. A self-preference effect is highly plausible here. Table 4 uses the same circular evaluation for downstream model answers. Independent assessment of downstream task performance is needed before this claim can be accepted.","section":"Section 4.2 and Tables 3, 4"},{"comment":"The self-evolution gains are small in most dimensions (typically 1-5 percentage points) and negative in one dimension, and Section 7.1 itself acknowledges \"limited performance gains\" from self-evolution. This evidence does not support the paper's statement in Section 1 that the approach may lead to models \"potentially surpassing their strong supervisors.\" The authors should either temper this claim or provide direct comparisons against GPT-4 outputs.","section":"Section 2.2.2 and Table 6"}],"minor_comments":[{"comment":"The Qwen2 reliability entry in Table 2 is shown as \"c%\" and should be 67.1% according to the abstract; additionally, the model name is written inconsistently as \"LLaMA3\" and \"LLaMa3\" across the manuscript.","section":"Section 4.1 and Table 2"},{"comment":"There is a typo in the opening sentence: \"o validate\" should be \"To validate.\"","section":"Section 4.3"},{"comment":"The caption contains a typo: \"effiectiveness\" should be \"effectiveness.\"","section":"Table 6 caption"},{"comment":"The pairwise comparison prompt always presents Answer A before Answer B; position bias should be controlled by randomizing the order of the two answers, as is standard in LLM-as-judge evaluations.","section":"Table 8"},{"comment":"The hyperparameters N=3, K=10, and beam size 10 are fixed with no sensitivity analysis; the authors should provide ablations or justify the chosen values.","section":"Section 3 (Model and parameter settings)"}],"recommendation":"reject","confidential_remarks":"The central evaluation is circular because GPT-4 is simultaneously the teacher, the critic, and the judge, and no independent validation is provided. The internal inconsistency in Table 6 further weakens confidence in the reported numbers. Substantial new experiments, including external evaluation and baseline comparisons, would be required to support the manuscript's claims; this is beyond a routine revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the method is more interesting than the evaluation. The weak-to-strong guideline distillation (weak model writes guidelines that a strong model follows) is a fresh twist on an old idea, and the contrastive self-training between beam-search and greedy outputs is also a new component. The paper is clearly written, includes ablations for each phase, and the authors are honest in Section 7 that the self-evolution gains are limited. Those are real strengths.\n\nThe problem is that the evidence for the headline numbers does not hold up. GPT-4 generates the distilled training data, provides the corrective feedback in iterative training, and then acts as the sole judge for all five evaluation criteria. The trained model has been optimized to produce outputs that GPT-4 approves of, so the 60–82% win rates could just reflect stylistic mimicry—longer, better-formatted, more GPT-4-like answers—rather than real domain competence. There is no human evaluation, no alternative judge, no objective metric, and no comparison against standard baselines like direct distillation or STaR. That is a load-bearing gap.\n\nThere is also an internal contradiction you should know about. Section 4.4 claims self-evolution improves all metrics, but Table 6 shows LLaMA3 completeness dropping from 52.1% to 47.9% after self-evolution. The limitation section admits 'limited performance gains,' which is more consistent with the table, but the text still overstates the result. That kind of mismatch, on a small effect, makes me trust the GPT-4 scores even less.\n\nThe method components are plausible and cheap enough that the idea might work, and the paper deserves a serious referee—I would send it to review and ask for major revision rather than desk-reject. But as it stands, the central claim is not evidenced. Add human eval, an independent judge or objective correctness, plus baselines and error bars, and the paper could be rescued.","headline":"A plausible three-phase recipe for domain specialization, but the evaluation is undermined by GPT-4 serving as both teacher and judge.","tokens_in":16070,"tokens_out":2642,"would_cite":false,"duration_ms":24544,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"METEOR, a three-phase training pipeline, turns general-purpose LLMs into domain-specific experts, with large gains on all five evaluated quality criteria.","keywords":["METEOR","weak-to-strong distillation","LLM self-evolution","domain adaptation","iterative training","GPT-4 as judge","contrastive self-training","knowledge distillation"],"falsifier":"Take a held-out set of domain questions and have human domain experts (or a different strong model) independently judge the same before/after answers. If the human experts find no consistent preference for METEOR-trained outputs, or if a non-GPT-4 judge shows much smaller gains, the central claim fails. Concretely: run the exact pipeline with GPT-4 as teacher and judge, then evaluate on a multiple-choice QA benchmark with known ground truth in the same domain and compare accuracy.","tokens_in":15061,"feed_emoji":"🧠","tokens_out":6331,"duration_ms":49555,"temperature":0.7,"pith_summary":"The paper claims that a general-purpose large language model can be turned into a domain expert through a three-stage training pipeline called METEOR. In the first stage, the weak model writes guidelines that a strong model (GPT-4) must follow when producing answers, and the resulting data fine-tunes the weak model. In the second stage, GPT-4 critiques the model's chain-of-thought answers and the model retrains on its corrected attempts. In the third stage, the model trains itself by preferring its own beam-search outputs over its greedy outputs, using contrastive learning. On Stack Overflow questions about machine learning, deep learning, NLP, and computer vision, both LLaMA3-8B-Chat and Qwen2-7B-Instruct show large jumps in GPT-4-judged accuracy, completeness, relevance, coherence, and reliability.","feed_headline":"METEOR lifts LLM domain scores by up to 80 points","feed_subtitle":"Three stages: weak-to-strong distillation, iterative feedback, self-training: chat models become domain experts.","key_machinery":"The central mechanism is the weak-to-strong guideline bridge: the weak model first generates a structured guideline for each domain question, and GPT-4 generates the answer by following that guideline. This aligns the strong model's output distribution with the weak model's own distribution, producing training data the weak model can actually learn from. The other two phases are the reflective refinement loop, where GPT-4 judges chain-of-thought attempts and the model retrains on corrected rationales, and the contrastive self-training step, where beam search (high FLOPs) outputs are treated as positive examples and greedy (low FLOPs) outputs as negative examples.","core_discovery":"The central discovery claim is that the weak-to-strong distillation step is what unlocks most of the gain. When the weak model supplies guidelines before GPT-4 writes an answer, the distilled data scores roughly 9.07 out of 10 versus 5.77 without guidelines, and models fine-tuned on guideline-distilled data score 9.04-9.22 versus 5.38-6.13. The paper argues that directly distilling answers from a strong model's distribution is inefficient because the weak model cannot absorb them; the guidelines bridge the distribution gap. The iterative training phase adds further gains by using GPT-4 feedback to correct the model's reasoning, and the self-evolution phase gives smaller but consistent improvements by training the model to imitate its own higher-compute outputs. The overall result is that the full pipeline lifts LLaMA3-8B-Chat's judged accuracy from 21.3% to 78.7% and Qwen2-7B-Instruct's from 31.6% to 68.4%, with similar gains on the other four criteria.","pith_inferences":["If the gains survive evaluation by a non-GPT-4 judge or human raters, the weak-to-strong guideline bridge could become a general technique for distilling any strong teacher into a smaller student model, not just for question-answering domains.","The modest gains from the self-evolution phase suggest that beam-search-versus-greedy contrast may be a weak training signal; using a range of beam widths or sampling temperatures could produce stronger self-improvement.","Because GPT-4 supplies the training data, the feedback, and the judge, an independent verification study with a different judge model or human domain experts would determine whether the reported improvements reflect genuine domain competence or stylistic mimicry of the teacher."],"forward_implications":["Each phase contributes to the final result, and the weak-to-strong distillation step alone roughly doubles the judged quality of the training data (9.07 vs 5.77).","Models trained with the full METEOR pipeline outperform the same base models without it on all five criteria, with the largest gains in reliability and coherence for LLaMA3.","The self-evolution phase improves model outputs even without external feedback, by training on the model's own higher-compute outputs, though gains are smaller than the guided phases.","The method transfers across two different base models (LLaMA3-8B-Chat and Qwen2-7B-Instruct) and across four domain categories (ML, DL, NLP, CV).","Because the pipeline only needs domain questions without human-annotated answers, it can be applied to new domains cheaply."],"supporting_citations":[{"why":"Provides the strong teacher and judge model GPT-4 used in all three phases for distillation, feedback, and evaluation.","marker":"OpenAI, 2023"},{"why":"Supplies the LLaMA3-8B-Chat base model that is evolved into a domain expert.","marker":"Dubey et al., 2024"},{"why":"Supplies the Qwen2-7B-Instruct base model that is evolved into a domain expert.","marker":"Yang et al., 2024a"},{"why":"Motivates the weak-to-strong strategy for aligning strong model outputs with weak model capabilities.","marker":"Burns et al., 2024"},{"why":"Provides the reflection mechanism used in the iterative training phase to generate corrective feedback.","marker":"Shinn et al., 2023"},{"why":"Establishes the LLM-as-a-judge evaluation method used to score all model outputs on five criteria.","marker":"Zheng et al., 2023"},{"why":"Supports the claim that higher inference FLOPs improve performance, motivating beam search in the self-evolution phase.","marker":"Snell et al., 2024"},{"why":"Provides chain-of-thought prompting used to generate rationales and answers in the iterative training phase.","marker":"Wei et al., 2022"},{"why":"Prior work on self-training with self-generated data in coding, which this paper extends to other domains.","marker":"Singh et al., 2024"}],"fun_headline_variants":["METEOR: weak-to-strong guidance lifts LLM scores by 80 pts","METEOR's key: weak teacher, strong student, +80 points","METEOR self-evolves LLMs: 21% to 79% accuracy jump","Guidance-first distillation: METEOR adds 78 points to LLMs","METEOR: three-stage evolution, up to 80-point LLM gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire evaluation assumes that GPT-4's pairwise judgments of accuracy, completeness, relevance, coherence, and reliability are a valid and unbiased measure of true domain expertise, even though GPT-4 also wrote the training data and the feedback.","fun_headline_variants_meta":{"raw":{"variants":["METEOR: weak-to-strong guidance lifts LLM scores by 80 pts","METEOR's key: weak teacher, strong student, +80 points","METEOR self-evolves LLMs: 21% to 79% accuracy jump","Guidance-first distillation: METEOR adds 78 points to LLMs","METEOR: three-stage evolution, up to 80-point LLM gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000318,"raw_usage":{"total_tokens":1752,"prompt_tokens":858,"completion_tokens":894,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":786}},"tokens_in":474,"tokens_out":894,"duration_ms":8564,"temperature":1.0,"reasoning_tokens":786,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:17:40.828195+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of domain questions and have human domain experts (or a different strong model) independently judge the same before/after answers. If the human experts find no consistent preference for METEOR-trained outputs, or if a non-GPT-4 judge shows much smaller gains, the central claim fails. Concretely: run the exact pipeline with GPT-4 as teacher and judge, then evaluate on a multiple-choice QA benchmark with known ground truth in the same domain and compare accuracy.","supporting_citations":[],"review_version":1}