{"id":"f7f267e3-13b9-4c3b-b8a9-0d0730ecbc73","arxiv_id":"2509.25148","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding a frozen POLAR-based similarity penalty to SFT, GRPO, or CHORD improves averaged instruction-following scores by up to 5.77 percent at 0.6B scale in the paper's reported runs.","lead":"This paper adds a fixed teacher-similarity score computed by a frozen POLAR model to standard LLM post-training losses, and reports improved instruction-following scores on small Qwen3 models. The authors call this adversarial anchoring, but no discriminator is trained and no minimax game is played, so the contribution is closer to reward shaping.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central mechanism is under-specified: the POLAR scalar coef cannot be turned into the claimed adversarial gradient g_ADV without an explicitly stated estimator for discrete text; if the released code detaches coef into the reward, the method is reward shaping, not adversarial anchoring.","rationale":"The paper's provided abstract (AAPA) and body (UniAPL) are internally inconsistent, and the 4B headline is contradicted by Tables 2 and 8: GRPO=68.11 versus A-SFT→A-GRPO=67.93, with the detailed table listing S→A-G=66.10, neither of which supports a 3.75% improvement. These are real problems, but they are secondary because they can be resolved by aligning the text and correcting tables. The deeper issue is that the paper's proposed mechanism is not defined as an optimization objective: Section 3.2 defines L_ADV through an expectation over πθ and then declares g_ADV = ∇θL_ADV, while Algorithm 1 only returns a scalar coef. Discrete token sampling breaks the autograd path. The standard resolutions are REINFORCE with a baseline, a Gumbel-Softmax surrogate, or treating coef as a reward; each choice changes what is actually tested. A reward bonus is not an adversarial gradient, a REINFORCE term needs variance control, and a straight-through estimator needs to be described. Table 4 claims that an independent adversarial loss works best, but that loss is never written down. Because the central claim is that adversarial anchoring, and not merely reward shaping, improves alignment, the missing gradient estimator is the load-bearing assumption, and the code inspection described above would settle it.","tokens_in":19258,"tokens_out":12211,"duration_ms":108837,"concrete_test":"Clone the released repository (github.com/IsFaqq/AAPA) and locate the tensor for L_ADV in the A-SFT and A-GRPO training scripts. Trace whether coef is detached before any multiplication with logπθ, whether it appears inside the advantage/reward, or whether a differentiable surrogate sampler is used. Then re-run the 0.6B SFT→A-GRPO configuration with coef detached (reward-only); if IFEval/MultiIF scores do not change materially, the reported gains do not come from the proposed adversarial gradient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is that the scalar coef returned by Algorithm 1 can enter the policy update as the 'adversarial gradient' g_ADV = ∇θL_ADV in Section 3.2. The paper never defines the estimator for this gradient when the response is discrete text. If coef is detached and added to the GRPO reward or advantage, the method is reward shaping, not an adversarial gradient; if coef is used in a REINFORCE-style term -coef·∇θ log πθ(y_s|x), the missing baseline makes the update high-variance and the formula should be written; if coef is backpropagated through a Gumbel-Softmax or straight-through sampler, that estimator must be stated. The ablation in Table 4 distinguishes 'adding to reward' from 'using as a separate adversarial loss' and reports the latter as best, but the separate loss is never written down. Without this equation, the numbers in Tables 2-4 cannot be attributed to the claimed mechanism, and the central claim collapses if the code implements reward shaping instead.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript (submitted under the title AAPA; the body title is UniAPL) proposes a plug-in adversarial anchoring term for LLM post-training. A frozen POLAR discriminator scores pairs of teacher and student responses, and the resulting scalar coefficient is added to SFT, GRPO, and CHORD objectives. The authors report consistent instruction-following gains, with headline improvements of 5.77% on Qwen3-0.6B and 3.75% on Qwen3-4B over GRPO, and they include ablations, response-length distributions, log-probability analyses, and a code release.","tokens_in":19527,"tokens_out":6187,"duration_ms":52270,"significance":"If the mechanism is well defined, the idea of a frozen offline discriminator as a distributional regularizer is attractive and practical: it avoids co-training an online discriminator and does not reuse target-benchmark labels in the training signal, so the central numerical claim is not circular in the narrow sense. The reported gains on Qwen3-0.6B, the ablation separating reward injection from a separate adversarial loss, and the public code are assets. However, the mechanism's central gradient is never defined for discrete text, and key reported numbers are mutually inconsistent, so the paper cannot currently support its headline claims.","major_comments":[{"comment":"The adversarial gradient g_ADV = ∇θ L_ADV is not computable as written. L_ADV = -E[Dϕ(x, y_teacher, y)] depends on y sampled from πθ, and Dϕ is a scalar function of a complete text; the manuscript never states how the gradient through the sampling process is estimated. If the POLAR coefficient is detached and added to the GRPO reward or advantage, the method is reward shaping, not an adversarial gradient; if a REINFORCE-style estimator is intended, the baseline and variance-reduction term must be written; if Gumbel-Softmax or straight-through is used, it must be stated. Table 4 distinguishes 'adding to the reward' from 'a separate adversarial loss' and reports the latter as best, but the separate loss is never defined. Without this estimator, the numbers in Tables 2–4 cannot be attributed to the claimed mechanism.","section":"Section 3.2 and Algorithm 1"},{"comment":"The mapping coef = 1 - 8(p - 0.5)^2 is inconsistent with the paper's description of p as a similarity score and with the stated goal of maximizing Dϕ. Under this mapping, coef = +1 at p = 0.5 and coef = -1 as p approaches 0 or 1, so both 'same' and 'different' receive the most negative anchoring weight; maximizing L_ADV would then push p away from 0.5, that is, away from the configuration described as indistinguishable. The semantics of p (distance vs. similarity) and the direction of the transformation must be corrected, otherwise the sign of the proposed anchor is reversed.","section":"Algorithm 1 and Appendix A.3"},{"comment":"The SFT→A-GRPO row for Qwen3-4B reports Avg = 68.11 in Table 2 but 66.10 in Table 8. In addition, the abstract's claim of a 3.75% improvement over GRPO is not traceable to any staged row in Table 2: GRPO Avg is 68.11, A-GRPO is 68.39, SFT→A-GRPO is 68.11, and A-SFT→A-GRPO is 67.93. Please correct the tables and specify which configuration and which baseline produce the 3.75% figure.","section":"Section 4.2, Table 2, and Appendix B.3 Table 8"},{"comment":"The claim that AAPA consistently improves the corresponding base objectives across model scales is not supported by the unified experiments: on Qwen3-4B, A-CHORD (Avg 69.27) is slightly below CHORD (Avg 69.30), with the overall decrease driven by drops in Coding and Chinese. The paper should either qualify the universality claim, report run-to-run variance or statistical significance, or provide the missing supporting results.","section":"Table 3 and Section 4.2"}],"minor_comments":[{"comment":"The submission title and abstract use AAPA, while the body title, Section 3, and Figure captions use UniAPL; please unify the naming throughout.","section":"Title and abstract"},{"comment":"There is a recurring typo, 'adversaril loss', which should be 'adversarial loss'.","section":"Section 4.1 and Section 4.3"},{"comment":"The table and text use inconsistent labels for the proposed method (RLVR+ADVcoef, wADV coef, and w Acoef); please clarify which setting corresponds to the separate adversarial loss described in setting (iii).","section":"Table 4 and caption"},{"comment":"Figure 2 states that UniAPL-0.6B is 'on par' with Qwen3-32B, but no Qwen3-32B benchmark numbers appear in Table 2; please add the supporting comparison or remove the claim.","section":"Figure 2"},{"comment":"The row 'With GT Response, NO KL' shows a sharp collapse in mathematics (GSM8K 12.64, MATH-500 8.34); the main text's remark that disabling KL 'may lead to collapse' should be quantified and explicitly tied to this row.","section":"Appendix B.2, Table 6"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript has a submission-id/title mismatch (AAPA vs. UniAPL), and the headline percentages do not currently match the reported tables. The undefined gradient estimator is the central technical gap; if the released code disambiguates the estimator, the authors should be asked to state it explicitly in the paper. I recommend a full numerical audit of Tables 2, 3, and 8 before any acceptance decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper (the arXiv listing says AAPA; the full text says UniAPL—already a red flag) has a reasonable empirical idea: take a frozen POLAR discriminator, compute a sentence-level coefficient comparing policy rollouts to offline expert responses, and add that as an extra term on top of SFT, GRPO, or CHORD. The 0.6B results are directionally consistent and the ablation that separates \"add to reward\" from \"use as a separate loss\" is a useful data point. If the implementation matches the description, this is a cheap plug-in that small-model practitioners would want.\n\nBut two load-bearing issues stand between this paper and a clean recommendation.\n\nFirst, the adversarial gradient is never actually defined for discrete text. Section 3.2 writes g_ADV = ∇θ L_ADV with L_ADV = -E[D], and D is the scalar from Algorithm 1. For sampled tokens, that gradient doesn't exist as written. If the coefficient is detached and used as a reward shaper, the method is reward shaping, not adversarial anchoring; if it enters through a REINFORCE term, that estimator needs to be written down; if through Gumbel-Softmax or similar, that also needs to be stated. The ablation in Table 4 claims the separate loss works best, but the formula for that separate loss is missing. This is not a nitpick—without the estimator, the numbers can't be attributed to the claimed mechanism.\n\nSecond, the headline 4B gain does not survive contact with the tables. The abstract says 3.75% over a strong GRPO baseline. In Table 2, GRPO on Qwen3-4B is 68.11, and A-SFT→A-GRPO is 67.93—that's a small drop, not a 3.75% gain. SFT→A-GRPO in the same table is 68.11, identical to GRPO. Worse, the detailed table in Appendix B.3 lists SFT→A-GRPO as 66.10, not 68.11. The main table and the appendix disagree, so the reader cannot reproduce the central claim from the paper's own data.\n\nThere are smaller issues: the \"adversarial\" framing is mostly rhetoric since the discriminator is frozen and no game is played; the unified SFT-as-reward derivation in Appendix A.1 is standard MLE repackaging; and there are no error bars on any of the instruction-following numbers, which are noisy.\n\nWho is this for? Practitioners who want a cheap anchor for small-model post-training. The idea deserves a serious referee, but the paper needs major revision: spell out the gradient estimator, fix the numbers, and reconcile the tables. Send it out, but expect the authors to do real work.","headline":"A cheap and interesting post-training plug-in for small models, undermined by an unspecified gradient estimator and a 4B headline gain that the tables don't support.","tokens_in":20067,"tokens_out":4863,"would_cite":false,"duration_ms":35746,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen discriminator scoring rollouts against offline expert answers improves SFT, GRPO, and CHORD on Qwen3-0.6B and 4B.","keywords":["adversarial preference learning","instruction following","GRPO","supervised fine-tuning","distributional mismatch","post-training alignment","large language models","discriminator regularization"],"falsifier":"Open the public implementation and locate where the discriminator coefficient is computed. If the coefficient is detached and used only as a scalar reward, reproduce the main A-GRPO result with a version that backpropagates through the sampled tokens; the central claim collapses if the only working version is detached.","tokens_in":19081,"feed_emoji":"🎯","tokens_out":11814,"duration_ms":94705,"temperature":0.7,"pith_summary":"This paper claims that the standard SFT-then-RL post-training recipe is inefficient because the policy drifts away from the expert demonstrations it was tuned on, while online RL explores without access to that dense expert knowledge. The proposed fix, AAPA (called UniAPL in the full text), is a plug-in term that compares each policy rollout with an offline expert response using a fixed discriminator and adds the resulting scalar as an adversarial anchoring gradient to SFT, GRPO, or CHORD. The paper reports that this term improves instruction-following benchmarks across Qwen3-0.6B and Qwen3-4B, with the staged configuration beating a strong GRPO baseline by 5.77% and 3.75%, respectively, while keeping general English, coding, math, and Chinese capabilities roughly intact. A sympathetic reader would take away that a frozen reference signal can regularize both imitation and preference optimization without extra teacher calls or discriminator updates during training.","feed_headline":"One frozen discriminator improves LLM alignment across SFT and RL","feed_subtitle":"Adding a fixed adversarial term to GRPO is reported to lift Qwen3-0.6B by 5.77% and Qwen3-4B by 3.75%.","key_machinery":"The load-bearing object is a frozen, reference-based discriminator adapted from POLAR. Given a prompt $x$, a teacher response $y_{\\mathrm{teacher}}$, and a student rollout $y$, it returns a scalar coefficient $coef = 1 - 8(\\sigma(r) - 0.5)^2$, where $r$ is the POLAR Bradley-Terry score and $\\sigma$ is the sigmoid. The paper treats this coefficient as the discriminator output $D_\\phi(x, y_{\\mathrm{teacher}}, y)$, feeds it into the adversarial loss $L_{\\mathrm{ADV}} = -\\mathbb{E}[D_\\phi(x, y_{\\mathrm{teacher}}, y)]$, and adds the resulting gradient $g_{\\mathrm{ADV}} = \\nabla_\\theta L_{\\mathrm{ADV}}$ to the base objective with a small weight $\\lambda_{\\mathrm{adv}}$. In the unified formulation, one update is written as the weighted sum of an imitation gradient, a preference-seeking gradient, and the adversarial grounding gradient; that sum is what the paper calls the mechanism that prevents ungrounded policy drift.","core_discovery":"The paper frames alignment as a constrained optimization problem: maximize a preference reward subject to staying close to an expert policy, and it argues that the sequential SFT-then-RL pipeline solves this problem badly. Its claim is that adding a sentence-level adversarial anchor makes the solution better: a frozen discriminator $D_\\phi(x, y_{\\mathrm{teacher}}, y)$ scores each student rollout against an offline teacher response, and the policy is trained to maximize that score through the adversarial loss $L_{\\mathrm{ADV}} = -\\mathbb{E}[D_\\phi(x, y_{\\mathrm{teacher}}, y)]$, while the original SFT, GRPO, or CHORD loss is kept. Across the reported instruction-following benchmarks, the staged A-SFT to A-GRPO configuration reaches an average of 43.31 on Qwen3-0.6B versus 40.93 for GRPO, and the paper reports 5.77% and 3.75% improvements over the GRPO baseline on the 0.6B and 4B Qwen3 models. The paper also shows that models trained with the anchor have response-length and log-probability distributions closer to the teacher than GRPO-trained models, which it reads as evidence of semantic grounding rather than reward hacking.","pith_inferences":["If the public implementation detaches the discriminator scalar and uses it only as a reward, the method reduces to a form of reward shaping, and the adversarial-gradient interpretation is a relabeling; checking that implementation is the decisive test.","Because the teacher is frozen and no minimax game is played, the method is a fixed reference-based regularizer rather than a true adversarial procedure; comparing it against simply adding a fixed reference reward to the verifiable reward would separate these two explanations.","The paper only tests students far smaller than the 235B teacher, so whether the anchor still helps when the student approaches the teacher is an open prediction, not a demonstrated result."],"forward_implications":["The same fixed anchor can be bolted onto SFT, GRPO, and CHORD without changing their pipelines, so the claimed gain is tied to the anchor, not to a specific optimizer.","In the reported experiments A-GRPO matches the instruction-following level of an SFT-then-GRPO sequence, implying the anchor can act as a surrogate warm-up.","The anchor does not require online teacher inference or discriminator co-training, so the cost per rollout is one forward pass through the frozen discriminator.","The mixed-batch unified objective lets post-training run as a single stage, removing the multi-run hand-off of staged pipelines while retaining the instruction-following gain."],"supporting_citations":[{"why":"Defines GRPO, the base RL objective that A-GRPO extends and the main baseline for the reported gains.","marker":"[8]"},{"why":"Supplies the frozen POLAR discriminator that the paper adapts to compute the adversarial coefficient.","marker":"[53]"},{"why":"Defines CHORD, the joint SFT/RL baseline to which the paper adds the anchor as A-CHORD.","marker":"[33]"},{"why":"Provides the AutoIF instruction-following training data.","marker":"[38]"},{"why":"Provides the IFevallike training data.","marker":"[39]"},{"why":"Provides the IFBench training data and the 38K subset used for distribution analysis.","marker":"[40]"},{"why":"Defines the IFEval instruction-following benchmark used in evaluation.","marker":"[41]"},{"why":"Defines the MultiIF benchmark used in evaluation.","marker":"[42]"},{"why":"Documents the Qwen3 models used as base models and as the teacher that produced offline expert responses.","marker":"[21]"}],"fun_headline_variants":["Frozen discriminator anchor boosts LLM alignment by up to 5.77%","Plug-in adversarial term lifts GRPO SFT and CHORD alignment","AAPA: one fixed discriminator improves post-training alignment","Add a frozen discriminator to LLM training, get better alignment","Adversarial anchoring without co-training: AAPA boosts LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the frozen discriminator's scalar output can be turned into a genuine gradient for the policy update, even though the paper does not specify how that gradient flows through discrete token sampling.","fun_headline_variants_meta":{"raw":{"variants":["Frozen discriminator anchor boosts LLM alignment by up to 5.77%","Plug-in adversarial term lifts GRPO SFT and CHORD alignment","AAPA: one fixed discriminator improves post-training alignment","Add a frozen discriminator to LLM training, get better alignment","Adversarial anchoring without co-training: AAPA boosts LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000979,"raw_usage":{"total_tokens":4225,"prompt_tokens":1081,"completion_tokens":3144,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":697,"completion_tokens_details":{"reasoning_tokens":3051}},"tokens_in":697,"tokens_out":3144,"duration_ms":21710,"temperature":1.0,"reasoning_tokens":3051,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:43:21.490788+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Open the public implementation and locate where the discriminator coefficient is computed. If the coefficient is detached and used only as a scalar reward, reproduce the main A-GRPO result with a version that backpropagates through the sampled tokens; the central claim collapses if the only working version is detached.","supporting_citations":[],"review_version":2}