{"id":"56e68419-f4f8-4a7f-a1d0-2abda51df7b6","arxiv_id":"2412.11145","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper proposes a definition and an attacker-learner-critic framework for aligning AI models that are stronger than human experts, based on scalable feedback instead of reliable human labels.","lead":"This position paper defines superalignment as learning from noisy, machine-generated labels when AI models outsmart human experts, and proposes a loop of attacker, learner, and critic models to keep such systems aligned. It matters because it gives researchers a shared map for aligning future superhuman AI, a problem that today's human-feedback methods cannot solve.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central feasibility claim depends on critic feedback being faithful and informative, but the paper's own cited evidence includes self-preference and model collapse; without a controlled noisy-critic test, the attacker-learner-critic loop is not shown to improve rather than amplify noise.","rationale":"The reader's weakest assumption—critic feedback must be faithful and learnable—is the right locus. My stress-test sharpens it: the paper's own Section 4.4 supplies evidence that self-improvement loops can collapse or lose diversity, and reference [49] indicates self-preference bias, so the cited 'early attempts' do not cleanly verify feasibility. This is an evidentiary gap rather than an internal inconsistency; the paper is a position paper and explicitly labels many problems open. The reader's CONDITIONAL verdict already captures this: the framework is a plausible research agenda but not a validated method. I recommend no change to the verdict. Agreement is partial because the reader emphasizes the critic being fallible or weaker, while I emphasize the loop's information closure and the mixed empirical evidence the paper itself cites; both reduce to the same load-bearing assumption, but the concrete test differs. The proposed test is feasible with existing benchmarks and would directly distinguish the oracle-feedback regime from the self-feedback regime that the paper's framework permits.","tokens_in":13757,"tokens_out":4352,"duration_ms":39745,"concrete_test":"Run the proposed loop on a ground-truth-verifiable task (e.g., MATH-500 or competitive-programming unit tests) for 5-10 iterations under three critic conditions: (i) an oracle/ground-truth critic, (ii) the same base model as the learner acting as self-critic, and (iii) an independent stronger model as critic. Measure task accuracy, output diversity, and critic agreement with ground truth each iteration. If condition (ii) or (iii) does not improve over iterations, or degrades relative to (i), the critic-faithfulness assumption is violated and the paper's automatic-iteration claim must be restricted to settings with reliable external supervision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the attacker-learner-critic pipeline 'can be automatically executed and iterated when it is started from some seed input' (Section 4) and that existing self-alignment, self-play, and self-refinement works 'partially verif[y] the feasibility' of the framework (Section 5). For that claim to hold, the critic's feedback must be faithful, informative, and learnable. The paper itself leaves this open: Section 4.3 asks 'how can we ensure and evaluate the faithfulness of a generated critic?', and Section 4.2 asks which critic forms will be easiest to learn from. This is not merely an open question, because the cited evidence is mixed: Section 4.4 cites model collapse [60] and output-diversity declines in self-refinement loops [61,62,63,64], and reference [49] shows LLM evaluators favor their own generations. When the attacker, learner, and critic are versions of the same foundation model, the loop can become informationally closed, amplifying the model's existing biases instead of providing an independent alignment signal. The paper does not bound the critic-error rate or diversity loss under which the loop improves rather than degrades, so the feasibility statement goes beyond what the cited works establish.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This position paper argues that as LLMs approach or exceed human performance on complex tasks, conventional human-feedback alignment becomes unscalable, and proposes a learning-theoretic definition of superalignment: designing alignment algorithms that learn from noisy labels (pointwise or pairwise) when tasks are too complex for human annotation and the model is stronger than human experts. It identifies three key research problems—weak-to-strong generalization, scalable oversight, and evaluation—and then presents a conceptual framework consisting of an attacker, a learner, and a critic. The attacker generates adversarial queries to expose learner weaknesses, the learner improves from scalable critic feedback with minimal human intervention, and the critic produces textual critiques. The paper surveys related work on self-alignment, self-play, and self-refinement, and concludes that these existing works 'partially verif[y] the feasibility' of the framework.","tokens_in":13890,"tokens_out":3126,"duration_ms":28920,"significance":"As a research agenda, the paper is a useful synthesis: it gives a clear framing of superalignment as a noisy-label learning problem, identifies concrete open problems (e.g., critic faithfulness, learnable feedback forms, automatic weakness discovery), and honestly acknowledges negative results such as model collapse and self-preference. The proposed attacker-learner-critic loop is an organizing framework rather than an empirical claim, and for a position paper that is acceptable provided the claims are carefully calibrated. The paper's main weakness is that its conclusion overstates the degree to which existing work verifies feasibility; the cited evidence is mixed, and the central assumption about critic quality is left explicitly open. The paper contains no experiments or proofs, but this is not a defect for a position paper if the framing and open problems are the contribution.","major_comments":[{"comment":"The conclusion that existing self-alignment, self-play, and self-refinement works 'partially verif[y] the feasibility of the framework' is not supported by the evidence cited in the paper itself. Section 4.4 cites model collapse [60] and declines in output diversity and OOD generalization in iterative self-refinement loops [61–64], and reference [49] shows that LLM evaluators favor their own generations. These results indicate that the loop can amplify noise or reduce diversity under conditions the paper does not specify. The feasibility claim should be softened to 'these works provide partial evidence in restricted settings where automatic verification is available' or accompanied by explicit conditions under which the attacker-learner-critic loop is expected to improve rather than degrade.","section":"Section 4.4 and Section 5"},{"comment":"The framework's central load-bearing assumption is that the critic can provide feedback that is simultaneously faithful, informative, and learnable by the learner. The paper itself lists this as an open problem: it asks 'how can we ensure and evaluate the faithfulness of a generated critic?' (Section 4.3) and 'how can the learner learn from such textual critics?' (Section 4.2). Because the attacker, learner, and critic can be versions of the same foundation model, the loop is informationally closed; if the critic is noisy or biased, the learner may amplify that bias rather than receive an independent alignment signal. The manuscript should either provide a minimal formal condition (e.g., a bound on critic error or a diversity-preservation mechanism) or explicitly reframe feasibility as an empirical hypothesis to be tested, rather than a claim partially verified by prior work.","section":"Sections 4.2–4.3"},{"comment":"The attacker is trained using a reward signal derived from the critic, but the paper does not discuss how to distinguish weaknesses of the learner from artifacts or biases of the critic. If the critic systematically down-ranks certain legitimate responses, the attacker will learn to exploit those critic errors, and the loop will optimize against a flawed reward rather than against genuine alignment failures. The paper should discuss this potential circularity, for example by proposing the use of multiple critics, human spot-checking of adversarial queries, or validation of attacker-generated queries on independent evaluators.","section":"Section 4.1"}],"minor_comments":[{"comment":"The word 'alignning' should be corrected to 'aligning' in the sentence describing PPO.","section":"Section 2.2"},{"comment":"In the sentence beginning 'however, it becomes much more complex in The setting of LLMs', 'The' should be lowercase.","section":"Section 3.1"},{"comment":"The phrase 'today's evaluation is heavily reliable on static evaluation' should be 'heavily relies on static evaluation'.","section":"Section 3.3"},{"comment":"References [29] and [35] are the same CriticGPT paper (McAleese et al., 2024) and should be merged or cross-referenced to avoid duplication.","section":"References"},{"comment":"The phrase 'that is worthy to study in near future' should read 'that is worthy of study in the near future'.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a position paper, so the lack of experiments is not itself a reason for rejection. However, the conclusion's feasibility claim goes beyond what the cited evidence supports, and the central critic-quality assumption is left unresolved. The authors should calibrate the language accordingly. I also note that several pieces of evidence for feasibility come from the authors' own prior systems (AutoDetect, CritiqueLLM, SPAR); while this is legitimate, independent replication or a more neutral framing would strengthen the paper. The manuscript fits the scope of a position paper in an information sciences venue, but the revision should make clear that the framework is a research agenda, not a verified mechanism."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good to have a clear position paper on superalignment. The definition is useful: superalignment as designing alignment algorithms that learn from noisy labels when the model is stronger than human experts. That reframes the problem in ML terms and makes it tractable. The attacker-learner-critic loop is a sensible decomposition, and it connects red teaming, scalable oversight, and self-improvement into one agenda.\n\nWhat the paper does well: it surveys the relevant literature accurately, including weak-to-strong generalization, scalable oversight, and evaluation. It also cites the negative results—model collapse and diversity decline in self-refinement loops—so it is not naive about the risks. The writing is clear and the open questions are honestly listed.\n\nThe soft spots are real but not fatal. First, the conclusion claims that existing self-alignment, self-play, and self-refinement works 'partially verify' the framework's feasibility. That overstates things. Most of those works rely on seed human data or ground-truth verifiers; they do not test the hard part of the framework, which is learning from critic feedback that is itself noisy and possibly biased. The paper's own Section 4.3 flags critic faithfulness as an open problem, so the gap is acknowledged, but the conclusion still sounds more confident than the evidence warrants. Second, the framework's novelty is mainly packaging; the components all exist in prior work. For a position paper, that is acceptable, but it should be clearer that the contribution is organizational, not a new mechanism.\n\nThe stress-test concern about information closure—attacker, learner, and critic being versions of the same model—is legitimate. The paper cites self-preference results [49] but does not discuss how the loop avoids amplifying the model's own biases. However, the paper explicitly lists this as a research problem, so I would not call it a fatal flaw, just an unresolved one.\n\nBottom line: this is a useful survey and agenda-setting document for alignment researchers, and it deserves serious peer review as a position paper. I would recommend acceptance after the authors tone down the 'partial verification' claim and sharpen the distinction between the proposed framework and the existing self-improvement works it cites.","headline":"A clear, honest position paper that defines superalignment as scalable learning from noisy labels and proposes an attacker-learner-critic loop; the framework is plausible but the conclusion slightly overstates how much existing self-improvement work verifies it.","tokens_in":14520,"tokens_out":2092,"would_cite":true,"duration_ms":18166,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This position paper argues that superalignment can be achieved by an automatic loop in which an attacker exposes a model's weaknesses, a critic supplies scalable feedback, and the learner improves itself with minimal human oversight.","keywords":["superalignment","superhuman intelligence","large language models","scalable feedback","weak-to-strong generalization","attacker-learner-critic loop","self-refinement","adversarial evaluation"],"falsifier":"A controlled experiment in which a strong learner is trained exclusively on critiques generated by a weaker critic on tasks too hard for humans would falsify the framework if the learner fails to improve or degrades over multiple rounds.","tokens_in":13459,"feed_emoji":"🤖","tokens_out":5146,"duration_ms":41813,"temperature":0.7,"pith_summary":"This position paper defines superalignment as the problem of designing alignment algorithms that can learn from noisy labels when tasks become too complex for human experts to annotate and the model is stronger than its human supervisors. It argues that superalignment can be achieved by an automatic pipeline with three modules: an attacker that generates adversarial queries to expose the learner's weaknesses, a learner that improves from scalable feedback, and a critic that produces faithful, learnable textual critiques, with humans providing only minimal supervision. The paper's load-bearing claim is that this pipeline can be automatically executed and iterated from some seed input, and that existing self-alignment, self-play, and self-refinement work already shows positive signs, partially verifying the framework's feasibility. If true, this would mean superhuman models could be kept aligned without reliable human annotation on every hard task.","feed_headline":"Attacker-learner-critic loop could align superhuman AI","feed_subtitle":"The paper argues that scalable feedback from a critic, with minimal human input, can keep superhuman models safe and aligned.","key_machinery":"The central mechanism is the attacker-learner-critic loop: an attacker writes adversarial queries that make the learner fail, a critic explains why the learner's responses are good or bad in natural language, and the learner updates itself from those textual critiques, with human experts occasionally intervening. The paper argues that textual critiques are especially promising because they do not require training a separate reward model, and it identifies weak-to-strong generalization, scalable oversight, and automated evaluation as the load-bearing capabilities that must hold for the loop to work.","core_discovery":"The central claim is that superalignment reduces to a scalable learning loop rather than a static alignment procedure. The paper proposes that a learner model can be continuously improved by an attacker that discovers its weaknesses and a critic that generates textual critiques of the learner's responses; the learner trains on these critiques together with minimal human feedback, and the process repeats automatically. The paper further claims that weak-to-strong generalization, scalable oversight, and automated evaluation are the key research problems underlying this loop, and that existing self-improvement methods can be seen as early attempts toward superalignment that show positive results. This is a conceptual framework, not an experimental demonstration, and the paper leaves open the detailed implementation of each module.","pith_inferences":["Editorial inference: the framework implies that the critic's faithfulness, not the learner's capacity, is the main bottleneck, so improving critique quality should directly improve alignment outcomes.","Editorial inference: the loop's long-run stability is not established; the model-collapse results the paper cites indicate that iterative training on generated data can degrade diversity and out-of-distribution performance unless fresh data or oversight enters.","Editorial inference: a direct test would be to run the loop on olympiad-level mathematics or competitive programming, where human labels are unreliable, and measure whether the learner's performance increases across rounds."],"forward_implications":["If the loop works, superhuman models can be aligned through iterative self-improvement rather than through human annotation of every difficult task.","Weak-to-strong generalization would let a weaker supervisor elicit and strengthen capabilities of a stronger model, reversing the usual teacher-student direction.","Textual critiques could replace or augment reward models, reducing the need for explicit reward-model training in alignment.","The loop's adversarial attack module would make evaluation dynamic and ongoing, catching weaknesses that static benchmarks miss.","The existence of self-alignment, self-play, and self-refinement results suggests the pipeline is at least partially realizable today."],"supporting_citations":[{"why":"Supplies the weak-to-strong generalization result that a strong model can be supervised by a weak model and recover much of its capability.","marker":"[20]"},{"why":"Provides evidence that a critic model can help humans spot flaws in responses, supporting scalable oversight.","marker":"[29]"},{"why":"Introduces a critic model that generates informative critiques for diverse generation tasks, grounding the critic module.","marker":"[34]"},{"why":"Shows an automated pipeline can detect weaknesses in LLMs, grounding the attacker module.","marker":"[37]"},{"why":"Presents a self-refinement framework with actor and refiner that improves through iterative training, grounding the learner-critic loop.","marker":"[40]"},{"why":"Shows bootstrapping from correct rationales can iteratively improve reasoning, an early self-improvement loop.","marker":"[55]"},{"why":"Demonstrates self-play fine-tuning can convert a weaker LLM to a stronger one, supporting the framework's feasibility.","marker":"[56]"},{"why":"Shows a single model can generate, critique, and refine its own output, a minimal instance of the critic-learner loop.","marker":"[57]"},{"why":"Formalizes the generation-verification gap driving self-refinement, the theoretical underpinning for the loop's improvement mechanism.","marker":"[65]"}],"fun_headline_variants":["Superalignment as a scalable learning loop","Attacker-learner-critic loop for superhuman AI","Scalable oversight with a critic feedback loop","Weak-to-strong generalization: superalignment key","A conceptual loop for aligning superhuman intelligence"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes the critic can produce feedback that is genuinely faithful and useful for the learner, even though the critic is fallible and may be weaker than the learner; if the critiques are noisy or unlearnable, the loop cannot keep the model aligned.","fun_headline_variants_meta":{"raw":{"variants":["Superalignment as a scalable learning loop","Attacker-learner-critic loop for superhuman AI","Scalable oversight with a critic feedback loop","Weak-to-strong generalization: superalignment key","A conceptual loop for aligning superhuman intelligence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000259,"raw_usage":{"total_tokens":1600,"prompt_tokens":973,"completion_tokens":627,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":557}},"tokens_in":589,"tokens_out":627,"duration_ms":5837,"temperature":1.0,"reasoning_tokens":557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:15:36.401536+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled experiment in which a strong learner is trained exclusively on critiques generated by a weaker critic on tasks too hard for humans would falsify the framework if the learner fails to improve or degrades over multiple rounds.","supporting_citations":[{"cited_title":"Unveiling the implicit toxicity in large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the weak-to-strong generalization result that a strong model can be supervised by a weak model and recover much of its capability."},{"cited_title":"Lm vs lm: Detecting factual errors via cross examination","cited_arxiv_id":null,"evidence_quote":"Provides evidence that a critic model can help humans spot flaws in responses, supporting scalable oversight."}],"review_version":1}