{"id":"5301fddd-bdf5-45a4-8bfd-55e6f91f0ef5","arxiv_id":"2411.11504","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey and position paper that reframes post-training of foundation models as a search, verify, and feedback pipeline called verifier engineering.","lead":"This paper proposes a new organizing framework, verifier engineering, for improving foundation models after training: search for candidate responses, verify them with automated checkers, and feed the results back into the model. It is a survey that maps many existing methods, such as RLHF and chain-of-thought, onto this three-stage loop.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Verify stage's reliability is load-bearing for the AGI claim, yet the paper itself admits weak correlations between verifier metrics and downstream performance; the PAC-based assertion in §2.1 is not a proof.","rationale":"The reader is right to flag §2.1's PAC statement as unsupported, but the sharpest problem is more general: the paper's own §6.2.2 admits that verifier metrics correlate only weakly with downstream task performance. That is an internal admission that the Verify stage—the core of the paradigm—may not yield reliable supervision. Because no new experiments or formal proofs are supplied, the condition needed for the AGI claim is exactly the condition left unexamined. I do not see a fatal internal inconsistency; the taxonomy and survey are useful, and the paper is best read as a position piece. The reader's CONDITIONAL verdict is therefore the right one, with the condition that the central claim be moderated to 'a promising direction' and the PAC-based reliability assertion either proved or replaced by explicit open questions. Hence verdict_should_be = UNCHANGED.","tokens_in":25415,"tokens_out":4178,"duration_ms":45787,"concrete_test":"Take a fixed set of verifiers (one ORM, one PRM, one LLM-as-a-judge) on an open-ended generation task without a programmatic ground-truth checker. Measure (a) each verifier's agreement with human labels and (b) the correlation between verifier scores and final downstream task performance after training with the same feedback algorithm. If high verifier accuracy does not correspond to improved downstream performance, the Verify stage's load-bearing assumption fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that verifier engineering is a fundamental pathway to general capability—requires that verification signals reliably improve the policy in the Search-Verify-Feedback loop. This is precisely the least secure point. Section 2.1 asserts, with no derivation, that 'Following Probably Approximately Correct Theory (PAC), even with imperfect sub-functions, we can achieve reliable overall evaluation by combining multiple weak verifiers' (Eq. 2). PAC guarantees bounds on in-family generalization under specific i.i.d. conditions; they do not imply that arbitrary ensembles of learned reward models and rule checkers produce a faithful reward function. More importantly, the paper's own discussion (§6.2.2) acknowledges that 'recent studies (Wen et al., 2024c) have revealed only weak correlations between existing verifier evaluation metrics and downstream task performance.' If verifier quality does not predict downstream improvement, the Verify stage cannot be assumed to supply the supervision needed for the feedback stage, and the AGI claim loses its main support. The paper provides no theoretical bridge from verifier accuracy to policy improvement, and no experimental demonstration that the proposed loop closes in a way that outperforms standard RLHF.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes \"verifier engineering\" as a post-training paradigm for foundation models, in which a suite of automated verifiers replaces or supplements human annotation as the main source of supervision. The paradigm is formalized as a goal-conditioned MDP whose reward function is decomposed into sub-functions, and the authors organize the field into three stages—Search, Verify, and Feedback—with a large table (Table 3) mapping many existing methods to these stages. The paper also revisits SFT, DPO, and RLHF from this perspective and concludes that verifier engineering is a fundamental pathway toward AGI.","tokens_in":25683,"tokens_out":2817,"duration_ms":29412,"significance":"If the framework is accepted, it offers a useful unifying vocabulary for post-training research and highlights a concrete design space (verifier construction, combination, and feedback) beyond the current reward-model-centric view of RLHF. The survey is broad, with extensive coverage of recent work, and the GC-MDP formalization provides a common notation for comparing methods. The paper ships a GitHub repository collecting the categorized references, which is a concrete resource. However, the central theoretical justification for the Verify stage rests on an unproven PAC-based claim, and the paper itself acknowledges evidence that verifier quality does not reliably predict downstream gains. These gaps directly affect the strength of the paradigm claim, making the paper more a position/taxonomy contribution than a demonstrated technical result.","major_comments":[{"comment":"The sentence \"Following Probably Approximately Correct Theory (PAC), even with imperfect sub-functions, we can achieve reliable overall evaluation by combining multiple weak verifiers\" is stated without derivation or a specific theorem reference. PAC theory gives generalization bounds for a learned hypothesis class under i.i.d. assumptions; it does not, by itself, imply that an arbitrary combination of learned reward models, rule checkers, and tools yields a reliable reward function R_g. This claim is load-bearing for the Verify stage and for the AGI claim. Please either provide a concrete theorem with conditions (e.g., independence assumptions, error-rate bounds, or a calibrated aggregation rule) under which F is reliable, or explicitly rephrase this as a heuristic design principle rather than a theoretical guarantee.","section":"§2.1, Eq. (2)"},{"comment":"The paper acknowledges that \"recent studies (Wen et al., 2024c) have revealed only weak correlations between existing verifier evaluation metrics and downstream task performance.\" This admission directly undercuts the assumption that the Verify stage supplies supervision that reliably improves the policy in the Search-Verify-Feedback loop. The paper does not provide a theoretical bridge from verifier accuracy to policy improvement, nor experimental evidence that the loop outperforms standard RLHF. To make the central claim defensible, the authors should either present such evidence or explicitly scope the claim to settings where verifier-policy alignment has been established, and discuss concrete failure modes such as reward hacking and over-optimization (which are only mentioned in passing in §6.2.3).","section":"§6.2.2"},{"comment":"The claim that verifier engineering \"constitutes a fundamental pathway toward achieving Artificial General Intelligence\" goes beyond what the survey supports. The paper provides a taxonomy and qualitative examples, but no argument that verifier engineering is necessary or sufficient for general intelligence, nor a definition of AGI against which the claim could be evaluated. This overclaim weakens the paper's credibility. I suggest reframing the statement as \"a promising and scalable paradigm for post-training foundation models\" and, if the AGI connection is to be retained, supporting it with a substantive argument about the scalability of supervision and its relation to general capability.","section":"Abstract and §7"}],"minor_comments":[{"comment":"The sentence \"The remarkable success of landmark projects such as ImageNet (Deng et al., 2009) and BERT (Brown et al., 2020a)\" misattributes BERT; Brown et al. 2020 is the GPT-3 paper. The correct citation for BERT is Devlin et al., 2018.","section":"§1, paragraph 4"},{"comment":"There are several typographical inconsistencies: \"STar\" should be \"STaR\", \"Constrative decoding\" should be \"Contrastive decoding\", and the use of both \"Self-Debug\" and \"Self-debug\" across rows should be unified.","section":"Table 3"},{"comment":"The caption says gray nodes represent sample paths not used in training, but it is not immediately clear from the figure which nodes are gray in each of (a), (b), and (c); please increase the contrast or label the paths explicitly.","section":"Figure 3 caption"},{"comment":"The text states that \"in large language models, the state transition is a deterministic function\" but then describes search as \"exploration in the action-state space.\" This is not contradictory, but it would be clearer to note that determinism is with respect to the transition given a chosen action, while exploration arises from the policy's stochastic action selection.","section":"§2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a position/survey contribution rather than a technical result, which is fine, but the PAC-based claim in §2.1 and the AGI framing in the abstract are the kind of overclaim that will draw justified criticism from reviewers. The heavy self-citation is noticeable but not inappropriate given the authors' own program of work on verifiers. If the journal is willing to accept a taxonomy paper, the revision path is clear; if the bar is for demonstrated technical novelty, the current version may be below it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a survey and position paper, not a new result. It organizes a large chunk of the post-training literature into a search–verify–feedback loop and calls the whole enterprise \"verifier engineering.\" That framing is genuinely handy. Table 3 alone, which maps dozens of known methods onto the three stages, is worth having. The discussion in §6.2.2 about open questions in verifier design is also honest—it explicitly admits that existing verifier evaluation metrics correlate only weakly with downstream performance. That kind of self-criticism is rare and should be credited.\n\nThe soft spots are real but not disqualifying for what the paper actually is. Section 2.1 asserts, with no derivation, that PAC theory lets you combine multiple weak verifiers into a reliable overall evaluation. That is a load-bearing claim for the Verify stage, and PAC guarantees do not transfer to arbitrary ensembles of learned reward models and rule checkers. The paper does not prove it, and the weak-correlation admission in §6.2.2 undercuts it further. So the AGI sentence in the abstract—that verifier engineering is a fundamental pathway to AGI—is rhetoric, not a conclusion supported by the text. The GC-MDP formalization is a standard restatement and does not add much, but it does give the survey a unified notation.\n\nSome cited works are from the authors' own group. That is mildly noticeable but not a problem: the works are directly relevant, and surveys naturally cite the authors' prior surveys and papers.\n\nWho should read this: anyone working on post-training, reward models, RLHF, or test-time compute who wants a map of the territory. It will not change what you build, but it may help you place your work relative to the field. The paper deserves a serious referee: it is well-organized, covers the literature broadly, and raises the right open questions. My recommendation is to send it to peer review, but with a clear request: tone down the AGI claim, and either prove the PAC-based verifier-combination assertion or remove it. As it stands, conditional accept as a survey.","headline":"A useful survey taxonomy that reframes post-training as search, verify, and feedback, but the AGI claim leans on an unproven PAC assertion and the paper's own admission of weak verifier-performance correlation.","tokens_in":26179,"tokens_out":1305,"would_cite":true,"duration_ms":14457,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that post-training foundation models should be rebuilt as a search–verify–feedback loop driven by automated verifiers rather than primarily by human annotations.","keywords":["verifier engineering","post-training","foundation models","large language models","search-verify-feedback loop","goal-conditioned Markov decision process","process reward models","reinforcement learning from human feedback"],"falsifier":"A concrete check: assemble a suite of independent weak verifiers, adversarially search for responses that the majority rates correct but human experts judge wrong, and test whether such responses are systematically accepted by the combined verifier; if they are, the reliability assumption fails.","tokens_in":25256,"feed_emoji":"✅","tokens_out":5849,"duration_ms":54270,"temperature":0.7,"pith_summary":"This paper proposes verifier engineering as the next post-training paradigm for foundation models, after feature engineering and data engineering. The central idea is to replace most human annotation with a suite of automated verifiers—rule checkers, code interpreters, reward models, and judges—that evaluate candidate outputs and feed the results back to the model. The paper organizes the field into three stages: search, which samples candidate responses; verify, which scores or checks them; and feedback, which updates the model through training or inference-time guidance. It formalizes the loop as a goal-conditioned Markov decision process in which search, verify, and feedback correspond to action selection, reward evaluation, and policy optimization. The authors argue that this closed loop is the route to general intelligence.","feed_headline":"Search, verify, feed back: the next AI post-training loop","feed_subtitle":"Verify-first loops could cut human annotation and unify RLHF, DPO, tree search, and self-correction under one engine.","key_machinery":"The load-bearing object is the goal-conditioned Markov decision process tuple $(S, A, T, G, R_g, p_g)$, with the reward decomposable as $R_g(s,a)=F(R_{g,i}(s,a)\\mid i \\in S_g)$, where each $R_{g,i}$ is a verifier for one capability dimension and $F$ combines them. Search corresponds to action selection in this MDP, verify to evaluation by the reward and verifier functions, and feedback to policy optimization. The paper's taxonomy of verifiers—by verification form (binary, score, rank, text), by granularity (token, thought, trajectory), by source (program-based or model-based), and by the need for extra training—delineates the design space from which verifier combinations are drawn. The combination step is justified by the paper's appeal to PAC theory, stated in Section 2.1, that ensembles of weak verifiers yield reliable overall evaluation.","core_discovery":"On the paper's own terms, the central claim is that the next scaling step for foundation models is not more data or more parameters but a new source of supervision: automated verifiers. The paper names this paradigm verifier engineering and defines it as a closed loop of search, verify, and feedback: search generates diverse candidate responses, verify scores or checks them with a combination of verifiers, and feedback turns those results into either parameter updates (supervised fine-tuning, preference learning, reinforcement learning) or inference-time guidance (verifier-guided search, self-correction). It formalizes the loop as a goal-conditioned Markov decision process in which the reward function is decomposed over goal-specific verifier sub-functions, and it grounds the claim that even imperfect sub-functions can be combined into a reliable overall evaluation by appealing to PAC theory. The paper further claims that existing methods—RLHF, DPO, SFT, tree-of-thought-style search, and self-refinement—are all instances of this loop, and that verifier engineering is a fundamental pathway toward general intelligence.","pith_inferences":["Editorial: a direct empirical test of the paradigm would be to build a full search–verify–feedback loop on a single capability, such as mathematical word problems, using only open-source verifiers, and compare its closed-loop gain against the same budget spent on human-annotated supervised data.","Editorial: the framework implies a systems-level research agenda—automatic routing of instructions to verifier combinations and conflict resolution among their verdicts—that the paper describes but does not implement.","Editorial: if verifier ensembles are reliable only when their errors are decorrelated, the paradigm inherits an implicit diversity requirement; measuring error correlation across verifiers would sharpen the design space."],"forward_implications":["Human annotation can shrink from dense labeling to sparse verification, because a suite of automated verifiers supplies the supervision for both training and inference-time updates.","Methods as different as SFT, DPO, RLHF, step-wise verifiers, and tree search become instances of a single search–verify–feedback loop, so improvements in one stage could transfer across all of them.","The binding constraint on further model improvement shifts from data collection to verifier design, routing, and combination, making systematic verifier evaluation a critical benchmark problem.","Closed-loop improvement becomes possible without human labels: models can generate candidates, verify them with code interpreters or rule checkers, and update or refine from those signals."],"supporting_citations":[{"why":"Supplies the canonical RLHF pipeline that the paper reinterprets as a linear search, reward-model verifier, and PPO feedback loop.","marker":"Ouyang et al., 2022a"},{"why":"Establishes process reward models and step-by-step verification, supporting the thought-level verify granularity and verifier-guided feedback.","marker":"Lightman et al., 2023a"},{"why":"Introduces outcome reward models and the practice of training verifiers for math word problems, grounding trajectory-level verification.","marker":"Cobbe et al., 2021"},{"why":"Supplies the PAC theory that the paper cites to justify combining multiple weak verifiers into a reliable overall evaluation.","marker":"Vapnik, 2000"},{"why":"Provides the goal-conditioned value-function formulation that the paper adapts into its GC-MDP formalization of verifier engineering.","marker":"Schaul et al., 2015"},{"why":"Provides the experiential co-learning example used to illustrate collaborative verifiers and verifier-aware feedback.","marker":"Qian et al., 2023"},{"why":"Shows rule-based verifiers selecting correct solutions for bootstrapped training, exemplifying verifier-guided imitation learning.","marker":"Zelikman et al., 2022b"},{"why":"Reports weak correlations between existing verifier evaluation metrics and downstream performance, motivating the paper's call for systematic verifier evaluation.","marker":"Wen et al., 2024c"}],"fun_headline_variants":["Verifier engineering: AI post-training's new supervision engine","Search, verify, feedback: the loop that could replace RLHF","Automated verifiers as scalable supervision for foundation models","One loop to train them all: search, verify, feedback in AI","The verifier loop: how AI can supervise itself post-training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that combining several imperfect verifiers—learned reward models, rule checkers, and code interpreters—yields a reliable evaluation signal, as the paper asserts by appealing to PAC theory.","fun_headline_variants_meta":{"raw":{"variants":["Verifier engineering: AI post-training's new supervision engine","Search, verify, feedback: the loop that could replace RLHF","Automated verifiers as scalable supervision for foundation models","One loop to train them all: search, verify, feedback in AI","The verifier loop: how AI can supervise itself post-training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1604,"prompt_tokens":899,"completion_tokens":705,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":619}},"tokens_in":515,"tokens_out":705,"duration_ms":7454,"temperature":1.0,"reasoning_tokens":619,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:25:23.513425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: assemble a suite of independent weak verifiers, adversarially search for responses that the majority rates correct but human experts judge wrong, and test whether such responses are systematically accepted by the combined verifier; if they are, the reliability assumption fails.","supporting_citations":[],"review_version":1}