{"id":"d962e4de-8671-415f-be45-a941a27cbacb","arxiv_id":"2501.12895","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TPO iteratively refines LLM responses at inference time using reward-model scores converted into textual critiques, letting an unaligned 70B model beat its aligned counterpart on several benchmarks.","lead":"This paper introduces Test-time Preference Optimization (TPO), a method that uses a reward model's feedback to iteratively revise an LLM's response during inference, without changing the model weights. Why a generalist should read: it offers a lightweight, on-the-fly alternative to retraining models for alignment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 'surpasses Instruct' is not yet established: on the only length-controlled metric reported (AlpacaEval LC), TPO is below Instruct, while the metrics where it wins (AlpacaEval raw WR, Arena-Hard) are length-sensitive raw win rates.","rationale":"The reader's verdict is CONDITIONAL, and my concern does not overturn that; it sharpens the condition. The reader's weakest_assumption is about the policy model's self-critique capability, which is a mechanism-level risk. My concern is about the validity of the benchmark evidence for the central claim. The two are related but distinct: even if the policy model perfectly interprets and acts on textual feedback, the headline 'surpasses Instruct' could still be an artifact of length bias in the chosen metrics. Table 1 provides internal evidence for this: on AlpacaEval, TPO beats Instruct on raw WR but loses on LC, and the case studies show TPO responses become visibly longer after revision. Arena-Hard has no length-controlled counterpart reported, so the largest headline gain (70.5 vs 59.0) is exactly the kind of raw win-rate that LC was designed to correct. This is a concrete, testable threat to the abstract's strongest claim. I recommend keeping the CONDITIONAL verdict but adding an explicit condition: report a length-controlled Arena-Hard evaluation (or an equivalent matched-length comparison). If the advantage survives, the central claim is substantially strengthened; if it vanishes, the paper's headline overstates what TPO achieves. I agree with the reader's broader evaluation concerns about circular reward-model metrics and missing compute-matched baselines, but those are not the single most load-bearing issue for this particular claim.","tokens_in":36150,"tokens_out":7073,"duration_ms":78539,"concrete_test":"Evaluate TPO-D2-N5 and Llama-3.1-70B-Instruct on Arena-Hard using a length-controlled protocol, e.g., re-use AlpacaEval's length-control regression on Arena-Hard judgments, or compare responses truncated/padded to matched token counts. If the length-controlled win rate of TPO over Instruct drops below 50% (or below the raw 70.5 advantage), the headline claim is a length artifact. Also report the same LC result for AlpacaEval to confirm the pattern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.2 and Table 1 support the abstract's strongest claim with AlpacaEval raw WR (39.5 vs 34.9) and Arena-Hard WR (70.5 vs 59.0). The same table contains a red flag: the length-controlled AlpacaEval LC for TPO-D2-N5 is 33.2, below Llama-3.1-70B-Instruct's 36.9. Thus the raw-WR advantage reverses when length is controlled. Arena-Hard is reported only as raw WR, with no length-controlled variant, and the paper's own case studies show optimized responses are systematically longer (added examples, caveats, resources, and 'more detailed' instructions). If Arena-Hard's 11.5-point advantage is similarly length-driven, the central claim reduces to a length effect rather than preference alignment. Because the abstract states the SFT model 'surpasses' Instruct without the LC caveat, the load-bearing evidence for that claim is exactly the metric most vulnerable to length bias. The missing control is not a minor robustness check: it determines whether TPO outperforms a well-aligned model or merely produces longer answers that LLM judges prefer.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"TPO is a test-time alignment method that, for each prompt, samples N responses, scores them with an external reward model, and then iteratively uses the highest- and lowest-scoring responses to generate textual critiques and improvement suggestions (\"textual loss\" and \"textual gradient\"), producing new candidates that are scored and cached. After up to D iterations, the highest-scoring cached response is returned. The paper evaluates TPO on Llama-3.1-70B-SFT, Llama-3.1-70B-Instruct, and Mistral-Small-Instruct-2409 across AlpacaEval 2, Arena-Hard, HH-RLHF, BeaverTails, XSTest, and MATH-500, reporting that TPO improves all models, that unaligned Llama-3.1-70B-SFT with TPO surpasses Llama-3.1-70B-Instruct on most metrics, and that TPO scales with width and depth and costs far less than training-time preference optimization. The code is publicly available.","tokens_in":36539,"tokens_out":6770,"duration_ms":68022,"significance":"If the empirical claims were fully established, TPO would be a valuable and practical test-time alignment alternative to training-time methods: it is parameter-free in the sense of not updating weights, is conceptually clean, and the paper honestly analyzes the prerequisite of instruction-following ability (Section 7.4) and includes a useful width/depth scaling study (Section 7.2). The release of code is a concrete community asset. However, the central headline claims—\"surpasses the aligned counterpart\" and the computational-efficiency advantage—are not yet conclusively supported because of length-controlled metric reversals, circular evaluation with the same reward model used for optimization, the absence of a compute-matched Best-of-N baseline on the main benchmarks, and a per-query versus one-time-training compute comparison that overstates the efficiency gain. The core idea is sound and the independent AlpacaEval/Arena-Hard results are promising, but the evidence currently falls short of the paper's strongest claims.","major_comments":[{"comment":"The abstract and introduction state that Llama-3.1-70B-SFT with a few TPO steps can \"surpass\" Llama-3.1-70B-Instruct, but the length-controlled result in Table 1 contradicts this for the default setting: TPO-D2-N5 (†) obtains AlpacaEval 2 LC of 33.2, below Llama-3.1-70B-Instruct's 36.9, while it wins only on raw WR (39.5 vs. 34.9) and on Arena-Hard raw WR (70.5 vs. 59.0). The Appendix D case studies show that TPO systematically produces longer, more detailed responses (adding examples, caveats, references, and extra advice). Raw win rates in AlpacaEval 2 and Arena-Hard are length-sensitive, so the reported advantage may reflect a length effect rather than better preference alignment. To support the \"surpasses Instruct\" claim, please report a length-controlled metric on Arena-Hard (or an equivalent bias-controlled evaluation) and state the LC result clearly in the abstract and introduction; otherwise the claim should be tempered.","section":"§6.2, Table 1; Abstract"},{"comment":"HH-RLHF and BeaverTails are evaluated using FsfairX-LLaMA3-RM-v0.1 (Section 5, \"Evaluation Benchmarks\"), which is the same reward model used by TPO for scoring, choosing chosen/rejected responses, and selecting the final response (Section 4.2). The reported gains on these two benchmarks thus largely measure how well TPO optimizes that particular reward model, not how well the outputs align with human preferences. This circularity makes the \"preference alignment\" claims on HH-RLHF and BeaverTails unsupported by the current evidence. Please either (i) evaluate with a held-out reward model or human judgments on these benchmarks, or (ii) explicitly re-label the columns as \"RM score\" and avoid presenting them as independent alignment evidence.","section":"§5, §6.2, Tables 1–2"},{"comment":"The paper claims TPO's search depth is more effective than pure Best-of-N, but no compute-matched BoN baseline is provided on the main benchmarks. Table 1 reports no BoN baseline at all, and Figure 6 only compares TPO-D2-N5 (15 total samples) against BoN-30 and BoN-60 on 100 randomly selected instances per dataset, using GPT-4 win-rate rather than the official benchmark scores. The relevant control is BoN-15 (or BoN with equivalent total samples/compute) evaluated on full AlpacaEval 2 and Arena-Hard, because BoN-15 already selects the single best of 15 initial samples by the same reward model. Without this control, the headline benchmark gains in Table 1 could simply reflect the reward-model-selected BoN component rather than the iterative textual refinement. Please add a BoN baseline with matched total samples to Tables 1 and 2 and to the width/depth analysis.","section":"§7.2, Figure 6; Table 1"},{"comment":"The computational-efficiency claim \"consuming only a neglectable fraction of the FLOPs\" is based on comparing the total training FLOPs of Llama-3.1-70B-DPO (72,840 PFLOPs) with the per-query FLOPs of TPO (9.3 PFLOPs/query). This is an apples-to-oranges comparison: training cost is one-time and amortized over the model's entire serving lifetime, while TPO's cost is paid for every query at test time. For a service that processes many queries, the cumulative TPO compute can exceed the training compute. To make the efficiency claim valid, please report (i) amortized per-query training cost (total training FLOPs divided by the number of queries served) or (ii) a clear statement that TPO is per-query and that the comparison is illustrative only for a single query. Without this, the \"lightweight\" positioning is overstated.","section":"§7.3, first paragraph"}],"minor_comments":[{"comment":"The abstract's \"after only a few TPO steps\" is vague: the default benchmark configuration uses D=2 and N=5, but the \"surpasses on all metrics\" result in Table 1 is from the D5-N20 ultra setting, which uses 20 samples and 5 iterations per query. Please clarify which configuration supports each claim.","section":"Abstract, §6.2"},{"comment":"Figure 6 uses only 100 instances per benchmark for the BoN comparison; this is a small sample for win-rate estimation. Please report confidence intervals or use the full benchmark sets to make the comparison more robust.","section":"§7.2, Figure 6"},{"comment":"The citation to \"Zhang et al.\" appears without a year and without a numbered reference callout in the text (the reference entry is present in the bibliography). Please add the proper citation and year.","section":"§2, §7.2"},{"comment":"Minor typographical issues include \"neglectable\" (should be \"negligible\") in §6.1 and §7.3, and the grammatically awkward sentence \"we report the accuracy score whether WildGuard classified the response as a refusal or compliance\" in §5.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim in the abstract is stronger than what the evidence supports: the length-controlled metric reverses, the HH-RLHF/BeaverTails evaluation is circular with respect to the optimization reward model, and no compute-matched BoN baseline is given on the main benchmarks. These are fixable with additional experiments and a revised framing. The method itself is plausible and the independent AlpacaEval/Arena-Hard improvements are encouraging, so I see this as a major-revision case rather than a rejection, provided the authors add the missing controls and adjust the claims accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: TPO is a clean, practical idea—use a reward model's scores to pick chosen/rejected responses, translate the contrast into a textual critique in the style of TextGrad, then iteratively refine and cache. The paper shows real gains on AlpacaEval 2 raw WR and Arena-Hard, and the 8B-model ablation is an honest demonstration of the prerequisite instruction-following assumption. The code is public. Credit where due: the chosen/rejected contrast in the textual loss is a new combination, the width-vs-depth scaling analysis is useful, and the body explicitly acknowledges the AlpacaEval LC exception.\n\nThe soft spots are real and not minor. First, HH-RLHF and BeaverTails scores are computed with the same reward model TPO optimizes against, and the final output is selected as the max-reward cache entry. Those columns mostly show successful reward-model optimization, not human alignment. Second, the headline 'surpasses Instruct' rests on length-sensitive metrics: raw AlpacaEval WR and Arena-Hard WR. The length-controlled LC is below Instruct (33.2 vs 36.9), and the case studies confirm TPO outputs get systematically longer—more examples, caveats, references. If Arena-Hard's 11.5-point advantage is similarly length-driven, the claim weakens to a length effect. Third, the BoN-30/60 comparison is on 100 instances with a GPT-4 judge, and it is not compute-matched; TPO spends extra model calls on critique and update.\n\nNone of this kills the idea. The method is sound, the independent-benchmark gains are non-trivial, and the paper is transparent about the main caveat. But the abstract overstates the result, and the evaluation needs a held-out reward model or human judgments for HH-RLHF/BeaverTails, plus a length-controlled analysis of Arena-Hard and a compute-matched BoN baseline.\n\nI'd send this to a serious referee. It deserves reading, and with revisions the claims can be made solid. For a reading group, it is a good case study in evaluation pitfalls. I would not cite it as evidence for the 'surpass' claim, but might cite it as related work on test-time alignment.","headline":"TPO is a sound test-time alignment method with real gains on independent benchmarks, but the 'surpasses Instruct' headline rests on length-sensitive metrics and is undercut by the length-controlled LC.","tokens_in":36939,"tokens_out":2476,"would_cite":true,"duration_ms":27843,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Test-Time Preference Optimization aligns LLM responses during inference by converting reward-model scores into textual critiques and iteratively refining the answer, with no parameter updates.","keywords":["test-time preference optimization","textual feedback","reward model","iterative refinement","inference-time alignment","LLM alignment without retraining","test-time compute scaling"],"falsifier":"Run TPO with reward scores randomly shuffled before choosing the chosen and rejected responses: if outputs still improve on preference benchmarks, the gains come from generic self-revision rather than reward-guided alignment, while a sharp drop shows the reward signal is load-bearing. A complementary check is to compare TPO outputs against human preference judgments instead of reward-model scores on a held-out set.","tokens_in":35928,"feed_emoji":"🎯","tokens_out":9984,"duration_ms":89801,"temperature":0.7,"pith_summary":"The paper proposes Test-Time Preference Optimization (TPO), a method that aligns a language model's outputs with human preferences during inference, leaving the model weights untouched. TPO samples several responses, uses a reward model to pick the best and worst, asks the model to write a textual critique of that pair, and then generates revised responses guided by that critique; repeating this loop progressively raises reward-model scores. The central empirical claim is that after only a few such steps, the unaligned Llama-3.1-70B-SFT model surpasses the RLHF-aligned Llama-3.1-70B-Instruct on Arena-Hard and several other benchmarks. This matters because if it holds, alignment becomes an on-the-fly, interpretable process that can adapt to new preferences or reward signals without retraining, at a fraction of the compute of DPO or RLHF.","feed_headline":"After a few test-time edits, an unaligned 70B beats its aligned twin","feed_subtitle":"TPO's text critiques lift Llama-3.1-70B-SFT past Llama-3.1-70B-Instruct on Arena-Hard, with no weight updates.","key_machinery":"The load-bearing object is the textual gradient loop, an analogy to gradient descent executed in natural language. The response $v$ is the variable to optimize; a loss prompt $P_{\\mathrm{loss}}$ produces a textual loss $L(x,v)$ by comparing the chosen response $v$ with the rejected response $\\hat{v}$; a gradient prompt $P_{\\mathrm{grad}}$ converts that loss into improvement instructions $\\partial L/\\partial v$; and an update prompt $P_{\\mathrm{update}}$ samples a fresh set of responses under those instructions. A reward model $R$ supplies the scores that select chosen and rejected responses and that terminate the loop when the best cached response is returned. The same textual machinery is used for both the loss and gradient steps, so the policy model's own instruction-following ability is the engine of the optimization.","core_discovery":"On the paper's own terms, TPO shows that preference alignment can be reframed as an optimization of the response variable rather than of the model parameters. The method maintains a cache of sampled responses paired with reward-model scores; each iteration picks the highest- and lowest-scoring responses as chosen and rejected, prompts the policy model to produce a textual loss explaining why the chosen response is better, converts that loss into a textual gradient of improvement suggestions, and samples new candidate responses under the revised context. This loop re-allocates probability mass toward high-reward outputs while keeping the policy fixed. With a maximum of two iterations and five samples per iteration, TPO lifts Llama-3.1-70B-SFT's Arena-Hard win rate from 44.1 to 70.5 against a 59.0 for Llama-3.1-70B-Instruct, and its AlpacaEval 2 win rate from 16.8 to 39.5 against 34.9, with the improvement requiring roughly 9.3 PFLOPs per query, less than 0.01% of the estimated training cost of the DPO baseline.","pith_inferences":["Editorial extension: TPO's textual gradient does not need to be a language-model critique; the same loop could consume user edits, code-execution errors, or tool feedback, turning TPO into a general test-time optimizer for interactive agents.","Editorial extension: because the loop maximizes a reward-model score, longer runs may over-optimize the proxy rather than human preference, so the paper's choice of few iterations likely acts as an implicit regularizer; a direct study of reward over-optimization would test how far depth can be pushed.","Editorial extension: the headline comparison relies on the reward model as a proxy for humans, so the practical claim is bounded by that proxy; using a stronger or more diverse reward ensemble could change which responses are chosen and rejected and thus the final output."],"forward_implications":["A model that has never seen RLHF or DPO training can be brought to or past the alignment of an Instruct model within two TPO iterations on several benchmarks.","Already-aligned models also improve under TPO, with Mistral-Small-Instruct-2409 (22B) reaching an AlpacaEval 2 LC of 53.4%, a level associated with much larger systems.","TPO's test-time compute scales in both width (samples per iteration) and depth (iterations), and two revision rounds outperform Best-of-N sampling that draws 30 or 60 fresh samples.","Because all updates are textual and parameter-free, TPO can be applied to a different reward model or preference signal without any retraining, just by swapping the reward model.","The approach degrades on weaker models such as Llama-3.1-8B-Instruct, indicating that instruction-following competence is a prerequisite rather than a bonus."],"supporting_citations":[{"why":"This reference supplies the text-gradient and update prompts that TPO adapts for its textual loss calculation.","marker":"Yuksekgonul et al., 2024"},{"why":"This reference provides the unaligned Llama-3.1-Tulu-3-70B-SFT model and the Tulu-3-8B reward model used for the headline result.","marker":"Lambert et al., 2024"},{"why":"This reference provides the Llama-3.1 base and Instruct models that serve as policy model and aligned baseline.","marker":"Dubey et al., 2024"},{"why":"This reference defines DPO, the training-time preference optimization method TPO is compared against.","marker":"Rafailov et al., 2024"},{"why":"This reference is the source of the FsfairX-LLaMA3-RM-v0.1 reward model used for scoring all policy models.","marker":"Dong et al., 2023"},{"why":"This reference defines AlpacaEval 2 and its length-controlled win rate, one of the paper's main evaluation metrics.","marker":"Dubois et al., 2024"},{"why":"This reference provides the Arena-Hard benchmark and its GPT-4-based win-rate evaluation.","marker":"Li et al., 2024b"},{"why":"This reference frames test-time compute scaling in terms of width and depth, which TPO's scaling analysis adopts.","marker":"Snell et al., 2024"},{"why":"This reference defines Best-of-N sampling and the MATH-500 benchmark used for comparison.","marker":"Lightman et al., 2024"}],"fun_headline_variants":["No weights updated: iterative text feedback aligns LLMs on the fly","Unaligned 70B surpasses aligned twin after few test-time edits","TPO: text-based rewards align LLMs at inference, no parameter updates","Test-time optimization: unaligned 70B beats aligned twin with text feedback","TPO: iterative textual feedback aligns LLMs without retraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the policy model can turn a reward-model critique into genuinely better text, and that the reward model's scores actually track what humans prefer; the paper itself shows the first half fails for a weaker 8B model, and if the second half fails the textual gradient would amplify the wrong signal.","fun_headline_variants_meta":{"raw":{"variants":["No weights updated: iterative text feedback aligns LLMs on the fly","Unaligned 70B surpasses aligned twin after few test-time edits","TPO: text-based rewards align LLMs at inference, no parameter updates","Test-time optimization: unaligned 70B beats aligned twin with text feedback","TPO: iterative textual feedback aligns LLMs without retraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001531,"raw_usage":{"total_tokens":6166,"prompt_tokens":1018,"completion_tokens":5148,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":5051}},"tokens_in":634,"tokens_out":5148,"duration_ms":30096,"temperature":1.0,"reasoning_tokens":5051,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:39:11.705221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TPO with reward scores randomly shuffled before choosing the chosen and rejected responses: if outputs still improve on preference benchmarks, the gains come from generic self-revision rather than reward-guided alignment, while a sharp drop shows the reward signal is load-bearing. A complementary check is to compare TPO outputs against human preference judgments instead of reward-model scores on a held-out set.","supporting_citations":[],"review_version":1}