{"id":"cd86eebc-1106-40bd-874a-8c4f4a46d10b","arxiv_id":"2505.20809","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"RePS, a reference-free bidirectional preference optimization objective, improves representation steering and suppression for Gemma models, outperforming language-modeling objectives and approaching prompting performance.","lead":"This paper introduces RePS, a new training objective for representation steering in language models that combines reference-free preference optimization with bidirectional positive and negative steering. Tests on Gemma models from 2B to 27B show RePS-trained steering vectors outperform language-modeling-trained vectors and narrow the gap with prompting, while staying robust to prompt-based jailbreak attacks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RePS trains negative steering with a null-projection intervention (Eq. 7) but suppresses at inference with negative factors on ΦSteer (Eq. 9); the suppression claims may not be grounded in the trained objective.","rationale":"I read the paper as making two intertwined central claims: (1) RePS improves concept steering over the language-modeling objective, and (2) RePS-trained interventions can be turned into robust suppression mechanisms. The reader's weakest assumption was the linear representation hypothesis, which is plausible but somewhat speculative given the empirical success of rank-1 SVs in the same benchmark. I find a more concrete and load-bearing issue in the suppression pipeline: the negative-steering objective in Eq. (6) optimizes an intervention ΦNull that is defined in Eq. (7), but the evaluation in Section 5.3 applies negative factors to the standard steering intervention ΦSteer rather than ΦNull. This mismatch means the suppression results are not directly testing what was trained. The paper also explicitly replaces ΦNull with negative factor sampling for LoRA and ReFT, acknowledging that the null-projection form is specific to SV; this makes the SV case stand out as needing justification. The concern is not that the results are false; both Lang.- and RePS-trained vectors are evaluated with the same negative-α protocol, so the comparative claims survive as empirical observations. Rather, the explanation and generalization of RePS's advantage in suppression are ungrounded. The central steering claim is otherwise supported by Table 1 for most cells, though the abstract's 'consistently' is overstated given the LoReFT underperformances at Gemma-2-2B L10 and Gemma-3-12B. A conditional acceptance remains appropriate: the paper should either evaluate ΦNull at inference or explicitly train with negative α on the same intervention, and clarify the scope of the suppression claims. The reader's linear-representation concern is reasonable but less decisive, since the empirical comparisons show SV working well enough in the tested settings; the intervention mismatch is a direct testable gap in the method as described.","tokens_in":42964,"tokens_out":5527,"duration_ms":65343,"concrete_test":"Run the suppression evaluation using the trained ΦNull intervention itself (from Eq. 7) on the Table 2 and Table 3 setups, and compare against the reported negative-α results. As a complementary check, train a RePS variant for rank-1 SV where the negative direction uses sampled negative α on ΦSteer (as done for LoRA/ReFT) rather than ΦNull; if this variant matches the current suppression scores, the mismatch is inconsequential, whereas if scores drop or differ substantially, the suppression claims must be re-attributed to an untested intervention.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's suppression experiments, which support major claims about robustness and jailbreak resistance, exhibit a train/inference mismatch for the rank-1 SV intervention. In Section 3.3, the negative steering loss in Eq. (6) is defined using ΦNull(hl) = hl − ReLU(hl·w1)/‖w1‖² w1 (Eq. 7), an asymmetric intervention that removes the projection onto the steering direction. However, Section 5.3 states that suppression is evaluated by applying negative coefficients α to ΦSteer as in Eq. (9), i.e., hl ← hl + α·w1 + b1 with α < 0. These are different operations: ΦNull is a norm-dependent projection removal, while negative α is a fixed translation along w1 with a bias term. The learned w1 is shared, so the negative-training signal may still shape a useful direction, but the paper never evaluates the intervention it actually trained for suppression, nor does it demonstrate that the two interventions are behaviorally equivalent. This matters because the suppression results (Tables 2 and 3, Figure 1) are used to conclude that RePS is a robust alternative to prompting; if the negative-α evaluation does not reflect the trained objective, the mechanism behind RePS's suppression gains is unverified and the comparison against Lang.-trained vectors under the same negative-α protocol is not explained by the paper's stated training objective.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RePS, a reference-free bidirectional preference-optimization objective for training low-rank representation-steering interventions (rank-1 steering vectors, LoRA, and LoReFT). RePS is derived from SimPO and BiPO and jointly trains positive steering (via the steering intervention) and negative/suppression behavior (via a null-projection intervention in Eq. 7). The authors evaluate RePS against the standard language-modeling objective and the BiPO baseline across Gemma-2/3 models from 2B to 27B on the AxBench concept-steering and suppression tasks, including prompt-based jailbreaking scenarios. They report that RePS outperforms existing steering methods trained with a language-modeling objective, narrows the gap with prompting, and is more robust than prompting for suppression under attack.","tokens_in":1504,"tokens_out":1904,"duration_ms":52749,"significance":"If the reported results hold, RePS would be a useful and practical contribution to representation steering: it is reference-free, parameter-efficient, and appears to improve suppression robustness for larger Gemma-3 models while maintaining interpretability. The paper provides a substantial evaluation across model scales and intervention types, and it ships a public code repository. However, the main comparative claims currently rest on single-seed numbers with no variance estimates, and the suppression evaluation does not match the intervention actually trained in the negative-steering objective; these issues are load-bearing for the paper's central claims and need to be addressed before the results can be regarded as established.","major_comments":[{"comment":"The claim that \"RePS consistently outperforms Lang.\" is contradicted by the reported LoReFT results on Gemma-3-12B/D100: Lang. scores 0.714 while RePS scores 0.651. The same table shows very small RePS advantages for SV (1.230 vs 1.219) and LoRA (0.950 vs 0.943) on this model. Given that the table reports no variance and the experiments use a single seed, I cannot tell whether these differences are meaningful, and the word \"consistently\" is not supported. Please add confidence intervals or replicate at least the critical comparisons over several seeds, and revise the overbroad claim.","section":"§5.2, Table 1"},{"comment":"The suppression evaluation does not test the intervention that the negative-steering loss actually trains. Eq. (6) defines the negative loss using ΦNull(hl) = hl − ReLU(hl·w1)/‖w1‖² w1, which removes the projection of the activation onto w1. Section 5.3, however, states that suppression is applied by using negative coefficients α in ΦSteer(hl, α) = hl + α·w1 + b1, a translation along w1 with a bias term. These are different operations, and the paper never evaluates ΦNull nor demonstrates that the two interventions are behaviorally equivalent. Since the suppression results in Tables 2 and 3 and Figure 1 are used to conclude that RePS is a robust alternative to prompting, this train/inference mismatch is load-bearing. The authors should either evaluate suppression with the actual ΦNull intervention used in training, or provide empirical evidence (e.g., on a development set) that negative-α ΦSteer and ΦNull produce equivalent suppression behavior for the trained vectors.","section":"§3.3 vs §5.3, Eqs. (6)--(7) and Eq. (9)"},{"comment":"The suppression comparisons are reported without any error bars, confidence intervals, or multiple seeds. For example, the 12B and 27B differences between RePS and Lang. in Table 2 (1.035 vs 0.912 and 1.031 vs 0.940) are plausible but could easily be within run-to-run variability, especially given that Appendix D says all experiments use a single seed and that the model-generation and LM-judge pipeline is stochastic. The robustness claims in Section 5.4 would be stronger if the authors provided variance estimates or replication for at least the key attack conditions.","section":"§5.3, Tables 2--3, Figure 1"}],"minor_comments":[{"comment":"The sentence \"We how take the RePS-trained interventions...\" contains a typo; it should likely read \"We now take...\".","section":"§5.3"},{"comment":"The phrase \"from from hl\" in the text defining ΦNull is a typo and should be corrected.","section":"§3.3, Eq. (7)"},{"comment":"The benchmark name \"AxBench\" is typeset inconsistently (e.g., AXBENCH, A XBENCH, AxBench); please standardize.","section":"§1 and throughout"},{"comment":"The stability analysis in Figure 12 is helpful, but it reports only the maximum steering-score difference across three runs for two settings; reporting the full distribution or standard deviations of the overall score would give a better sense of evaluation noise.","section":"Appendix D"},{"comment":"The table note \"Φr=1 SV is rank-1 and has the fewest trainable parameters\" is useful, but the main text could also state explicitly which layers were used for each model and whether the same supervised layer search was applied to all objectives with identical budgets.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The central idea of RePS seems sound and the paper is a strong empirical study, but the suppression claims rely on an evaluation that does not match the trained negative intervention, and the single-seed, no-variance reporting makes the \"consistently better\" claims fragile. I would be willing to accept after the authors either evaluate ΦNull or demonstrate equivalence with negative-α ΦSteer, and after adding variance estimates for the key comparisons. The LoReFT 12B reversal in Table 1 should also be acknowledged and discussed rather than glossed over."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zhengxuan et al. propose RePS, a reference-free bidirectional preference objective for steering vectors and low-rank adapters. The core idea—SimPO-style length-normalized preference loss applied in both positive and negative directions, with random steering factors during training—is new, and the evaluation is broad: four Gemma sizes, three intervention families, budget-controlled hyperparameter search, and honest appendices covering stability, ablations, and failure modes. The steering results are credible: on SV and LoRA, RePS beats the LM objective on most datasets and noticeably narrows the gap with prompting.\n\nThe main soft spot is the suppression story. Section 3.3 defines negative steering with a null-projection intervention ΦNull(hl) = hl − ReLU(hl·w1)/||w1||² w1 (Eq. 7). Section 5.3 then evaluates suppression by applying negative coefficients α to ΦSteer (Eq. 9). Those are different operations. The paper never evaluates the intervention it actually trained for suppression, and it does not show that negative-α on ΦSteer approximates ΦNull. The jailbreak-resilience claims rest on this untested protocol. This is fixable—evaluate ΦNull directly or justify the equivalence—but as written the mechanism is unverified.\n\nSecond, the abstract and Section 5.2 claim RePS is 'consistently better' than Lang. Table 1 contradicts this: LoReFT with Lang gets 0.714 on Gemma-3-12B D100 versus 0.651 for RePS, and on 27B both collapse but in different ways. The claim should be qualified to specific intervention types and model sizes.\n\nThird, the main tables report no variance and all experiments use a single seed. The appendix's stability analysis (max difference 0.05 across three runs) is reassuring but not a substitute for reporting error bars on central comparisons.\n\nFourth, the equations in Section 3.3 are garbled in the version I read, so I could not verify the math directly; the surrounding text is clear enough, but this needs to be cleaned up. Also, the code link points to the axbench repo, not to RePS; actual code release would help.\n\nThe linear-representation assumption is worth stating more openly as a limitation, but it is not a flaw by itself.\n\nBottom line: this is a solid paper with a real new objective and broad evaluation. It deserves serious refereeing. The referee should push on the suppression evaluation protocol, the consistency claim, and the need for variance reporting and code. My recommendation: accept for review, conditional on those revisions.","headline":"RePS is a genuinely new steering objective with credible steering gains, but the suppression claims rest on a train/inference mismatch that needs fixing.","tokens_in":43776,"tokens_out":3349,"would_cite":true,"duration_ms":32654,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reference-free preference steering beats LM training on Gemma models.","keywords":["representation steering","preference optimization","steering vectors","concept suppression","jailbreaking robustness","low-rank adaptation","reference-free objective","linear representation hypothesis"],"falsifier":"Run the same 500 AxBench concepts through RePS training while measuring how linearly separable the steered and unsteered activations are with a linear probe at the intervened layer; if a large fraction of concepts show low linear-probe separability yet still steer well, the linear-representation premise is not what carries the method. A direct alternative is to train RePS on a task set constructed to be nonlinear in representation space (e.g., disjunctive multi-concept targets) and compare rank-1 steering vectors with rank-4 LoRA interventions on steering and suppression scores.","tokens_in":42774,"feed_emoji":"🎛️","tokens_out":5783,"duration_ms":56251,"temperature":0.7,"pith_summary":"Representation steering — editing a model's internal activations at inference time to add or remove a behavior — has lagged behind prompting. This paper claims the shortfall is a training-objective problem, not a method problem: steering interventions trained with a language-modeling loss miss the preference signal that instruct-tuned models are built on. The authors propose RePS, a reference-free, bidirectional preference objective that rewards steered behavior under positive intervention and punishes it under negative intervention, and train three low-rank interventions (steering vectors, LoRA, LoReFT) with it. On the AxBench concept tasks across Gemma models from 2B to 27B, RePS consistently beats the language-modeling objective and the earlier BiPO baseline, substantially narrowing the gap with prompting. For suppressing a concept, RePS-trained rank-1 steering vectors match the LM objective on Gemma-2 and beat it on larger Gemma-3 models, and they resist prompt-based jailbreaks that defeat prompt defenses.","feed_headline":"Reference-free preference steering beats LM training on Gemma models","feed_subtitle":"Small rank-1 steering vectors trained with a preference objective nearly match prompting and resist jailbreak-style attacks.","key_machinery":"The load-bearing object is the RePS loss, summed over two log-sigmoid terms: one for positive steering (increase the likelihood of the steered response under the learned intervention) and one for negative steering (increase the likelihood of the original response under a null projection that removes the steering direction). The null projection, $\\Phi_{\\mathrm{Null}}(h_l) = h_l - \\frac{\\mathrm{ReLU}(h_l \\cdot w_1)}{\\lVert w_1\\rVert^2} w_1$, is the mechanism that turns a single learned direction into a suppression operator without retraining. A training-time factor-sampling trick, in which steering factors are drawn from a fixed set rather than fixed to one value, stabilizes optimization and is what makes preference training of interventions practical across model scales.","core_discovery":"The central discovery is that a preference-optimization objective of a specific shape—asymmetric, reference-free, and bidirectional—makes rank-1 representation interventions genuinely competitive with prompting. Positive steering uses a SimPO-style length-normalized likelihood margin in which the steered response is up-weighted by a factor that grows when the unsteered model finds it improbable; negative steering is trained against a null intervention that projects out the learned steering direction. The same learned vector therefore transfers from eliciting a concept to suppressing it, and suppression survives both explicit instruction-following attacks and many-shot jailbreaking, where text-based defenses collapse. The result is evidence that steering performance is bounded less by the capacity of rank-1 interventions than by the training objective applied to them.","pith_inferences":["Because RePS closes much of the gap with prompting using only a rank-1 vector, the paper's logic extends to a prediction: as base models get larger and better at following instructions, preference-trained interventions should continue to improve, possibly overtaking prompting on steering where prompt defenses leak system instructions.","The null-projection trick suggests a principled way to build 'erase' operators for any learned direction; one could extend it to higher-rank interventions such as LoRA by projecting onto the null spaces of multiple learned directions, which the paper did not test.","Using RePS as a refinement step on top of unsupervised directions (e.g., difference-in-means vectors or SAE features) might inherit both the interpretability of those features and the preference-aligned steering strength; this is a direct testable extension.","The resistance of intervention-based suppression to prompt attacks implies a defense-in-depth design: an invisible, auditable intervention could guard system prompts against user-induced leakage, since prompt-append defenses in the paper are shown to leak the system prompt under many-shot attacks."],"forward_implications":["RePS-trained rank-1 steering vectors beat language-modeling-trained vectors on steering across all three intervention types and all four Gemma sizes tested, with the widest margins on the smaller Gemma-2 models.","For concept suppression, RePS-trained steering vectors outperform LM-objective-trained ones on Gemma-3-12B and 27B and match them on Gemma-2-2B and 9B, indicating the advantage grows with model scale.","RePS suppression is substantially more resistant than prompting to instruction-following and many-shot jailbreaking attacks; on larger models the prompt-based defenses degrade while the intervention-based defense stays roughly constant.","The gap between representation steering and prompting in the AxBench steering scores is narrowed but not closed, with RePS surpassing prompt scores on some Gemma-2 concept subsets and remaining below prompt scores on most.","Factor sampling during training lowers the variance of steering scores across hyperparameter runs, which the paper credits with making the preference objective practical to tune."],"supporting_citations":[{"why":"Supplies the AxBench benchmark, dataset, evaluation protocol, and baseline numbers that RePS must beat.","marker":"[Wu et al., 2025]"},{"why":"SimPO is the source of the reference-free reward and length normalization used in the RePS positive and negative deltas.","marker":"[Meng et al., 2024]"},{"why":"BiPO is the prior bidirectional preference objective for steering vectors, the main baseline, and the starting point RePS modifies.","marker":"[Cao et al., 2024]"},{"why":"Activation addition defines the rank-1 steering vector intervention and the steering-factor extrapolation RePS trains with.","marker":"[Turner et al., 2023a]"},{"why":"The linear representation hypothesis grounds the rank-1 steering vector parameterization and the null-projection suppression in Eq. (7).","marker":"[Park et al., 2024]"},{"why":"Orthogonal negation provides the idea of removing a direction by projection, used in the negative-steering null intervention.","marker":"[Widdows, 2003]"},{"why":"LoRA is one of the three intervention parameterizations trained under RePS.","marker":"[Hu et al., 2022]"},{"why":"ReFT/LoReFT is the third intervention parameterization trained under RePS.","marker":"[Wu et al., 2024]"},{"why":"Many-shot jailbreaking is the attack used to test suppression robustness in Section 5.4.","marker":"[Anil et al., 2024]"}],"fun_headline_variants":["RePS steering beats LM training, nearly matches prompting","Reference-free RePS resists jailbreaks, rivals prompting","Rank-1 RePS vectors outperform LM-training steering on Gemma","Reference-free steering nearly matches prompting, defeats jailbreaks","New steering objective beats prompting at jailbreak resistance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Each steering concept can be represented by a single direction in the residual stream of the model, an assumption that the rank-1 steering vector and the null-projection suppression both rely on and that the paper does not separately validate across its 500 concepts.","fun_headline_variants_meta":{"raw":{"variants":["RePS steering beats LM training, nearly matches prompting","Reference-free RePS resists jailbreaks, rivals prompting","Rank-1 RePS vectors outperform LM-training steering on Gemma","Reference-free steering nearly matches prompting, defeats jailbreaks","New steering objective beats prompting at jailbreak resistance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001337,"raw_usage":{"total_tokens":5405,"prompt_tokens":887,"completion_tokens":4518,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":4439}},"tokens_in":503,"tokens_out":4518,"duration_ms":35215,"temperature":1.0,"reasoning_tokens":4439,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:46:09.326114+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 500 AxBench concepts through RePS training while measuring how linearly separable the steered and unsteered activations are with a linear probe at the intervened layer; if a large fraction of concepts show low linear-probe separability yet still steer well, the linear-representation premise is not what carries the method. A direct alternative is to train RePS on a task set constructed to be nonlinear in representation space (e.g., disjunctive multi-concept targets) and compare rank-1 steering vectors with rank-4 LoRA interventions on steering and suppression scores.","supporting_citations":[{"cited_title":"Orthogonal negation in vector spaces for modelling word-meanings and document retrieval","cited_arxiv_id":null,"evidence_quote":"Orthogonal negation provides the idea of removing a direction by projection, used in the negative-steering null intervention."},{"cited_title":"Bowman, Ethan Perez, Roger Baker Grosse, and David Duvenaud","cited_arxiv_id":null,"evidence_quote":"Many-shot jailbreaking is the attack used to test suppression robustness in Section 5.4."}],"review_version":1}