{"id":"d43f5a66-416d-4c70-b055-49ba5870bcf4","arxiv_id":"2607.18259","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"PCS improves steering direction accuracy by adaptively sampling the intervention coefficient from a cosine-similarity-conditioned Gaussian, but its evaluation is partly circular because the optimal coefficient is chosen on the same logit-difference metric used for scoring.","lead":"PCS is a new inference-time method for steering large language models by sampling a steering strength from a Gaussian distribution tuned by how similar a prompt's concept vector is to a stored steering vector. The reported gains are large, but the evaluation selects the steering strength using the same logit-difference metric it then scores, so the main empirical claim needs stronger validation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported PCS gains may be an artifact of per-prompt λ selection using the same logit-difference metric on which accuracy is measured; the cosine-conditioned calibration mechanism itself is not isolated.","rationale":"I read the paper's central claim as: the cosine-similarity-conditioned Gaussian coefficient is a better calibration mechanism than fixed-strength steering, as evidenced by Table 1c. For that claim to hold, PCS's advantage must be attributable to the (µ, σ) mapping, not to the per-prompt search over λ. The stated algorithm does the latter (§3.4), and the metrics reward exactly that (§4, B.2.1, B.2.5). I therefore treat the evaluation protocol as the most load-bearing issue. The reader's formal weakest_assumption (cosine similarity as predictor of optimal λ) is plausible, and Table 9's negative-similarity best pairs support it, but even if cosine similarity were a perfect predictor, the current comparison would not demonstrate it because of the oracle search. I also note absence of accessible code in the provided text, but the decisive check is computational. I agree with the reader's rejection: the central empirical claim is not established as stated. If the no-oracle ablation later shows retained gains, the verdict could be revised toward conditional acceptance.","tokens_in":23765,"tokens_out":7327,"duration_ms":56926,"concrete_test":"Run a no-oracle ablation of PCS on the same ten datasets: in Algorithm 1 / §3.4, remove the candidate-evaluation step and return either the distribution mean µ or one sample drawn before any LD computation, for every prompt. Keep all other hyperparameters (Table 4), layers, splits, and baselines identical. If the ten-dataset average margin over CAA/Λ1 remains near the reported +24.8/+16.8 pp, the calibration claim is supported; if the margin collapses toward 0, the reported gains are artifacts of per-prompt oracle selection. A secondary check would rescore outputs with an independent target-concept classifier rather than LD, but the no-oracle ablation alone decides the circularity concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Evaluation circularity in the headline comparison is the load-bearing weakness. Section 3.4 states that PCS 'sample[s] k candidate values from the adaptive Gaussian distribution and evaluate[s] each by measuring the resulting change in logit difference.' Direction accuracy and effectiveness (§4.1, B.2.1, B.2.5) are then defined by whether LD moves in the intended direction (with a 0.2 threshold). PCS therefore receives per-prompt, test-time access to the exact objective used to score it, while CAA and Λ1 use fixed strengths. The Table 1c margins (+24.8/+16.8 pp) conflate oracle search over λ with the contribution of similarity-conditioned calibration. The same selection-on-the-outcome problem appears in the restriction to 'steerable examples'—prompts selected by the same logit-shift criterion—before reporting accuracy. The reader's weakest_assumption about negative cosine-similarity pairs (Table 9) is relevant, but secondary: even on positive-similarity examples the current protocol cannot distinguish calibrated sampling from test-time optimization. Appendix tables (e.g., Table 18 vs Table 1c) also contain inconsistent numbers, reinforcing that the quantitative claim is not yet reliably supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Probabilistic Concept-Aware Steering (PCS), an inference-time steering method for LLMs. For each prompt, PCS computes the cosine similarity between a concept vector and a steering vector, maps that similarity to the mean and variance of a Gaussian over the steering coefficient λ, samples k candidate λ values, evaluates each by its induced change in logit difference, and applies the best one through an affine hidden-state intervention. The authors claim large gains in direction accuracy over fixed-strength baselines (CAA and a discrete grid Λ1), improved steering scores, reduced variance, and lower inference cost, and they report experiments across multiple model families and ten datasets.","tokens_in":24132,"tokens_out":6307,"duration_ms":55685,"significance":"If the central mechanism were cleanly isolated, PCS would be a useful contribution: it is training-free, model-agnostic, addresses the real limitation of fixed-strength steering, and the paper provides code and uses public datasets. However, the headline evaluation grants PCS per-prompt access to the same logit-difference signal on which it is scored, while the baselines use fixed strengths. The reported margins therefore do not establish that cosine-similarity-conditioned calibration is the cause of the improvement. The inconsistent numbers between the main table and the appendix further undermine the quantitative claim. The idea is defensible and the evaluation flaw is fixable, but as written the central empirical claim is not supported.","major_comments":[{"comment":"The main comparison in Table 1c is circular. Section 3.4 states that PCS 'sample[s] k candidate values from the adaptive Gaussian distribution and evaluate[s] each by measuring the resulting change in logit difference.' Direction accuracy is then defined by whether the logit difference shifts in the intended direction (Appendix B.2.1, B.2.5, threshold 0.2). Thus PCS receives online access to the exact metric on which it is scored, while CAA and Λ1 use fixed strengths. The reported +24.8/+16.8 percentage-point margins conflate per-prompt oracle search over λ with the contribution of similarity-conditioned sampling. A fair comparison must include a baseline that also optimizes λ per prompt over the same candidate set, or must select λ on training prompts and evaluate on held-out prompts.","section":"§3.4, §4, Appendix B.2.1/B.2.5"},{"comment":"Accuracy is reported on a 'steerable' subset defined by significant logit shifts (s ≥ 0.2). Because PCS selects λ specifically to maximize the logit-difference shift, membership in this subset is itself influenced by the optimization objective. Filtering on the outcome variable after optimizing the same outcome can inflate measured accuracy. Please report metrics on the full test set, and if the subset analysis is retained, define steerability by an independent criterion or by an out-of-sample λ.","section":"§4, Evaluation Metrics"},{"comment":"The main and appendix tables contradict each other. For example, SA-Gen is 72.6% in Table 1c but 68.2% in Table 18; Coord-Ver is 72.6% vs. 56.7%; Corr-Less is 69.2% vs. 77.6%; Corr-More is 77.1% vs. 76.6%. Since Table 18 is presumably the supporting evidence for Table 1c, the quantitative central claim is not reliably established. The authors must reconcile these numbers and state which are final.","section":"Table 1c vs. Table 18"},{"comment":"The mathematical description is internally inconsistent. Equation (2) defines (µ, σ) = (1−s)(µ0, σ0) + s(µs, σs) and mentions a κ downscaling, but Algorithm 1 uses µ = t + sα and σ = d·(w + (1−s)/v) or d + (1−s)·a, with no κ and no (µ0, σ0), (µs, σs). These quantities are never defined. This ambiguity blocks reproduction and must be fixed by unifying the notation and defining every symbol.","section":"§3.4 vs. Algorithm 1"},{"comment":"The threshold τ = 0.2 is imported from SEMSCORE, where it reflects correlation with human quality judgments for textual similarity. It is then applied to a different quantity—logit-difference steerability—without evidence that the two thresholds coincide. Since τ changes the sampling schedule, the paper should provide a calibration curve or ablation over τ on steering outcomes, rather than relying on an analogy to an unrelated metric.","section":"§3.5, Appendix B.1"}],"minor_comments":[{"comment":"The Introduction contains a duplicated paragraph beginning 'As large language models (LLMs) become increasingly prevalent...' which repeats several sentences verbatim.","section":"Section 1"},{"comment":"There are typos such as 'address address' and 'Our figue in the experiment section...' Please proofread.","section":"Section 2/3"},{"comment":"The SFT row is malformed ('SFT –88.885.8 50.6') and does not align with the model columns.","section":"Table 1a"},{"comment":"The sentence 'It can be proven that SA-NN achieves 77.1% accuracy...' is not a proof; it is an empirical observation. Please rephrase.","section":"§4.3.1"},{"comment":"The top direction-accuracy pairings include negative cosine similarities (e.g., SA-Gen with Corr-More at −0.42). This appears to conflict with the claim that cosine similarity is a reliable guide to intervention strength; please discuss this explicitly, for example by reporting the λ distribution on negative-similarity pairs.","section":"Table 9"}],"recommendation":"major_revision","confidential_remarks":"The reader's rejection is understandable and the circularity concern is real. I am recommending major revision rather than reject because the flaw is an evaluation-design problem that can be fixed with additional experiments: a per-prompt-oracle baseline, held-out λ selection, and full-set reporting. If the authors cannot show gains against a per-prompt oracle of comparable strength, the contribution reduces to a heuristic for choosing λ, and the paper should be reframed accordingly. The inconsistent tables must also be resolved before any positive recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you care about steering-vector calibration. The core idea — sampling the steering coefficient λ from a Gaussian whose mean and variance are interpolated by cosine similarity between the prompt's concept vector and the steering vector (Eq. 2, Algorithm 1) — is genuinely new and not in the cited baselines. It is also a sensible thing to try: fixed-strength steering is known to be brittle, and the paper attacks a real problem without fine-tuning, in a model-agnostic way. The authors also acknowledge real limitations: PCS is in-domain, requires a concept vector per prompt, and leaves an anti-steerable minority even at extreme λ.\n\nThe soft spot is load-bearing, and it is exactly what the stress-test flags. In §3.4, PCS samples k candidate λ values and evaluates each \"by measuring the resulting change in logit difference.\" Direction accuracy and effectiveness are then defined by whether that same logit difference moves in the intended direction, with a 0.2 threshold. So PCS receives per-prompt, test-time access to the exact objective used to score it, while the CAA and Λ1 baselines use fixed strengths. The headline margins (+24.8/+16.8 pp) therefore conflate oracle search over λ with the contribution of the similarity-conditioned calibration. Restricting evaluation to a post-hoc \"steerable\" subset selected by the same logit-shift criterion compounds the problem. This is not a minor methodological tic; it means the central empirical claim is not yet supported.\n\nThere are also smaller inconsistencies that reinforce this. Headline numbers in Table 1c do not match the appendix tables: Corr-Less is 69.2% in Table 1c but 77.6% in Table 18, Coord-Ver 72.6% vs 56.7%. And Table 9 shows top-performing pairs with negative cosine similarity (e.g., SA-Gen with Corr-More at −0.42), which undercuts the assumption that similarity semantics cleanly predict steering efficacy. The paper has many free hyperparameters and no accessible code, so even the positive results would be hard to verify.\n\nThat said, this is not a crackpot paper. The evaluation is flawed, but the underlying question — can cosine similarity between concept and steering directions guide coefficient selection? — is legitimate and the mechanism is worth testing with a cleaner protocol. A serious referee could help the authors redesign the evaluation: compare against random search over λ, report accuracy on all prompts rather than the steerable subset, or use a separate validation set for λ selection.\n\nMy recommendation: send it to peer review, not desk reject. The idea deserves referee time, but the quantitative claims as presented should not be cited.","headline":"PCS offers a plausible calibration idea, but the reported gains are not trustworthy: λ is tuned per prompt on the same logit-difference metric used to score accuracy.","tokens_in":24574,"tokens_out":2009,"would_cite":false,"duration_ms":19032,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PCS replaces the fixed coefficient in activation steering with a similarity-conditioned probabilistic one and reports higher direction accuracy on all ten datasets tested.","keywords":["steering vectors","activation engineering","inference-time intervention","probabilistic strength calibration","cosine similarity","concept-aware steering","logit difference","semantic alignment"],"falsifier":"Run PCS while replacing s with shuffled or constant similarities (the paper's own constant-similarity ablation gives a 6.1-point drop); if a broader set of datasets shows no drop or a drop that vanishes, the similarity-conditioned calibration is not doing the work. Also test whether the reported reverse-U peak at moderate similarity reproduces on held-out or out-of-distribution prompts; a monotone relationship would contradict the proposed sweet-spot mechanism.","tokens_in":23677,"feed_emoji":"🧭","tokens_out":8094,"duration_ms":68565,"temperature":0.7,"pith_summary":"This paper sets out to show that the biggest practical weakness of steering vectors—directions in a model's internal activation space that push output toward a concept—can be fixed by treating the push strength as a random sample from a Gaussian whose mean and spread are set by how similar the prompt's concept direction is to the steering direction. If that works, LLM behavior could be controlled at inference time without retuning or parameter updates, which would make fine-grained semantic control cheaper and more trustworthy. The authors report that this method, PCS, beats fixed-strength baselines on every dataset considered, by 24.8 and 16.8 percentage points on average, while cutting variance roughly in half and running with near-zero setup cost. The authors also concede that about a quarter of prompts remain resistant and that PCS currently needs a concept vector for every prompt (Appendix B.7), so the win is real but bounded.","feed_headline":"Similarity-tuned steering beats fixed strengths on every test set","feed_subtitle":"A similarity-shaped Gaussian for steering strength lifts direction accuracy by ~25 points over fixed-strength baselines","key_machinery":"The central mechanism is probabilistic strength calibration: lambda, the scalar multiplying the steering vector, is drawn from N(mu,sigma^2) where both parameters are conditioned on cosine similarity between the prompt's concept vector and the steering vector. This same similarity also decides between a confident low-variance schedule and an exploratory high-variance schedule at threshold tau=0.2, with the sampled lambda then applied through an affine hidden-state transformation h'=[I|lambda v_sv]h~ at one mid-transformer layer. The machinery matters because it turns a global hyperparameter (steering strength) into a per-prompt, geometry-aware quantity, and the paper's empirical case is that","core_discovery":"On its own terms, the paper advances one claim: the optimal steering strength for an LLM intervention is not a constant but a sample from a Gaussian distribution whose parameters interpolate between the orthogonality and full-alignment cases according to cosine similarity s between the steering vector and the prompt concept vector, (mu,sigma) = (1-s)(mu0,sigma0) + s(mus,sigmas), with extra variance when s falls below tau=0.2 and hard clipping on both parameters. The injected intervention is affine, h' = [I lambda v_sv] h~, applied at a chosen mid-layer. Empirically, PCS reports higher direction accuracy than fixed-strength steering baselines on all ten datasets, average gains of +24.8 and +1","pith_inferences":["Because the paper's own top-performing pair can have negative cosine similarity (SA-Gen with Corr-More at -0.42), raw cosine may not be the best calibration signal; a learned or ensemble estimate of expected steering efficacy could improve on the interpolation equation.","The reverse-U between inter-dataset similarity and direction accuracy suggests that steering success is a function of semantic distance, so a practical system could precompute sweet-spot strengths per source-target pair instead of sampling at inference.","PCS's dependence on a per-prompt concept vector is the main barrier to open-ended use; plugging in zero-shot concept detection or using the model's own intermediate representation as the concept vector is a natural extension the authors flag as future work.","The variance reduction from roughly 48-50 to 28.6 hints that the method is not just shifting means but stabilizing behavior; testing whether this stability persists on out-of-distribution prompts would clarify the generalization claim."],"forward_implications":["A single cosine-similarity measurement could stand in for per-task grid search, since the Gaussian parameters are fixed across concept domains (t=3.0, d=0.5, tau=0.2).","Interventions become dramatically cheaper: no offline optimization phase per concept (153.4 ms setup versus 600+ seconds for a parameter-updating baseline) and stable runtime.","Small models benefit most; PCS steering scores exceed supervised fine-tuning on 1B and 3B parameter models, which matters for resource-constrained deployment.","Practitioners should avoid very large lambda: the paper finds extreme scaling creates semantic drift and incoherent outputs even when logit-difference metrics look better.","A residual set of prompts (roughly 25%) remains anti-steerable under any reasonable strength, so steering-vector control cannot be the only safety mechanism."],"fun_headline_variants":["Gaussian steering strength beats fixed baselines by 25 points","Similarity-guided Gaussian beats fixed strengths on 10 datasets","Adaptive steering strength: Gaussian beats fixed by 25 pts","Probability-based steering gains 25 points over fixed methods","PCS: Gaussian strength beats fixed on all 10 test sets"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The framework assumes cosine similarity between the steering vector and a prompt's concept vector predicts how much steering force that prompt needs; if that mapping is unreliable, the adaptive distribution loses its justification.","fun_headline_variants_meta":{"raw":{"variants":["Gaussian steering strength beats fixed baselines by 25 points","Similarity-guided Gaussian beats fixed strengths on 10 datasets","Adaptive steering strength: Gaussian beats fixed by 25 pts","Probability-based steering gains 25 points over fixed methods","PCS: Gaussian strength beats fixed on all 10 test sets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000741,"raw_usage":{"total_tokens":3099,"prompt_tokens":655,"completion_tokens":2444,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":399,"completion_tokens_details":{"reasoning_tokens":2360}},"tokens_in":399,"tokens_out":2444,"duration_ms":13802,"temperature":1.0,"reasoning_tokens":2360,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T13:52:04.779194+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PCS while replacing s with shuffled or constant similarities (the paper's own constant-similarity ablation gives a 6.1-point drop); if a broader set of datasets shows no drop or a drop that vanishes, the similarity-conditioned calibration is not doing the work. Also test whether the reported reverse-U peak at moderate similarity reproduces on held-out or out-of-distribution prompts; a monotone relationship would contradict the proposed sweet-spot mechanism.","supporting_citations":[],"review_version":1}