{"id":"8ef54672-7326-4121-a904-df2ccc6c6944","arxiv_id":"2608.08514","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Independent reproduction and stress-tests show RPC's test-time aggregation reproduces its original grid but gains nothing significant on new domains, while LCF's logic editing is a real but weak signal that does not transfer.","lead":"This paper independently reruns two methods for making LLM reasoning more reliable, across new models and task domains. It finds that test-time probability aggregation reproduces its original results, while logic-representation editing shows only a weak, non-controllable effect that hurts two of four models.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LCF-negative half of the asymmetry depends on an unverified from-scratch reimplementation; without original code or a hyperparameter sensitivity check, the comparison to the exact RPC reproduction is not apples-to-apples.","rationale":"The reader's weakest_assumption identifies exactly this concern: the LCF negative result depends on a from-scratch reimplementation whose fidelity cannot be verified without original code. My read agrees and treats it as the load-bearing condition for the paper's central asymmetry. The paper itself honestly flags the risk in Limitations, which is why I do not recommend a harsher verdict: the authors do not claim to disprove LCF universally, only that their faithful reimplementation fails. However, the abstract and conclusion state the asymmetry in stronger terms ('LCF's logic-representation editing rests on a real but weak, locally-encoded signal'), which goes beyond the qualified limitation. A sensitivity sweep over the inferred hyperparameters would test whether the negative result is robust or an artifact of a single configuration choice. Until that is done, or original LCF artifacts are obtained, the CONDITIONAL verdict stands: the asymmetry is plausible but not settled. I do not see a separate load-bearing concern in the RPC half, since the exact match on the authors' released paths is a genuine internal replication and the new-domain null results are reported with appropriate power bounds.","tokens_in":18199,"tokens_out":6028,"duration_ms":72200,"concrete_test":"Run a sensitivity sweep on Qwen3-8B over the unverified LCF components: projector width (e.g., 1024, 2048, 4096), InfoNCE temperature (0.05, 0.1, 0.2), edited layer range (single best sub-layer vs. the paper's stated range), and intervention scale eta, training five seeds per configuration. If any plausible configuration yields a per-item DeltaProb gain with p < 0.05 or probe accuracy above 0.9, then the 'weak and uncontrollable' conclusion is not robust to reimplementation choices. If the original LCF code or checkpoints become available, rerunning the same eval harness with the original projectors would be the definitive check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is an asymmetry: RPC is safe and reproduces exactly, while LCF is weak, not controllable, and harmful. The RPC side reuses the authors' released aggregation code and reasoning paths (Section 4), so exact reproduction is expected. The LCF side is a from-scratch re-implementation with no public code: projector dimensions (d -> 2048 -> 1024), InfoNCE temperature (tau = 0.1), intervention scale eta, and edited layer range are inferred from the paper, and the unreleased validity discriminator is replaced by a GPT-4 judge and DistilBERT classifier (Section 5.2). The paper's own Limitations section concedes that the implementation 'may differ in unstated details' and that the conclusions are 'negative results under a faithful re-implementation, not proof that no implementation can reproduce the paper.' Every LCF-negative number—the 0.82 probe separability, the null controllability sweep, and the significant DeltaProb drops on Llama-2 and Mistral—comes from this unverified pipeline. If any unstated detail of the original recipe differs (e.g., the correct edited layer range or intervention scale), the observed non-controllability and DeltaProb decreases could be artifacts of the reimplementation rather than properties of LCF. Because the central asymmetry is built on comparing an exact RPC reproduction with an inferred LCF reimplementation, reimplementation fidelity is the single most load-bearing condition for the conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an independent reproduction and stress-test of two LLM reliability methods: RPC, a test-time probability aggregation method, and LCF, a logic-representation editing method. RPC is reproduced using the authors' released code and reasoning paths, matching the published accuracy/ECE grid, and is then extended to four new task domains plus three public benchmarks; the edge over self-consistency is never significant, and a BIRD budget sweep shows a small, non-significant gain at K=32 that reverses to −0.25 at n=200. LCF, for which no public code exists, is re-implemented from scratch; the paper finds a weak probe-separable validity signal (0.82 at the best sub-layer), no controllability under additive or projective interventions, a non-significant positive DeltaProb effect on one model, and significant DeltaProb drops on two of four models. A training-free contrastive steering variant also fails except under a narrow matched-distribution condition. The conclusion is an asymmetry: RPC is safe and reproducible, whereas LCF's logic editing is a real but weak, locally encoded signal that is decodable but not controllable, and not model-agnostic. The limitations section candidly acknowledges small samples, model-generated labels with 28% judge agreement, and uncertainty about LCF reimplementation fidelity.","tokens_in":18458,"tokens_out":3670,"duration_ms":41281,"significance":"If upheld, the paper provides a valuable independent check on two influential methods: it shows that RPC's aggregate behavior transfers without measurable harm across domains, while LCF's headline effects do not reproduce as a model-agnostic intervention at 7–8B scale. The RPC side is a faithful reproduction using the authors' own code and data, with machine-checkable numbers that match the published grid; the released code, data, and experiment logs are a concrete strength. The LCF side is honest in its limitations, explicitly flagging the unverified reimplementation and the noisy validity labels. The main value is therefore the asymmetry claim: aggregation is the safer, more trustworthy intervention. However, the LCF-negative half of that asymmetry depends on an unverified from-scratch reimplementation and on labels of uncertain validity, which limits how strongly the conclusion can be drawn. The paper is most useful as a careful, reproducible stress-test of RPC and as a cautionary, clearly hedged negative result for LCF.","major_comments":[{"comment":"The LCF negative results are load-bearing for the central asymmetry, but they rest entirely on a reimplementation whose fidelity is unverified. The manuscript infers projector dimensions (d→2048→1024), InfoNCE temperature (tau=0.1), the edited layer range, and the intervention scale eta from the paper text, and substitutes an unreleased validity discriminator with a GPT-4 judge and DistilBERT classifier. The Limitations concede that the implementation 'may differ in unstated details,' but the conclusion's asymmetry ('logic-representation editing rests on a real but weak, locally-encoded signal... decodable yet not controllable') is stated without this caveat at full strength. Every LCF number in Table 5, Figure 3, and Table 9 comes from this unverified pipeline. Because the RPC side of the comparison is an exact reproduction using the authors' released code, the two sides are not compared on equal footing. The paper should either add a hyperparameter sensitivity analysis around eta, layer range, and projector architecture, or clearly reframe the LCF conclusions as 'under our faithful re-implementation, not as a property of the original method.' As written, the asymmetry claim outruns the evidence.","section":"Section 5.2 and Limitations"},{"comment":"The probe claim that a 'logic-validity direction is real but weak' is confounded by the construction of the valid/invalid labels. Valid conclusions are generated by GPT-4o-mini while invalid ones come from LFUD's fallacious conclusions; an independent judge accepts only 28% of the generated 'valid' conclusions as strictly valid. The probe could therefore be separating surface style (e.g., formality, wording patterns) rather than logical validity. The Limitations say this 'weakens, not strengthens, our reading of the signal,' but that argument only works if the style confound is orthogonal to logic; nothing in the paper establishes that. A concrete control would be to probe on a style-matched invalid set (e.g., generated invalid conclusions) or to compare against a bag-of-words or embedding-distance baseline. Without such a control, the 0.82 separability does not establish that the signal is 'logic' at all, which is a central premise for the conclusion that the signal is 'real but weak.'","section":"Section 5.2, Figure 3, and Limitations"},{"comment":"The conclusion that RPC 'never significantly degraded accuracy' and is therefore 'the safer intervention' overstates what the statistics support. The power analysis in the Limitations shows that at K=32 on BIRD (n=80) the study has 80% power only for gaps of at least 5.0 accuracy points, and the observed +2.5 gap reverses to −0.25 at n=200. A non-significant difference at these sample sizes does not establish safety; it only bounds the effect size. The phrase 'never significantly degraded' is technically true but the prose in the Conclusion ('aggregation is the safer intervention') reads as a stronger claim. The asymmetry should be restated as 'no significant evidence of harm at these evaluation sizes, and the only observed positive gain disappears with more data,' which is still a meaningful result but less categorical than the current wording.","section":"Section 6, Table 4, and Limitations"}],"minor_comments":[{"comment":"The abstract reports the BIRD enlargement reversal as '−0.25' in text and elsewhere as '−0.25' but the full text, including the Conclusion, uses '−0.25' consistently; however, in Table 4 the caption refers to a '+2.5 lead' and the Limitations say '−0.25', which is fine, but the abstract's phrase 'its edge over self-consistency is never significant' is contradicted in the same sentence by the BIRD K=32 p=0.16; clarify that the edge is non-significant rather than absent.","section":"Abstract and Section 6"},{"comment":"The K=32 row is a single deterministic run with no seed variance, whereas K=8 and K=16 report 95% CIs over 20 seeds; the caption does explain this, but it would be clearer to mark the K=32 row explicitly as 'no CI' in the table itself to avoid reader confusion.","section":"Table 4"},{"comment":"The phrase 'we find no evidence of fabrication; we find that the headline leans on an unauditable component' is a useful clarification, but it appears only in the results section; the same nuance would be worth one sentence in the Limitations where the discriminator substitution is discussed.","section":"Section 5.2 and Limitations"},{"comment":"The LFUD split is described as '45:5:17' for conclusion generation and '540/204' for RPC MCQ; the relationship between these two splits is not explained in the main text. Please clarify whether the 204-item LCF evaluation is a subset of the 540-item MCQ or a separate split.","section":"Section 3 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"This is a well-conducted reproducibility study with honest limitations, but the central asymmetry claim currently depends on an unverified reimplementation of LCF. The authors already acknowledge this in the Limitations; the revision should make the caveat prominent in the conclusion and add a sensitivity analysis or a more explicit reframing. I would not reject: the RPC reproduction is exact and the new-domain results are useful, and the LCF negative results, if properly qualified, are a valuable contribution. However, the comparison is not yet apples-to-apples, and the paper's own power analysis undercuts the 'safer intervention' wording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: the RPC half is a clean, exact reproduction with useful new-domain and budget results, and the LCF half is a carefully hedged negative result that lands as plausible but provisional. If you cite one paper on whether RPC beats self-consistency at 7–8B scale, or on whether LCF transfers across models, this is now the first independent data point.\n\nThe paper does several things well. It reproduces Zhou et al.'s RPC grid exactly on the released paths, which is the baseline any reviewer should want. The four new domains and the BIRD K-scaling sweep are genuinely new, and the BIRD reversal from +2.5 at n=80 to −0.25 at n=200 is a good direct test of a small-sample fluctuation. The LCF re-implementation is the first independent attempt, and the probe design separates separability from controllability, showing the logic direction is weak (0.82 vs 0.95 for a semantic control) and not a causal lever. The Limitations section is unusually candid: the authors flag underpowered sets, unverified re-implementation details, noisy labels, and the 28% judge agreement on generated valid items. That honesty is real credit.\n\nThe soft spots are where the stress-test lands. The central asymmetry compares an exact RPC reproduction against an inferred LCF re-implementation. Projector dimensions, InfoNCE temperature, edited layer range, and η are taken from the paper; the unreleased discriminator is replaced by a GPT-4 judge and DistilBERT. Every LCF-negative number comes from this pipeline. The authors concede this in the Limitations, but it means the non-controllability and the DeltaProb drops on Llama-2 and Mistral could still be artifacts of unstated recipe differences. The label noise cuts the same direction: if the “valid” conclusions are themselves shaky, the weak probe signal and failed steering are hard to interpret as pure properties of LCF. And the sample sizes really are small—80% power only for 5-point gaps—so the RPC non-significance also bounds effects rather than excluding them.\n\nStill, the paper never overclaims. It says the LCF conclusions are negative results under a faithful re-implementation, not disproof. The math and the data handling look careful; the citation pattern is appropriate; the code and logs are shipped. This deserves a serious referee. The main requests should be: release the LCF re-implementation code, do a sensitivity analysis on the inferred hyperparameters, and either human-validate a sample of the generated valid labels or report results under both label sets. With those, the asymmetry could move from directional to settled.\n\nRecommendation: send it to peer review. It is exactly the kind of independent reproduction the field needs, even if the headline is a conditional negative.","headline":"An honest RPC reproduction plus a plausible but unverified LCF negative result; the asymmetry is directional, not settled.","tokens_in":19031,"tokens_out":1379,"would_cite":true,"duration_ms":17348,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Independent rerun: aggregation reproduces, logic steering does not","keywords":["LLM reasoning reliability","reproducibility study","test-time aggregation","self-consistency","logic representation editing","representation probing","activation steering","calibration"],"falsifier":"Run the original LCF code, or a version whose projector, contrastive objective, edited layers, and validity discriminator are supplied by the original authors, on Llama-2 and Mistral with the same 204-item fallacy task: if DeltaProb rises significantly on either model instead of falling, the paper's central negative claim is false. Likewise, a further independent dataset draw on BIRD at K=32, which has already turned a +2.5 lead into -0.25 when the sample grew from n=80 to n=200, would settle whether any aggregation edge exists.","tokens_in":17944,"feed_emoji":"🧠","tokens_out":7185,"duration_ms":67729,"temperature":0.7,"pith_summary":"The paper independently re-runs two published methods that claim to make large language model reasoning more reliable: a test-time aggregation rule that combines token probabilities with self-consistency votes (RPC), and a representation-editing pipeline that trains projectors to separate 'content' from 'logic' in hidden states and shifts the logic part toward valid conclusions (LCF). The aim is to test whether results reported by each method's own authors survive a faithful re-implementation and transfer across domains and models. It finds an asymmetry: RPC reproduces the original math grid exactly and never significantly hurts accuracy on new domains, but its edge over plain self-consistency never reaches significance; LCF's logic-validity direction is real but weak, is not controllable by the tested interventions, and significantly lowers the probability margin on two of four models. A reader choosing between the two mechanisms would trust the output-side aggregation more, at least at the 7-8B model sizes and small evaluation budgets tested.","feed_headline":"Independent rerun: aggregation reproduces; logic steering doesn't","feed_subtitle":"The probability-voting method matches its published numbers; the logic-subspace edit helps only one of four models.","key_machinery":"The argument is carried by two concrete mechanisms at opposite ends of the model. RPC is an aggregation rule over K sampled chain-of-thought paths: it weights each path by its mean token probability, combines the weighted votes with self-consistency, and prunes a low-probability component using a Weibull mixture. LCF trains projectors of shape $d \\to 2048 \\to 1024$ that split a hidden state into content and logic vectors, a cross-attention decoder that reconstructs the state, and an InfoNCE contrastive objective that pulls logic vectors of valid conclusions together; at inference the logic vector is shifted toward the 'valid' region with a scale $\\eta$ and decoded back into the residual stream. The paper also uses a held-out linear probe to measure how strongly logic validity and a semantic control attribute are linearly encoded; the probe's 0.82-versus-0.95 contrast is the evidence that the logic signal is decodable but weak, and the additive-intervention sweep is the evidence that it is not controllable.","core_discovery":"On its own terms, the paper's central claim is that the two reliability mechanisms do not have the same status under independent testing. RPC's published pipeline, run on the original authors' reasoning paths, reproduces the reported grid nearly exactly (accuracy/ECE 26.15/12.32 versus the paper's 26.11/12.37, with the perplexity and self-consistency baselines also matching), and on four new domains plus three public benchmarks its difference from self-consistency is never significant (paired $p \\ge 0.28$), with one K=32 lead of +2.5 accuracy points on BIRD reversing to -0.25 when the sample is enlarged to n=200. LCF's premise is partially confirmed: a held-out linear probe separates valid from invalid conclusions at 0.82 accuracy at the best sub-layer, versus 0.95 for a semantic control, showing a real but weak and localized logic-validity direction. But shifting along that direction—whether by the trained projector, a supervised best-layer direction, or a random same-norm direction—does not move behavior, the DeltaProb gain on Qwen3 is not significant ($p=0.56$), and the same recipe significantly reduces DeltaProb on Llama-2 and Mistral. The paper concludes that test-time aggregation is the safer intervention, while logic-representation editing is not a reliably reproducible, model-agnostic reliability fix at this scale.","pith_inferences":["A step the paper leaves implicit: the benefit of RPC-style aggregation appears only when path probabilities carry information about correctness, so one could measure the confidence-correctness correlation on a task before adopting aggregation.","Because the valid-conclusion labels in the LCF study were machine-generated and only 28% were accepted by an independent judge, the 0.82 separability may overestimate the logic signal; human-validated labels could yield a weaker or stronger estimate.","At 7-8B scale, model capability appears to dominate intervention effects: the only model helped was the one that already had the target competence, suggesting that logic-steering research should either move to stronger models or focus on unlocking existing capability.","If the original LCF recipe, including its discriminator, were released and restored controllability, the asymmetry would narrow; testing that is a direct way to decide whether the negative result is about LCF itself or about reproducibility."],"forward_implications":["If the asymmetry holds, users can apply RPC-style aggregation to 7-8B models without fear of significant accuracy loss, but should not expect a large gain over self-consistency on new domains.","LCF-style logic editing should not be treated as a model-agnostic reliability fix: at this scale it can significantly reduce the model's confidence in valid options, even where a decodable logic direction exists.","A reported gain that rests on an unreleased validity discriminator cannot be independently checked; any claim of near-perfect control needs the discriminator or an auditable substitute before it can be reproduced.","Separability of a concept direction does not imply controllability: a direction can be linearly decodable yet adding or subtracting it changes behavior no more than a random direction does.","The BIRD budget sweep shows that apparent gains at small sample sizes can vanish with more data, so aggregation advantages should be reported with power bounds."],"supporting_citations":[{"why":"Supplies the RPC aggregation method and the original evaluation grid that this paper reproduces.","marker":"Zhou et al., 2025"},{"why":"Supplies the LCF projector, contrastive, and intervention pipeline that this paper re-implements from scratch.","marker":"Wu et al., 2025"},{"why":"Defines self-consistency, the baseline that RPC must beat and the comparison anchor for the new domains.","marker":"Wang et al., 2023c"},{"why":"Provides the LFUD fallacy corpus used as the shared task for both RPC and LCF experiments.","marker":"Li et al., 2024"},{"why":"Provides the K-CAST conditional activation steering recipe tested in the model-agnostic logic-steering attempt.","marker":"Valentino et al., 2026"},{"why":"Provides the BIRD text-to-SQL benchmark used for the RPC sample-budget scaling study.","marker":"Li et al., 2023a"}],"fun_headline_variants":["Aggregation reproduces; logic steering fails on 3 of 4 models","LLM reliability: voting works, logic editing doesn't","Independent test: probability aggregation holds, logic edits don't","Rerun shows voting method solid, logic steering shaky"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's negative verdict on LCF assumes that its reimplementation matches the original unpublished recipe—the projector size, training objective, layer range, intervention strength, and validity judge—so if any of those details differ, the failures could be artifacts of the reimplementation rather than properties of the method.","fun_headline_variants_meta":{"raw":{"variants":["Aggregation reproduces; logic steering fails on 3 of 4 models","LLM reliability: voting works, logic editing doesn't","Independent test: probability aggregation holds, logic edits don't","Rerun shows voting method solid, logic steering shaky"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001083,"raw_usage":{"total_tokens":4655,"prompt_tokens":1198,"completion_tokens":3457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":814,"completion_tokens_details":{"reasoning_tokens":3386}},"tokens_in":814,"tokens_out":3457,"duration_ms":25683,"temperature":1.0,"reasoning_tokens":3386,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:33:21.986466+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the original LCF code, or a version whose projector, contrastive objective, edited layers, and validity discriminator are supplied by the original authors, on Llama-2 and Mistral with the same 204-item fallacy task: if DeltaProb rises significantly on either model instead of falling, the paper's central negative claim is false. Likewise, a further independent dataset draw on BIRD at K=32, which has already turned a +2.5 lead into -0.25 when the sample grew from n=80 to n=200, would settle whether any aggregation edge exists.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RPC aggregation method and the original evaluation grid that this paper reproduces."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the K-CAST conditional activation steering recipe tested in the model-agnostic logic-steering attempt."}],"review_version":1}