{"id":"d16ee9fb-511a-4fbd-8a62-d5cda60b5f89","arxiv_id":"2510.09278","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A consistency reward parsed by a 7B LLM, plus a two-stage refine-then-monitor pipeline and reformulated easy questions, improves MCQ-RL accuracy-with-consistency in law and medicine (58.9 vs 51.4 average Acc+).","lead":"CLARITY trains legal and medical LLMs on multiple-choice questions by rewarding consistency between the model's chain-of-thought and its final answer, using a small 7B LLM as the judge instead of an expensive process-reward model. The method lifts a combined correct-and-consistent score by 7.5 points over standard RL and is the kind of cheap supervision signal that could change how expert-domain models are fine-tuned.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 7B consistency checker's precision is unmeasured; if it false-flags valid reasoning, the reward teaches style rather than reasoning.","rationale":"The reader's weakest_assumption and my analysis converge: the reliability of Qwen-2.5-7B as a consistency checker is the linchpin. The paper provides credible empirical scaffolding - code, ablations, reward-hacking analysis, and case studies - so I do not recommend rejection. However, the absence of a direct precision/false-positive measurement for the 7B checker leaves a real hole: the 94% detection rate is recall against DeepSeek, and DeepSeek-human agreement itself is only r=0.61. The oracle-replacement test directly targets whether checker noise changes final outcomes. Because this is an addressable empirical question, the reader's CONDITIONAL verdict already captures the appropriate stance, and my read does not move it.","tokens_in":26769,"tokens_out":5249,"duration_ms":506288,"concrete_test":"Re-run CLARITY 7B on JEC-QA and MedQA with the consistency reward computed by an oracle checker (e.g., DeepSeek-V3 or human labels on a subsample) instead of Qwen-2.5-7B, keeping all other hyperparameters and the two-stage pipeline identical. If final Acc+ or Cons% changes by more than 2 points on either domain, the Qwen checker's noise is consequential and the central claim is weakened; if results are within 2 points, the 7B checker is sufficient and the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a small 7B general-purpose LLM can supervise expert MCQ training through a consistency reward. For that to hold, the reward model's parse of 'believed-correct options' (Algorithm 1, lines 32-33) must be both accurate and non-gameable. The paper's evidence bounds recall, not precision: Appendix B.1 reports DeepSeek-V3-human agreement r=0.610, and Section 5.3 reports Qwen-2.5-7B detects 94% of DeepSeek-flagged inconsistencies. Neither number measures how often the 7B checker falsely flags consistent reasoning as inconsistent or mis-parses options in domain-specific phrasing (legal Chinese, medical English). The tiny-checker failure (Figure 5) shows reward hacking is a real failure mode; nothing in the paper shows the 7B checker is far enough from that regime. If its false-positive rate is nontrivial, the stage-2 reward (Algorithm 1 line 35) punishes valid reasoning, and the model can learn to satisfy the checker's heuristics rather than reason - making the approach a style-enforcer. This is the load-bearing soft spot, separate from the secondary framing issue that stage-2 also includes answer reward (Algorithm 1 line 34).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLARITY, a reinforcement-learning framework for multiple-choice question (MCQ) training in expert domains such as law and medicine. CLARITY replaces expensive process reward models with a small general-purpose LLM (Qwen-2.5-7B) that parses the policy's believed-correct options from its chain of thought and penalizes mismatches between those parsed judgments and the final answer. The training uses a two-stage refine-then-monitor pipeline with a structure reward in stage 1 and a consistency reward plus answer reward in stage 2, together with a dynamic data-reformulation strategy that deconstructs easy examples into propositions and regroups them into harder questions. Experiments on JEC-QA and MedQA report improvements in a compound \"Acc+\" (correct-and-consistent) metric of 7.5 percentage points and in consistency of 16.5 percentage points, plus gains on out-of-domain open-ended tasks and in human evaluations of coherence and professionalism. The authors claim this shows that small, general-purpose LLMs can guide expert-model training using only consistency supervision, without large teacher models or expert-annotated corpora.","tokens_in":26974,"tokens_out":6820,"duration_ms":60424,"significance":"If the central claim is sustained, CLARITY is a valuable contribution: it offers a relatively cheap, domain-agnostic way to inject reasoning-quality supervision into MCQ reinforcement learning, where outcome-only rewards are known to produce superficially correct but internally inconsistent chains of thought. The paper includes useful analyses of failure modes (over-exclusion, over-selection, dissociated answers), open-source code, and human evaluation. However, the evidence as presented does not fully support the strongest claims. The headline accuracy gain is a compound metric, not raw accuracy; the reliability of the 7B consistency checker is measured only indirectly; and the pipeline still relies on DeepSeek-V3 for data augmentation and evaluation, undercutting the \"small-model-only\" framing. These issues are load-bearing because the method's value depends on the consistency reward being both accurate and non-gameable.","major_comments":[{"comment":"The headline \"accuracy +7.5%\" is the compound Acc+ (correct and consistent) metric, not raw accuracy. In Table 1, CLARITY-7B's average raw accuracy is 61.3 vs. 61.1 for Standard RL (+0.2 pp), and legal accuracy drops from 58.5 to 56.4. The abstract and §5.1 should state explicitly that the 7.5-point gain is in Acc+, and the paper should report raw accuracy and significance separately.","section":"Abstract, §5.1, Table 1"},{"comment":"The consistency reward's reliability is not established. The paper reports that Qwen-2.5-7B detects 94% of DeepSeek-V3-flagged inconsistencies, and that human–DeepSeek agreement is only r=0.610 (Pearson). Neither number measures the 7B checker's precision or false-positive rate on valid reasoning. If the 7B checker false-flags legitimate legal/medical phrasing, stage-2 reward (line 35) punishes correct reasoning and the policy can learn to satisfy the checker's heuristics rather than reason. Please provide a precision/false-positive measurement against human-annotated labels and demonstrate robustness to paraphrasing and domain-specific language.","section":"Algorithm 1 lines 32–35; §5.3; Appendix B.1"},{"comment":"The claim that CLARITY uses only a small, general-purpose LLM is not accurate for the full pipeline. Section 4 states that DeepSeek-V3 is used for \"data augmentation and consistency evaluation,\" and Algorithm 1 line 4 invokes LLM_Refine for reformulation. Thus the method still requires a large LLM for data reformulation; only the consistency reward itself is 7B. This should be scoped carefully in the abstract and conclusion, or the augmentation component should be ablated to show what can be achieved with the 7B checker alone.","section":"Section 4; Algorithm 1 line 4"},{"comment":"No error bars, multiple seeds, or significance tests are reported. Given that the raw-accuracy gap between CLARITY-7B and Standard RL is only 0.2 pp on average, the headline gains in Acc+ and consistency could reflect run-to-run variance. Please report at least 3 seeds with standard deviations and, where possible, paired significance tests.","section":"Tables 1–2, §5"},{"comment":"The sentence \"models that even surpass large commercial systems like GPT-4o\" is not supported by the tables. In Table 1, CLARITY-7B has lower average Acc+ (58.9 vs. 61.3) and much lower medical Acc+ (63.3 vs. 83.0) than GPT-4o; in Table 2 the average Acc+ is 52.0 vs. 52.2. The claim should be tempered or substantiated with a specific comparison.","section":"§5.1, Tables 1–2"}],"minor_comments":[{"comment":"Typo: \"to to better exploit\" should be \"to better exploit.\"","section":"Abstract"},{"comment":"Typo: \"reinforencement\" should be \"reinforcement.\"","section":"Related Work"},{"comment":"The metric \"Acc+\" is defined only in §4; please define it in the table caption or a footnote, and clarify that it is not the same as raw accuracy.","section":"Table 1 caption"},{"comment":"The axes of Figures 5 and 6 are not labeled. In particular, the y-axis of Figure 6 (response frequency/length) should specify what is plotted.","section":"Figures 5–6"},{"comment":"The table captions \"True Value\" and \"False Value\" are awkward; use \"Correct\" and \"Incorrect\" or similar. Also, the absence of an answer-reward row for Stage-1 is confusing given Algorithm 1 lines 23–24; consider aligning the notation.","section":"Table 7"},{"comment":"Typo: \"underoptimal\" should be \"suboptimal.\" Also, \"Rouge-L\" should be \"ROUGE-L.\"","section":"Appendix C.1"}],"recommendation":"major_revision","confidential_remarks":"The central idea is promising, but the evidence needs strengthening before the paper can be accepted. The main issues are the accuracy-metric conflation, the unmeasured precision of the 7B consistency checker, the undisclosed reliance on DeepSeek-V3 for data augmentation, and the absence of variance/error analysis. The overclaim relative to GPT-4o should also be corrected. I would lean toward asking for a revised version with additional experiments (precision evaluation, multiple seeds) rather than rejection, because the core contribution is defensible and the failure modes are acknowledged in the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core idea: use a small general-purpose LLM to parse which options the policy's CoT endorses, penalize mismatch with the final answer, and wrap that in a two-stage refine-then-monitor pipeline. That's a genuinely cheap reasoning-quality signal for MCQ RL, and it's the paper's main contribution.\n\nThe paper does several things well. The pilot study documenting that standard RL raises accuracy but drops consistency (7% to 31% logical fallacies) is a useful data point. The tiny-checker ablation (0.5B/1.5B) shows reward hacking is real and motivates the two-stage design. They include ablations for the reformulation pipeline, a rule-based consistency failure case, case studies, and open source code. The dynamic data reformulation — deconstructing options, polishing with an LLM, regrouping by pass rate — is a reasonable extension of curriculum ideas, not a huge leap.\n\nThe soft spots are real but not fatal. First, the headline '+7.5% accuracy' is the Acc+ compound metric; raw average accuracy is +0.2pp over standard RL, and legal raw accuracy actually drops from 58.5 to 56.4. That framing should be revised even if Acc+ is the right headline metric for the paper's thesis. Second, there are no error bars, seeds, or significance tests, so the stability of the gains is unknown. Third, the consistency reward is LLM-judged; human agreement with DeepSeek-V3 is r=0.61, which is moderate, and the paper only reports recall (94% of DeepSeek-flagged inconsistencies caught) without measuring precision. The stress-test concern is legitimate: if the 7B checker false-flags valid but unusual reasoning, the reward becomes a style enforcer. That's the biggest unaddressed risk and it's fixable with a small annotation study. Fourth, the title and abstract say 'consistency alone' but stage-2 includes an answer reward (Algorithm 1, line 34), so it's consistency + outcome. Minor: the text claims CLARITY surpasses GPT-4o, but Table 1's average Acc+ is 58.9 vs 61.3; that's an easy correction.\n\nWho's this for: anyone doing RL on MCQ-heavy domains who wants a cheap way to keep reasoning coherent without a PRM. It's a solid, useful paper, not a landmark, and its weaknesses are addressable. I'd send it to peer review, asking for multi-seed runs, raw accuracy reported separately, and a precision measurement for the consistency checker before acceptance.","headline":"A cheap consistency-reward recipe for MCQ RL that mostly delivers, but the headline accuracy gain is the compound metric, the consistency checker's precision is unmeasured, and 'consistency alone' overstates the stage-2 reward.","tokens_in":27594,"tokens_out":3133,"would_cite":true,"duration_ms":31076,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that supervising a model only on whether its chain of thought agrees with its final answer — using a small, general-purpose language model — produces multiple-choice reasoning that is both more consistent and more accurate","keywords":["consistency-aware RL","multiple-choice question training","reasoning consistency","reward hacking","process reward model","data reformulation","law and medicine LLMs","small-model supervision"],"falsifier":"Take 100 responses from a CLARITY-trained model on held-out MCQs; have human experts mark each response as consistent or not, and compute the agreement between human labels and the small checker's consistency verdict. If agreement falls far below the paper's reported human-reference-model correlation (r ≈ 0.61), or if a stronger checker flags many consistent-but-wrong answers that received full reward, the core mechanism — consistency as a faithful proxy for reasoning quality — is not doing the work.","tokens_in":26541,"feed_emoji":"🧠","tokens_out":5257,"duration_ms":48559,"temperature":0.7,"pith_summary":"Standard reinforcement learning on multiple-choice data can raise final-answer accuracy while degrading the reasoning behind it; the paper reports logical inconsistency rising from 7% to 31% in a pilot. CLARITY is proposed as a remedy: a small, general-purpose language model reads the model's chain of thought, extracts the options the reasoning endorses, and penalizes any mismatch with the final answer. A two-stage refine-then-monitor scheme and a reformulation of easy questions into harder ones push consistency and accuracy up, with reported gains of 16.5% and 7.5% over baselines. The broader claim is that reasoning consistency alone — cheap to check, with no domain expertise — is enough of a signal to train expert models in law and medicine and to transfer to open-ended tasks.","feed_headline":"Consistency reward lifts MCQ reasoning 16.5% and accuracy 7.5%","feed_subtitle":"A small 7B model checks whether a model's reasoning and final answer agree—no expert labels needed.","key_machinery":"The load-bearing object is the consistency-aware reward: a separate small LLM parses the believed-correct options from the policy's chain of thought, and the reward penalizes any mismatch between that parsed set and the final answer. Around it, CLARITY wraps a two-stage pipeline — stage one using a structural reward to force transparent option-wise reasoning, stage two dropping that constraint while applying the consistency reward plus a strict answer reward — and a dynamic data reformulation that deconstructs easy questions into propositions, polishes and diversifies them, and regroups them into new harder questions. The consistency signal is what makes a weak checker sufficient: it only ne","core_discovery":"The paper's central claim is that the consistency of a model's judgment about each option — not the factual correctness of the reasoning itself — is a sufficient reward signal for MCQ reinforcement learning in expert domains. Using only a 7B general-purpose LLM to parse the believed-correct options from the chain of thought and penalize disagreement with the final answer, CLARITY reports a 16.5% improvement in response consistency and a 7.5% improvement in reliable accuracy over outcome-based RL baselines. The paper also claims that this consistency-only supervision transfers to out-of-domain open-ended benchmarks and improves human-rated professionalism, coherence, and readability. The larg","pith_inferences":["If consistency is the causal mechanism, the same supervision should transfer to open-ended tasks with a checkable final answer, such as evidence-based fact-checking or code with tests, where a small model could check that intermediate claims support the final claim; the paper only demonstrates MCQ-to-open-ended transfer.","Because weaker checkers fail badly, a natural extension is to distill the 7B checker's consistency judgments into a smaller classifier; if the distilled checker preserves the 7B model's detection rate, the training overhead could drop well below the paper's reported 8-hour-to-over-1-day increase.","Measured against human consistency labels rather than the same kind of checker that provides the reward, the reported 16.5% gain could shrink or grow; a direct comparison would clarify whether the reward teaches genuine logical coherence or checker-pleasing phrasing.","The data reformulation strategy's alpha schedule is a coarse knob; one could test whether adaptively varying the number of options or mixing domains further improves generalization, which the paper lists as future work."],"forward_implications":["Outcome-only RL on MCQs can make answers more accurate while making reasoning less consistent, so accuracy alone is an unsafe training signal.","A 7B general-purpose LLM can substitute for large or domain-trained process reward models when the supervision target is consistency rather than step-level correctness.","Consistency supervision improves both consistency (16.5%) and reliable accuracy (7.5%), where 'reliable' means the final answer is both correct and consistent with the reasoning.","Trained models generalize to open-ended out-of-domain tasks, indicating the benefit is not mere memorization of MCQ format.","The two-stage refine-then-monitor design prevents the reward hacking that appears when the checker is too weak (0.5B/1.5B models)."],"fun_headline_variants":["Consistency-only RL lifts expert reasoning 16.5% in MCQs","A 7B model supervises expert LLMs via reasoning consistency","No PRMs: consistency check alone improves MCQ accuracy 7.5%","Small model guides experts by rewarding consistent reasoning","Consistency as reward: cheaper expert training, +16.5% quality"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole method hinges on the small 7B checker reliably extracting the options the policy's reasoning endorses and doing so without drifting on domain-specific phrasing; if it misreads those endorsements, the reward punishes style rather than bad reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Consistency-only RL lifts expert reasoning 16.5% in MCQs","A 7B model supervises expert LLMs via reasoning consistency","No PRMs: consistency check alone improves MCQ accuracy 7.5%","Small model guides experts by rewarding consistent reasoning","Consistency as reward: cheaper expert training, +16.5% quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000423,"raw_usage":{"total_tokens":1999,"prompt_tokens":724,"completion_tokens":1275,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":468,"completion_tokens_details":{"reasoning_tokens":1183}},"tokens_in":468,"tokens_out":1275,"duration_ms":10635,"temperature":1.0,"reasoning_tokens":1183,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:28:42.470481+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take 100 responses from a CLARITY-trained model on held-out MCQs; have human experts mark each response as consistent or not, and compute the agreement between human labels and the small checker's consistency verdict. If agreement falls far below the paper's reported human-reference-model correlation (r ≈ 0.61), or if a stronger checker flags many consistent-but-wrong answers that received full reward, the core mechanism — consistency as a faithful proxy for reasoning quality — is not doing the work.","supporting_citations":[],"review_version":1}