{"id":"48df39d5-2459-40e7-aaf5-d876ad776b47","arxiv_id":"2506.01116","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"ChemAU adds a position penalty to token-level uncertainty estimates so that flagged reasoning steps are corrected by a fine-tuned chemistry model, reporting improved accuracy on GPQA, MMLU-Pro, and SuperGPQA chemistry questions.","lead":"This paper presents ChemAU, a framework that pairs a general large language model with a fine-tuned chemistry model, using a position-weighted uncertainty score to decide when to inject domain knowledge during chain-of-thought reasoning. The authors report accuracy gains on three chemistry QA benchmarks, but the core uncertainty formula is internally inconsistent and no code or data is released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The adaptive uncertainty mechanism is numerically inert: with alpha = -0.08 and theta = -1.5, Eq. (2) flags every reasoning step in the paper's own example, so the claimed step-selective behavior cannot occur.","rationale":"The reader's REJECT is justified, and the most load-bearing defect is in the uncertainty rule itself rather than only in the undisclosed domain model. Eq. (2) with the stated hyperparameters cannot selectively flag steps: for valid token probabilities the max negative-log term is nonnegative, the position penalty is nonpositive, and a threshold of -1.5 is below every attainable score in the paper's short-chain examples. The threshold therefore never binds. This is an internal inconsistency, not merely a disagreement with existing consensus, and it directly undermines the central claimed contribution of adaptive, step-level identification of chemistry knowledge gaps. If the threshold is never active, the framework as actually parameterized is always invoke the domain model, and the empirical improvements, if real, cannot be attributed to uncertainty estimation; the ablation comparing step-wise versus whole-chain processing tests decomposition, not selection. I also note that the sign of the position term contradicts Section 3.3's statement that earlier steps should be judged more strictly. The undisclosed training data for the 1.5B domain model remains a second-order concern: it affects transferability and reproducibility, but the formula defect is decisive on the manuscript's own terms. A minimal arithmetic check on the example from Figure 3 would settle the issue; no full rerun of the benchmark is needed. Because the central mechanism is internally inconsistent, the paper does not support its central claim as written, so I concur with REJECT. The reader's stated weakest assumption focuses on domain-model data overlap, but their rationale contains the same Eq. (2) concern, hence partial agreement.","tokens_in":15334,"tokens_out":5477,"duration_ms":67897,"concrete_test":"Take the three-step reasoning chain from Section 3.3 and Figure 3, assign all p_ij = 1, the most favorable case for a low uncertainty score, and evaluate Eq. (2) with alpha = -0.08, L_R = 3, and theta = -1.5. Report U_1, U_2, and U_3. If all three exceed theta, the threshold is inert on the paper's own motivating example. To make the test conclusive, also re-run the Figure 5 comparison under an always-trigger policy; if reported accuracy is unchanged, the adaptive component contributes nothing and the central claim fails.","verdict_should_be":"REJECT","load_bearing_attack":"Eq. (2) in Section 3.1 defines U_i = max_j -log(p_ij) + alpha(L_R - i), with p_ij treated as token probabilities, and Appendix B sets alpha = -0.08, theta = -1.5. Since -log(p_ij) >= 0 for probabilities in (0,1], the first term is nonnegative and the position term alpha(L_R - i) is nonpositive. For the paper's own 3-step example in Section 3.3 and Figure 3, every step has U_i >= -0.16, which is greater than theta = -1.5 for all valid token probabilities. Thus the condition U_i > theta is automatically satisfied, and no reasoning step can ever be classified as certain. For any chain with L_R <= 19 the same vacuity holds; for longer chains the threshold only begins to bind at the extreme where every token has probability 1. Additionally, the design claim in Section 3.3 is that earlier reasoning steps should receive stricter scrutiny, but with alpha < 0 the penalty alpha(L_R - i) is largest in magnitude for the earliest step, lowering its uncertainty; the sign is backwards. Thus Eq. (2) as parameterized cannot implement the adaptive, step-selective trigger the paper claims. The reported ChemAU gains therefore cannot be attributed to identifying high-uncertainty steps; the mechanism collapses to always consulting the domain model in the demonstrated setting, and the comparison against Max(-log p) in Figure 5 does not test selective detection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ChemAU, a framework that augments a general LLM's chain-of-thought reasoning on chemistry problems with an adaptive, position-weighted uncertainty estimator. The estimator is intended to identify specific reasoning steps that are likely to contain chemistry knowledge errors, after which a fine-tuned 1.5B chemistry domain model is invoked to supply corrected knowledge and the LLM regenerates the remainder of the chain. The method is evaluated on chemistry subsets of GPQA, MMLU-Pro, and SuperGPQA with three base LLMs (Qwen2.5-7B-Instruct, LLaMA-3-8B-Instruct, DeepSeek-R1-Distill-Qwen-14B), reporting accuracy gains over vanilla CoT and RAG baselines, alongside ablations on the domain model and on step-wise versus chain-level uncertainty.","tokens_in":15606,"tokens_out":6965,"duration_ms":67529,"significance":"If the claimed adaptive uncertainty mechanism genuinely identified the steps that need domain knowledge, ChemAU would be a useful contribution to LLM reasoning in specialized scientific domains. The paper includes a multi-model, multi-dataset evaluation and two ablation studies, which are welcome. However, the central uncertainty formula as parameterized is numerically incapable of performing the claimed step-selective detection: with alpha = -0.08 and theta = -1.5, the trigger condition U_i > theta is satisfied for every step in the paper's own example and for essentially any chain of realistic length. The reported accuracy gains therefore cannot be attributed to adaptive, step-selective uncertainty estimation, and the paper's central claim is not supported by its evidence.","major_comments":[{"comment":"With alpha = -0.08 and theta = -1.5, the decision rule U_i > theta is vacuous for any practical reasoning chain. Since -log(p_ij) >= 0 for probabilities in (0,1], U_i is bounded below by alpha(L_R - i). In the paper's own 3-step example, U_1 >= -0.16, U_2 >= -0.08, and U_3 >= 0, so all three steps are above theta = -1.5 and are flagged as uncertain. For any chain with L_R <= 18 steps, the minimum possible value of U_i is alpha(17) = -1.36, which is still greater than -1.5, so every step is flagged. The threshold can only begin to bind for chains of 20 or more steps, and only when every token in the first step has probability exactly 1. Thus the mechanism does not implement the claimed step-selective triggering; it always consults the domain model, and the comparison against Max(-log p) in Figure 5 does not test selective detection.","section":"Section 3.1, Eq. (2), Appendix B"},{"comment":"The paper states in Section 3.3 that 'the certainty for earlier reasoning steps should be stricter' and that the method 'assigns higher uncertainty values to reasoning steps at earlier positions'. Equation (2) with alpha = -0.08 produces the opposite: for i < L_R, the additive term alpha(L_R - i) is negative and is largest in magnitude for the smallest i, so earlier steps receive a larger negative penalty and therefore a lower U_i. The sign of the position term is thus inconsistent with the design rationale stated in the paper.","section":"Section 3.3 versus Section 3.1"},{"comment":"The specialized chemistry domain model is created by instruction fine-tuning Qwen2.5-1.5B-Instruct on an 'author-constructed chemistry knowledge dataset', but the dataset's size, provenance, construction process, and quality controls are not described. Without this information, and without any analysis of potential overlap between the domain model's training data and the evaluation benchmarks, the reported accuracy improvements cannot be attributed to the uncertainty-driven correction mechanism rather than to training-data leakage or to the domain model's general knowledge. This is a load-bearing gap for the paper's attribution claim.","section":"Section 3.2 and Section 4.1"}],"minor_comments":[{"comment":"The paper uses 'logit' and 'probability' interchangeably. The values 0.72, 0.81, and 1.0 in Figure 1 are probabilities, not logits (which can be any real number). Please clarify whether p_ij in Eq. (2) denotes token probabilities or logits, and use consistent terminology throughout.","section":"Figure 1 and Section 3.1"},{"comment":"The paper does not specify whether the step index i starts at 0 or 1, which changes the value of L_R - i. Please state the indexing convention.","section":"Eq. (2)"},{"comment":"The legend in Figure 5 uses 'Max(p)' while the text and Eq. (1) define 'Max(-log p)'. This inconsistency makes the baseline hard to interpret.","section":"Figure 5"},{"comment":"The main text reports that removing the domain model causes accuracy to drop 'up to 16.39%', while the caption of Figure 6 states 'up to 14.87%'. These numbers should be reconciled.","section":"Section 5"},{"comment":"The claim that ChemAU is 'the first framework to introduce a model collaboration strategy for chemistry reasoning tasks' is too strong and is not supported by a systematic literature search; it is likely to be contested.","section":"Section 1"}],"recommendation":"reject","confidential_remarks":"The core problem is that the reported hyperparameters (alpha = -0.08, theta = -1.5) make the uncertainty trigger mathematically invalid, so the experimental results as reported do not test the adaptive mechanism. Even if the parameters were corrected, the missing details of the domain-model training set would need to be supplied and the experiments rerun. The current manuscript does not support its central claim, and I would recommend rejection, though a substantially revised version with corrected formulas and complete experimental details could merit reconsideration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the paper's empirical story is plausible but its central mechanism, as written, cannot implement selective triggering, and there is no code/data or description of the domain-model training set to fall back on. The genuinely new piece is the framework combining step-wise uncertainty estimation with a fine-tuned chemistry domain model to inject corrections into an LLM reasoning loop. That is a reasonable engineering idea, and the evaluation covers three LLMs and three datasets with the right ablations: with vs without the domain model, and step-wise vs chain-level injection. If the numbers are accurate, always consulting a domain model per step may be a useful recipe.\n\nThe problem is Eq. (2). With alpha = -0.08 and theta = -1.5, the position penalty is nonpositive and -log(p) is nonnegative, so every reasoning step in a chain of up to 19 steps has U_i well above -1.5. The condition U_i > theta is satisfied for every step; no step can ever be classified certain. Furthermore, the paper's design claim in Section 3.3 is that earlier steps should receive stricter (higher) uncertainty, but a negative alpha reduces the uncertainty of earlier steps the most. The sign is backwards. The mechanism is therefore numerically inert and contradicts the stated design. That is a load-bearing flaw: the paper's contribution is supposed to be adaptive, step-selective uncertainty estimation, and Figure 5 compares against Max(-log p) on that basis. As written, the comparison tests an always-trigger policy, not a selective one.\n\nOther soft spots: the 'logit rise' phenomenon rests on one anecdotal example; the domain model's training data is undisclosed; there are no error bars, significance tests, code, or data. The reference list is appropriate and the writing is clear. The limitation paragraph honestly notes that the method only applies to open-source models.\n\nWho this is for: practitioners building chemistry QA pipelines may find the injection strategy useful, but as a scientific claim about uncertainty estimation the paper is not supportable in its current form. I would not accept it for publication or send it to referees as is. I would encourage the authors to fix the parameterization (or reframe the method as always consulting the domain model), release the domain-model training data and evaluation code, and show that the threshold actually selects a subset of steps. A revised version with those changes deserves serious review.","headline":"The empirical idea is plausible, but the adaptive uncertainty formula as parameterized cannot trigger selectively, which collapses the central claim.","tokens_in":16177,"tokens_out":4150,"would_cite":false,"duration_ms":42715,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ChemAU improves LLM chemistry reasoning by scoring each reasoning step and triggering a fine-tuned chemistry model to correct uncertain steps.","keywords":["chemistry reasoning","uncertainty estimation","chain-of-thought","adaptive uncertainty","domain-specific model","knowledge supplementation","large language models"],"falsifier":"Run ChemAU on a chemistry benchmark whose questions have been checked not to overlap with the author-constructed training data for the domain model; if the accuracy gain over the base LLM largely disappears, the improvement comes from memorized benchmark content rather than from uncertainty-guided knowledge supplementation.","tokens_in":15111,"feed_emoji":"🧪","tokens_out":9216,"duration_ms":86757,"temperature":0.7,"pith_summary":"ChemAU is a reasoning framework that makes general-purpose large language models more reliable on chemistry questions. It has the model write out its chain of thought, scores each step for uncertainty, and whenever a step looks unreliable it sends that step to a small chemistry-specialized model, which returns the correct domain knowledge before the main model continues. The paper claims this uncertainty-triggered correction raises accuracy on chemistry questions from GPQA, MMLU-Pro, and SuperGPQA across three open-source LLMs, and improves the quality of uncertainty estimates themselves. A sympathetic reader should care because the approach targets the specific knowledge gaps that cause chemistry hallucinations rather than regenerating whole answers or retrieving broad background text.","feed_headline":"Step-level uncertainty checks lift LLM chemistry scores","feed_subtitle":"A position-weighted uncertainty rule triggers a specialist chemistry model only where reasoning steps go wrong.","key_machinery":"The load-bearing mechanism is the adaptive uncertainty estimator. It scores the $i$-th reasoning step of a chain with $L_R$ steps as $U_i = \\max_j -\\log(p_{ij}) + \\alpha (L_R - i)$, using $\\alpha = -0.08$ and triggering further processing when $U_i > \\theta$ with $\\theta = -1.5$. The position term $L_R - i$ is what makes the estimate adaptive: the paper argues that chemistry tokens become artificially more probable as reasoning proceeds, so earlier steps need a different treatment from later ones. When a step triggers, the framework decomposes it into atomic knowledge points, asks a fine-tuned Qwen2.5-1.5B chemistry model to supply corrected knowledge, and reintroduces that knowledge with the confirmed earlier steps into the general LLM to regenerate the rest of the chain.","core_discovery":"The paper's central claim is that chemistry reasoning fails not because general LLMs cannot reason, but because they lack precise chemistry knowledge at particular steps, and that those steps can be located by a position-aware uncertainty signal. The authors observe that chemistry-specific tokens start with low logits and rise as the reasoning chain grows, so a fixed uncertainty threshold misclassifies steps. Their adaptive estimator adjusts each step's uncertainty by its position in the chain; a step above the trigger threshold is decomposed into atomic chemistry-knowledge points, corrected by a fine-tuned chemistry domain model, and then fed back into the general model to resume reasoning. The reported result is consistent accuracy gains across three backbones, with the largest improvement on MMLU-Pro for LLaMA-3, and ablation results indicating both the domain model and the step-wise trigger are necessary.","pith_inferences":["The same position-dependent logit inflation could occur in other terminology-heavy domains such as law or medicine, where a fixed-threshold uncertainty estimator would face the same failure mode.","A direct extension would swap the fine-tuned chemistry model for a public knowledge source or a tool-based verifier, while keeping the step-level uncertainty trigger.","Because the domain model's training corpus is not described, a natural next test is to rebuild ChemAU with a public chemistry dataset and check whether the accuracy gains persist.","For black-box LLMs, approximating token logits by sampling many continuations could test whether the adaptive position term still helps without direct logit access."],"forward_implications":["ChemAU improves chemistry accuracy across all three tested open-source models, with LLaMA-3 reaching 53.56% on MMLU-Pro, above the unaided 14B DeepSeek-R1 model.","Corrections are localized to single reasoning steps, so the framework can repair a chain without regenerating or rescoring the whole response.","The method requires token logits, so it applies to any open-source LLM and does not need multiple sampled generations for uncertainty.","The ablation results imply that telling the general model a step is uncertain is not enough; the externally supplied chemistry knowledge is what fixes the error.","Retrieval-augmented generation with broad knowledge is expected to trail ChemAU, because imprecise retrieved knowledge can mislead the reasoning chain."],"supporting_citations":[{"why":"Supplies the Max(-log p) uncertainty baseline that the adaptive estimator modifies and outperforms.","marker":"Manakul et al. [2023]"},{"why":"Defines the semantic-contribution weighting baseline that the paper argues fails on chemistry-specific tokens.","marker":"Duan et al. [2023]"},{"why":"Provides the Qwen2.5 models used both as reasoning backbones and as the base for the fine-tuned chemistry domain model.","marker":"Yang et al. [2024]"},{"why":"Contributes the GPQA benchmark's chemistry questions used for evaluation.","marker":"Rein et al. [2024]"},{"why":"Contributes the MMLU-Pro chemistry questions used for evaluation.","marker":"Wang et al. [2024]"},{"why":"Contributes the SuperGPQA chemistry questions used for evaluation.","marker":"Du et al. [2025]"},{"why":"Defines the chain-of-thought prompting baseline against which ChemAU's gains are measured.","marker":"Wei et al. [2022]"}],"fun_headline_variants":["Adaptive uncertainty rules repair LLM chemistry steps","ChemAU targets step-wise doubt to fix chemistry LLMs","Position-scaled uncertainty lifts LLM chemistry accuracy","Step-aware uncertainty triggers chemistry expert rescue","Reasoning-position uncertainty sharpens LLM chemistry"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the fine-tuned 1.5B chemistry model gives correct corrections whenever the uncertainty trigger fires, and that a flagged step really is a knowledge gap rather than a benign low-probability token.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive uncertainty rules repair LLM chemistry steps","ChemAU targets step-wise doubt to fix chemistry LLMs","Position-scaled uncertainty lifts LLM chemistry accuracy","Step-aware uncertainty triggers chemistry expert rescue","Reasoning-position uncertainty sharpens LLM chemistry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000141,"raw_usage":{"total_tokens":1142,"prompt_tokens":904,"completion_tokens":238,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":166}},"tokens_in":520,"tokens_out":238,"duration_ms":3053,"temperature":1.0,"reasoning_tokens":166,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:50:51.432839+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ChemAU on a chemistry benchmark whose questions have been checked not to overlap with the author-constructed training data for the domain model; if the accuracy gain over the base LLM largely disappears, the improvement comes from memorized benchmark content rather than from uncertainty-guided knowledge supplementation.","supporting_citations":[],"review_version":1}