{"id":"3af28498-bf8f-4b17-8b21-07ed540b0141","arxiv_id":"2509.09705","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Small open-source LLMs give inconsistent answers on 20% to 50% of multiple-choice questions even at low temperature, while 50B-80B models are consistent on over 90% of questions.","lead":"This study repeated the same multiple-choice questions 10 times across 26 open-source language models and measured how often each model gives the same answer. Small models (2B-8B parameters) were consistent on only about half to two-thirds of questions at low temperatures, while larger models were consistent on most questions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Medium-model consistency advantage may be inflated by benchmark contamination; the paper's own limitations concede this, so the scale-based conclusion is not yet established.","rationale":"The reader's weakest assumption—that contamination and rogue memorization are negligible—is the most load-bearing point. The paper's own Sections 6 and 7 acknowledge the issue, but the central scale comparison is not robust to it. My proposed test is the natural next step: filter contaminated items and recompute the medium-model S/T values. If the high consistency persists on clean items, the conclusion is supported; if not, the main claim needs substantial qualification. The reader's CONDITIONAL verdict already requires addressing this and other issues, so no verdict change is needed. I agree with the reader's assessment and would keep the verdict conditional pending the contamination check.","tokens_in":16739,"tokens_out":6729,"duration_ms":73875,"concrete_test":"Run a contamination filter such as LLMsanitize (Ravaut et al., 2024) or exact/n-gram overlap against the reported training corpora for Llama-3.3-70B, Qwen2.5-72B, and Mixtral-8x7B on all MMLU-Redux prompts. Recompute S/T, RWS, and average accuracy on only the flagged-clean subset. If S/T remains at or above 95% on clean items, the contamination concern is resolved; if it falls toward the 50%-80% range, the scale-based consistency claim collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that limited consistency is chiefly a small-model problem rests on the assumption that the near-ceiling S/T values of the three medium models reflect genuine answer stability rather than memorized reproduction. Section 6 lists benchmark contamination as a key limitation and the Conclusion repeats that it 'may have had significant impacts,' but no quantitative check is provided. Because Llama-3.3-70B, Qwen2.5-72B, and Mixtral-8x7B were trained on internet-scale data and MMLU-Redux is derived from the extremely widely copied MMLU questions, many items are plausibly present in training. Rogue memorization would cause the model to emit the same correct letter on every repetition, mechanically inflating S/T and RWS. This is not an external objection: the only evidence for the size-based conclusion is Table 1's medium-model S/T of 87%-99%, which is exactly the quantity contamination would inflate. Without a contamination-filtered recomputation, the 50B-80B versus 2B-8B contrast is confounded, and the paper's headline 'much higher consistency for medium models' cannot be taken as established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies answer consistency of open-source LLMs when the same multiple-choice question is repeated 10 times. It proposes a definition of c-answer consistency through an oracle-guessing analogy, operationalizes it as a SURE/UNSURE classification (9 or 10 identical choices out of 10), and reports a two-number summary RWS | S/T for each model and temperature. Experiments cover MMLU-Redux and MedQA across 26 small (2B-8B) and medium (50B-80B) models at temperatures 0.3, 0.7, and 1.0. The central empirical claims are that small models produce consistent answers on only about 50%-80% of questions at low temperature, that medium models show much higher consistency (about 87%-99% S/T on MMLU-Redux), and that the accuracy of consistently answered questions (RWS) correlates with overall accuracy, especially at low temperature. The paper also introduces consistency plots as a visualization tool.","tokens_in":17108,"tokens_out":8530,"duration_ms":85212,"significance":"If the reported consistency range for small models is accurate, the paper provides practically important evidence that single-run benchmark scores can mask substantial trial-to-trial variability in small open LLMs. The RWS|S/T representation and the consistency plot are simple and potentially useful tools for reliability evaluation, and the use of open models avoids closed-API confounds such as caching. The paper explicitly reports machine-generated raw repetition data rather than fitting a model to a target. However, the headline scale-based conclusion and some of the aggregate statistics are currently not established because of data-reporting inconsistencies and because the medium-model S/T values, which carry the size comparison, are exactly the values most vulnerable to the benchmark-contamination confound that the paper itself acknowledges in Section 6.","major_comments":[{"comment":"The empirical tables contain arithmetic inconsistencies that make the reported RWS/S/T values unreliable as printed. In Table 1, the Llama-3-8B row at t=0.3 lists SURE&right=43%, UNSURE&right=18%, UNSURE&wrong=29%, and SURE&wrong=104%, which is impossible and does not sum to 100%. In Table 2, BioMistral-7B at t=1.0 has SURE&right=7% and SURE&wrong=28%, so RWS should be 7/(7+28)=0.20, but the table reports 0.02. Because RWS and S/T are the paper's central quantities, the tables must be regenerated and checked before the empirical claims can be assessed.","section":"Tables 1-2"},{"comment":"The aggregate S/T report is inconsistent with Table 1. The text states that for 'all 9 small models' at t=0.3 the average S/T is 69% with standard deviation 11%, implying a range of 58%-80%. The nine small-model S/T values in Table 1 at t=0.3 are 53, 79, 74, 78, 78, 63, 52, 78, and 24; their average is 64%, not 69%. The reported value is obtained only by silently dropping Granite-3.1-2b-base, the lowest cell. Similarly, Section 4.3 says the granite S/T range is 52%-75% after ignoring that model, but Table 1 shows 78% for two granite instruct models. If the outlier is excluded, the exclusion must be stated and justified; otherwise the aggregate and the '50%-80%' range need revision.","section":"Section 4.5"},{"comment":"The paper's own limitations section concedes that benchmark contamination 'may have inflated the number of consistently corrected answers due to rogue memorization, especially for medium models.' This is not a peripheral caveat: the sole evidence for the claim that 'limited consistency is mostly an issue of small models' is the 96%-99% S/T of the three medium models in Table 1, and contamination would mechanically inflate exactly that quantity if the model has memorized the benchmark item. The paper does not provide a contamination-filtered recomputation, a memorization diagnostic, or any analysis separating stable generation from memorized reproduction. Without such a check, the size-based conclusion is confounded. A concrete test is needed, for example removing items that a model can reproduce from a masked or paraphrased prompt, or applying an existing contamination audit to MMLU-Redu","section":"Sections 6-7"},{"comment":"The derivation does not establish the stated equivalence between 'oracle guessing correctly at a 0.99 rate' and 'answering at least 9 of the 10 repetitions with the same choice.' The binomial calculation concerns the number of correct repetitions p, not the number of identical choices. Later, SURE is defined by identical answers, which may be wrong. The jump from 'correct with probability 0.99' to 'same letter 9 or 10 times' is informal, so the '0.99-consistency' label is not justified by the stated oracle model. Please either give a precise probabilistic model (e.g., a latent-answer model with probability c of emitting the same latent choice) or rename the threshold to avoid implying a false equivalence.","section":"Section 3.1, Eqs. (1)-(3)"}],"minor_comments":[{"comment":"Typo: 'MMLM-Redux' should be 'MMLU-Redux'.","section":"Introduction"},{"comment":"The reference mapping is garbled: Mixtral-8x7b-instruct is attributed to Qwen et al. (2025), while Qwen2-5-72b-instruct is attributed to Jiang et al. (2024). Please correct.","section":"Appendix A, Table 3"},{"comment":"The paper says top-K sampling was used but only specifies 'default top-K search inference parameters.' The actual top-k value should be stated, since it directly affects the degree of nondeterminism.","section":"Section 4.1"},{"comment":"The text says 'all the accuracy standard deviations are extremely small,' but Table 1 shows Mixtral-8x7b-instruct with stdev 0.093 and 0.105. Clarify whether this statement refers only to the MedQA results.","section":"Section 5.1"},{"comment":"The quadrant names (USEFUL, RELIABLE but inconsistent, etc.) are introduced after the plots are shown. Consider defining them formally in the text before the figures, or present them as illustrative labels rather than definitions.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful empirical core, but the printed data tables contain impossible values, the aggregate statistics silently exclude an outlier, and the central size-based conclusion is vulnerable to the contamination caveat the authors themselves raise. I do not see these as unfixable: corrected tables, transparent aggregation, and a contamination-sensitive analysis would make the contribution publishable. The requested revision is substantial, hence major_revision rather than minor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper gives you a genuinely useful empirical map of how inconsistent small open LLMs are on multiple-choice benchmarks, and the headline 50-80% S/T range at low temperature holds up for the models they tested. The medium-model contrast is plausible but not fully established, largely because of the contamination problem the authors themselves concede.\n\nWhat's new: a multi-temperature, 10-repetition consistency study across 26 open models, with a compact RWS|S/T representation and the consistency plot. That is a modest but real extension of the MV metric from Pezeshkpour and Hruschka, and the finetuned-vs-base MedQA comparison is a nice addition. The measurements are straightforward and not fitted to any target; the paper reports standard deviations and the consistency data are internally coherent. The granite results, trained with more curated data, are a good partial control.\n\nSoft spots, in proportion: Table 1 has a 104% cell for Llama-3-8B at 0.3, which makes you wonder about the rest of the table until you recheck. Section 4.3 quietly drops 3.1-2b-base as an outlier with no rule for doing so; that exclusion should be justified and the summary range adjusted or defended. The oracle derivation is a bit hand-wavy: it moves between \"answers consistently\" and \"answers correctly\" without making the target clear, though a model can be consistently wrong. The 0.99 threshold and SURE/UNSURE labels are arbitrary but not load-bearing.\n\nThe bigger issue is contamination. The paper's own limitations section says the medium-model high consistency \"may have been inflated... due to rogue memorization,\" and the conclusion repeats that it \"may have had significant impacts.\" Since the only evidence for the size-based claim is the 87-99% S/T of the three medium models, and those models were trained on internet-scale data that almost certainly includes MMLU-derived text, the \"much higher consistency\" conclusion is not yet established. This is not an external nitpick; the paper flags it, then moves on. A contamination-filtered recomputation, or at least a quantitative sensitivity check using the known MMLU-Redux contamination discussions, would make the claim stand. Without it, the small-model result is solid and the medium-model result is provisional.\n\nAlso, no code or data is released, which limits independent verification. That matters for a measurement paper.\n\nVerdict: this deserves a serious referee. The central small-model finding is important and reproducible in principle, and the flaws are fixable. I would send it to review, asking for the table fix, explicit outlier handling, a contamination analysis, and ideally a release of the repetition data. For a reading group on LLM evaluation reliability, it is worth a session. I would cite the small-model S/T numbers with the contamination caveat attached.","headline":"Useful empirical map of small-model answer inconsistency, but the medium-model contrast is provisional until contamination is addressed; needs one solid revision, not a desk reject.","tokens_in":859,"tokens_out":760,"would_cite":true,"duration_ms":27204,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Small LLMs answer the same multiple-choice question consistently only 50–80% of the time at low temperature, while 50B–80B models exceed 95%.","keywords":["answer consistency","non-determinism","small language models","multiple-choice benchmarks","inference temperature","MMLU-Redux","MedQA","oracle guessing"],"falsifier":"Repeat the 10-trial protocol on a question set verified to be absent from each model's training data. If medium models' S/T drops below 90% while small models stay in the 50–80% range, the scale gap is contamination; if small models rise above 90% at temperature 0.3, the reported inconsistency is a sampling artifact, not a capacity limit.","tokens_in":16721,"feed_emoji":"🎲","tokens_out":4620,"duration_ms":40905,"temperature":0.7,"pith_summary":"This paper tries to establish that small language models (2B–8B parameters) are far less deterministic than their average scores suggest: when asked the same multiple-choice question ten times, they produce the same answer only about half to four-fifths of the time, even at low inference temperatures. Medium models (50B–80B) stay consistent on more than 95% of questions, pointing to a scale-linked reliability gap. The authors argue that accuracy on consistently answered questions tracks overall accuracy, so a model can decline to answer when it is not consistent and raise precision. The work matters because customer-service, medical, and safety-critical uses of LLMs assume that identical inputs get identical answers, an assumption these results call into question.","feed_headline":"Small LLMs flip-flop on 20–50% of benchmark answers","feed_subtitle":"Ten repetitions show 2B–8B models stay consistent on only 50–80% of questions; 50B–80B models top 95%.","key_machinery":"The machinery is a formal definition of answer consistency as equivalence to an oracle guessing at a 0.99 success rate. For ten repetitions of a 4- or 5-choice question, this reduces to the rule: answer the same way in at least 9 of 10 trials (SURE) or not (UNSURE). The paper packages results as a pair, RWS | S/T—the share of SURE answers that are correct, and the share of questions that are SURE—and visualizes the temperature path in a consistency plot with quadrants labeled by reliability and consistency.","core_discovery":"The paper's central discovery is a quantified consistency gap: across 26 open-weight models on MMLU-Redux and MedQA, small models (2B–8B) achieve 0.99-consistency—defined as giving the same answer in at least 9 of 10 repetitions—on only 50–80% of questions at temperature 0.3, with the share falling as temperature rises. Medium models (50B–80B) reach S/T values of 87–99%. Accuracy among consistent answers (RWS) correlates with overall accuracy at low temperature (R² = 0.488 on MMLU-Redux, 0.831 on MedQA at t=0.3), so filtering out inconsistent questions can raise precision. Finetuned medical models can exceed their base models in both accuracy and consistency, as medllama3-v20 shows with 96%","pith_inferences":["If consistency tracks reliability, a cheap runtime gate could ask the model the same question two or three times and abstain when answers diverge; the paper's RWS|S/T curves give a direct way to price that gate's false-abstention rate.","The 9-of-10 threshold inherits the oracle's 0.99 success rate; choosing a stricter or laxer bar (e.g., 10-of-10 or 7-of-10) would compress or widen the reported consistency ranges and could be tuned per application risk tolerance.","A stronger test than identical repetition would be paraphrase consistency—asking semantically equivalent rewordings—which the paper lists as future work; if paraphrase consistency is lower than repetition consistency, the 50–80% band is an upper bound on real-world reliability.","The scale-linked gap suggests a testable hypothesis: intermediate sizes (e.g., 13B–30B) should interpolate between the 50–80% and >95% bands, providing a curve that could separate capacity effects from contamination effects."],"forward_implications":["Single-run evaluation scores for small models overstate their reliability; a model averaging 60% accuracy may flip answers on a large share of those it gets right.","Requiring 0.99-consistency before trusting an answer lets an application trade coverage for precision: on MedQA, Bio-Medical-Llama-3-8B improves RWS to 0.79 while covering only 60% of questions.","Low temperature (0.3) maximizes the share of consistently answered questions, while higher temperature (1.0) increases accuracy among those consistent answers on MMLU-Redux—a trade-off that flips for MedQA, where low temperature is best on both axes.","The 50–80% S/T band reproduces across the granite family, which was trained to avoid benchmark contamination, suggesting the inconsistency is not an artifact of a single model or benchmark."],"supporting_citations":[{"why":"Supplies the MMLU-Redux general-knowledge benchmark that anchors the main consistency measurements.","marker":"(Gema et al., 2025)"},{"why":"Supplies the MedQA medical-exam benchmark used to test whether results generalize beyond general knowledge.","marker":"(Jin et al., 2021)"},{"why":"Describes the top-K sampling decoding strategy and its non-determinism, the generation method used in all experiments.","marker":"(Song et al., 2024)"},{"why":"Provides granite models trained on curated data to reduce benchmark contamination, serving as a low-contamination check on the consistency findings.","marker":"(Granite Team, 2024)"},{"why":"Defines the MV majority-vote metric that the paper's SURE/UNSURE criterion builds on and compares against.","marker":"(Pezeshkpour and Hruschka, 2024)"},{"why":"Introduces the rogue-memorization concept that the limitations section cites as a potential inflation source for medium-model consistency.","marker":"(Cavalin et al., 2024)"},{"why":"Prior consistency evaluation that reported average accuracy across seeds; the paper extends it to per-question repetition behavior at multiple temperatures.","marker":"(Nalbandyan et al., 2025)"}],"fun_headline_variants":["Small LLMs change answers on 20-50% of repeated questions","Repeated benchmark: small LLMs only 50-80% consistent","Small model answers don't stick: 50-80% match on second try","Small LLMs: only half to 80% of benchmark answers stay put","Temperature up, small LLM consistency down: 50-80% on repeat quizzes"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The results assume that benchmark contamination and rogue memorization are negligible; if medium models memorized MMLU-Redux, their near-perfect consistency reflects memory, not stable reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Small LLMs change answers on 20-50% of repeated questions","Repeated benchmark: small LLMs only 50-80% consistent","Small model answers don't stick: 50-80% match on second try","Small LLMs: only half to 80% of benchmark answers stay put","Temperature up, small LLM consistency down: 50-80% on repeat quizzes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001417,"raw_usage":{"total_tokens":5569,"prompt_tokens":768,"completion_tokens":4801,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":4699}},"tokens_in":512,"tokens_out":4801,"duration_ms":29355,"temperature":1.0,"reasoning_tokens":4699,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:24:37.485127+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the 10-trial protocol on a question set verified to be absent from each model's training data. If medium models' S/T drops below 90% while small models stay in the 50–80% range, the scale gap is contamination; if small models rise above 90% at temperature 0.3, the reported inconsistency is a sampling artifact, not a capacity limit.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides granite models trained on curated data to reduce benchmark contamination, serving as a low-contamination check on the consistency findings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the rogue-memorization concept that the limitations section cites as a potential inflation source for medium-model consistency."}],"review_version":1}