{"id":"18357744-8a3d-485b-a4ac-f7b9d57eb2f2","arxiv_id":"2502.03711","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MultiQ&A automatically rephrases questions and crowdsources answers from LLM agents, showing gpt-3.5-turbo is fairly stable on extractive and multiple-choice QA but more variable on free-form answers.","lead":"This paper introduces MultiQ&A, a pipeline that automatically rewrites questions, has independent LLM agents answer the rewrites, and measures how consistent the answers are. It is a large-scale robustness test for language models that could help institutions probe a model's reliability before deployment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4's Abstractive TruthfulQA validation row reports Base=52.4, Mode=28.1, Worst=61.8, which is impossible under the paper's own definition Ω≤A; the main empirical table therefore contains a mathematical contradiction and needs correction before the robustness claim can be assessed.","rationale":"Good faith: the paper proposes a scalable, well-motivated framework, and the scale of the experiment (1.9M perturbations, 2.3M answers) is substantial. The reader's CONDITIONAL verdict is reasonable, and their semantic-preservation concern is genuine: since gpt-3.5-turbo generates both the perturbations and the answers, observed answer variability could reflect changed question meaning rather than model inconsistency. However, my stress-test pass found a more decisive, internal problem. The impossible row is not an opinion about external validity; it contradicts the paper's own equations. Even if the row is a one-off typo, it demonstrates that the published results have not been audited against the stated definitions, and without code/data release there is no way for a reader to verify any of the 376,201-question aggregate. Consequently, I would move from CONDITIONAL to REJECT as currently written: the central robustness claim rests on a table that contains at least one mathematically impossible entry. If the authors provide corrected tables and raw answer logs, the framework could be re-evaluated; the semantic-preservation validation should also be added.","tokens_in":16075,"tokens_out":11307,"duration_ms":128639,"concrete_test":"Recompute the Abstractive TruthfulQA validation set rows from the raw answer logs: for each of the 807 questions, score the original answer and each of the five perturbed answers using the paper's abstractive scoring rule (semantic similarity to correct options). Then compute A = fraction of original answers correct, Ŷ = fraction of plurality choices correct, and Ω = fraction of questions where all six answers are correct. If the recomputed Ω is ≤ A (as required) and differs from 61.8, Table 4 has an error; if the impossible values persist, the metric pipeline or table is fundamentally broken. Publish the per-question answer sets so this check is independently repeatable.","verdict_should_be":"REJECT","load_bearing_attack":"Section 5.1 defines Worst-case Robustness as Ω = (1/n) Σ_j min_i m(f(T_i(x_j)), y_j), with T_0 the identity, so for every sample the min over raters is ≤ the original question's correctness indicator. Thus Ω ≤ A must hold in every row of Table 4. The Abstractive TruthfulQA validation row reports A=52.4, Mode=28.1, and Ω=61.8, violating Ω ≤ min(A, Mode) and the stated inequality Ω ≤ min(Ŷ,A). This is not a statistical anomaly but a mathematical impossibility for the published pipeline: the reported numbers cannot be produced by the definitions in §5.1. The most likely causes are a column misalignment, a different abstractive scoring convention applied inconsistently, or a transcription error in a central results table. Any of these undermines confidence in Table 4 and the aggregate Table 5, which are the sole support for the headline claim that gpt-3.5-turbo is 'relatively robust and consistent under perturbations.' The row must be recomputed from raw per-question answers, and the audit trail released, before the central claim can be evaluated.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MultiQ&A, a pipeline in which a gpt-3.5-turbo Query Rewriter generates five lexical perturbations of each input question, a set of gpt-3.5-turbo Answer Generator agents answers the original and perturbed questions, and an Aggregator computes accuracy, worst/best-case robustness, plurality-vote accuracy, and agreement/reliability statistics. The pipeline is applied to 12 QA datasets across extractive, multiple-choice, and abstractive formats, yielding roughly 1.9 million perturbed questions and 2.3 million answers. The central claim is that gpt-3.5-turbo remains 'relatively robust and consistent under perturbations' and that MultiQ&A can serve as a scalable framework for institutional LLM stress-testing.","tokens_in":16353,"tokens_out":3869,"duration_ms":39714,"significance":"If the empirical claims held, MultiQ&A would be a useful large-scale robustness evaluation tool: it combines standard psychometric and agreement metrics, covers diverse QA formats, and ships a concrete agent-based pipeline. The paper's strengths are its scale, the breadth of datasets, and the explicitness of the metric definitions in §5. However, the central empirical claim rests on Table 4, which contains a row that is impossible under the paper's own definitions, and on two unverified assumptions: that perturbations preserve semantics and that the answering agents are independent of the perturbation generator. Until these are addressed, the paper does not support its headline conclusion that gpt-3.5-turbo is relatively robust and consistent under perturbations.","major_comments":[{"comment":"The Abstractive TruthfulQA validation row reports Base=52.4, Mode=28.1, Worst=61.8. Under the definition in §5.1, Ω = (1/n) Σ_j min_i m(f(T_i(x_j)), y_j), and T_0 is the identity transformation, so for every sample the minimum over raters is no greater than the correctness indicator of the original question. Therefore Ω ≤ A must hold in every row. The paper itself states the stronger hierarchy Ω ≤ min(Ŷ, A) ≤ max(Ŷ, A) ≤ O. The reported Worst=61.8 exceeds both Base=52.4 and Mode=28.1, which is a mathematical impossibility, not a statistical anomaly. This row must be recomputed from raw per-question responses, and the audit trail released, before the robustness claim in Table 5 and the abstract can be evaluated.","section":"§5.1, Table 4"},{"comment":"The load-bearing assumption that perturbed questions preserve the semantic intent of q0 is asserted but never validated. The Query Rewriter prompt asks for 'radically different ways' of rewriting, and Table 3's own caption concedes that 'variations can predispose gpt-3.5-turbo to hallucinations.' No human evaluation, entailment/paraphrase classifier, or post-hoc semantic consistency check is reported. Consequently, answer variability across perturbations may reflect changed question meaning rather than model inconsistency, which would invalidate the interpretation of Ω, O, and the agreement metrics as measuring robustness to meaning-preserving reformulations.","section":"§3.1, Table 3 caption"},{"comment":"The Answer Generator agents are described as 'independent,' but they are all instances of gpt-3.5-turbo, and the Query Rewriter that produces the perturbations is also gpt-3.5-turbo. Independence is only at the level of separate prompts and separate API calls; there is no evidence that the answers are conditionally independent given the question, nor that the perturbation-generation process is independent of the answer-generation process in a way that the robustness metrics require. The reported robustness scores should be framed as measuring the self-consistency of a single model under its own rewrites, not as evidence about model robustness generally. A concrete fix would be to cross-perturb with one model and answer with another, or to report agreement statistics across multiple base models.","section":"§3.2"},{"comment":"Two sampling issues affect the generalizability of the aggregate numbers. First, MathQA train is evaluated on only 693 of 29,800 samples, with no description of how this subset was selected or whether it is representative; the MathQA row is also an outlier on every metric, so this selection matters. Second, §6.4 reports 2,293 cases where gpt-3.5-turbo failed due to content filtering, but the paper does not state whether these cases were excluded from the per-dataset and aggregate metric computations, nor how their exclusion biases the reported robustness and agreement numbers. The paper should report the number of excluded samples per dataset split and recompute Table 5 under both inclusion and exclusion policies.","section":"§4.2, Table 4, §6.4"},{"comment":"The abstractive QA scoring protocols are heterogeneous and may not be comparable across datasets: TruthfulQA is scored by semantic similarity with correct options, WikiQA by a cosine similarity threshold of 60%, SciQ by approximate Levenshtein distance, and SQuADv2/TriviaQA/HotpotQA by exact or fuzzy matching. These thresholds and matching functions are not justified, and the differences can dominate the reported accuracy and robustness numbers. Table 5's aggregation across abstractive datasets therefore mixes incompatible correctness definitions; the paper should either use one shared scoring protocol or clearly separate datasets by scoring method in the aggregate analysis.","section":"§4.3, Table 2"}],"minor_comments":[{"comment":"The notation for the plurality metric is confusing: Ŷ is defined as a sum of indicator values but is described as 'the ensemble answer generated by plurality voting'; it is actually the accuracy of the mode, not the mode itself. Please rename the quantity or clarify the definition.","section":"§5.1"},{"comment":"The 'answer-critic model' used for re-ranking is mentioned without a citation or specification. Please state the model, its training data, and how semantic alignment is measured.","section":"§3.3"},{"comment":"The text states Cronbach's α ≈ 42.8% on the MathQA test split, but Table 4 reports α = 45.2 for MathQA test. Please reconcile the numbers.","section":"§6.2 vs Table 4"},{"comment":"The introduction says the study covers 12 QA datasets, while the conclusion says '13 datasets'; Table 1 lists 12 datasets, but SQuADv2 is used in both extractive and abstractive modes, so the counting should be made consistent.","section":"§1, §7"},{"comment":"The verbal description of Cronbach's α reverses the roles of the variance terms: 'σ2_y is the variance in scores across v+1 raters for each sample' should be the item variance, and 'σ2_x is the variance in total correct responses per rater' should be the total score variance. Please align the prose with the formula.","section":"§5.3"},{"comment":"There is a typo: 'even with with significant prompt alterations' should read 'even with significant prompt alterations.'","section":"§6.1"},{"comment":"The split label 'Extn' appears to be a truncated form of 'Extractive'; please use the full label for clarity.","section":"Table 4"},{"comment":"For TruthfulQA abstractive scoring by 'semantic similarity with correct options,' the similarity measure and threshold are not specified; please provide them or cite the exact implementation.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The impossible row in Table 4 (Abstractive TruthfulQA, Worst=61.8 > Base=52.4) should be verified against raw logs before any further review. If the raw per-question answers cannot be released or recomputed, I would consider this a rejection-level error, because the central empirical table is the sole support for the paper's headline robustness claim. The other major concerns (semantic validity and model self-independence) are methodologically fixable but require additional experiments or at least a substantial re-framing of the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a serious large-scale robustness evaluation, but its central results table contains a row that is mathematically impossible under the paper's own definitions. That has to be fixed before the main claim can be trusted.\n\nWhat's genuinely useful: MultiQ&A integrates well-known pieces—LLM-generated paraphrases, self-consistency voting, entropy and agreement metrics—into one pipeline and runs it at a scale (1.9M perturbations, 2.3M answers) that few academic groups can match. The metric definitions in §5.1 are clearly stated, the relationships between accuracy, worst-case, best-case, and plurality voting are spelled out, and the taxonomy across extractive, multiple-choice, and abstractive QA is sensible. The content-filter failure analysis in §6.4 is a nice touch.\n\nNow the soft spots, in order of severity.\n\nFirst, the Abstractive TruthfulQA validation row in Table 4 reports Base=52.4, Mode=28.1, Worst=61.8. From §5.1, Worst-case Ω is the per-sample minimum over raters including the identity T_0, so Ω ≤ A and Ω ≤ Ŷ must hold for every row. Here Ω beats both A and Mode, which cannot happen. This isn't a statistical anomaly; it's a contradiction with the published equations. Most likely a column misalignment or transcription error, but it taints Table 4 and the aggregate Table 5, which are the sole support for the headline robustness claim. The authors need to recompute that row and release the per-question audit trail.\n\nSecond, the same model, gpt-3.5-turbo, generates the perturbations and the answers. The \"independent agents\" are only independent at the prompt level, so the robustness scores partly reflect the model's consistency with itself. That's a real circularity burden.\n\nThird, perturbation semantic quality is never validated. Table 3's own caption concedes variations can predispose the model to hallucinations. Without a human or external check, the distinction between \"robust under paraphrase\" and \"sensitive to meaning change\" is blurred.\n\nSmaller issues: MathQA train uses 693 of 29,800 questions, a tiny slice; no code or data is released despite the institutional-adoption framing; and the paper says 12 datasets in the abstract but 13 in the conclusion.\n\nThe underlying idea is worth taking seriously, and the scale is a real contribution. But as written, the impossible row means the paper is not yet citable. A serious referee should see it after that row is recomputed and the data trail is provided.","headline":"A large-scale, well-integrated robustness pipeline whose main results table contains a mathematically impossible row, so the headline claim is currently unverified.","tokens_in":16844,"tokens_out":3729,"would_cite":false,"duration_ms":35771,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MultiQ&A claims that gpt-3.5-turbo answer cohorts stay largely consistent across 1.9 million rephrased questions, and that the disagreements quantify hallucinations for pre-deployment auditing.","keywords":["LLM robustness","question perturbations","crowdsourcing","hallucination","ensemble voting","answer consistency","question answering","gpt-3.5-turbo"],"falsifier":"A hand-labeled sample of a few hundred perturbed question pairs, asking whether the ground-truth answer remains the same for each pair, would settle it: if a nontrivial share of paraphrases have a different correct answer, the robustness numbers conflate meaning change with inconsistency.","tokens_in":15887,"feed_emoji":"🧪","tokens_out":4743,"duration_ms":42524,"temperature":0.7,"pith_summary":"MultiQ&A is a stress test for question-answering LLMs: it takes a question, rewrites it into several lexical variants, answers each variant with a separate model agent, and compares the answers. The paper applies this to 376,201 questions, generating 1.9 million perturbations and 2.3 million answers across extractive, multiple-choice, and abstractive QA. It reports that gpt-3.5-turbo is largely consistent under these perturbations, while worst-case robustness is weakest in free-form answering and collapses on arithmetic reasoning. The aim is to give institutions a scalable way to measure confidence, consistency, and hallucination before deployment.","feed_headline":"GPT-3.5 stays consistent under 1.9M rephrased questions","feed_subtitle":"An automated pipeline rewrites questions, answers each variant, and flags disagreements to expose hallucinations.","key_machinery":"Three cooperating gpt-3.5-turbo agents: a Query Rewriter that produces v+1 paraphrases (including the identity) at temperature 1.0; independent Answer Generators that answer each paraphrase with task-specific prompts; and an Aggregator that clusters answers with Sentence-BERT, re-ranks them with a cross-encoder, and computes agreement metrics (Fleiss's kappa, entropy certainty, Gibbs' M2) plus supervised robustness metrics (worst-case $\\Omega$, best-case O, plurality vote). The pipeline treats answer variability across paraphrases as the signal of instability.","core_discovery":"The central claim is that an ensemble of independently prompted gpt-3.5-turbo agents answering the same question in several phrasings remains relatively robust and consistent, and that the disagreements between agents can be quantified to expose hallucinations. The paper packages this as MultiQ&A, a five-step pipeline that perturbs, answers, quantifies, plurality-votes, and visualizes, measuring robustness with worst-case and best-case accuracy, Fleiss's kappa, entropy-based certainty, and Cronbach's alpha. Across extractive, multiple-choice, and abstractive tasks, the paper finds a clear ordering: extractive outperforms multiple choice outperforms abstractive, with MathQA as an outlier where consistency collapses. The system also logged 2,293 content-filter refusals, which it suggests can enrich adversarial datasets for safety testing.","pith_inferences":["Because the same model both rewrites and answers the questions, the reported 'robustness' may partly reflect gpt-3.5-turbo's own paraphrasing tendencies rather than intrinsic semantic stability; a cross-model rewriter-answerer split could test this.","The metrics are computed from a small cohort of six phrasings per question, so increasing the number of perturbations or using human-verified paraphrases could materially change the measured consistency.","The 2,293 content-filter refusals could serve as a focused probe of safety behavior under paraphrase, an angle the paper starts but does not develop.","The ordinal ranking by scenario suggests a testable recipe for deployment: give the model extractive context when possible, constrain choices when not, and add extra monitoring for generative and numeric tasks."],"forward_implications":["If MultiQ&A is right, institutions can audit an LLM's consistency before deployment by perturbing their own real questions at scale.","The extractive greater than multiple choice greater than abstractive ordering implies that context and constrained choices are protective, and free-form generation is where hallucinations concentrate.","Plurality voting across paraphrases gives a practical ensemble answer that is usually close to or better than the baseline answer.","The MathQA outlier shows that arithmetic tasks need separate scrutiny, since model consistency is not uniform across domains.","Automated perturbation can also generate adversarial examples and content-filter triggers for safety evaluation."],"supporting_citations":[{"why":"Documents that neural NLP models are sensitive to textual perturbations, motivating the need for a robustness stress test.","marker":"Zhang et al. 2022"},{"why":"Argues that existing benchmarks do not adequately assess robustness, the gap MultiQ&A targets.","marker":"Moradi and Samwald 2021"},{"why":"Supplies Sentence-BERT, used to cluster and re-rank generated answers by semantic similarity.","marker":"Reimers and Gurevych 2019"},{"why":"The retrieval-augmented generation baseline that MultiQ&A contrasts with as an alternative hallucination mitigation.","marker":"Lewis et al. 2021"},{"why":"Self-consistency sampling inspires the ensemble and plurality-voting aggregation of multiple answers.","marker":"Wang et al. 2023"},{"why":"Provides the kappa agreement benchmarks used to interpret the model's consistency as substantial.","marker":"Landis and Koch 1977"},{"why":"Supplies the alpha reliability coefficient used to measure internal consistency across perturbation raters.","marker":"Cronbach 1951"},{"why":"Provides SQuADv2, the extractive QA dataset central to the performance and agreement results.","marker":"Rajpurkar et al. 2016"}],"fun_headline_variants":["MultiQ&A: 1.9M perturbed questions test GPT-3.5 consistency","Automated crowdsourcing reveals GPT-3.5 answer stability","1.9M rephrasings, 2.3M answers: GPT-3.5 holds up","System quantifies LLM hallucinations via question perturbations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole measurement rests on the assumption that the rewrites preserve the original question's meaning, so that answer differences count as model inconsistency rather than new questions; the paper never independently verifies that assumption.","fun_headline_variants_meta":{"raw":{"variants":["MultiQ&A: 1.9M perturbed questions test GPT-3.5 consistency","Automated crowdsourcing reveals GPT-3.5 answer stability","1.9M rephrasings, 2.3M answers: GPT-3.5 holds up","System quantifies LLM hallucinations via question perturbations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1401,"prompt_tokens":871,"completion_tokens":530,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":444}},"tokens_in":487,"tokens_out":530,"duration_ms":5062,"temperature":1.0,"reasoning_tokens":444,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T01:00:18.863848+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A hand-labeled sample of a few hundred perturbed question pairs, asking whether the ground-truth answer remains the same for each pair, would settle it: if a nontrivial share of paraphrases have a different correct answer, the robustness numbers conflate meaning change with inconsistency.","supporting_citations":[],"review_version":1}