{"id":"37f111b8-be68-45fa-8758-9964129ff102","arxiv_id":"2512.09066","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ORCA predicts the distribution of human correctness ratings for open-ended audio QA answers and matches or beats LLM judges while also estimating annotator disagreement.","lead":"ORCA is a small neural judge that predicts both the expected correctness and the expected disagreement of human raters for open-ended answers from audio AI models. It offers a cheap, reproducible alternative to expensive LLM judges for audio QA evaluation, if its promised data and models are released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains rely on human-corrected rationales and a filtered subset; the abstract's unseen-benchmark result (0.85) is absent from the paper, so the practical generalization claim is unsupported.","rationale":"The reader's conditional verdict is appropriate. My concern is not that the Table 2 result is internally wrong, but that its external validity is not established: ORCA is evaluated on a heavily curated subset with corrected rationales, while the practical claim—a lightweight replacement for LLM judges on new audio QA benchmarks—requires operating on raw inputs. The missing unseen-benchmark experiment (promised in the abstract as 0.85, absent from §5/§6) is a concrete, checkable gap. If the missing experiment is supplied and reproduces the claim, the paper's central promise is materially supported. If not, the paper should be scoped to the curated benchmark setting. Since the reader already recommended conditional acceptance pending these issues, my read does not move the verdict; it sharpens the condition.","tokens_in":15147,"tokens_out":9736,"duration_ms":103228,"concrete_test":"Run the best clamped ORCA (Gemma3-12B) on a held-out audio QA benchmark (e.g., AIR-Bench or AudioBench) with only Stage-1a raw Gemini rationales and Whisper transcripts, no Stage-3 human corrections or filtering, and compare Spearman ρ and MAE_μ against Gemini-2.5-Flash and the claimed 0.85. If the 0.85 result was already computed, it must be reported with full experimental details; if it was not, the abstract and conclusions should be revised to claim only within-benchmark generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central Table 2 numbers are computed on the 2,459 question-answer pairs that survived the Stage 3 filter (§5.2), using rationales and reference answers that were human-corrected after annotators flagged 30.5% of questions as having insufficient rationale (Table 4). In deployment, ORCA would receive raw Gemini-generated rationales and uncorrected references, not the curated versions used for training and testing. The paper reports no evaluation on uncorrected inputs, and no evaluation on a third benchmark. The abstract asserts 'generalize to unseen benchmarks with a score of 0.85,' but §5.3 defines only unseen-question and unseen-LALM splits, both within the same two corrected benchmarks; no such 0.85 result appears in the text or tables. The advertised capability of replacing LLM judges therefore depends on a human-in-the-loop curation pipeline whose contribution is not quantified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ORCA, a text-only framework that models human correctness judgments for open-ended audio question answering as a Beta distribution, predicting both the mean and the variance (disagreement) of ratings. The authors collect human annotations through a three-stage pipeline: Gemini/Whisper-based context generation, human rating with structured feedback, and expert human-AI correction. They then train small open-weight language models (OLMo, Gemma, Llama) with an MLP head to predict Beta parameters from question, reference, rationale, transcript, and candidate answer text. Experiments on two audio QA benchmarks (MMAU test-mini and MMAR) report a best Spearman correlation of 0.9103 with mean human judgments for the clamped Gemma3-12B ORCA model, outperforming several LLM-judge baselines including Gemini-2.5-Flash, while also providing variance estimates that LLM judges do not. The paper claims that ORCA generalizes to unseen LALM response styles and identifies ambiguous benchmark items via predicted variance.","tokens_in":15445,"tokens_out":2613,"duration_ms":29468,"significance":"If substantiated, ORCA would be a practically valuable contribution: a lightweight, reproducible, open-weight evaluator for open-ended audio QA that goes beyond mean scores to capture annotator disagreement. The release of models, code, and curated annotations is a concrete benefit to the community, and the distributional modeling of human judgment variability is a timely direction. The method is conceptually simple and the training cost is low (≈15 minutes per run on a single GPU). The manuscript also includes useful input-ablation analysis and a study of training on LLM-judge versus human labels. However, the strength of these claims is currently undermined by several load-bearing gaps between the abstract, the reported evaluation protocol, and the data curation decisions, which must be addressed before the contribution can be accepted as stated.","major_comments":[{"comment":"The abstract claims that ORCA models 'generalize to unseen benchmarks with a score of 0.85' and that the method employs 'curriculum learning'. Neither claim appears anywhere in the body. Section 5.3 defines only two evaluation scenarios—unseen questions and unseen LALMs—both within the same two corrected benchmarks (MMAU and MMAR). No third benchmark is used, and Table 2's best Spearman is 0.9103, not 0.85. The '0.85' number and curriculum learning are unsupported and must either be added with a concrete experimental definition or removed from the abstract.","section":"Abstract and §5.3–6.1"},{"comment":"All headline numbers are computed on the 2,459 question-answer pairs that survived the Stage 3 filter, after post hoc removal of 1,121 pairs and 3,150 ratings (32% of the total). The justification that the removed ratings had lower agreement (α=0.59) is circular: the paper's own motivation is that genuinely ambiguous items produce low agreement. Removing them selectively discards the hard cases that ORCA is supposed to model. The paper should report results on the original 3,580 pairs, or otherwise treat the filtering as a modeling choice and ablate it, showing that ORCA still outperforms LLM judges without the filter.","section":"§5.2 and Table 2"},{"comment":"The clamping post-processing threshold is 'optimized on the development set to maximize ρ+τ−MAE_μ'. Since the same dev set is used for the reported results, the clamped ORCA numbers in Table 2 include a tuned free parameter. The paper should report unclamped results for all ORCA models, and clarify how the threshold would be set in a deployment scenario without access to human ratings. Without this, the comparison between clamped ORCA and the unclamped LLM judges is not parameter-free.","section":"§5.5 and Table 2"},{"comment":"There is a circularity concern that is central to the claimed validity of the gold standard. Human annotators (and ORCA, and the LLM judges) all receive the same Gemini-generated rationale as part of the input. The paper itself notes that Gemini sometimes produces non-informative rationales and that 30.5% of questions were flagged as having insufficient rationale, leading to human corrections in Stage 3. However, the reported evaluation is run on the corrected rationales, and ORCA's input always includes that corrected rationale. Thus the high agreement between ORCA and human ratings may largely reflect shared reliance on the same textual rationale, rather than independent assessment of the audio. The paper should provide an ablation where ORCA is trained/evaluated without rationales (or with audio-only human ratings) to quantify how much of the reported correlation is attributable to th","section":"§3.1–§3.2 and §4"},{"comment":"The paper motivates ORCA as a replacement for LLM judges in real-world audio QA evaluation, but the models are trained and tested on the post-correction dataset, in which rationales and reference answers were manually fixed by six domain experts. In deployment, ORCA would receive raw Gemini-generated rationales and uncorrected benchmark references. No experiment evaluates this setting. The authors should test ORCA on the pre-correction data (e.g., the original rationales before Stage 3) and report the performance drop, if any. Unless this is done, the claimed practical advantage over LLM judges is not established.","section":"§5.2, §5.6, and deployment claim"}],"minor_comments":[{"comment":"The abstract and Section 1 use different annotation counts: 11,721 vs. 11,721 in Section 5.2 is consistent, but the abstract says '9,663 annotations' in the reader's take? Actually the manuscript abstract in the provided text says 9,663; the body says 11,721. Check consistency: §5.2 reports 11,721 total annotations, while the abstract says 9,663. Please reconcile.","section":"Abstract and Introduction"},{"comment":"Typo: 'Krippendorff' is misspelled as 'Krip-pendorff' in §2.2? Check reference. Also the citation format for Krippendorff (2019) is inconsistent in the text.","section":"§5.2 and Table 1"},{"comment":"The figure label 'LLM-Judge Correctness score [1,5]' appears twice, once for humans and once for LLM judges. Please disambiguate the annotation source.","section":"Figure 2"},{"comment":"Typo in Figure 7 prompt: 'trasncript' should be 'transcript'. Also Appendix B.1 says 'Gemini-2.5-Flash' while Figure 6 says 'Gemini-2.0-Flash'—please align.","section":"Appendix B"},{"comment":"The held-out LALM experiment uses only OLMo2-7B clamped, and Figure 3's caption says 'OLMo-7B' while the text says 'OLMo2-7B'. Please make model names consistent throughout.","section":"§6.2"},{"comment":"The two-level stratification description is clear, but it would help to report the number of unique questions in the test set of each split and the overlap of question categories across splits, to assess the difficulty of the unseen-question scenario.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The abstract contains claims (unseen benchmarks, curriculum learning) that are not present in the body; this is a serious reporting issue that the editor should require the authors to fix. The central methodology is promising, but the post-hoc filtering and dev-set-tuned clamping must be addressed before the numbers can be trusted. The circularity concern regarding the shared Gemini rationale is real and should be probed with an ablation. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core contribution is real: a lightweight, open-weight judge for audio QA that predicts both mean and variance of human correctness judgments, plus a substantial human annotation effort (three-stage pipeline, 37 annotators, ~11.7k ratings). The held-out-LALM experiment is a genuinely useful test of response-style generalization, and the input ablation showing the question matters most is a nice sanity check. I'd call the Beta-MLE modeling itself standard — that's been done in NLP — but the audio QA application and the curated dataset are new enough to matter to people in this subfield.\n\nThe problem is the packaging. The abstract makes claims that simply don't appear in the body: no \"curriculum learning\" anywhere in the experiments, no \"unseen benchmark\" with a 0.85 score, and the counts disagree (9,663 annotations across 3,699 pairs on three benchmarks vs. the body's 11,721 across 3,580 on two). That's not a rounding error; it's a red flag that the abstract was written to sell the paper rather than summarize it.\n\nMore substantively, the headline Table 2 numbers are computed on the 2,459 pairs that survived the Stage 3 filter, using rationales and reference answers that human experts corrected. In deployment, ORCA would receive the raw Gemini-generated rationales and uncorrected references. The paper reports no evaluation on uncorrected inputs, so the central claim that ORCA can replace LLM judges is untested in the realistic setting. The filtering itself removes 32% of the rated pairs; the authors show the filtered-out ratings had low agreement (alpha 0.59), which is a reasonable defense, but they should still report ORCA's performance on the unfiltered set as a sensitivity check.\n\nThe circularity concern is mild but worth acknowledging: humans rated answers while reading the same Gemini rationale that ORCA takes as input, so part of the target is shaped by the input. The expert corrections and the human judgments themselves provide independent signal, so I don't think it's fatal, but it deserves a sentence of caveat.\n\nThe clamping threshold is tuned on the dev set. Minor, but it's a tuned hyperparameter and should be reported as such.\n\nBottom line: this is a useful dataset and a plausible framework, but the paper currently oversells itself. I'd send it to peer review — the resource and the question are serious — but the authors need to reconcile the abstract with the body, and either evaluate on uncorrected rationales or clearly scope the claims to the corrected-pipeline setting. This is a revise-and-resubmit, not a reject.","headline":"The dataset and Beta-judge idea are real, but the abstract overclaims and the headline numbers rest on human-corrected rationales; the paper needs revision before its practical claims hold.","tokens_in":15911,"tokens_out":2666,"would_cite":true,"duration_ms":28342,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ORCA predicts both the expected correctness and the disagreement of human judgments for open-ended audio question answering, using a Beta distribution over ratings, and a small text-only model matches or beats large LLM judges while providi","keywords":["audio question answering","answer correctness assessment","LLM-as-a-judge","Beta distribution","uncertainty quantification","annotation disagreement","LALM evaluation","open-ended response evaluation"],"falsifier":"Construct a set of audio QA items where the audio contains decisive information that the transcript and rationale systematically omit or misrepresent—for example, a sound event mislabeled in the rationale with no transcript available—and check whether ORCA's ratings and confidence track human ratings made with access to the audio. If ORCA's accuracy and calibration collapse on these items while human judges who can listen still agree, the text-only grounding premise fails.","tokens_in":15132,"feed_emoji":"🎧","tokens_out":3499,"duration_ms":36617,"temperature":0.7,"pith_summary":"ORCA is a lightweight, text-only evaluator for open-ended audio question answering that predicts the full distribution of human correctness ratings, not just the average. It models ratings with a Beta distribution, outputting both the expected score and the variance, so it can tell consensus from genuine disagreement. Trained on human annotations collected through a three-stage pipeline with structured feedback and expert correction, ORCA reaches a Spearman correlation of about 0.91 with average human scores on unseen questions, matching or beating much larger LLM judges while requiring a single forward pass. The paper argues that this makes audio QA evaluation cheaper, more reproducible, and able to flag ambiguous or flawed benchmark items instead of hiding them in a mean score.","feed_headline":"Audio QA judge predicts human disagreement, not just scores","feed_subtitle":"ORCA matches big LLM judges on audio QA with far less compute, and its predicted variance exposes ambiguous questions.","key_machinery":"The central object is the Beta distribution over normalized correctness ratings. A small pre-trained transformer encodes the concatenated text inputs, and an MLP head predicts log α and log β. The mean α/(α+β) serves as the point estimate of correctness, while the variance αβ/((α+β)^2(α+β+1)) serves as the uncertainty estimate. Training maximizes the log-likelihood of each individual human rating, treating every rating as a sample from the predicted distribution. A clamping post-processing step sharpens predictions that are near zero or one and have low predicted variance.","core_discovery":"On the paper's own terms, ORCA establishes that answer correctness for open-ended audio QA can be assessed reliably by a small text-only model—fed the question, reference answer, a machine-generated rationale, and a speech transcript—when the model is trained to predict a Beta distribution over human ratings. The model outputs log α and log β, whose mean and variance match human average scores and human disagreement. In experiments on two audio QA benchmarks with ratings from 15 different audio language models, the best ORCA configuration reaches a Spearman correlation of 0.91 with mean human judgments on unseen-question splits and also achieves low mean-absolute error on predicted variance,","pith_inferences":["If ORCA's variance estimates are trustworthy, the same Beta-distribution head could be adapted to any evaluation or self-assessment task, letting models report calibrated confidence instead of just a score.","The three-stage human-AI refinement loop implicitly measures benchmark quality: the paper's correction statistics suggest existing audio QA benchmarks contain substantial shares of ambiguous or flawed questions, which distributional evaluation surfaces.","A natural extension the paper does not pursue is active learning: use ORCA's predicted variance to decide which items need additional human annotation, concentrating effort where disagreement is predicted to be high.","The text-only premise could be stress-tested on categories where audio carries information invisible to transcripts and rationales, such as emotional nuance or non-speech sound texture—the exact cases where human annotators most often disagree."],"forward_implications":["Open-ended audio QA evaluation no longer requires an audio model as judge, avoiding the circularity of using audio-language models to rate other audio-language models.","Benchmark reports can include a per-item disagreement score, letting researchers identify ambiguous, poorly written, or under-specified questions rather than averaging over them.","A single forward pass through a small open-weight model replaces multi-step LLM judging, cutting computational cost and improving reproducibility.","The curated dataset of thousands of human ratings across many audio-language models can serve as a reusable evaluation resource for the community.","LLM-judge ratings, despite being noisier, can be used for pretraining, pointing toward reduced human annotation burden for future benchmarks."],"fun_headline_variants":["Lightweight audio QA judge also predicts ambiguity","Small text-only model scores open-ended audio answers","Audio QA: tiny judge matches big LLMs on correctness","Predicting human disagreement in open-ended audio QA","ORCA: efficient audio QA assessment with variance insight"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole evaluation—for annotators, ORCA, and LLM judges—rests on the assumption that the text-only context (question, reference answer, machine-generated rationale, and transcript) carries enough of the audio's content for correctness to be judged; the paper itself acknowledges in Stage 1a that automatically generated rationales can be non-informative or based on textual cues rather than on the audio.","fun_headline_variants_meta":{"raw":{"variants":["Lightweight audio QA judge also predicts ambiguity","Small text-only model scores open-ended audio answers","Audio QA: tiny judge matches big LLMs on correctness","Predicting human disagreement in open-ended audio QA","ORCA: efficient audio QA assessment with variance insight"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000396,"raw_usage":{"total_tokens":1897,"prompt_tokens":714,"completion_tokens":1183,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":1110}},"tokens_in":458,"tokens_out":1183,"duration_ms":12233,"temperature":1.0,"reasoning_tokens":1110,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T19:33:31.266460+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a set of audio QA items where the audio contains decisive information that the transcript and rationale systematically omit or misrepresent—for example, a sound event mislabeled in the rationale with no transcript available—and check whether ORCA's ratings and confidence track human ratings made with access to the audio. If ORCA's accuracy and calibration collapse on these items while human judges who can listen still agree, the text-only grounding premise fails.","supporting_citations":[],"review_version":1}