{"id":"69c8de90-bfbf-4a91-9824-308c43a29d42","arxiv_id":"2607.06452","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":5,"one_line_summary":"A question-type-specific LLM ensemble and multi-agent pipeline achieved competitive results on BioASQ 14b Task B, including first place in the factoid subtask of Batch 4.","lead":"This paper describes a competition system for biomedical question answering that uses different LLM pipelines for yes/no, factoid, and list questions, combining ensemble voting, chain-of-thought in-context learning, and multi-agent verification. A smart generalist might read it to understand how question-type-specific prompt engineering and agent collaboration affect LLM performance on domain-specific QA benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"glm-5.2","headline":"The paper's evidence cannot support its causal claim about strategy effectiveness: official results are per-metric maximums across five variants, and development experiments show proposed strategies underperforming baselines on yes/no and factoid CoT-ICL without analysis.","rationale":"The reader correctly identified the two core issues: per-metric maximum reporting and the yes/no baseline outperformance. I extend this by noting that the factoid development experiments also show a proposed strategy (CoT-ICL) hurting performance for GPT-5.2, and that the cross-batch list improvement narrative is confounded by variant tuning rather than pipeline refinement. These are all facets of the same fundamental problem: the paper's evidence structure cannot support causal attribution to its design choices.\n\nThe per-metric maximum issue is the most load-bearing because it affects the official results, which are the paper's primary evidence. The authors acknowledge it in §6 but continue to use the maximum-based numbers in the abstract, conclusion, and effectiveness arguments. The development experiment issues are secondary but compound the problem: even the controlled comparisons meant to justify strategy selection show mixed or negative results.\n\nThe paper does have legitimate strengths: it is a real competition system with official evaluation, the multi-agent list pipeline is a reasonable design, and the prompt templates in Appendix A provide transparency. The factoid Batch 4 first-place result is a genuine achievement. But 'first place in one batch on one metric' is thin evidence for the broad claim that the three type-specific strategies are effective, especially when the development experiments contradict that claim for yes/no and partially contradict it for factoid.\n\nThe CONDITIONAL verdict is appropriate. The work is legitimate but the claims about strategy effectiveness are not adequately supported by the evidence presented. A single-variant Table 4 would clarify how much of the reported performance comes from the system design versus variant selection.","tokens_in":14400,"tokens_out":4666,"duration_ms":158773,"concrete_test":"Recompute Table 4 using a single fixed variant (e.g., ku_dmis1) across all four batches and all metrics, and place it alongside the per-metric maximum. If single-variant factoid MRR and list F-measure drop by more than 15% relative compared to the per-metric maximum, the headline numbers are substantially inflated by variant selection and the claim of strategy effectiveness is not supported by the official results. This is feasible from existing submission data since each variant's per-batch scores are already recorded by the BioASQ evaluation system.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that type-specific strategies—snippet shuffling for yes/no, CoT-ICL for factoid, multi-agent for list—are 'effective' for biomedical QA. Two evidence sources should support this, but neither can bear the weight.\n\nFirst, Table 4 reports per-metric maximums across five submission variants (ku_dmis1–5), not any single system's performance. The authors acknowledge this in §6, but the abstract and conclusion still present these numbers as evidence of effectiveness. Table 5 shows that different variants are best for different metrics and batches: for factoid MRR, the best variant shifts from ku_dmis1 (Batch 1) to ku_dmis2–5 (Batch 3) to ku_dmis3,5 (Batch 4). No single variant achieves all reported numbers, so headline performance is partly an artifact of variant selection.\n\nSecond, the development experiments (Tables 1–3) do not consistently show the proposed strategies outperforming baselines. For yes/no (Table 1), the previous baseline achieves 1.0000 macro F1, strictly outperforming all GPT-4o variants (0.9444) and GPT-5 variants (0.9244). The paper states it 'extended this baseline with self-reflection, repeated inference, and a verification agent to improve robustness' but the extended system performs worse, with no analysis. For factoid (Table 2), GPT-5.2 with CoT-ICL2 drops to 0.4230 MRR from 0.5320 without CoT—CoT-ICL actively hurts this model. The paper selects GPT-4o+ICL2 (0.5580 MRR) for the final system but does not explain why CoT-ICL helps one model and hurts another.\n\nThird, the paper attributes cross-batch list improvement (F-measure 0.2734→0.5652) to 'the addition and refinement of the collaborative list-answer pipeline' (§5.3), but the variants differ in aggregation thresholds, not pipeline structure. The improvement could reflect variant tuning to batch characteristics rather than pipeline effectiveness.\n\nCombined, these issues mean the paper cannot demonstrate that its specific design choices—rather than variant selection, model ca","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This paper presents the ku_dmis system for BioASQ 14b Task B, which routes questions by type (yes/no, factoid, list) to different LLM-based inference strategies. For yes/no questions, the system uses snippet shuffling with majority voting and selective verification. For factoid questions, it combines full-snippet input with retrieval-based chain-of-thought in-context learning and multi-model consensus filtering. For list questions, it employs a four-agent collaborative pipeline (evidence analyst, reasoning, verification, supervisor). Development experiments on BioASQ 13b informed strategy selection, and the final system was evaluated on the official BioASQ 14b test set across four batches, achieving competitive results including first place in factoid MRR for Batch 4.","tokens_in":14711,"tokens_out":1904,"duration_ms":129866,"significance":"The paper provides a clear, well-structured system description for a competitive shared-task submission. The answer-type-aware routing design is sensible, and the list-question multi-agent pipeline is described in sufficient detail (including prompt templates in Appendix A) to be reproducible in principle. The official BioASQ 14b evaluation provides an external benchmark with independently prepared gold answers. The variant analysis in Table 5 offers useful insight into which design choices matter for which question types. The paper is appropriate in scope for a CLEF working notes contribution.","major_comments":[{"comment":"§5.1, Table 4 and Abstract/Conclusion: Table 4 reports per-metric maximums across five submission variants (ku_dmis1–5), not the performance of any single fixed system. The authors acknowledge this in §6, but the abstract ('our framework showed competitive performance across multiple batches') and conclusion ('the system achieved average scores of 0.9110... 0.4626... 0.4439') present these numbers as if they describe one system. Table 5 confirms that no single variant achieves all reported values. This framing overstates the evidence for the central claim. The abstract and conclusion should either (a) report a single representative variant's scores, or (b) explicitly state in both places that the numbers are per-metric maxima across variants, not a single system's performance.","section":null},{"comment":"§4.1, Table 1: The 'previous baseline' (the authors' own BioASQ 13b system [8]) achieves 1.0000 macro F1 on yes/no questions, strictly outperforming all proposed GPT-4o variants (0.9444) and GPT-5 variants (0.9244). The paper states it 'extended this baseline with self-reflection, repeated inference, and a verification agent to improve robustness,' yet the extended system performs worse. The paper offers no analysis of why the proposed yes/no strategies underperform the baseline on the development set. Since the yes/no subsystem is a load-bearing component of the type-aware framework claim, this gap should be addressed: either explain the discrepancy (e.g., test-set size, baseline overfitting to 13b) or qualify the yes/no effectiveness claim.","section":null},{"comment":"§4.2, Table 2: For GPT-5.2, adding CoT-ICL2 reduces MRR from 0.5320 to 0.4230, a substantial degradation. The paper selects GPT-4o+ICL2 (0.5580 MRR) for the final factoid system, which is reasonable, but the text in §4.2 states that 'in-context learning and full-snippet input were effective in the BioASQ-13b development experiments' without acknowledging that CoT-ICL actively hurt the strongest single model (GPT-5.2). The claim that CoT-ICL is effective for factoid QA should be qualified to note that its benefit is model-dependent.","section":null}],"minor_comments":[{"comment":"Table 1: The yes/no test set appears to contain 17 questions (0.9412 accuracy = 16/17). This small sample size should be noted, as it affects the reliability of the development comparisons.","section":null},{"comment":"§4.1: The final yes/no system uses GPT-5 Thinking, GPT-5 Pro, GPT-4o, and Gemini 3.0 Thinking, but Table 1 reports development results for GPT-4o, GPT-5.2, GPT-5.4, Claude Sonnet-4.6, and Opus-4.6. The relationship between the development models and the deployed models is unclear and should be clarified.","section":null},{"comment":"§4.4: The submission variants are described only vaguely ('small changes in prompting, answer normalization, and aggregation thresholds'). A table listing the specific differences between ku_dmis1–5 would improve reproducibility.","section":null},{"comment":"Figure 1: The 'Selective Verification' box in the yes/no branch and the 'Supervisor Agent' boxes are shown but their connection to the output is not clearly drawn. The figure could be improved.","section":null},{"comment":"§5.1: The paper attributes list-question improvement across batches to 'the addition and refinement of the collaborative list-answer pipeline,' but does not clarify whether the pipeline itself changed between batches or whether the improvement reflects test-set difficulty differences. This should be disambiguated.","section":null},{"comment":"References [28]–[38]: Several model references point to product announcement pages rather than technical reports. Where model cards or technical reports exist, they should be cited instead.","section":null}],"recommendation":"major_revision","confidential_remarks":"This is a CLEF working notes paper, where the rigor bar is typically lower than for archival venues. However, the mismatch between the per-metric-maximum reporting and the single-system framing in the abstract/conclusion is a substantive issue that should be corrected before publication. The unaddressed yes/no baseline discrepancy is also notable given that it is the authors' own prior system. I would not expect these to require extensive revision, but they do need explicit acknowledgment and qualification in the abstract and conclusion."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive review. All three major comments identify legitimate issues in how results and claims are framed. We agree with each point and will revise the manuscript accordingly: (1) the abstract and conclusion will explicitly state that reported numbers are per-metric maxima across submission variants, not a single system's performance; (2) we will add analysis of why the proposed yes/no strategies underperform the previous baseline on the development set, with appropriate qualification of the yes/no effectiveness claim; and (3) we will qualify the CoT-ICL effectiveness claim for factoid QA to note that its benefit is model-dependent, acknowledging the GPT-5.2 degradation. No standing objections remain.","responses":[{"response":"The referee is correct. The abstract and conclusion currently present per-metric maxima across five submission variants as if they describe a single system, which is misleading. We acknowledge this framing issue. While §6 (Limitations) does note that Table 4 reports per-metric maxima rather than a single fixed system's scores, this qualification should appear in the abstract and conclusion as well, not only in the limitations section. We will revise both the abstract and conclusion to explicitly state that the reported numbers are per-metric maxima across submission variants, not the performance of any single fixed system. We will retain the per-metric maximum presentation in Table 4 itself (as the caption already states this), but will ensure the abstract and conclusion are transparent about this. We considered the alternative of reporting a single representative variant's scores, but since different variants were optimal for different question types and batches, no single variant is truly representative, and reporting one would understate the range of strategies explored. Explicitly labeling the numbers as per-metric maxima is the more honest approach.","revision_made":"yes","referee_comment":"§5.1, Table 4 and Abstract/Conclusion: Table 4 reports per-metric maximums across five submission variants (ku_dmis1–5), not the performance of any single fixed system. The abstract and conclusion present these numbers as if they describe one system, overstating the evidence for the central claim. The abstract and conclusion should either report a single representative variant's scores or explicitly state that the numbers are per-metric maxima across variants."},{"response":"The referee correctly identifies a gap in our analysis. The previous baseline achieving 1.0000 macro F1 on the BioASQ-13b yes/no development set while all proposed variants score lower is a striking result that we should have addressed. We will add analysis of this discrepancy in the revised manuscript. The most likely explanation is overfitting to the 13b test set: the previous baseline was specifically developed and tuned on 13b data, and the yes/no subset is small (17 questions, as indicated by the granularity of the scores), making perfect performance plausible but not necessarily generalizable. The proposed extensions (self-reflection, repeated inference, verification agent) were designed for robustness on unseen data rather than to maximize performance on the 13b development set specifically. We will add this explanation to §4.1 and qualify the yes/no effectiveness claim to note that the proposed strategies did not improve over the previous baseline on the development set, while noting that the development set size limits the reliability of comparisons. We will also adjust the language in §4.1 to avoid implying that the extensions were demonstrated to improve yes/no performance on the development set.","revision_made":"yes","referee_comment":"§4.1, Table 1: The previous baseline (authors' own BioASQ 13b system [8]) achieves 1.0000 macro F1 on yes/no questions, strictly outperforming all proposed GPT-4o variants (0.9444) and GPT-5 variants (0.9244). The paper states it extended this baseline with self-reflection, repeated inference, and a verification agent to improve robustness, yet the extended system performs worse. No analysis is offered for why the proposed yes/no strategies underperform the baseline on the development set."},{"response":"The referee is correct. The current text in §4.2 makes an unqualified claim that CoT-ICL was effective, when Table 2 clearly shows that CoT-ICL2 reduced GPT-5.2's MRR from 0.5320 to 0.4230—a substantial degradation for the strongest single model. This is an important omission. We will revise §4.2 to explicitly acknowledge that the benefit of CoT-ICL is model-dependent: it improved GPT-4o's MRR from 0.4420 to 0.5580 but degraded GPT-5.2's MRR from 0.5320 to 0.4230. We will qualify the general claim about CoT-ICL effectiveness to note this model dependency and explain that the final factoid system's design—using GPT-4o with ICL2 as a primary component while incorporating multiple models with consensus filtering—was motivated in part by this observation. The revised text will present a more balanced picture of when CoT-ICL helps and when it hurts.","revision_made":"yes","referee_comment":"§4.2, Table 2: For GPT-5.2, adding CoT-ICL2 reduces MRR from 0.5320 to 0.4230, a substantial degradation. The text states that 'in-context learning and full-snippet input were effective in the BioASQ-13b development experiments' without acknowledging that CoT-ICL actively hurt the strongest single model (GPT-5.2). The claim that CoT-ICL is effective for factoid QA should be qualified to note that its benefit is model-dependent."}],"tokens_in":14344,"tokens_out":1262,"duration_ms":190244,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Two things matter here. First, this is a BioASQ 14b competition system description that combines known techniques (ensemble voting, CoT ICL, multi-agent verification) in a question-type-specific pipeline, with official evaluation results including a first-place factoid MRR finish in Batch 4. Second, the headline numbers in Table 4 are per-metric maximums across five submission variants, not a single system's performance — and the authors acknowledge this in the limitations section but still frame the abstract and conclusion as evidence of strategy effectiveness. That gap is the main problem.","headline":"Competition system paper with honest reporting but a per-metric-maximum presentation problem that undermines the effectiveness claim","tokens_in":15400,"tokens_out":178,"would_cite":false,"duration_ms":154058,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Type-Aware Pipelines Beat One-Size-Fits-All for Biomedical QA","keywords":["biomedical question answering","question-type-aware routing","large language models","multi-agent systems","ensemble methods","chain-of-thought prompting","BioASQ"],"falsifier":"If a single uniform prompting strategy applied across all question types matched or exceeded the type-routed system on the same set of BioASQ 14b questions, the central design claim would be undermined.","tokens_in":14531,"feed_emoji":"🧬","tokens_out":752,"duration_ms":136075,"temperature":0.7,"pith_summary":"This paper argues that biomedical question answering systems should route questions to different inference pipelines based on the expected answer type (yes/no, factoid, or list) rather than using a single prompting strategy for all questions. The authors build and deploy a system for the BioASQ 14b challenge that uses snippet-shuffled majority voting for binary questions, chain-of-thought in-context learning with multi-model consensus for factoid questions, and a four-agent collaboration pipeline for list questions. The central claim is that matching the inference mechanism to the structural demands of the answer type yields more reliable results than a uniform approach.","feed_headline":"Type-Aware LLM Pipelines Beat One-Size-Fits-All for Biomedical QA","feed_subtitle":"Routing questions by answer type to specialized ensemble and agent pipelines yields competitive results on BioASQ 14b, including first place","key_machinery":"The central mechanism is a type router that dispatches each question to one of three specialized pipelines: (1) a yes/no branch using snippet-order shuffling across four LLMs with selective verification on disagreements, (2) a factoid branch using full-snippet input with BioBERT-retrieved chain-of-thought demonstrations and cross-model consensus filtering, and (3) a list branch using a four-agent chain (Evidence Analyst, Reasoning Agent, Verification Agent, Supervisor) that separates evidence extraction, candidate generation, auditing, and final aggregation.","core_discovery":"The paper demonstrates that a type-routed architecture—combining ensemble voting for yes/no, retrieval-based in-context learning with consensus filtering for factoid, and a sequential multi-agent verification pipeline for list questions—achieves competitive performance on BioASQ 14b, including first place on the factoid subtask of Batch 4 by MRR. The development experiments on BioASQ 13b show that the best strategy varies by question type: ensemble voting stabilized yes/no decisions, in-context learning improved factoid entity identification, and agent-based architectures outperformed single-pass inference on list questions.","pith_inferences":[],"forward_implications":["Type-aware routing could be extended to additional answer formats such as summary or explanatory questions, each with its own specialized pipeline.","The multi-agent verification architecture for list questions could be adapted to other domains requiring high-precision entity extraction from scattered evidence, such as legal or financial document analysis.","The selective verification trigger on yes/no questions—only invoking agents when models disagree—suggests a cost-efficient pattern for deploying expensive verification only on uncertain cases.","Retrieval-based in-context learning with verified chain-of-thought exemplars could be generalized as a technique for improving surface-form matching in any exact-answer evaluation setting."],"fun_headline_variants":["Type-routed LLM pipelines outperform single-strategy approaches in biomedical QA","Specialized LLM pipelines per question type achieve top BioASQ 14b factoid score","Question-type routing improves biomedical QA on BioASQ 14b factoid subtask","Routing biomedical questions to type-specific LLM agents on BioASQ 14b","Adapting LLM inference by answer type improves robustness in biomedical QA"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The headline results in the main results table report the maximum score per metric across five separately submitted system variants, not the performance of any single fixed configuration; this means no one system achieves all the reported numbers simultaneously, and the relative contribution of each design choice cannot be cleanly isolated from variant selection.","fun_headline_variants_meta":{"raw":{"variants":["Type-routed LLM pipelines outperform single-strategy approaches in biomedical QA","Specialized LLM pipelines per question type achieve top BioASQ 14b factoid score","Question-type routing improves biomedical QA on BioASQ 14b factoid subtask","Routing biomedical questions to type-specific LLM agents on BioASQ 14b","Adapting LLM inference by answer type improves robustness in biomedical QA"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1654,"prompt_tokens":570,"completion_tokens":1084,"prompt_tokens_details":null},"tokens_in":570,"tokens_out":1084,"duration_ms":46666,"temperature":1.0,"reasoning_tokens":931,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T05:07:53.285078+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a single uniform prompting strategy applied across all question types matched or exceeded the type-routed system on the same set of BioASQ 14b questions, the central design claim would be undermined.","supporting_citations":[],"review_version":1}