{"id":"28a30711-3b69-4db2-86df-56dafad86175","arxiv_id":"2611.16425","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"low","formal_verification":"none","parameter_count":11,"one_line_summary":"The RMIT-ADM+S team submitted seven RAG systems to TREC 2026; their best system tied the track maximum on Batch-1 vital coverage, while an open-weight system was strongly preferred by human judges despite lower coverage and at a fraction of the cost.","lead":"This paper describes seven retrieval-augmented generation (RAG) systems submitted to the TREC 2026 RAG track by the RMIT-ADM+S team. The study highlights how automated coverage metrics and human preferences can rank the same systems differently, and shows that a lower-cost open-weight model can be competitive on human preference despite lower automated coverage.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Batch-1 SxS preference for owa-glm5 may depend heavily on topic 68; sensitivity analysis not reported.","rationale":"The paper’s central claims (six of seven systems above median, best ties track max, open-weight preferred in Batch 1) are well-supported by the official NIST evaluation. My identified concern targets only the robustness of the Batch-1 human preference result, which the paper already presents with caveats (small N, protocol differences, per-topic outliers). The reader’s weakest_assumption already flags the context-dependence of the human evaluation; my concern adds a specific analytical gap within that context. Even if the sensitivity test weakens the preference claim, the paper’s overll contribution as a system description and empirical comparison remains valid and appropriately hedged. The verdict should stay unchanged because the core finding about coverage and the cost comparison are unaffected, and the paper does not overclaim generalizability. I agree partially with the reader: the evaluation measures are load-bearing, but the key missing check is not about measurement validity but about sensitivity of the primary significant result to single outliers. This is a legitimate stress-t that would improve the paper, but it does not invalidate the overll verdict.","tokens_in":18091,"tokens_out":3447,"duration_ms":28919,"concrete_test":"Recompute the Batch-1 Bradley-Terry rating and topic-level sign-flip paired test after excluding topic rag2026-68 (school safety). If the p-value increases to >0.05, the claimed significance is not robust to removal of a single outlier; report the new BT ratings and comment on the fragility of the result. If the p-value remains ≤0.05, the preference claim is stable; this would strengthen the paper's argument.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's claim that the open-weight model owa-glm5 was \"significantly preferred\" over sol-research-first in Batch-1 human side-by-side evaluation (sign-flip p=0.006) is a headline result supporting the competitive potential of open-weight models. However, the per-topic breakdown reveals that topic 68 (school safety) is an extreme outlier: owa-glm5 won 13 of 17 comparisons on this topic despite far lower coverage, while on other topics preferences were mixed or reversed. With only 17 topics total, a single topic can dominate the aggregate Bradley-Terry rating and the sign-flip test. The paper acknowledges per-topic variation (Section 5.3) but does not conduct a leave-one-out or jackknife sensitivity analysis to verify that the significance survives excluding this influential case. If the p-value rises above 0.05 after removing topic 68, the claim of robust preference would be undermined, weakening the paper's central narrative about open-weight competitiveness. This is not a flaw in the NIST evaluation but an analytical omission: without sensitivity testing, the reader cannot assess how much of the headline preference rests on one exceptional topic.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper describes the RMIT-ADM+S team's participation in the TREC 2026 RAG track, presenting seven runs that combine proprietary and open-weight models with different retrieval strategies, writer models, and agent workflows. The main empirical findings are: (1) six of seven runs exceed the global track median on all official coverage measures; (2) the open-weight system owa-glm5 is significantly preferred by human assessors in Batch-1 side-by-side evaluation despite lower automated coverage, at a fraction of the cost of the proprietary leader; (3) the preference direction reverses in Batch-2, though not significantly; (4) writer choice has the largest effect in pre-submission ablations, while hybrid reciprocal rank fusion offers modest quality and efficiency gains over other retrieval strategies. The paper explicitly cautions that automated coverage and human preference can diverge, that per-topic variation is substantial, and that the internal evaluations are sensitive to judge choice and not directly comparable to official results.","tokens_in":18432,"tokens_out":2604,"duration_ms":21262,"significance":"If the results are robust, the paper provides notable evidence that open-weight models can be competitive on human-preference dimensions in an agentic RAG task, and it systematically documents the divergence between automated coverage metrics and holistic human evaluation. The ablation study (writer choice vs. structural changes) and the cost analysis are practically useful for practitioners building similar systems. The findings also underscore the need for multiple evaluation signals rather than a single metric. The reliance on official NIST evaluation strengthens external validity, though the small number of topics (17 Batch-1) limits precision of the head-to-head preference conclusion.","major_comments":[{"comment":"The headline claim that owa-glm5 is 'significantly preferred' in Batch-1 (sign-flip p=0.066) is an important finding, but it rests on only 17 topics. The paper itself highlights topic 58 as an extreme outlier (13/17 comparisons won on that topic despite far lower coverage). The authors should report a leave-one-out sensitivity analysis to verify that the p-value remains below 0.05 when topic 58 is excluded. Without this, readers cannot assess whether the aggregate significance is driven by a single exceptional case, especially given the small sample size.","section":"5.2 / 5.3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Section 4.1 demonstrates strong sensitivity of pairwise LLM judging to the choice of judge model: win rates against the organizer baseline range from 17.5% to 45.0% depending on the judge. Yet the internal arena against the baseline (Table3) uses a single judge (GPT-5.6-Luna), which Section 4.1 shows can produce substantially different results than other judges. The paper should either add a caveat that the Table3 results are specific to this judge and may not generalize, or report agreement across multiple judges as done in Section 4.1.","section":"4.1 / Table 3","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The phrase 'Hybrid RRF' appears to contain a typo ('Hybrid' should be 'Hybrid').","section":"4.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The caption contains a typo: 'Human-Intuition Rubric' is misspelled as 'Human-Intuition Rubric'.","section":"Figure 3 caption","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The notation 'vital_strict' and 'all_strict' in Table 4 could be confusing; consider adding a footnote that 'strict' scoring means no partial credit for items ('vital' and 'all' items are defined in the track guidelines).","section":"5.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"References are well-formatted and appear complete. The code repository link is provided and seems operational.","section":"General","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This is a well‑documented TREC 2026 RAG track participation report. The paper does what a good system description should: it describes seven configurations, runs pre‑submission ablations with clear caveats about judge sensitivity and topic size, and reportes official coverage and human preference results with appropriate statistics (Tukey HSD, sign‑flip tests). The cost breakdown is useful, and the discussion of divergence between automated coverage and human preference is honest. No new algorithm or theory, but the empirical findings are grounded in an external evaluation by NIST, which gives them weight.\n\nThe main news is that the open‑weight GLM‑5 system was significantly preferred by assessors in Batch‑1 side‑by‑side, at one‑fifth the cost of the proprietary system. The paper’s own per‑topic analysis (Figure 3) shows that Topic 68 drives much of that preference: owa‑glm5 won 13/17 comparisons there despite low coverage. The authors acknowledge this case but do not run a leave‑one‑out sensitivity test. With only 17 topics, a single outlier can inflate the aggregate Bradley‑Terry rating and the p‑value. The stress‑test concern is therefore valid: without that check, the headline claim is less robust than it appears, even though the authors flag the per‑topic variation. This is a moderate analytical omission, not a fatal flaw – the paper already says readability differences are confounded, so it is cautious about interpretation. Still, adding a jackknife would have strengthened the paper.\n\nThe paper’s other findings (writer dominance, hybrid retrieval efficiency, coverage vs. human preference divergence) are consistent with prior work and stated with appropriate modesty. The internal evaluation caveats are thorough. The worflow diagrams and factor catalog are helpful for practitioners.\n\nWho is this for? Researchers building agentic RAG systems, especially those following TREC tracks. It does not claim more than it delivers. A desk reject would be wrong; the paper deserves a serious referee and would benefit from a request to supply the sensitivity analysis for the Batch‑1 preference claim. I would bring it to a reading group only if the group is focused on system engineering rather than fundamental advances, and I would not directly cite it, but it is a solid piece of empirical work.","headline":"A solid TREC system report with transparent ablations and honest caveats; the open‑weight preference claim is real but the paper could have checked its robustness to a single dominant topic.","tokens_in":18965,"tokens_out":1800,"duration_ms":16129},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Open-weight agent beats proprietary model in human preference, but coverage tells a different story","keywords":["retrieval-augmented generation","TREC RAG","agentic retrieval","Reciprocal Rank Fusion","open-weight models","human evaluation","coverage metrics","side-by-side preference"],"falsifier":"A controlled experiment that fixes answer length and reading level across the open-weight and proprietary systems while preserving their other differences (tone, uncertainty handling, citation style) could falsify the claim that readability contributed to the observed human preference; if the preference persists after controlling for those factors, then other unmeasured stylistic or structural features must be at play.","tokens_in":21234,"feed_emoji":"🤖","tokens_out":2758,"duration_ms":63295,"temperature":0.7,"pith_summary":"This TREC 2026 RAG track participation report argues that open-weight research agents can match or exceed proprietary systems in human preference at a fraction of the generation cost, while automated coverage metrics may rank the same systems in opposite order. The best coverage was achieved by a hybrid-retrieval system that tied the track maximum on Batch-1 vital coverage. However, in the same Batch-1 evaluation, the open-weight owa-glm5 was significantly preferred by human assessors despite lower coverage scores, at roughly one-fifth the generation cost. The authors conclude that there is no single best architecture for agentic RAG and that evaluation methods—automated versus human, holistic versus nugget-guided—are themselves consequential design choices.","feed_headline":"Open-weight agent wins human preference at fractional cost","feed_subtitle":"At one-fifth the cost, GLM-5 beat GPT-5.6 in holistic human preference; coverage metrics gave opposite ranking.","key_machinery":"The experimental design compares seven research agents built on a shared staged-evidence architecture that varies the writer model (proprietary GPT-5.6 variants vs open-weight GLM-5), retrieval method (semantic, keyword, hybrid Reciprocal Rank Fusion with κ=60), and planning components (pre-flight briefs, blind obligation scouts, post-draft review). The evaluation machinery pairs automated coverage metrics (vital_strict and all_strict over three rubric and nugget collections) with human side-by-side Bradley-Terry preferences from two batches using different assessment protocols. The divergence between these two evaluation signals carries the paper's main argument about the sensitivity of system rankings to choice of assessment method.","core_discovery":"The paper's central empirical claim is that six of seven submitted systems exceeded the TREC 2026 RAG track median across all three official rubric and nugget collections. The proprietary brief-hybrid system tied the global maximum for Batch-1 vital coverage at 0.892. More surprisingly, the open-weight owa-glm5 (based on GLM-5) was significantly preferred in Batch-1 human side-by-side evaluation (Bradley-Terry rating 1212.8 vs 1062.7, p=0.006), even though it scored lower on automated coverage. This preference reversed in Batch-2 under a nugget-guided protocol, though the difference was not statistically significant. The paper identifies writer model choice as the largest effect among measured development ablations and hybrid Reciprocal Rank Fusion as the most efficient retrieval design. The authors conclude that evaluation protocol can reverse system rankings and that no single architecture dominates.","pith_inferences":["The combined cost advantage and reading-ease advantage of owa-glm5 suggest that applications prioritizing human satisfaction and readability over exhaustive recall—such as consumer-facing summarisation—may particularly benefit from open-weight models.","The review-pass failure on topic 65 (a 21-word collapsed revision) highlights a brittle step in agentic RAG pipelines; future work could design explicit review instructions or fallback mechanisms that prevent such collapse.","The strong correlation between answer length and coverage under automated nugget evaluation (Spearman 0.72) raises the question of whether nugget metrics inadvertently reward length; an information-density-adjusted metric would be worth exploring.","The paper's reader-response analysis suggests that human preference may be partly driven by style and reading ease, not just factual completeness; a controlled experiment that varies these factors independently could separate their contributions."],"forward_implications":["Open-weight models can provide competitive quality for agentic RAG tasks at substantially lower generation cost ($0.52 vs $2.82 per topic in this study), lowering the barrier for deployment.","Future RAG evaluations should incorporate multiple complementary signals—automated coverage, human preference, and perhaps readability—rather than relying on any single metric.","The design choice of writer model had a larger measured effect than any structural pipeline change, suggesting that effort on model selection may yield more consistent gains than complex orchestration.","Hybrid retrieval with Reciprocal Rank Fusion and adjacent-chunk expansion improved efficiency (14% fewer searches, 8% fewer tokens) without sacrificing coverage, making it a pragmatic default.","The reversal of human preferences between holistic and nugget-guided protocols implies that tracks must explicitly specify which quality dimension they value, as rankings are not transferable."],"supporting_citations":[{"why":"Defines the TREC 2026 RAG track task, evaluation protocols, and the organizer's baseline system.","marker":"[23]"},{"why":"Supplies the ClimbMix-400B-shuffle corpus, the document collection for all submitted runs.","marker":"[9]"},{"why":"Provides the Reciprocal Rank Fusion method used in the best-coverage runs brief-hybrid and owa-glm5.","marker":"[8]"},{"why":"Supplies the Bradley–Terry model used to analyze human side-by-side preferences.","marker":"[3]"},{"why":"Introduces the AutoNuggetizer framework used for automated nugget evaluation.","marker":"[17]"},{"why":"Provides the nugget-recall methodology used for Batch-2 comparison.","marker":"[18]"},{"why":"Cited to support the sensitivity of LLM-based judges to evaluator and presentation choices, a key limitation noted by the authors.","marker":"[10]"},{"why":"Informs the paragraph-aware chunking strategy used to build the search index.","marker":"[31]"}],"fun_headline_variants":["Open-weight agent wins human preference at 1/5 cost","GLM-5 beats GPT-5.6 in human eval despite lower coverage","TREC RAG: Six of seven runs beat median; open model wins","Coverage metrics flip: human side-by-side favors open-weight","Fractional-cost open model tops human preference in RAG"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper assumes that the official NIST coverage scores and human side-by-side Bradley–Terry ratings are valid and reliable measures of answer quality, despite the authors acknowledging that automated and human signals can diverge, that the two human batches used different topics and protocols, and that automated judges are sensitive to evaluator choice and presentation order.","fun_headline_variants_meta":{"raw":{"variants":["Open-weight agent wins human preference at 1/5 cost","GLM-5 beats GPT-5.6 in human eval despite lower coverage","TREC RAG: Six of seven runs beat median; open model wins","Coverage metrics flip: human side-by-side favors open-weight","Fractional-cost open model tops human preference in RAG"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000532,"raw_usage":{"cost":0.000531608,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000531608,"upstream_inference_prompt_cost":0.000101528,"upstream_inference_completions_cost":0.00043008},"total_tokens":2466,"prompt_tokens":930,"completion_tokens":1536,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":1443}},"tokens_in":930,"tokens_out":1536,"cost_source":"openrouter_usage","duration_ms":12215,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":1443,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T15:14:48.557646+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"A controlled experiment that fixes answer length and reading level across the open-weight and proprietary systems while preserving their other differences (tone, uncertainty handling, citation style) could falsify the claim that readability contributed to the observed human preference; if the preference persists after controlling for those factors, then other unmeasured stylistic or structural features must be at play.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the TREC 2026 RAG track task, evaluation protocols, and the organizer's baseline system."},{"cited_title":"Nemotron-CLIMB: Clustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training , url =","cited_arxiv_id":null,"evidence_quote":"Supplies the ClimbMix-400B-shuffle corpus, the document collection for all submitted runs."},{"cited_title":"Reciprocal rank fusion outperforms condorcet and individual rank learning methods","cited_arxiv_id":null,"evidence_quote":"Provides the Reciprocal Rank Fusion method used in the best-coverage runs brief-hybrid and owa-glm5."},{"cited_title":"Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons","cited_arxiv_id":null,"evidence_quote":"Supplies the Bradley–Terry model used to analyze human side-by-side preferences."},{"cited_title":"Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework","cited_arxiv_id":"2411.09607","evidence_quote":"Introduces the AutoNuggetizer framework used for automated nugget evaluation."},{"cited_title":"The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models","cited_arxiv_id":null,"evidence_quote":"Provides the nugget-recall methodology used for Batch-2 comparison."},{"cited_title":"URLhttps://www.cs.unh.edu/ ~dietz/papers/ dietz2025principles.pdf","cited_arxiv_id":null,"evidence_quote":"Cited to support the sensitivity of LLM-based judges to evaluator and presentation choices, a key limitation noted by the authors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Informs the paragraph-aware chunking strategy used to build the search index."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}