Pith.

Information Retrieval · arXiv:2611.16425 v1 · submitted 12 Nov 2026 · 33 references

RMIT-ADM+S at TREC RAG 2026: Research Agents, Retriever Ablations, and Open-Weight Replication

Oleg Zendel, Kun Ran, Chenglong Ma, Nguyen Khoi Vo, Shuai Wang

The paper says

“We describe the RMIT-ADM+S team's participation in the TREC 2026 Retrieval-Augmented Generation (RAG) track using the ClimbMix corpus. We submitted seven research-agent runs using proprietary OpenAI and open-weight models, exploring different retrieval strategies, writer models, and agent workflows.”

more of the abstract

Pre-submission ablations identified writer selection and adjacent-chunk expansion as the main factors affecting answer quality. In the official NIST evaluation, six of seven runs exceeded the track median across all three rubric/nugget collections. Our best-performing run, brief-hybrid, achieved the highest all-item coverage among our submissions and tied the track maximum (0.892) for Batch-1 vital coverage. Human side-by-side preferences differed across evaluation batches: the open-weight owa-glm5 led Batch 1 despite lower automated coverage, while sol-research-first led Batch 2. These findings highlight differences between automated coverage measures and human preferences, as well as the potential of lower-cost open-weight models for research-agent systems.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. A solid TREC system report with transparent ablations and honest caveats; the open‑weight preference claim is real but the paper could have checked its robustness to a single dominant topic. This line is wrong

2 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 33 references say.

The paper says, at 5.2 / 5.3

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The headline claim that owa-glm5 is 'significantly preferred' in Batch-1 (sign-flip p=0.066) is an important finding, but it rests on only 17 topics. The paper itself highlights topic 58 as an extreme outlier (13/17 comparisons won on that topic despite far lower coverage).

more of objection 1

The authors should report a leave-one-out sensitivity analysis to verify that the p-value remains below 0.05 when topic 58 is excluded. Without this, readers cannot assess whether the aggregate significance is driven by a single exceptional case, especially given the small sample size.

The paper says, at 4.1 / Table 3

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

Section 4.1 demonstrates strong sensitivity of pairwise LLM judging to the choice of judge model: win rates against the organizer baseline range from 17.5% to 45.0% depending on the judge. Yet the internal arena against the baseline (Table3) uses a single judge (GPT-5.6-Luna), which Section 4.1 shows can produce substantially different results than other judges.

more of objection 2

The paper should either add a caveat that the Table3 results are specific to this judge and may not generalize, or report agreement across multiple judges as done in Section 4.1.

4 minor objections

The paper says, at 4.2

Pith says: minor objection 1 of 4

The phrase 'Hybrid RRF' appears to contain a typo ('Hybrid' should be 'Hybrid').

The paper says, at Figure 3 caption

Pith says: minor objection 2 of 4

The caption contains a typo: 'Human-Intuition Rubric' is misspelled as 'Human-Intuition Rubric'.

show 2 more

The paper says, at 5.1

Pith says: minor objection 3 of 4

The notation 'vital_strict' and 'all_strict' in Table 4 could be confusing; consider adding a footnote that 'strict' scoring means no partial credit for items ('vital' and 'all' items are defined in the track guidelines).

The paper says, at General

Pith says: minor objection 4 of 4

References are well-formatted and appear complete. The code repository link is provided and seems operational.

What would settle it

Recompute the Batch-1 Bradley-Terry rating and topic-level sign-flip paired test after excluding topic rag2026-68 (school safety). If the p-value increases to >0.05, the claimed significance is not robust to removal of a single outlier; report the new BT ratings and comment on the fragility of the result. If the p-value remains ≤0.05, the preference claim is stable; this would strengthen the paper's argument.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up confidence high
  2. Skeptic partly agrees 29 s · 18,091 tokens read, 3,447 written
  3. Editor wrote the verdict above 16 s · 18,965 tokens read, 1,800 written
  4. Referee minor revision: 2 major objections, 4 minor 21 s · 18,432 tokens read, 2,604 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This is a well‑documented TREC 2026 RAG track participation report. The paper does what a good system description should: it describes seven configurations, runs pre‑submission ablations with clear caveats about judge sensitivity and topic size, and reportes official coverage and human preference results with appropriate statistics (Tukey HSD, sign‑flip tests). The cost breakdown is useful, and the discussion of divergence between automated coverage and human preference is honest. No new algorithm or theory, but the empirical findings are grounded in an external evaluation by NIST, which gives them weight.

The main news is that the open‑weight GLM‑5 system was significantly preferred by assessors in Batch‑1 side‑by‑side, at one‑fifth the cost of the proprietary system. The paper’s own per‑topic analysis (Figure 3) shows that Topic 68 drives much of that preference: owa‑glm5 won 13/17 comparisons there despite low coverage. The authors acknowledge this case but do not run a leave‑one‑out sensitivity test. With only 17 topics, a single outlier can inflate the aggregate Bradley‑Terry rating and the p‑value. The stress‑test concern is therefore valid: without that check, the headline claim is less robust than it appears, even though the authors flag the per‑topic variation. This is a moderate analytical omission, not a fatal flaw – the paper already says readability differences are confounded, so it is cautious about interpretation. Still, adding a jackknife would have strengthened the paper.

The paper’s other findings (writer dominance, hybrid retrieval efficiency, coverage vs. human preference divergence) are consistent with prior work and stated with appropriate modesty. The internal evaluation caveats are thorough. The worflow diagrams and factor catalog are helpful for practitioners.

Who is this for? Researchers building agentic RAG systems, especially those following TREC tracks. It does not claim more than it delivers. A desk reject would be wrong; the paper deserves a serious referee and would benefit from a request to supply the sensitivity analysis for the Batch‑1 preference claim. I would bring it to a reading group only if the group is focused on system engineering rather than fundamental advances, and I would not directly cite it, but it is a solid piece of empirical work.

The strongest objection

The paper's claim that the open-weight model owa-glm5 was "significantly preferred" over sol-research-first in Batch-1 human side-by-side evaluation (sign-flip p=0.006) is a headline result supporting the competitive potential of open-weight models. However, the per-topic breakdown reveals that topic 68 (school safety) is an extreme outlier: owa-glm5 won 13 of 17 comparisons on this topic despite far lower coverage, while on other topics preferences were mixed or reversed. With only 17 topics total, a single topic can dominate the aggregate Bradley-Terry rating and the sign-flip test. The paper acknowledges per-topic variation (Section 5.3) but does not conduct a leave-one-out or jackknife sensitivity analysis to verify that the significance survives excluding this influential case. If the p-value rises above 0.05 after removing topic 68, the claim of robust preference would be undermined, weakening the paper's central narrative about open-weight competitiveness. This is not a flaw in the NIST evaluation but an analytical omission: without sensitivity testing, the reader cannot assess how much of the headline preference rests on one exceptional topic.

The paper’s central claims (six of seven systems above median, best ties track max, open-weight preferred in Batch 1) are well-supported by the official NIST evaluation. My identified concern targets only the robustness of the Batch-1 human preference result, which the paper already presents with caveats (small N, protocol differences, per-topic outliers). The reader’s weakest_assumption already flags the context-dependence of the human evaluation; my concern adds a specific analytical gap within that context. Even if the sensitivity test weakens the preference claim, the paper’s overll contribution as a system description and empirical comparison remains valid and appropriately hedged. The verdict should stay unchanged because the core finding about coverage and the cost comparison are unaffected, and the paper does not overclaim generalizability. I agree partially with the reader: the evaluation measures are load-bearing, but the key missing check is not about measurement validity but about sensitivity of the primary significant result to single outliers. This is a legitimate stress-t that would improve the paper, but it does not invalidate the overll verdict.

The referee's report

If the results are robust, the paper provides notable evidence that open-weight models can be competitive on human-preference dimensions in an agentic RAG task, and it systematically documents the divergence between automated coverage metrics and holistic human evaluation. The ablation study (writer choice vs. structural changes) and the cost analysis are practically useful for practitioners building similar systems. The findings also underscore the need for multiple evaluation signals rather than a single metric. The reliance on official NIST evaluation strengthens external validity, though the small number of topics (17 Batch-1) limits precision of the head-to-head preference conclusion.

Pith's reading of the paper (a machine's summary, not the authors')

This TREC 2026 RAG track participation report argues that open-weight research agents can match or exceed proprietary systems in human preference at a fraction of the generation cost, while automated coverage metrics may rank the same systems in opposite order. The best coverage was achieved by a hybrid-retrieval system that tied the track maximum on Batch-1 vital coverage. However, in the same Batch-1 evaluation, the open-weight owa-glm5 was significantly preferred by human assessors despite lower coverage scores, at roughly one-fifth the generation cost. The authors conclude that there is no single best architecture for agentic RAG and that evaluation methods—automated versus human, holistic versus nugget-guided—are themselves consequential design choices.

What carries the argument. The experimental design compares seven research agents built on a shared staged-evidence architecture that varies the writer model (proprietary GPT-5.6 variants vs open-weight GLM-5), retrieval method (semantic, keyword, hybrid Reciprocal Rank Fusion with κ=60), and planning components (pre-flight briefs, blind obligation scouts, post-draft review). The evaluation machinery pairs automated coverage metrics (vital_strict and all_strict over three rubric and nugget collections) with human side-by-side Bradley-Terry preferences from two batches using different assessment protocols. The divergence between these two evaluation signals carries the paper's main argument about the sensitivity of system rankings to choice of assessment method.

Core claim. The paper's central empirical claim is that six of seven submitted systems exceeded the TREC 2026 RAG track median across all three official rubric and nugget collections. The proprietary brief-hybrid system tied the global maximum for Batch-1 vital coverage at 0.892. More surprisingly, the open-weight owa-glm5 (based on GLM-5) was significantly preferred in Batch-1 human side-by-side evaluation (Bradley-Terry rating 1212.8 vs 1062.7, p=0.006), even though it scored lower on automated coverage. This preference reversed in Batch-2 under a nugget-guided protocol, though the difference was not statistically significant. The paper identifies writer model choice as the largest effect among measured development ablations and hybrid Reciprocal Rank Fusion as the most efficient retrieval design. The authors conclude that evaluation protocol can reverse system rankings and that no single architecture dominates.

The premise it rests on. The paper assumes that the official NIST coverage scores and human side-by-side Bradley–Terry ratings are valid and reliable measures of answer quality, despite the authors acknowledging that automated and human signals can diverge, that the two human batches used different topics and protocols, and that automated judges are sensitive to evaluator choice and presentation order.

If the paper is right (Pith's inferences, not the authors')

Open-weight models can provide competitive quality for agentic RAG tasks at substantially lower generation cost ($0.52 vs $2.82 per topic in this study), lowering the barrier for deployment.

Future RAG evaluations should incorporate multiple complementary signals—automated coverage, human preference, and perhaps readability—rather than relying on any single metric.

The design choice of writer model had a larger measured effect than any structural pipeline change, suggesting that effort on model selection may yield more consistent gains than complex orchestration.

Hybrid retrieval with Reciprocal Rank Fusion and adjacent-chunk expansion improved efficiency (14% fewer searches, 8% fewer tokens) without sacrificing coverage, making it a pragmatic default.

The reversal of human preferences between holistic and nugget-guided protocols implies that tracks must explicitly specify which quality dimension they value, as rankings are not transferable.

The combined cost advantage and reading-ease advantage of owa-glm5 suggest that applications prioritizing human satisfaction and readability over exhaustive recall—such as consumer-facing summarisation—may particularly benefit from open-weight models.

The review-pass failure on topic 65 (a 21-word collapsed revision) highlights a brittle step in agentic RAG pipelines; future work could design explicit review instructions or fallback mechanisms that prevent such collapse.

The strong correlation between answer length and coverage under automated nugget evaluation (Spearman 0.72) raises the question of whether nugget metrics inadvertently reward length; an information-density-adjusted metric would be worth exploring.

The paper's reader-response analysis suggests that human preference may be partly driven by style and reading ease, not just factual completeness; a controlled experiment that varies these factors independently could separate their contributions.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

We describe the RMIT-ADM+S team's participation in the TREC 2026 Retrieval-Augmented Generation (RAG) track using the ClimbMix corpus. We submitted seven research-agent runs using proprietary OpenAI and open-weight models, exploring different retrieval strategies, writer models, and agent workflows. Pre-submission ablations identified writer selection and adjacent-chunk expansion as the main factors affecting answer quality. In the official NIST evaluation, six of seven runs exceeded the track median across all three rubric/nugget collections. Our best-performing run, brief-hybrid, achieved the highest all-item coverage among our submissions and tied the track maximum (0.892) for Batch-1 vital coverage. Human side-by-side preferences differed across evaluation batches: the open-weight owa-glm5 led Batch 1 despite lower automated coverage, while sol-research-first led Batch 2. These findings highlight differences between automated coverage measures and human preferences, as well as the potential of lower-cost open-weight models for research-agent systems.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-13

The authors' main claim, as the classifier picked it out 1.00

“These findings highlight differences between automated coverage measures and human preferences, as well as the potential of lower-cost open-weight models for research-agent systems.”

How the main claim is stated: hedged possibility 0.65

What the abstract itself states, as probabilities: claims first 0.02 · states limitation 0.72 · human participants 0.28 · controlled comparison 0.46 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.77 · mentions code or data availability 0.05

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
We describe the RMIT-ADM+S team's participation in the TREC 2026 Retrieval-Augmented Generation (RAG) track using the ClimbMix corpus. contribution 0.82 0.92 0.02 asserted 0.98
We submitted seven research-agent runs using proprietary OpenAI and open-weight models, exploring different retrieval strategies, writer models, and agent workflows. method 0.56 0.97 0.04 asserted 1.00
Pre-submission ablations identified writer selection and adjacent-chunk expansion as the main factors affecting answer quality. result 0.60 0.96 0.08 asserted 1.00
In the official NIST evaluation, six of seven runs exceeded the track median across all three rubric/nugget collections. result 1.00 0.95 0.13 asserted 1.00
Our best-performing run, brief-hybrid, achieved the highest all-item coverage among our submissions and tied the track maximum (0.892) for Batch-1 vital coverage. result 1.00 0.97 0.12 asserted 1.00
Human side-by-side preferences differed across evaluation batches: the open-weight owa-glm5 led Batch 1 despite lower automated coverage, while sol-research-first led Batch 2. result 1.00 0.95 0.09 asserted 1.00
These findings highlight differences between automated coverage measures and human preferences, as well as the potential of lower-cost open-weight models for research-agent systems. interpretation 1.00 0.83 0.78 hedged possibility 0.65

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.98

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 5,633 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (3)
Figure 1 of the paper
Figure 1. Control flows for three submissions: our leading coverage run (
Figure 2 of the paper
Figure 2. Official human side-by-side Bradley–Terry ratings
Figure 3 of the paper
Figure 3. Per-topic official all_strict coverage difference (Δ) relative to the TREC 2026 RAG track median across Batch 1 (𝑛 = 17, left) and Batch 2 manual nuggets (𝑛 = 25, right), sorted within each panel by brief-hybrid Δ. Shaded bars show the difference between the track maximum and median. 5.3 Per-Topic Divergence

Figures are the authors', reproduced from the paper.

References (33)
  1. [1] Nuha Abu Onq, Chenglong Ma, Mark Sanderson, and Falk Scholer. 2026. Under- standing and Modeling Heterogeneous Search Behavior. InProceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26). 4–16. doi:10.1145/3805... on Pithdoi:10.1145/3805712.3809618
  2. [2] Mohammad Kalim Akram, Saba Sturua, Nastia Havriushenko, Quentin Herreros, Michael Günther, Maximilian Werk, and Han Xiao. 2026. jina-embeddings-v5-text: Compact and Robust Text Embedding Models using Task-Targeted Distillation. InProceedings of the 49th International ACM SIGIR... on Pithdoi:10.1145/3805712
  3. [3] Ralph Allan Bradley and Milton E. Terry. 1952. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons.Biometrika39, 3/4 (1952), 324–345. doi:10.2307/2334029 on Pithdoi:10.2307/2334029
  4. [4] Jaime Carbonell and Jade Goldstein. 1998. The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. InProceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, US... on Pithdoi:10.1145/290941
  5. [5] Haodong Chen, Shuai Wang, Yu Yin, Shengyao Zhuang, Guido Zuccon, and Teerapong Leelanupab. 2026. ITER: Interaction-aware retrieval for agentic search. arXiv preprint arXiv:2608.27912(2026) on PithPith's reviewarXiv
  6. [6] Charles L. A. Clarke and Gordon V. Cormack. 2000. Shortest-Substring Retrieval and Ranking.ACM Transactions on Information Systems18, 1 (2000), 44–78. doi:10.1145/333135.333137 on Pithdoi:10.1145/333135.333137
  7. [7] Charles L. A. Clarke and Mark D. Smucker. 2026. Boolean Queries Are All You Need? arXiv:2607.11362 https://arxiv.org/abs/2607.11362 on PithPith's reviewarXiv
  8. [8] Gordon V. Cormack, Charles L. A. Clarke, and Stefan Buettcher. 2009. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. InProceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’09)... on Pithdoi:10.1145/1571941.1572114
  9. [9] Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Yingyan Lin, Jan Kautz, and Pavlo Molchanov. 2025. Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model P... on Pithdoi:10.52202/085713-0865
  10. [10] Laura Dietz, Oleg Zendel, Peter Bailey, Charles L. A. Clarke, Ellese Cotterill, Jeff Dalton, Faegheh Hasibi, Mark Sanderson, and Nick Craswell. 2025. Principles and Guidelines for the Use of LLM Judges. InProceedings of the 2025 International ACM SIGIR Conference on Innovative... on Pithdoi:10.1145/3731120.3744588
  11. [11] Naghmeh Farzi, Tim Hagen, Eugene Yang, Maik Fröbe, Ronak Pradeep, Hos- sein A. Rahmani, Xi Wang, Oleg Zendel, Martin Potthast, and Laura Dietz. 2026. Auto-Judge: A Cross-Task Benchmark for Comparing LLM Judges for Citation- Grounded RAG Systems. InProceedings of the 49th Inter... on Pithdoi:10.1145/3805712.3808601
  12. [12] Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise Zero-Shot Dense Retrieval without Relevance Labels. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, T... on Pith
  13. [13] Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Tevatron: An Efficient and Flexible Toolkit for Neural Retrieval. InProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’23). ACM, New York, NY, USA, 31... on Pithdoi:10.1145/3539618
  14. [14] Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri, Ravishankar Krishnawamy, and Rohan Kadekodi. 2019. DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node. InAdvances in Neural Information Pro- cessing Systems, H. Wallach, H. Larochelle, A... on Pith
  15. [15] J. Peter Kincaid, Robert P. Fishburne, Jr., Richard L. Rogers, and Brad S. Chissom. 1975.Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Technical Re- port Research Branch Report 8-75.... on Pith
  16. [16] Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python Toolkit for Reproducible In- formation Retrieval Research with Sparse and Dense Representations. InPro- ceedings of the 44th International ACM SIGIR Conferen... on Pithdoi:10.1145/3404835.3463238
  17. [17] Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2024. Initial Nugget Evaluation Results for the 8 RMIT-ADM+S at TREC RAG 2026: Research Agents, Retriever Ablations, and Open-Weight Replication TREC 2026, November 2026, Gaithersburg,... on PithPith's reviewarXiv
  18. [18] Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, and Jimmy Lin. 2025. The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models. InProceedings of the 48th International ACM SIGIR... on Pithdoi:10.1145/3726302.3730090
  19. [19] Kun Ran, Marwah Alaofi, Danula Hettiachchi, Chenglong Ma, Khoi Nguyen Dinh Anh, Khoi Vo Nguyen, Sachin Pathiyan Cherumanal, Lida Rashidi, Falk Scholer, Damiano Spina, et al . 2026. RMIT-ADM+ S at the MMU-RAG NeurIPS 2025 Competition.arXiv preprint arXiv:2602.20735(2026) on PitharXiv
  20. [20] Kun Ran, Shuoqi Sun, Khoi Nguyen Dinh Anh, Damiano Spina, and Oleg Zendel on Pith
show 13 more
  1. [21] David Rau, Hervé Déjean, Nadezhda Chirkova, Thibault Formal, Shuai Wang, Stéphane Clinchant, and Vassilina Nikoulina. 2024. BERGEN: A Benchmarking Li- brary for Retrieval-Augmented Generation. InFindings of the Association for Com- putational Linguistics: EMNLP 2024. 7640–7663... on Pithdoi:10.18653/v1/2024.findings-
  2. [22] Manasi Sharma, Chen Bo Calvin Zhang, Chaithanya Bandi, Clinton Wang, Ankit Aich, Huy Nghiem, Tahseen Rabbani, Ye Htet, Brian Jang, Sumana Basu, Aishwarya Balwani, Denis Peskoff, Marcos Ayestaran, Sean M. Hendryx, Brad Kenstler, and Bing Liu. 2025. ResearchRubrics: A Benchmark ... on PitharXiv
  3. [23] TREC RAG Track Organizers. 2026. TREC Retrieval-Augmented Generation (RAG) Track. https://trec-rag.github.io/. Track website; accessed 2026-11-10 on Pith
  4. [24] John W Tukey. 1949. Comparing individual means in the analysis of variance. Biometrics5, 2 (1949), 99–114. doi:10.2307/3001913 on Pithdoi:10.2307/3001913
  5. [25] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation Learning with Contrastive Predictive Coding. arXiv:1807.03748 https://arxiv.org/abs/1807. 03748 on PithPith's reviewarXiv
  6. [26] Ellen M. Voorhees. 2003. Overview of the TREC 2003 Question Answering Track. InProceedings of the Twelfth Text REtrieval Conference (TREC 2003). 54–68. https://trec.nist.gov/pubs/trec12/papers/QA.OVERVIEW.pdf on Pith
  7. [27] Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the Use of Lucene for Information Retrieval Research. InProceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’17). ACM, New York, NY, USA, 1253–1256... on Pithdoi:10.1145/3077136.3080721
  8. [28] Oleg Zendel, Sara Fahad Dawood Al Lawati, Lida Rashidi, Falk Scholer, and Mark Sanderson. 2025. A Comparative Analysis of Linguistic and Retrieval Diversity in LLM-Generated Search Queries. InProceedings of the 34th ACM International Conference on Information and Knowledge Man... on Pithdoi:10.1145/3746252.3761382
  9. [29] Oleg Zendel, J. Shane Culpepper, Falk Scholer, and Paul Thomas. 2024. Enhancing Human Annotation: Leveraging Large Language Models and Efficient Batch Processing. InProceedings of the 2024 Conference on Human Information Interaction and Retrieval (CHIIR ’24). 340–345. doi:10.1... on Pithdoi:10.1145/3627508.3638322
  10. [30] Yuqi Zhou, Sunhao Dai, Changle Qu, Liang Pang, Jun Xu, and Ji-Rong Wen on Pith
  11. [31] Yongjie Zhou, Shuai Wang, Bevan Koopman, and Guido Zuccon. 2026. Beyond Chunk-Then-Embed: A Comprehensive Taxonomy and Evaluation of Document Chunking Strategies for Information Retrieval. InProceedings of the 49th In- ternational ACM SIGIR Conference on Research and Developme... on Pithdoi:10.1145/3805712.3808575
  12. [2025] RMIT-ADM+ S at the SIGIR 2025 LiveRAG Challenge.arXiv preprint arXiv:2506.14516(2025) on PithPith's reviewarXiv
  13. [2026] Learning to Retrieve from Agent Trajectories. InProceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26). 2544–2555. doi:10.1145/3805712.3809675 on Pithdoi:10.1145/3805712.3809675

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment