JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment

Bhaskar Mitra; Emine Yilmaz; Hossein A. Rahmani; Nick Craswell

REVIEW 3 major objections 5 minor 26 references

JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read JudgeBlender claims that a panel of small open language models can judge search relevance as reliably as a single large proprietary model.

desk verdict A plausible but under-powered result: ensembling small open LLMs lands in the same kappa range as GPT-4o judges, but the headline edge rests on a 0.01 gap over 25 queries with no significance testing. read the letter →

arxiv 2412.13268 v1 pith:GRDN3UBC submitted 2024-12-17 cs.IR

Hossein A. Rahmani , Emine Yilmaz , Nick Craswell , Bhaskar Mitra This is my paper · ORCID

classification cs.IR

keywords LLM-as-a-JudgeautomaticrelevanceassessmentensemblelearningLLMBlenderPromptBlendermajorityvotingsmallopen-sourcemodelsTRECDL2023

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes JudgeBlender, an ensemble method that produces search relevance labels by pooling judgments from several small open-weights language models, either by varying the prompt to one model (PromptBlender) or by using several distinct models (LLMBlender). The authors aim to show that such a panel, aggregated by majority or average voting, yields relevance judgments that track human labels as well as a single large commercial judge such as GPT-4o. On the LLMJudge benchmark, LLMBlender with majority voting and random tie-breaking reaches Cohen's kappa 0.2619 against human labels, slightly above the best GPT-4o baseline RelExp (0.2519) and above the LLMJudge-challenge winner MultiCriteria (0.1829). If this holds, evaluation of retrieval systems could become cheaper and more reproducible because large proprietary models are not required.

What carries the argument

The machinery is a panel of evaluators plus an aggregator. PromptBlender uses one model, Meta-Llama-3-8B, with three distinct prompts: a direct relevance-grading prompt, a criteria-decomposition prompt, and a binary-then-score two-step prompt. LLMBlender uses Mistral-7B, Gemma-7B, and Llama-3-8B, each attached to a different prompt family from the same set. The per-judge scores are pooled by majority voting, with four tie-breaking rules (random, max, min, average), or by average voting. Majority voting is the component that lifts the ensemble above its individual members; the paper reports LLMBlender's best Cohen's kappa under random tie-breaking and its most balanced per-level agreement under average tie-breaking.

What would settle it

Compute per-query Cohen's kappa for LLMBlender with majority voting and random tie-breaking, and for GPT-4o-based RelExp, on the LLMJudge test set: with only 25 queries, dropping a handful of topics could erase the 0.2619 versus 0.2519 margin. The decisive check is to repeat the vote on TREC DL 2019-2021 qrels or a non-web collection; if the open-model ensemble does not match or exceed the single GPT-4o judge there, the claim that large models are unnecessary loses its support.

Watch

Extended reading notes

Core claim

JudgeBlender's central claim is that ensembling matters more than raw model size: a jury of small open models produces automatic relevance judgments whose agreement with human assessors is competitive with, and on the headline metric slightly better than, a single large proprietary model. The paper demonstrates this by comparing label-level agreement (Cohen's kappa and Krippendorff's alpha) and system-ranking agreement (Kendall's tau and Spearman's rho) on the LLMJudge test set, and by showing that both PromptBlender and LLMBlender improve over their individual members. The strongest result is LLMBlender with majority voting and random tie-breaking, at kappa 0.2619, compared with 0.2519 for GPT-4o-based RelExp and 0.1829 for MultiCriteria. The paper also reports that the best JudgeBlender variants are more balanced across all four relevance levels and show less systematic over- or under-estimation for systems grouped by the language model they use.

Load-bearing premise

The whole comparison rests on one benchmark: the LLMJudge test set contains only 25 queries drawn from TREC DL 2023, so the claim that very large models are often unnecessary presumes that one small collection represents relevance judgment across domains, query types, and document scales.

Editorial extensions

If this is right

Automatic relevance-label generation for retrieval evaluation can be run locally on open 7-8 billion parameter models instead of through expensive commercial APIs.
Prompt diversity alone is a cheaper form of ensembling: PromptBlender's aggregated judgments beat any of its three individual prompts.
System rankings computed from JudgeBlender judgments closely track official human-based TREC rankings, supporting its use for evaluating shared-task submissions.
The best JudgeBlender variants distribute TREC DL 2023 systems evenly around the diagonal, indicating less model-family bias than single-judge methods.
The optimal configuration depends on the tie-breaking rule and evaluation metric, so JudgeBlender is best treated as a family of methods rather than a single fixed recipe.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next experiment is to run the same panel on TREC DL 2019-2021 and on non-web test collections; the claim that very large models are unnecessary stands or falls on whether the kappa advantage survives beyond the current 25-query test set.
Because the best LLMBlender result uses random tie-breaking, the practical recipe is unstable at the margin; a deterministic second-stage judge or a learned aggregator could convert the small kappa gain into a more reliable one.
The economic case may be the stronger one: even at parity, replacing GPT-4o with locally run open models removes per-query API costs, data-leakage exposure, and output nondeterminism from the evaluation pipeline.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful thing here is concrete: a simple majority vote across three 7-8B open models reaches Cohen's kappa 0.2619 on the LLMJudge test set, nudging past the best GPT-4o baseline (RelExp, 0.2519) and clearly beating MultiCriteria (0.1829). The paper also contributes a nice bias analysis showing that the ensemble judges are more even-handed across GPT-, T5-, and other-based systems than the single-model baselines. That bias result is genuinely valuable and arguably more robust than the kappa headline.

The paper is clearly written, compares against a sensible set of baselines including fine-tuned models, and ships prompts on GitHub. The novelty claim for LLM ensembles specifically for relevance assessment looks correct against the cited literature; Verga et al. is about generation evaluation, not relevance judging.

The soft spots are real and load-bearing. The test collection is 25 queries, and kappa is computed over pooled qrels with no query-level clustering, no confidence intervals, no significance tests. The winning configuration, MV(Rnd.), resolves ties randomly, so the reported 0.2619 is one stochastic draw. The family of aggregator variants spans only 0.2502 to 0.2619, which brackets the RelExp baseline, and the best variant was selected post hoc from ten options. In other words, the 0.01 kappa advantage is indistinguishable from noise. The paper does acknowledge the single-dataset limitation in the conclusion, but the missing robustness analysis is more damaging than the single dataset per se. A minor internal inconsistency: Section 5.3 treats MV(Avg.) as the best variant while Table 2 gives MV(Rnd.) the highest kappa, which suggests the 'best' label is not computed against a fixed criterion.

Who is this for: IR evaluation researchers and anyone using LLM judges in practice. The method is cheap, open, and reproducible, and the bias analysis should survive the statistical critique. But nobody should cite the kappa number as evidence that large models are unnecessary until it is replicated on more collections with proper significance testing and a pre-specified aggregation rule.

Recommendation: yes, this deserves a serious referee. The question matters, the method is simple enough to verify, and the bias analysis is a real contribution. A referee should push for significance testing, multiple datasets, and a less post hoc selection story.

Referee Report

3 major / 5 minor

Summary. The paper proposes JudgeBlender, an ensemble framework for automatic relevance assessment with two variants: PromptBlender (one LLM, multiple prompts) and LLMBlender (multiple LLMs, one prompt each). Judgments are aggregated by majority voting with four tie-breaking strategies (random, max, min, average) or by averaging scores. The framework is evaluated on the LLMJudge benchmark derived from TREC DL 2023, reporting Cohen's kappa, Krippendorff's alpha, Kendall's tau, Spearman's rho, system ranking correlation, and per-level inter-annotator agreement. The central claim is that ensembling small open-source 7B-8B models reaches or exceeds the label correlation of a single GPT-4o-based judge, so 'very large models are often unnecessary' for relevance assessment. The experiments are limited to a single test collection with 25 queries.

Significance. If the result is statistically robust, the paper makes a practically useful contribution: it suggests that cost-effective, reproducible, open-weight models can substitute for expensive proprietary judges in relevance labeling. The work uses a public benchmark, reports multiple agreement and ranking metrics, analyzes bias across system types, and provides prompts on GitHub, which are concrete reproducibility strengths. However, the headline kappa advantage over GPT-4o rests on a very small test set and is not supported by significance testing, confidence intervals, or query-level analysis; the significance of the claim is therefore conditional on the robustness analysis that the paper currently lacks.

major comments (3)

[§5.1, Table 2] The central comparison, LLMBlender-MV(Rnd.) with kappa 0.2619 versus RelExp with 0.2519, is based on only 25 test queries (Table 1). No significance test, confidence interval, or query-level clustering is reported; Cohen's kappa is computed over the pooled qrels, which inflates the effective sample size. The paper should provide a bootstrap or permutation-based confidence interval and ideally a query-stratified analysis, because the observed Δκ = 0.010 could easily arise from sampling noise.
[§4.2, Table 2] The best-performing variant is selected post hoc from ten aggregator/tie-break configurations that all share the same underlying judge scores, so the highlighted value is not an independent observation. Moreover, the highlighted MV(Rnd.) variant resolves ties randomly, making the reported 0.2619 one realization of a stochastic procedure. The authors should report the mean and variance over multiple random seeds, or use a deterministic tie-break chosen on the development set, to support the claim that ensembling small models is genuinely competitive with GPT-4o.
[§5.3] The text states that 'the best-performing variant of JudgeBlender, LLMBlender-MV(Avg.)' shows consistent agreement, but Table 2 reports the highest kappa for LLMBlender-MV(Rnd.) (0.2619), the highest alpha for LLMBlender-AV (0.4832), and different best values for ranking metrics. This inconsistency makes it unclear which variant is claimed to be best and risks cherry-picking. The authors should define a single selection criterion (e.g., kappa on the dev set) and align the textual claims with the table.

minor comments (5)

[§4.5] The text contains typos and duplicated words, e.g., 'prposed by Thomas et al.' and 'the prompt used by byMultiCriteria method'; these should be corrected.
[Figure 3] The caption labels the fourth subplot as '(b)' instead of '(d)', which is confusing for readers.
[Table 3] The column header 'PromptBlender - MV(Av g.)' has an odd spacing in the displayed table; the formatting should be fixed.
[§4.3] The model names are used inconsistently: Section 4.3 says 'Meta-Llama-3-8B' and 'Llama-3-8B', while Table 2 reports 'Llama-3-8B-Instruct' for PromptBlender; the exact model variants should be stated consistently.
[§6] The limitation section acknowledges the single-dataset scope but does not mention the absence of significance testing; adding this caveat would make the conclusions more appropriately cautious.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: JudgeBlender's zero-shot ensemble judgments are compared directly against external TREC DL 2023 human qrels, with no fitted parameter or self-referential derivation chain.

full rationale

The paper's central derivation is self-contained and non-circular. JudgeBlender takes pre-trained, open-source LLMs (Mistral-7B, Gemma-7B, Llama-3-8B) with fixed prompts, aggregates their relevance scores via majority or average voting, and measures agreement against human labels from the LLMJudge dataset, which is built on TREC DL 2023 qrels. No parameter is fitted to the target labels, and no claimed result is defined in terms of the outcome it is supposed to derive. The self-citations to the LLMJudge benchmark [18], SynDL [17], and Rahmani et al. [14] are not load-bearing in a circular way: the underlying human judgments and system metadata are externally anchored to TREC, and the cited bias-analysis methodology is used as a measurement tool rather than as a premise that entails the conclusion. The main limitations are robustness issues rather than circularity: the best aggregator and tie-breaking variant (LLMBlender with majority voting and random tie-breaking) was selected after examining the test set, the test collection has only 25 queries, and no significance or confidence intervals are reported. These concerns affect the strength and generality of the claim that 'very large models are often unnecessary,' but they do not make the derivation equivalent to its inputs. The framework's outputs are genuinely produced by the ensemble's zero-shot judgments, and the reported kappa values are not forced by construction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The empirical result depends on no fitted constants, but it relies on hand-chosen panel composition and test-set selection of the voting rule, plus the assumption that the LLMJudge benchmark is representative. There are no new postulated entities or forces.

free parameters (2)

Aggregation tie-breaking strategy = MV(Avg.) for PromptBlender, MV(Avg.)/MV(Rnd.) for LLMBlender
The paper evaluates four tie-break strategies and average voting, then highlights the best variant per aggregator in Table 2, effectively selecting the strategy on the test set.
Panel composition, model-prompt pairings = Mistral-7B with MultiCriteria prompt, Gemma-7B with Thomas prompt, Llama-3-8B with Sun prompt
The assignment of prompts to models in LLMBlender is hand-crafted in Section 4.5 without justification or sensitivity analysis.

assumptions (3)

domain assumption Human relevance judgments (TREC DL 2023 qrels) are the ground truth for relevance.
Section 4.1 treats the human labels in the LLMJudge dataset as the reference against which all methods are compared.
domain assumption The LLMJudge benchmark (25 test queries) is representative of relevance assessment tasks.
The conclusion that large models are often unnecessary is drawn from this single small benchmark in Sections 5 and 6.
standard math Cohen's kappa, Krippendorff's alpha, Kendall's tau, and Spearman's rho are appropriate agreement metrics for evaluating judge quality.
Section 4.4 adopts these metrics without justification; they are standard in IR evaluation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment." pith.science (2026). https://pith.science/paper/GRDN3UBC

@misc{pith2026241213268,
  author       = {Pith},
  title        = {Pith review of: JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GRDN3UBC}},
  note         = {Machine review of arXiv:2412.13268}
}

read the original abstract

The effective training and evaluation of retrieval systems require a substantial amount of relevance judgments, which are traditionally collected from human assessors -- a process that is both costly and time-consuming. Large Language Models (LLMs) have shown promise in generating relevance labels for search tasks, offering a potential alternative to manual assessments. Current approaches often rely on a single LLM, such as GPT-4, which, despite being effective, are expensive and prone to intra-model biases that can favour systems leveraging similar models. In this work, we introduce JudgeBlender, a framework that employs smaller, open-source models to provide relevance judgments by combining evaluations across multiple LLMs (LLMBlender) or multiple prompts (PromptBlender). By leveraging the LLMJudge benchmark [18], we compare JudgeBlender with state-of-the-art methods and the top performers in the LLMJudge challenge. Our results show that JudgeBlender achieves competitive performance, demonstrating that very large models are often unnecessary for reliable relevance assessments.

Figures

Figures reproduced from arXiv: 2412.13268 by the authors.

**Figure 1.** The PromptBlender evaluation uses an LLM to grade how relevant a passage is to a query. The evaluation is done by multiple prompting to an LLM then aggregating the scores based on an aggregation function (Section 4.2). ments [3,10,2]. Typically, these methods combine models through weighted contributions or by aggregating their diverse outputs, demonstrating the effectiveness of incorporating multiple perspectives… view at source ↗

**Figure 2.** The LLMBlender evaluation uses multiple LLMs to grade how relevant a passage is to a query. The evaluation is done by prompting several LLMs, then aggregating the scores based on an aggregation function (Section 4.2). niques to the domain of relevance assessment, we address the precision and contextual understanding necessary for high-quality relevance annotations. 3 JudgeBlender JudgeBlender is based on using an en… view at source ↗

**Figure 3.** Scatter plots of the effectiveness of TREC Deep Learning track 2023 [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 14 canonical work pages

[1]

Abbasiantaeb, Z., Meng, C., Azzopardi, L., Aliannejadi, M.: Can we use large language models to fill relevance judgment holes? arXiv preprint arXiv:2405.05600 (2024)

arXiv 2024
[2]

In: 2019 Seventh International Symposium on Computing and Networking Workshops (CANDARW)

Aniol, A., Pietron, M., Duda, J.: Ensemble approach for natural language question answering problem. In: 2019 Seventh International Symposium on Computing and Networking Workshops (CANDARW). pp. 180–183. IEEE (2019)

work page 2019
[3]

Machine learning24, 123–140 (1996)

Breiman, L.: Bagging predictors. Machine learning24, 123–140 (1996)

1996
[4]

In: Text REtrieval Conference (TREC)

Craswell, N., Mitra, B., Yilmaz, E., Rahmani, H.A., Campos, D., Lin, J., Voorhees, E.M., Soboroff, I.: Overview of the trec 2023 deep learning track. In: Text REtrieval Conference (TREC). NIST, TREC (February 2024)

work page 2023
[5]

In: International work- shop on multiple classifier systems

Dietterich, T.G.: Ensemble methods in machine learning. In: International work- shop on multiple classifier systems. pp. 1–15. Springer (2000)

2000
[6]

In: Proceedings of the 2023 ACM SIGIR Interna- tional Conference on Theory of Information Retrieval

Faggioli, G., Dietz, L., Clarke, C.L., Demartini, G., Hagen, M., Hauff, C., Kando, N., Kanoulas, E., Potthast, M., Stein, B., et al.: Perspectives on large language models for relevance judgment. In: Proceedings of the 2023 ACM SIGIR Interna- tional Conference on Theory of Information Retrieval. pp. 39–50 (2023)

work page 2023
[7]

Best in Tau@LLMJudge: Criteria-Based Relevance Evaluation with Llama3

Farzi, N., Dietz, L.: Best in tau@ llmjudge: Criteria-based relevance evaluation with llama3. arXiv preprint arXiv:2410.14044 (2024) JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment 13

work page Pith review arXiv 2024
[8]

In: Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval

Farzi, N., Dietz, L.: Pencils down! automatic rubric-based evaluation of re- trieve/generate systems. In: Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval. pp. 175–184 (2024)

work page 2024

Show all 26 references

[9]

arXiv preprint arXiv:2007.01282 (2020)

Izacard, G., Grave, E.: Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282 (2020)

2020 arXiv
[10]

In: Proceedings of the 16th ACM SIGKDD international con- ference on Knowledge discovery and data mining

Jahrer, M., Töscher, A., Legenstein, R.: Combining predictions for accurate rec- ommender systems. In: Proceedings of the 16th ACM SIGKDD international con- ference on Knowledge discovery and data mining. pp. 693–702 (2010)

2010
[11]

In: Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL 2023) (2023)

Jiang, D., Ren, X., Lin, B.Y.: Llm-blender: Ensembling large language models with pairwise comparison and generative fusion. In: Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL 2023) (2023)

2023
[12]

arXiv preprint arXiv:2404.01012 (2024)

Meng, C., Arabzadeh, N., Askari, A., Aliannejadi, M., de Rijke, M.: Query perfor- mance prediction using relevance judgments generated by large language models. arXiv preprint arXiv:2404.01012 (2024)

2024 arXiv
[13]

Ensemble machine learning: Methods and applica- tions pp

Polikar, R.: Ensemble learning. Ensemble machine learning: Methods and applica- tions pp. 1–34 (2012)

2012
[14]

In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

Rahmani, H.A., Craswell, N., Yilmaz, E., Mitra, B., Campos, D.: Synthetic test collections for retrieval evaluation. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 2647–2651 (2024)

2024
[15]

Rahmani,H.A.,Siro,C.,Aliannejadi,M.,Craswell,N.,Clarke,C.L.A.,Faggioli,G., Mitra, B., Thomas, P., Yilmaz, E.: Llm4eval: Large language model for evaluation inir.In:Proceedingsofthe47thInternationalACMSIGIRConferenceonResearch and Development in Information Retrieval. p. 3040–3...

2024
[16]

arXiv preprint arXiv:2408.05388 (2024)

Rahmani, H.A., Siro, C., Aliannejadi, M., Craswell, N., Clarke, C.L., Faggioli, G., Mitra, B., Thomas, P., Yilmaz, E.: Report on the 1st workshop on large language model for evaluation in information retrieval (llm4eval 2024) at sigir 2024. arXiv preprint arXiv:2408.05388 (2024)

2024 arXiv
[17]

arXiv preprint arXiv:2408.16312 (2024)

Rahmani, H.A., Wang, X., Yilmaz, E., Craswell, N., Mitra, B., Thomas, P.: Syndl: A large-scale synthetic test collection for passage retrieval. arXiv preprint arXiv:2408.16312 (2024)

2024 arXiv
[18]

arXiv preprint arXiv:2408.08896 (2024)

Rahmani, H.A., Yilmaz, E., Craswell, N., Mitra, B., Thomas, P., Clarke, C.L., Aliannejadi, M., Siro, C., Faggioli, G.: Llmjudge: Llms for relevance judgments. arXiv preprint arXiv:2408.08896 (2024)

2024 arXiv
[19]

arXiv preprint arXiv:2210.08779 (2022)

Ravaut, M., Joty, S., Chen, N.F.: Towards summary candidates fusion. arXiv preprint arXiv:2210.08779 (2022)

2022 arXiv
[20]

Wiley interdisciplinary reviews: data mining and knowledge discovery8(4), e1249 (2018)

Sagi, O., Rokach, L.: Ensemble learning: A survey. Wiley interdisciplinary reviews: data mining and knowledge discovery8(4), e1249 (2018)

2018
[21]

arXiv preprint arXiv:2304.09542 (2023)

Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D., Ren, Z.: Is chatgpt good at search? investigating large language models as re-ranking agents. arXiv preprint arXiv:2304.09542 (2023)

2023 arXiv
[22]

In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

Thomas, P., Spielman, S., Craswell, N., Mitra, B.: Large language models can accurately predict searcher preferences. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 1930–1940 (2024)

2024
[23]

arXiv preprint arXiv:2405.04727 (2024)

Upadhyay, S., Kamalloo, E., Lin, J.: Llms can patch up missing relevance judg- ments in evaluation. arXiv preprint arXiv:2405.04727 (2024)

2024 arXiv
[24]

arXiv preprint arXiv:2406.06519 (2024) 14 H

Upadhyay, S., Pradeep, R., Thakur, N., Craswell, N., Lin, J.: Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor. arXiv preprint arXiv:2406.06519 (2024) 14 H. A. Rahmani et al

2024 arXiv
[25]

arXiv preprint arXiv:2404.18796 (2024)

Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., Lewis, P.: Replacing judges with juries: Evaluating llm gener- ations with a panel of diverse models. arXiv preprint arXiv:2404.18796 (2024)

2024 arXiv
[26]

In: Pro- ceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

Xu, J., Li, H.: Adarank: a boosting algorithm for information retrieval. In: Pro- ceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. pp. 391–398 (2007)

2007

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Abbasiantaeb, Z., Meng, C., Azzopardi, L., Aliannejadi, M.: Can we use large language models to fill relevance judgment holes? arXiv preprint arXiv:2405.05600 (2024)

arXiv 2024

[2] [2]

In: 2019 Seventh International Symposium on Computing and Networking Workshops (CANDARW)

Aniol, A., Pietron, M., Duda, J.: Ensemble approach for natural language question answering problem. In: 2019 Seventh International Symposium on Computing and Networking Workshops (CANDARW). pp. 180–183. IEEE (2019)

work page 2019

[3] [3]

Machine learning24, 123–140 (1996)

Breiman, L.: Bagging predictors. Machine learning24, 123–140 (1996)

1996

[4] [4]

In: Text REtrieval Conference (TREC)

Craswell, N., Mitra, B., Yilmaz, E., Rahmani, H.A., Campos, D., Lin, J., Voorhees, E.M., Soboroff, I.: Overview of the trec 2023 deep learning track. In: Text REtrieval Conference (TREC). NIST, TREC (February 2024)

work page 2023

[5] [5]

In: International work- shop on multiple classifier systems

Dietterich, T.G.: Ensemble methods in machine learning. In: International work- shop on multiple classifier systems. pp. 1–15. Springer (2000)

2000

[6] [6]

In: Proceedings of the 2023 ACM SIGIR Interna- tional Conference on Theory of Information Retrieval

Faggioli, G., Dietz, L., Clarke, C.L., Demartini, G., Hagen, M., Hauff, C., Kando, N., Kanoulas, E., Potthast, M., Stein, B., et al.: Perspectives on large language models for relevance judgment. In: Proceedings of the 2023 ACM SIGIR Interna- tional Conference on Theory of Information Retrieval. pp. 39–50 (2023)

work page 2023

[7] [7]

Best in Tau@LLMJudge: Criteria-Based Relevance Evaluation with Llama3

Farzi, N., Dietz, L.: Best in tau@ llmjudge: Criteria-based relevance evaluation with llama3. arXiv preprint arXiv:2410.14044 (2024) JudgeBlender: Ensembling Judgments for Automatic Relevance Assessment 13

work page Pith review arXiv 2024

[8] [8]

In: Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval

Farzi, N., Dietz, L.: Pencils down! automatic rubric-based evaluation of re- trieve/generate systems. In: Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval. pp. 175–184 (2024)

work page 2024