Does UMBRELA Work on Other LLMs?

Laura Dietz; Naghmeh Farzi

REVIEW 2 major objections 5 minor 13 references

Does UMBRELA Work on Other LLMs?

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper reports that UMBRELA's relevance-judging framework transfers across language models, with the open-weight DeepSeek V3 matching GPT-4o, although per-label agreement degrades sharply in smaller models.

desk verdict A clean, honest reproduction showing DeepSeek V3 can stand in for GPT-4o in UMBRELA, with the caveat that the GPT-4o baseline is copied rather than re-run. read the letter →

arxiv 2507.09483 v1 pith:NIS5BKWD submitted 2025-07-13 cs.IR

Naghmeh Farzi , Laura Dietz This is my paper

classification cs.IR

keywords UMBRELALLM-as-a-judgerelevanceassessmentzero-shotpromptingreproducibilityCohen'skappaleaderboardrankcorrelationTRECDeepLearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the UMBRELA relevance-judging framework, which was built and tested with GPT-4o, keeps working when the underlying language model is replaced. The authors run the same zero-shot prompt on DeepSeek V3, LLaMA-3.3-70B, LLaMA-3-8B, and FLAN-T5-large and compare them with the original GPT-4o results on three TREC Deep Learning collections. They find that DeepSeek V3 performs at essentially the same level as GPT-4o, and that all models, down to a 783M-parameter model, produce leaderboards that closely match human-judged rankings. The sensitive part is per-label agreement: Cohen's kappa drops steeply for smaller models, which matters when labels are used to build training sets rather than just order systems. The practical upshot is that UMBRELA is not locked to one proprietary model, but the model choice should depend on whether the goal is ranking systems or labeling individual passages.

What carries the argument

The load-bearing object is UMBRELA's zero-shot DNA (Descriptive, Narrative, and Aspects) prompt, which asks the model to score a query-passage pair on a four-point relevance scale and to emit the answer only as '##final score: [score]'. The prompt names two aspects—alignment with the query's underlying intent and passage trustworthiness—and asks the model to weigh them into a final integer score. The strict output format is part of the mechanism, because the framework's usability depends on rule-based parsing; the paper shows that smaller models deviate from this format more often, up to about 1.9% invalid outputs for LLaMA-3-8B, and that a fallback parser is needed. The evaluation machinery is a pair of metrics: Spearman's rho and Kendall's tau for leaderboard rank correlation, and Cohen's kappa in four-point and binarized forms for per-label agreement.

What would settle it

Re-run UMBRELA with GPT-4o on the same cleaned DL 2023 collection, using the same prompt, temperature-zero decoding, and parsing function, and compare the resulting rho, tau, and kappa against the copied Table 2 numbers; a deviation beyond the paper's stated 0.005 threshold for rank correlations or 0.01 for Cohen's kappa would show the baseline comparison was not controlled.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that UMBRELA generalizes across LLM families with a clear scale split. DeepSeek V3, an open-weight model, lands within noise of GPT-4o on almost every metric and dataset, outperforming it on some rank-correlation values; GPT-4o remains the strongest in several per-label comparisons. LLaMA-3.3-70B places third, LLaMA-3-8B is weaker still, and FLAN-T5-large trails in per-label agreement while still keeping rank correlations around 0.86-0.97. Because the four-point scale and binary Cohen's kappa figures collapse for small models while rank correlations stay high, the paper concludes that leaderboard evaluation and document-level labeling are different tasks with different model requirements.

Load-bearing premise

The comparison assumes the GPT-4o numbers copied from the original paper were produced under identical dataset versions, prompt text, and decoding settings, so that any difference in Table 2 is caused only by the LLM.

Editorial extensions

If this is right

If the central claim holds, UMBRELA no longer depends on GPT-4o: an open-weight model with comparable capability, DeepSeek V3, can serve as a drop-in replacement for relevance labeling.
Research groups without access to proprietary APIs can still produce human-like relevance judgments and valid system rankings using publicly available weights.
System leaderboards generated by any tested model, including FLAN-T5-large, closely resemble the official human-judged rankings, so small and cheap models can support retrieval-system comparison.
For applications that need exact labels per document, such as synthetic training data, the paper says only large models, GPT-4o or DeepSeek V3, should be used.
Prompt wording matters less than model choice; the two tested prompts differed little on LLaMA-3-8B, so gains from prompt tuning are bounded relative to gains from model scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the same scale split likely transfers to other LLM-judge frameworks that rely on strict output formatting; format-following ability may be a hidden confound in any comparison of judge LLMs.
Editorial inference: since DeepSeek V3 was trained partly on GPT-4o-generated data, the comparable performance may reflect distillation rather than independent generalization; a test on a model family with no GPT-4o-derived training data would separate these explanations.
Editorial inference: a practical extension would be to measure cost-quality tradeoffs with an explicit budget, since the paper notes diminishing returns but does not price API or compute usage.
Editorial inference: one could test whether the per-label gap closes with a prompt tuned for small models; the paper hints shorter prompts helped FLAN-T5-large, so prompt-model co-adaptation is a plausible fix.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful thing here is the empirical pattern: across three TREC DL collections, leaderboard rank correlations stay high even when per-label agreement collapses as model size drops. That is a practical, non-obvious result, and it is the paper's real contribution. The headline numbers for DeepSeek V3 are also new and believable: it is close to GPT-4o on both rank and per-label metrics, and the authors are appropriately careful about saying "very comparable" rather than "better."

The paper is transparent about what it reuses, re-implements, and copies. The prompt is verbatim from the UMBRELA repo; the GPT-4o baseline is copied from Table 2 of the original paper; and they report invalid output rates and parsing fallbacks, which is good practice. The appendix examples of output format drift are useful for anyone building on UMBRELA.

The soft spots are real but not fatal. The copied GPT-4o baseline is the main one: because those numbers come from a different pipeline, possibly a different DL 2023 dataset version (the footnote points to LLMJudge data rather than the exact Upadhyay et al. split), and possibly different decoding settings, the DeepSeek-vs-GPT-4o comparison is not fully controlled. The stress-test note is right that this is the load-bearing comparison. Also, single runs with temperature 0 give no variance estimates, and the significance thresholds (<=0.005 for rho/tau, <=0.01 for kappa) are asserted without justification. These weaken the paper but do not sink it: the rank-vs-per-label sensitivity pattern is consistent across all three datasets and does not depend on the copied baseline.

Who is this for? IR researchers who use or consider UMBRELA for evaluation, especially those who want an open-weight alternative to GPT-4o or who need a rule of thumb about when smaller models suffice. It does not change retrieval theory, but it is honest operational progress.

Recommendation: deserve a serious referee. The key fix is simple and important: re-run GPT-4o through the same harness, or at least justify why the copied numbers are directly comparable. Adding bootstrap confidence intervals or a few seeds would strengthen the significance discussion. As is, I would conditionally accept with that revision required.

Referee Report

2 major / 5 minor

Summary. The paper reproduces the UMBRELA LLM-based relevance assessment framework on a range of open-weight and API-based LLMs (DeepSeek V3, LLaMA-3.3-70B, LLaMA-3-8B, FLAN-T5-large) and compares their leaderboard rank correlation and per-label agreement with human judgments on TREC DL 2019, 2020, and 2023. The central claims are that DeepSeek V3 performs comparably to GPT-4o, that per-label agreement (Cohen's kappa) drops sharply with model scale while rank correlations (Spearman's rho, Kendall's tau) remain high, and that smaller models may suffice for system ranking but not for label-level assessment. The GPT-4o baseline is copied directly from Table 2 of the original UMBRELA paper rather than re-run in the same pipeline, and the evaluation uses single runs with ad hoc significance thresholds.

Significance. If the central comparison to GPT-4o is validated, the results are practically significant: they suggest that the open-weight DeepSeek V3 can replace GPT-4o in UMBRELA without meaningful degradation, and they provide evidence that per-label agreement is more sensitive to model scale than leaderboard rank correlation. The paper is transparent about its pipeline, reports invalid output rates, and includes worked examples and appendix samples, which aids reproducibility. However, the uncontrolled GPT-4o baseline and the absence of uncertainty quantification weaken the strength of the headline claims.

major comments (2)

[Section 2.2, Table 2] The GPT-4o baseline is copied verbatim from Table 2 of Upadhyay et al. [13] rather than produced by the authors' own pipeline. All other rows in Table 2 are generated with the authors' dataset files, parsing logic (Section 3.6), and API/local inference settings. The DL 2023 data used here is the LLMJudge challenge data (footnote 4), which may not match the exact version used to produce the copied GPT-4o row. Consequently, differences in Table 2—including the headline 'very comparable' DeepSeek V3 result—cannot be attributed solely to the choice of LLM. The authors should re-run GPT-4o in the same harness, or at least verify parity on a shared subsample, before asserting that DeepSeek V3 matches GPT-4o.
[Section 2.6, Table 2] The significance thresholds introduced in Section 2.6 (differences of at most 0.005 for rho/tau and 0.01 for kappa are treated as not significant) are arbitrary and are applied to single-run results with no confidence intervals or repeated trials. In Table 2, the DL 2023 Spearman rho difference between GPT-4o and DeepSeek V3 is 0.004, which falls within the stated equivalence threshold, yet the text in Section 3.1 says DeepSeek 'outperforms' GPT-4o in rho and tau. The tau difference of 0.018 is above the threshold, so the evidence is mixed. Without bootstrap or multiple runs, the fine-grained comparisons and the bold 'best or equivalent' markings are not statistically supported. The authors should provide variance estimates or temper the comparative claims.

minor comments (5)

[Table 2 caption] The caption contains typos: 'Corelation' should be 'Correlation', and '01-vs-34' should be '01-vs-23' to match the binarization defined in Section 2.6 (labels 0/1 vs 2/3).
[Section 2.2] The first bullet says the prompt is reused 'form the original paper'; 'form' should be 'from'. The second bullet says 'we implement our own code to load TREC DL datasets and experiments'; 'experiments' should be 'experiment' or 'run experiments'.
[Section 3.6] Two wording issues: 'as merely the results if a less capable model' should be 'as merely the results of a less capable model', and 'the function can handles minor formatting variations' should be 'can handle'.
[Section 3.5] The RQ5 conclusion that prompt choice matters less than LLM choice is based on only one model (LLaMA-3-8B) and two prompt variants. This is a narrow basis for a general claim; the authors should soften the wording or add more models/prompts to support the relative-impact statement.
[Section 2.5] The 'architectural leak' estimate for GPT-4 is irrelevant to the comparison and also refers to GPT-4 rather than GPT-4o; it should be removed or replaced with a citable source.

Circularity Check

0 steps flagged · score 1.0 of 10

No meaningful circularity: the copied GPT-4o baseline is prior external work and the only self-citation is peripheral.

full rationale

The paper makes no fitted-parameter claim and no prediction that reduces to its inputs by construction. Its core evaluation compares independently generated UMBRELA labels from DeepSeek V3, LLaMA-3.3-70B, LLaMA-3-8B, and FLAN-T5-large against human TREC DL judgments, using standard external metrics such as Spearman's rho, Kendall's tau, and Cohen's kappa. The GPT-4o comparison row is copied from Upadhyay et al. [13] as described in Section 2.2, which is prior external work rather than the authors' own result; copying a baseline is a reproducibility choice, not a self-referential derivation, so any mismatch in dataset version, prompt revision, or decoding settings is a controlled-comparison risk, not circularity. The one self-citation, Farzi and Dietz [6] in Section 3.3, is used only to report that shorter prompts performed better for FLAN-T5-large in their earlier study; that observation is peripheral to the main research questions and does not establish the central claims. The paper's own tables are anchored to human labels and the LLMJudge benchmark, so the central assertion that UMBRELA transfers across LLMs has independent content. Score 1 reflects no meaningful circularity, with a minor caveat about relying on an external GPT-4o baseline for the headline comparison.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on external gold labels, copied baselines, and hand-set thresholds. No free parameters are fitted to data, and no new entities are introduced. The largest epistemic load is the copied GPT-4o baseline and the absence of variance estimates.

free parameters (3)

Rank correlation significance threshold = 0.005
Declared cutoff in Section 2.6 for treating differences in Spearman rho and Kendall tau as not significant; no bootstrapping or significance test supports the value, and it directly supports the comparable performance conclusion.
Cohen's kappa significance threshold = 0.01
Declared cutoff for scale and binary kappa in Section 2.6; used to call differences non-significant without variance estimates.
Parser fallback default label = 0
Section 3.6: outputs matching none of the expected patterns are assigned relevance 0, which can depress agreement estimates for models with format errors.

assumptions (5)

domain assumption Human NIST labels on TREC DL 2019/2020/2023 are treated as ground truth for relevance.
Section 2.3 defines evaluation as closeness to human-annotated judgments without questioning their reliability.
ad hoc to paper The GPT-4o baseline copied from the original UMBRELA paper is comparable to the new runs.
Section 2.2 states that GPT-4o numbers are taken unchanged; validity depends on identical datasets, prompt versions, decoding, and parsing between the two studies.
ad hoc to paper Metric differences within the stated thresholds are not significant.
Section 2.6 sets at most 0.005 for rho and tau and at most 0.01 for kappa as non-significant with no supporting statistical test.
domain assumption Deterministic decoding and unversioned API endpoints give stable, comparable outputs.
Section 2.5 uses temperature 0 and the Together API without pinning model versions or dates, so outputs are assumed stable.
standard math Cohen's kappa, Spearman's rho, and Kendall's tau are valid summary measures for this ordinal-label and system-ranking task.
Used in Section 2.6 without derivation; standard statistics literature supports them, so this is a low-risk background assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Does UMBRELA Work on Other LLMs?." pith.science (2026). https://pith.science/paper/NIS5BKWD

@misc{pith2026250709483,
  author       = {Pith},
  title        = {Pith review of: Does UMBRELA Work on Other LLMs?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NIS5BKWD}},
  note         = {Machine review of arXiv:2507.09483}
}

read the original abstract

We reproduce the UMBRELA LLM Judge evaluation framework across a range of large language models (LLMs) to assess its generalizability beyond the original study. Our investigation evaluates how LLM choice affects relevance assessment accuracy, focusing on leaderboard rank correlation and per-label agreement metrics. Results demonstrate that UMBRELA with DeepSeek V3 obtains very comparable performance to GPT-4o (used in original work). For LLaMA-3.3-70B we obtain slightly lower performance, which further degrades with smaller LLMs.

Figures

Figures reproduced from arXiv: 2507.09483 by the authors.

**Figure 2.** The zero-shot basic prompt for relevance assess [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Effect of model scale on rank correlation metrics. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Effect of model scale on per-label agreement. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages

[13]

Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor, 2024

Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, and Jimmy Lin. Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor, 2024

work page 2024
[1]

Charles L. A. Clarke and Laura Dietz. Llm-based relevance assessment still can’t replace human relevance assessment, 2024

work page 2024
[2]

Overview of the trec 2020 deep learning track

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the trec 2020 deep learning track. arXiv preprint arXiv:2102.07662, 2021

arXiv 2020
[3]

Overview of the trec 2019 deep learning track

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820, 2020

arXiv 2019
[4]

Zhang, Han Bao, Han- wei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bing-Li Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dong-Li Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Han- wei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Hua...

arXiv 2024
[5]

Guglielmo Faggioli, Laura Dietz, Charles L. A. Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, N. Kando, E. Kanoulas, Martin Potthast, Benno Stein, and Henning Wachsmuth. Perspectives on large language models for relevance judgment. Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval , 2023

work page 2023
[6]

Pencils down! automatic rubric-based evaluation of retrieve/generate systems

Naghmeh Farzi and Laura Dietz. Pencils down! automatic rubric-based evaluation of retrieve/generate systems. In Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval , pages 175–184, 2024

work page 2024
[7]

M. G. Kendall. A new measure of rank correlation. Biometrika, 30:81–93, 1938

work page 1938

Show all 13 references

[8]

Gpt-4o system card, 2024

OpenAI. Gpt-4o system card, 2024. arXiv:2410.21276

2024 arXiv
[9]

Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L

Hossein A. Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L. A. Clarke, Mohammad Aliannejadi, Clemencia Siro, and Guglielmo Fag- gioli. LLMJudge: LLMs for Relevance Judgments, August 2024. arXiv:2408.08896

2024 arXiv
[10]

Is chatgpt good at search? investigating large language models as re-ranking agent

Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agent. ArXiv, abs/2304.09542, 2023

2023 arXiv
[11]

Llm-assisted relevance assessments: When should we ask llms for help? arXiv preprint arXiv:2411.06877, 2024

Rikiya Takehi, Ellen M Voorhees, Tetsuya Sakai, and Ian Soboroff. Llm-assisted relevance assessments: When should we ask llms for help? arXiv preprint arXiv:2411.06877, 2024

2024 arXiv
[12]

Large language models can accurately predict searcher preferences

Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. Large language models can accurately predict searcher preferences. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1930–1940, 2024

1930

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [13]

Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor, 2024

Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, and Jimmy Lin. Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor, 2024

work page 2024

[2] [1]

Charles L. A. Clarke and Laura Dietz. Llm-based relevance assessment still can’t replace human relevance assessment, 2024

work page 2024

[3] [2]

Overview of the trec 2020 deep learning track

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the trec 2020 deep learning track. arXiv preprint arXiv:2102.07662, 2021

arXiv 2020

[4] [3]

Overview of the trec 2019 deep learning track

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820, 2020

arXiv 2019

[5] [4]

Zhang, Han Bao, Han- wei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bing-Li Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dong-Li Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Han- wei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Hua...

arXiv 2024

[6] [5]

Guglielmo Faggioli, Laura Dietz, Charles L. A. Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, N. Kando, E. Kanoulas, Martin Potthast, Benno Stein, and Henning Wachsmuth. Perspectives on large language models for relevance judgment. Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval , 2023

work page 2023

[7] [6]

Pencils down! automatic rubric-based evaluation of retrieve/generate systems

Naghmeh Farzi and Laura Dietz. Pencils down! automatic rubric-based evaluation of retrieve/generate systems. In Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval , pages 175–184, 2024

work page 2024

[8] [7]

M. G. Kendall. A new measure of rank correlation. Biometrika, 30:81–93, 1938

work page 1938