Pith. sign in

REVIEW 4 major objections 5 minor 22 references

When LLMs Disagree: Diagnosing Relevance Filtering Bias and Retrieval Divergence in SDG Search

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that when two large language models filter scholarly abstracts for relevance to UN Sustainable Development Goals, their disagreement is systematic: the models favor different vocabularies, surface different top-20 ranked…

desk verdict Useful empirical caution about LLM filtering in SDG search, with a solid core but fixable statistical and internal-consistency flaws plus missing prompt/parser artifacts. read the letter →

arxiv 2507.02139 v1 pith:UKMDYULT submitted 2025-07-02 cs.IR cs.AIcs.DL

William A. Ingram , Bipasha Banerjee , Edward A. Fox This is my paper
classification cs.IRcs.AIcs.DL
keywords LargeLanguageModelsInformationRetrievalSustainableDevelopmentGoalsRelevanceFilteringModelDisagreementDivergenceEvaluationMetricsBoolean
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the disagreements between two large language models used to filter research abstracts for relevance to UN Sustainable Development Goals (SDGs) are noise or signal. It shows they are signal: LLaMA and Qwen agree on about 84% of decisions, but the remaining disagreement cases have consistent lexical signatures (for example, one model favors clinical terms and the other molecular terms in health research), they change which documents reach the top of a ranked list even when the scoring function is identical, and a simple classifier can tell which model produced a label with AUC above 0.74. If correct, this means LLM-based relevance filtering is not a neutral preprocessing step: the choice of model reshapes which evidence gets retrieved, and that variability is structured enough to be predicted and audited. The paper's contribution is to treat classification disagreement itself as the object of analysis, rather than as labeling error to be averaged away.

What carries the argument

The load-bearing object is the directional disagreement subset: abstracts that exactly one model labels Relevant. Over these subsets the paper runs four analyses: (1) TF-IDF contrastive lexicography with permutation testing and Benjamini–Hochberg correction to establish that the models favor different vocabularies; (2) KL divergence between the normalized term distributions of the two directional subsets, giving a single scalar (1.04–1.60) for the lexical shift; (3) retrieval simulation in which both models' relevant sets are ranked by cosine similarity to a shared centroid or representative-term query over the same TF-IDF space, isolating the effect of filtering from ranking; and (4) a logistic-regression learnability test on TF-IDF features with five-fold cross-validated AUC, showing disagreement is predictable. The shared TF-IDF space and the controlled generation parameters (temperature 0, do_sample False) are what make the observed divergence attributable to model-specific relevance criteria rather than to sampling or scoring differences.

What would settle it

Re-run the labeling pipeline invoking both models through the same chat-template mechanism and have a human review a random sample of parsed labels; if the disagreement rate drops markedly, the lexical clusters blur, and the classifier AUC falls toward 0.5, the systematic patterns were artifacts of invocation rather than model-specific relevance criteria.

Watch

Extended reading notes

Core claim

The central claim is that model disagreement in LLM-based relevance filtering is systematic, not random. On a corpus of 46,755 scholarly abstract-SDG pairs from SDGs 1, 3, and 7, two open-weight LLMs labeled with identical fixed prompts agree on 83.6% of decisions but with only moderate chance-corrected agreement (Cohen's $\kappa = 0.467$), and the 16.4% disagreement is concentrated in interpretable directions: permutation tests after FDR correction show that LLaMA-relevant documents are enriched for terms like health, care, and insurance in SDG 1 while Qwen-relevant documents are enriched for inequality, tax, and wealth, with parallel clinical-versus-molecular and systems-versus-electrochemistry splits in SDGs 3 and 7. When both models' relevant subsets are ranked against the same TF-IDF query over the shared disagreement pool, the top-20 compositions diverge sharply (19 vs. 1 for SDG 7 under centroid ranking), and a logistic regression trained only on TF-IDF features can predict which model labeled a document with mean AUCs of 0.759 (SDG 1), 0.762 (SDG 3), and 0.746 (SDG 7). The paper explicitly does not claim one model is right; it claims that filtering disagreement propagates into retrieval divergence and is learnable from surface features.

Load-bearing premise

The models must actually be receiving and following the same instructions: LLaMA prompts were wrapped with its chat-template utility while Qwen prompts were passed as raw message dictionaries, and every label was parsed automatically with no human check, so part of the 'model-specific' disagreement could in principle come from prompt-wrapping or parsing artifacts.

Editorial extensions

If this is right

  • In a retrieval pipeline that uses a single LLM as a relevance filter before ranking, the set of documents surfaced depends on which LLM is chosen, even when the prompt and ranking logic are held fixed.
  • Disagreement has interpretable direction (e.g., clinical vs. molecular in health, systems vs. materials in energy), so audits of LLM-filtered corpora can target the specific term clusters each model favors.
  • Because a simple TF-IDF classifier can predict which model labeled a document with AUC above 0.74, disagreement sets can be flagged automatically for adjudication or ensemble filtering without access to human ground truth.
  • In thematic search tasks like SDG mapping, agreement statistics alone are misleading: high raw agreement (up to 89.6%) can coincide with low kappa and with substantial divergence in what is retrievable.
  • Retrieval divergence under shared scoring shows that filtering, not ranking, is where model choice exerts its main influence on top-k outcomes in borderline cases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same structured-disagreement phenomenon should be expected in other value-laden thematic searches, such as systematic-review screening or grant-portfolio mapping, where relevance is multidimensional and no gold labels exist; the paper's methods transfer directly to those settings.
  • If disagreement is learnable from lexical features, then sampling only the disagreement set for human review is a potentially far more efficient way to build ground-truth labels than random sampling, since those are exactly the cases where model output is contested.
  • A natural next test the paper does not run is to collect expert judgments on a sample of disagreement cases; such data could reveal whether one model's lexical preference aligns better with expert SDG contribution judgments, turning 'bias' from a retrieval-divergence descriptor into a correctness claim.
  • The observed split between clinical and molecular terms in SDG 3 suggests the models may be optimizing for different implicit definitions of 'contribution' — one favoring direct patient-level impact and the other favoring mechanistic knowledge; reconciling these definitions is a scientific question, not just a modeling artifact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies binary relevance labeling disagreement between two open-weight LLMs (LLaMA 3.1-8B and Qwen 2.5-7B) over scholarly abstracts mapped to SDGs 1, 3, and 7. It quantifies disagreement (about 16% of cases), contrasts the lexical profiles of the directional disagreement subsets via TF-IDF and permutation tests, simulates top-k retrieval over the disagreement pool under centroid- and query-based scoring, and trains logistic-regression classifiers to predict which model labeled a document relevant (AUCs above 0.74). The central claim is that disagreement is systematic rather than random, that it reflects model-specific relevance criteria under controlled prompting, and that it propagates into divergent retrieval outputs.

Significance. If the central claim holds, the paper makes a useful contribution to LLM-based relevance labeling in IR: it reframes inter-model disagreement as a diagnosable signal rather than noise, and it connects labeling variability to concrete retrieval consequences in a policy-relevant domain (SDG analysis). The study is also commendable for avoiding the common pitfall of treating LLM labels as ground truth, and for explicitly scoping its conclusions to the two models and three SDGs considered. However, the main empirical supports for 'systematic, not random' rest on a permutation-testing procedure with selection on the same data, and on an invocation pipeline with unequal chat-template formatting and unvalidated automatic parsing. These issues must be resolved before the paper's headline conclusions can be accepted.

major comments (4)
  1. [§5.2.2, Table 2] The permutation test is applied only to the top 200 terms ranked by absolute TF-IDF difference computed on the full disagreement data, and the Benjamini-Hochberg FDR correction is then applied to those selected terms only. This is a form of selective inference: under the global null, the selected terms are the most extreme by construction, so the reported FDR-corrected p-values do not control the error rate for the hypothesis tested. Consequently, the claim that 'in all three SDGs, the top-ranked terms by TF-IDF difference were found to be statistically significant after FDR correction' is not supported by the reported procedure. The authors should either test all terms with the FDR correction, or use a two-stage procedure in which the selection terms are chosen on a training split and tested on a holdout split, or apply a conditional/selective inference method.
  2. [§4.1, §3] The paper states in Section 3 that 'The LLMs are given identical prompts,' but Section 4.1 describes different invocation formats: LLaMA prompts were constructed with apply_chat_template, while Qwen prompts were passed directly as message dictionaries. These formats inject different system tokens and template-level instructions, so the effective prompts are not identical. Additionally, 'All labels were parsed automatically without human intervention' with no parse-success rate or validation sample reported; if one model's outputs contain refusals, extra text, or differently positioned labels, the parser could assign labels based on formatting artifacts. Because every downstream analysis (Sections 5.2-5.4) conditions on these parsed labels, the observed lexical patterns and AUC results cannot yet be attributed to model-specific relevance criteria. The authors should release the exact prompts, invocation code, and parser, and provide a human-checked sample of disagreement labels (or at minimum a parse-success rate and a random validation sample).
  3. [§5.3, Table 3] The retrieval-divergence result is close to definitional. After partitioning the disagreement pool into LLaMA-only relevant and Qwen-only relevant sets, ranking within each disjoint set under a shared scorer will generally produce different top-k lists even if the relevance labels were assigned at random. Table 3 therefore does not demonstrate retrieval divergence beyond the trivial fact that the candidate sets differ. To support RQ3, the authors need a null model or baseline, such as comparing the observed top-k overlap to the expected overlap under random partitions of the disagreement pool with the same set sizes, or reporting the overlap between top-k lists drawn from the full corpus with and without each model's filter. Without such a baseline, the claim that disagreement 'produces divergent top-ranked outputs' is not an empirical finding.
  4. [§4.3.1 vs. §5.3] The centroid query is defined inconsistently. Section 4.3.1 states that the centroid is computed from 'the entire SDG-specific abstract corpus,' while Section 5.3 states that 'the query was defined as the mean TF-IDF vector across all abstracts in the disagreement subset.' These definitions produce different query vectors and thus different rankings, so the retrieval results are not reproducible as reported. Similarly, Section 4.3 says a TF-IDF vectorizer is fit on the combined disagreement corpus, while Section 5.3 says the representation is 'trained on the full corpus'; this discrepancy also affects the cosine-similarity computations. The authors should clarify which corpus was used for vectorizer fitting and for centroid/query construction, and rerun or re-report the experiments consistently.
minor comments (5)
  1. [§5.2.4] The sentence 'These patterns are consistent with the models’ respective distributions over other SDGs as well' reads awkwardly; it presumably means the patterns are consistent across the other SDG analyses, not that the models have distributions over SDGs. Please rephrase.
  2. [Table 2] The SDG 3 column is not sorted consistently by the reported difference values (tavr +0.007 appears before risk +0.007, and coronary +0.006 appears after valve +0.005). Also, 'tavr' should be expanded on first use or written in title case.
  3. [§4.2 and §5.2.2] Section 4.2 says permutation tests are conducted 'for each term,' while Section 5.2.2 restricts testing to the top 200 terms. Please state the restriction in the methodology section as well, and make the permutation count consistent (9,999 in the text vs. n=9999 in the Table 2 caption is fine, but the current phrasing is confusing).
  4. [§4.3.2] The representative queries are selected as the eight terms with highest mean TF-IDF from the full SDG-specific corpus, but the method does not state whether the TF-IDF model used for this selection is the same as the one used to vectorize the disagreement matrix. Please make the shared/separate use of the vectorizer explicit.
  5. [§5.4] The AUC results are reported as mean ± standard deviation over five folds, but it is not stated whether the TF-IDF vectorizer was fit inside the cross-validation loop. If the vectorizer is fit on the full disagreement set before splitting, the cross-validated AUC may be optimistically biased due to feature leakage.

Circularity Check

1 steps flagged · score 4.0 of 10

Retrieval-divergence result in Section 5.3 is definitional, but the lexical and learnability findings are independent; no load-bearing circularity.

  1. self definitional [Section 5.3.1 (Findings), method in Section 4.3.1]
    "When documents labeled as relevant by one model and non-relevant by another are ranked using a shared scoring function, the resulting top-k sets differ by model. Because the input pool is fixed and scoring is held constant, these differences arise from earlier filtering decisions that determine which documents are present in the disagreement set."

    By construction, the disagreement set contains documents labeled Relevant by exactly one model, so each model's candidate pool is its own Relevant subset and the two pools are disjoint. Ranking two disjoint pools with any scoring function yields disjoint top-k lists, so the observed 'divergence' is an immediate consequence of the subset construction rather than an empirical discovery about scoring behavior. The paper's own caveat that the differences arise from earlier filtering decisions restates the premise; the shared scoring function is irrelevant to whether the lists differ.

full rationale

The central lexical-contrast (Section 5.2) and learnability (Section 5.4) results are self-contained empirical analyses: they use permutation tests, KL divergence, and cross-validated logistic regression on TF-IDF features to show that disagreement subsets are linguistically separable. These measurements do not reduce to the model labels by construction; they are genuine evidence for structured disagreement. The only definitional step is the retrieval-divergence result in Section 5.3, where top-k lists are drawn from disjoint model-specific relevant subsets, making their difference guaranteed by the definition of disagreement. The paper explicitly acknowledges this, and it does not rely on that result to establish systematicity, yet the abstract advertises 'divergent top-ranked outputs under shared scoring functions' as a joint finding alongside the empirical ones. Because one of the three advertised findings is analytic, a moderate score is warranted. No load-bearing self-citations or imported uniqueness theorems were found; reference [8] is merely motivational. The prompt-format differences between LLaMA and Qwen noted in Section 4.1 are a validity concern but not a circular step.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim relies on several unverified modeling choices: manual thresholds for TF-IDF and permutation tests, a selected-term significance test that is statistically invalid, and an unstated assumption that different chat templates and unvalidated parsing do not create the disagreement. No invented entities are introduced.

free parameters (6)
  • top_N_terms_for_permutation_test = 200
    Hand-chosen cutoff in Section 5.2.2; terms are selected by the same TF-IDF difference being tested, which inflates significance.
  • min_df = 5
    TF-IDF vocabulary cutoff used in Sections 4.2 and 5.2.1.
  • max_df = 0.95
    TF-IDF vocabulary cutoff used in Sections 4.2 and 5.2.1.
  • top_k_retrieval = 20
    Retrieval depth compared in Section 5.3; affects Table 3 counts.
  • representative_query_terms_per_SDG = 8 terms per SDG
    Selected as the highest mean TF-IDF terms from the full SDG corpus in Section 4.3.2; these data-derived terms define the query-based retrieval experiment.
  • permutation_iterations = 9999
    Number of permutations used in Section 5.2.2; a standard choice.
assumptions (5)
  • domain assumption LLaMA and Qwen outputs are comparable despite different chat template handling
    Section 4.1: LLaMA uses apply_chat_template while Qwen uses raw message dicts; the paper treats the prompts as identical.
  • domain assumption Automatic binary label parsing is error-free
    Section 4.1: labels are parsed automatically with no human validation, and no validation sample is reported.
  • domain assumption TF-IDF lexical features adequately capture systematic disagreement
    Sections 4.2 and 5.2 use only surface lexical features; semantic or embedding-based variation is not tested.
  • ad hoc to paper Permutation testing on the top 200 terms selected by the same statistic is valid
    Section 5.2.2: selection on the same data invalidates the reported p-values, yet the paper's significance claims rely on it.
  • domain assumption The Elsevier Boolean-query corpus is an appropriate relevance pool for the research questions
    Section 3 builds the corpus from Scopus queries with no human ground truth, which the paper acknowledges.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When LLMs Disagree: Diagnosing Relevance Filtering Bias and Retrieval Divergence in SDG Search." pith.science (2026). https://pith.science/paper/UKMDYULT

@misc{pith2026250702139,
  author       = {Pith},
  title        = {Pith review of: When LLMs Disagree: Diagnosing Relevance Filtering Bias and Retrieval Divergence in SDG Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UKMDYULT}},
  note         = {Machine review of arXiv:2507.02139}
}
read the original abstract

Large language models (LLMs) are increasingly used to assign document relevance labels in information retrieval pipelines, especially in domains lacking human-labeled data. However, different models often disagree on borderline cases, raising concerns about how such disagreement affects downstream retrieval. This study examines labeling disagreement between two open-weight LLMs, LLaMA and Qwen, on a corpus of scholarly abstracts related to Sustainable Development Goals (SDGs) 1, 3, and 7. We isolate disagreement subsets and examine their lexical properties, rank-order behavior, and classification predictability. Our results show that model disagreement is systematic, not random: disagreement cases exhibit consistent lexical patterns, produce divergent top-ranked outputs under shared scoring functions, and are distinguishable with AUCs above 0.74 using simple classifiers. These findings suggest that LLM-based filtering introduces structured variability in document retrieval, even under controlled prompting and shared ranking logic. We propose using classification disagreement as an object of analysis in retrieval evaluation, particularly in policy-relevant or thematic search tasks.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 9 canonical work pages

  1. [1]

    Zahra Abbasiantaeb, Chuan Meng, Leif Azzopardi, and Mohammad Aliannejadi

  2. [2]

    Saling, Mark Sanderson, Falk Scholer, Damiano Spina, and Ryen W

    Marwah Alaofi, Luke Gallagher, Dana Mckay, Lauren L. Saling, Mark Sanderson, Falk Scholer, Damiano Spina, and Ryen W. White. 2022. Where Do Queries Come From?. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’22) . Association for Computing Machinery, New York, NY, USA, 2850–2862. d...

    arXiv 2022

  3. [3]

    Negar Arabzadeh and Charles L. A. Clarke. 2024. A Comparison of Methods for Evaluating Generative IR. CoRR abs/2404.04044 (2024). doi:10.48550/arXiv.2404. 04044 arXiv:2404.04044 [cs]

    arXiv doi:10.48550/arxiv.2404 2024

  4. [4]

    Alexandre Bedard-Vallee, Chris James, and Guillaume Roberge. 2023. Elsevier 2023 Sustainable Development Goals (SDGs) Mapping . doi:10.17632/y2zyy9vwzy.1

    work page doi:10.17632/y2zyy9vwzy.1 2023

  5. [5]

    Lotufo, and Jayr Alencar Pereira

    Mirelle Bueno, Eduardo Seiti de Oliveira, Rodrigo Nogueira, Roberto A. Lotufo, and Jayr Alencar Pereira. 2024. Quati: A Brazilian Portuguese Information Retrieval Dataset from Native Speakers. CoRR abs/2404.06976 (2024). doi:10. 48550/arXiv.2404.06976 arXiv:2404.06976 [cs]

    arXiv doi:10.48550/arxiv.2404.06976 2024

  6. [6]

    Gabriel de Jesus and Sérgio Nunes. 2024. Exploring Large Language Models for Relevance Judgments in Tetun. CoRR abs/2406.07299 (2024). doi:10.48550/arXiv. 2406.07299 arXiv:2406.07299 [cs]

    work page Pith review arXiv doi:10.48550/arxiv.2406.07299 2024

  7. [7]

    Guglielmo Faggioli, Laura Dietz, Charles L. A. Clarke, Gianluca Demartini, Matthias Hagen, Claudia Hauff, Noriko Kando, Evangelos Kanoulas, Martin Potthast, Benno Stein, and Henning Wachsmuth. 2023. Perspectives on Large Lan- guage Models for Relevance Judgment. InProceedings of the 2023 ACM SIGIR Inter- national Conference on Theory of Information Retrie...

    work page arXiv 2023

  8. [8]

    Ingram, Bipasha Banerjee, and Edward A

    William A. Ingram, Bipasha Banerjee, and Edward A. Fox. 2024. Agentic AI for Improving Precision in Identifying Contributions to Sustainable Development Goals. In Proceedings of the 2024 IEEE International Conference on Big Data (Big- Data). IEEE, Washington, DC, USA, 8677–8679. doi:10.1109/BigData62323.2024. 10825072 Added to IEEE Xplore: 16 January 2025

    arXiv 2024

Show all 22 references
  1. [9]

    Bamini Jayabalasingham, Roy Boverhof, Kevin Agnew, and Lisette Klein. 2019. Identifying Research Supporting the United Nations Sustainable Development Goals . doi:10.17632/87txkw7khs.1

    2019 doi

  2. [10]

    Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021. Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators’ Disagreement. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Proce...

    2021 doi

  3. [11]

    Michael Li, Jianping Sun, and Xianming Tan. 2024. Evaluating the Effective- ness of Large Language Models in Abstract Screening: A Comparative Analysis. Systematic Reviews 13, 1 (aug 2024), 219. doi:10.1186/s13643-024-02609-x

    2024 doi

  4. [12]

    Sean MacAvaney and Luca Soldaini. 2023. One-Shot Labeling for Automatic Rel- evance Estimation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’23) . Association for Computing Machinery, New York, NY, U...

    2023 doi

  5. [13]

    Chuan Meng, Negar Arabzadeh, Arian Askari, Mohammad Aliannejadi, and Maarten de Rijke. 2024. Query Performance Prediction Using Relevance Judg- ments Generated by Large Language Models. CoRR abs/2404.01012 (2024). doi:10.48550/arXiv.2404.01012 arXiv:2404.01012 [cs]

    2024 doi arXiv

  6. [14]

    Meta. 2024. meta-llama/Llama-3.1-8B-Instruct. https://huggingface.co/meta- llama/Llama-3.1-8B-Instruct Accessed: 2025-04-02

    2024

  7. [15]

    Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. Transactions of the Association for Computational Linguistics 10 (2022), 92–110. doi:10.1162/tacl_a_00449

    2022 doi

  8. [16]

    Ian Soboroff. 2024. Don’t Use LLMs to Make Relevance Judgments. CoRR abs/2409.15133 (sep 2024). doi:10.48550/arXiv.2409.15133 arXiv:abs/2409.15133

    2024 doi arXiv

  9. [17]

    Qwen Team. 2024. Qwen/Qwen2.5-7B-Instruct. https://huggingface.co/Qwen/ Qwen2.5-7B-Instruct Accessed: 2025-04-02

    2024

  10. [18]

    THE reporters. 2024. Impact Rankings 2024: Methodology . https: //www.timeshighereducation.com/world-university-rankings/impact- rankings-2024-methodology

    2024

  11. [19]

    Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large Lan- guage Models Can Accurately Predict Searcher Preferences.CoRR abs/2309.10621 (2024). doi:10.48550/arXiv.2309.10621 arXiv:2309.10621 [cs]

    2024 doi arXiv

  12. [20]

    Voorhees

    Ellen M. Voorhees. 2002. The Philosophy of Information Retrieval Evaluation. In Evaluation of Cross-Language Information Retrieval Systems , Carol Peters, Martin Braschler, Julio Gonzalo, and Michael Kluck (Eds.). Springer, Berlin, Heidelberg, 355–370. doi:10.1007/3-540-45691-0_34

    2002 doi

  13. [21]

    Fanghua Ye, Meng Fang, Shenghui Li, and Emine Yilmaz. 2023. Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewrit- ing. In Findings of the Association for Computational Linguistics: EMNLP 2023 , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). A...

    2023 doi

  14. [2024]

    doi:10.48550/arXiv.2405.05600 arXiv:2405.05600 [cs]

    Can We Use Large Language Models to Fill Relevance Judgment Holes? CoRR abs/2405.05600 (2024). doi:10.48550/arXiv.2405.05600 arXiv:2405.05600 [cs]

    2024 doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.