{"id":"d1eddee1-4656-4e9d-854c-3aa90270be37","arxiv_id":"2507.02139","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Two LLMs disagree on about 16% of SDG relevance labels, and the disagreement is lexically systematic and changes top-20 retrieval results.","lead":"This paper compares two open-weight AI models, LLaMA and Qwen, as they label academic abstracts for relevance to UN Sustainable Development Goals. It finds the models disagree on 16% of documents and that their disagreements follow consistent word-use patterns, which changes which papers rise to the top of a search.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4.1's unequal chat-template application and unvalidated automatic parsing may confound the observed disagreement, so the central claim's 'controlled prompting' condition is not established.","rationale":"The reader's weakest-assumption analysis already identified the same load-bearing concern: the two models were invoked with different chat-template paths, and labels were automatically parsed without human validation. I agree that this is the most load-bearing issue because it sits upstream of every result. The paper's abstract asserts that disagreement is systematic 'even under controlled prompting'; if the two models were not actually given token-identical instructions, or if parsing errors create systematic label noise, then the lexical patterns, the ranking asymmetries, and the AUC values could all reflect invocation artifacts rather than model-specific relevance criteria. This does not mean the paper is worthless—the observed disagreement is still real under the specific pipeline used—but the central claim's interpretation as evidence about LLM filtering behavior requires the confound to be eliminated. The reader's CONDITIONAL verdict is exactly right: the concern is addressable by releasing code/data and adding validation, but until then the claim is not fully established. I considered the permutation-test selection bias in §5.2.2 (top-200 terms chosen by observed difference before testing) and the partly definitional nature of the retrieval-top-k comparison in §5.3, but these are secondary: the AUC result provides independent support for learnability, and the retrieval divergence, while perhaps under-analyzed, does not undercut the central claim as fundamentally as an unverified label pipeline. No ad hominem is intended; the concern is about the evidence chain, not the authors' conduct.","tokens_in":12293,"tokens_out":6971,"duration_ms":90576,"concrete_test":"Release the exact system/user prompts, the two message-construction code paths, and the parsing script. Then (1) rerun both models with the same invocation style (e.g., both via apply_chat_template, or both as raw message dictionaries) and recompute the disagreement rate, Table 2 terms, and §5.4 AUCs; if the disagreement pattern and learnability largely disappear, the reported effect is a template artifact. (2) Manually or rule-validate a random sample of at least 200 disagreement labels per SDG against the raw generation outputs, estimating parse-error rates and checking whether label flips are due to formatting variations; if parse errors are negligible and results persist under identical invocation, the confound is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that model disagreement is systematic, not random, reflecting model-specific relevance criteria under controlled prompting—requires that the automatically parsed binary labels faithfully represent each model's judgment. Section 4.1 introduces two confounds. First, LLaMA prompts were constructed with apply_chat_template while Qwen prompts were passed directly as message dictionaries; despite the paper's statement of 'identical prompts' in Section 3, the two models therefore receive different effective prompt formats, including model-specific system tokens and template-injected instructions. Second, 'All labels were parsed automatically without human intervention' with no reported parse-success rate or validation sample. If one model's outputs systematically contain extra text, refusals, or differently positioned labels, the parser could assign labels based on format, not relevance. Since every downstream analysis—lexical contrast (§5.2), retrieval divergence (§5.3), and the AUC learnability result (§5.4)—conditions on these parsed labels, any template or parsing artifact would masquerade as structured, model-specific disagreement. Without released prompts, invocation code, parser code, and a human-checked sample of disagreement labels, the observed lexical patterns and AUCs above 0.74 cannot be attributed to genuine differences in relevance criteria rather than to artifacts of the invocation pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies binary relevance labeling disagreement between two open-weight LLMs (LLaMA 3.1-8B and Qwen 2.5-7B) over scholarly abstracts mapped to SDGs 1, 3, and 7. It quantifies disagreement (about 16% of cases), contrasts the lexical profiles of the directional disagreement subsets via TF-IDF and permutation tests, simulates top-k retrieval over the disagreement pool under centroid- and query-based scoring, and trains logistic-regression classifiers to predict which model labeled a document relevant (AUCs above 0.74). The central claim is that disagreement is systematic rather than random, that it reflects model-specific relevance criteria under controlled prompting, and that it propagates into divergent retrieval outputs.","tokens_in":12555,"tokens_out":3585,"duration_ms":40765,"significance":"If the central claim holds, the paper makes a useful contribution to LLM-based relevance labeling in IR: it reframes inter-model disagreement as a diagnosable signal rather than noise, and it connects labeling variability to concrete retrieval consequences in a policy-relevant domain (SDG analysis). The study is also commendable for avoiding the common pitfall of treating LLM labels as ground truth, and for explicitly scoping its conclusions to the two models and three SDGs considered. However, the main empirical supports for 'systematic, not random' rest on a permutation-testing procedure with selection on the same data, and on an invocation pipeline with unequal chat-template formatting and unvalidated automatic parsing. These issues must be resolved before the paper's headline conclusions can be accepted.","major_comments":[{"comment":"The permutation test is applied only to the top 200 terms ranked by absolute TF-IDF difference computed on the full disagreement data, and the Benjamini-Hochberg FDR correction is then applied to those selected terms only. This is a form of selective inference: under the global null, the selected terms are the most extreme by construction, so the reported FDR-corrected p-values do not control the error rate for the hypothesis tested. Consequently, the claim that 'in all three SDGs, the top-ranked terms by TF-IDF difference were found to be statistically significant after FDR correction' is not supported by the reported procedure. The authors should either test all terms with the FDR correction, or use a two-stage procedure in which the selection terms are chosen on a training split and tested on a holdout split, or apply a conditional/selective inference method.","section":"§5.2.2, Table 2"},{"comment":"The paper states in Section 3 that 'The LLMs are given identical prompts,' but Section 4.1 describes different invocation formats: LLaMA prompts were constructed with apply_chat_template, while Qwen prompts were passed directly as message dictionaries. These formats inject different system tokens and template-level instructions, so the effective prompts are not identical. Additionally, 'All labels were parsed automatically without human intervention' with no parse-success rate or validation sample reported; if one model's outputs contain refusals, extra text, or differently positioned labels, the parser could assign labels based on formatting artifacts. Because every downstream analysis (Sections 5.2-5.4) conditions on these parsed labels, the observed lexical patterns and AUC results cannot yet be attributed to model-specific relevance criteria. The authors should release the exact prompts, invocation code, and parser, and provide a human-checked sample of disagreement labels (or at minimum a parse-success rate and a random validation sample).","section":"§4.1, §3"},{"comment":"The retrieval-divergence result is close to definitional. After partitioning the disagreement pool into LLaMA-only relevant and Qwen-only relevant sets, ranking within each disjoint set under a shared scorer will generally produce different top-k lists even if the relevance labels were assigned at random. Table 3 therefore does not demonstrate retrieval divergence beyond the trivial fact that the candidate sets differ. To support RQ3, the authors need a null model or baseline, such as comparing the observed top-k overlap to the expected overlap under random partitions of the disagreement pool with the same set sizes, or reporting the overlap between top-k lists drawn from the full corpus with and without each model's filter. Without such a baseline, the claim that disagreement 'produces divergent top-ranked outputs' is not an empirical finding.","section":"§5.3, Table 3"},{"comment":"The centroid query is defined inconsistently. Section 4.3.1 states that the centroid is computed from 'the entire SDG-specific abstract corpus,' while Section 5.3 states that 'the query was defined as the mean TF-IDF vector across all abstracts in the disagreement subset.' These definitions produce different query vectors and thus different rankings, so the retrieval results are not reproducible as reported. Similarly, Section 4.3 says a TF-IDF vectorizer is fit on the combined disagreement corpus, while Section 5.3 says the representation is 'trained on the full corpus'; this discrepancy also affects the cosine-similarity computations. The authors should clarify which corpus was used for vectorizer fitting and for centroid/query construction, and rerun or re-report the experiments consistently.","section":"§4.3.1 vs. §5.3"}],"minor_comments":[{"comment":"The sentence 'These patterns are consistent with the models’ respective distributions over other SDGs as well' reads awkwardly; it presumably means the patterns are consistent across the other SDG analyses, not that the models have distributions over SDGs. Please rephrase.","section":"§5.2.4"},{"comment":"The SDG 3 column is not sorted consistently by the reported difference values (tavr +0.007 appears before risk +0.007, and coronary +0.006 appears after valve +0.005). Also, 'tavr' should be expanded on first use or written in title case.","section":"Table 2"},{"comment":"Section 4.2 says permutation tests are conducted 'for each term,' while Section 5.2.2 restricts testing to the top 200 terms. Please state the restriction in the methodology section as well, and make the permutation count consistent (9,999 in the text vs. n=9999 in the Table 2 caption is fine, but the current phrasing is confusing).","section":"§4.2 and §5.2.2"},{"comment":"The representative queries are selected as the eight terms with highest mean TF-IDF from the full SDG-specific corpus, but the method does not state whether the TF-IDF model used for this selection is the same as the one used to vectorize the disagreement matrix. Please make the shared/separate use of the vectorizer explicit.","section":"§4.3.2"},{"comment":"The AUC results are reported as mean ± standard deviation over five folds, but it is not stated whether the TF-IDF vectorizer was fit inside the cross-validation loop. If the vectorizer is fit on the full disagreement set before splitting, the cross-validated AUC may be optimistically biased due to feature leakage.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant and timely topic for the LLM4Eval workshop, and the authors are transparent about the exploratory nature of parts of the analysis. The main technical issues—selective permutation testing and the confounded invocation pipeline—are fixable within the scope of a revised manuscript. I would also encourage the authors to make the code, prompts, and parsed labels available, as the claims depend heavily on the exact formatting and parsing steps. If the authors address the selective-inference problem and provide a human-validated sample of parsed labels, the paper could become a solid contribution; as it stands, the headline conclusions are not yet fully supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is a worthwhile empirical caution about using LLMs as relevance filters in SDG bibliometric search. Two open-weight models (LLaMA 3.1-8B, Qwen 2.5-7B) disagree on roughly 16% of labels across SDGs 1/3/7, the disagreement is lexically interpretable (clinical vs molecular in health; systems vs electrochemistry in energy), and a simple TF-IDF logistic regression separates the two sides at AUC ~0.75. Those numbers are new and useful. The paper also does the right thing by treating disagreement as a diagnostic signal rather than noise, and by refusing to claim one model is 'right.'\n\nThe main soft spots are real but fixable. First, the permutation test in §5.2.2 restricts to the top 200 terms by absolute TF-IDF difference before computing p-values on the same data. That selection invalidates the FDR-corrected p-values for those terms. The KL divergences and the AUC learnability result are independent evidence and carry the argument, so this isn't fatal, but the reported p-values overstate certainty.\n\nSecond, the retrieval section contradicts itself. §4.3.1 says the centroid query is the centroid of the full SDG corpus, while §5.3 says it is the mean of the disagreement subset. The two sections also disagree on whether the TF-IDF representation was fit on the disagreement corpus or the full corpus. That undermines the exact numbers in Table 3 until fixed. And the broad retrieval-divergence phenomenon is partly definitional: if you rank within each model's relevant set under a shared scorer, top-k lists will differ whenever the sets differ. The more interesting content is the directional asymmetry (e.g., SDG7: 19 vs 1), but that is presented without any uncertainty or significance assessment.\n\nThird, the prompting/parsing pipeline is not auditable. LLaMA was prompted with apply_chat_template, Qwen with direct message dictionaries; labels were parsed automatically with no reported parse-success rate or human validation, and no code/data are released. The stress-test concern about invocation artifacts is legitimate: if parse errors correlate with output format, they would masquerade as model-specific disagreement. The paper's own framing ('controlled prompting') requires this to be ruled out.\n\nBottom line: with the stats fixed, the contradiction resolved, and a small human-checked sample of parsed labels released, this is a solid workshop paper. It deserves a serious referee and, after revision, publication. I would bring it to reading group; the methodological discussion is valuable even if you don't work on SDGs.","headline":"Useful empirical caution about LLM filtering in SDG search, with a solid core but fixable statistical and internal-consistency flaws plus missing prompt/parser artifacts.","tokens_in":13091,"tokens_out":3012,"would_cite":true,"duration_ms":32953,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that when two large language models filter scholarly abstracts for relevance to UN Sustainable Development Goals, their disagreement is systematic: the models favor different vocabularies, surface different top-20 ranked…","keywords":["Large Language Models","Information Retrieval","Sustainable Development Goals","Relevance Filtering","Model Disagreement","Retrieval Divergence","Evaluation Metrics","Boolean Retrieval"],"falsifier":"Re-run the labeling pipeline invoking both models through the same chat-template mechanism and have a human review a random sample of parsed labels; if the disagreement rate drops markedly, the lexical clusters blur, and the classifier AUC falls toward 0.5, the systematic patterns were artifacts of invocation rather than model-specific relevance criteria.","tokens_in":12082,"feed_emoji":"🔍","tokens_out":10006,"duration_ms":93987,"temperature":0.7,"pith_summary":"This paper asks whether the disagreements between two large language models used to filter research abstracts for relevance to UN Sustainable Development Goals (SDGs) are noise or signal. It shows they are signal: LLaMA and Qwen agree on about 84% of decisions, but the remaining disagreement cases have consistent lexical signatures (for example, one model favors clinical terms and the other molecular terms in health research), they change which documents reach the top of a ranked list even when the scoring function is identical, and a simple classifier can tell which model produced a label with AUC above 0.74. If correct, this means LLM-based relevance filtering is not a neutral preprocessing step: the choice of model reshapes which evidence gets retrieved, and that variability is structured enough to be predicted and audited. The paper's contribution is to treat classification disagreement itself as the object of analysis, rather than as labeling error to be averaged away.","feed_headline":"SDG search results shift with the LLM doing the filtering","feed_subtitle":"Two open-weight LLMs disagree on 16% of abstracts in ways a simple classifier can predict.","key_machinery":"The load-bearing object is the directional disagreement subset: abstracts that exactly one model labels Relevant. Over these subsets the paper runs four analyses: (1) TF-IDF contrastive lexicography with permutation testing and Benjamini–Hochberg correction to establish that the models favor different vocabularies; (2) KL divergence between the normalized term distributions of the two directional subsets, giving a single scalar (1.04–1.60) for the lexical shift; (3) retrieval simulation in which both models' relevant sets are ranked by cosine similarity to a shared centroid or representative-term query over the same TF-IDF space, isolating the effect of filtering from ranking; and (4) a logistic-regression learnability test on TF-IDF features with five-fold cross-validated AUC, showing disagreement is predictable. The shared TF-IDF space and the controlled generation parameters (temperature 0, do_sample False) are what make the observed divergence attributable to model-specific relevance criteria rather than to sampling or scoring differences.","core_discovery":"The central claim is that model disagreement in LLM-based relevance filtering is systematic, not random. On a corpus of 46,755 scholarly abstract-SDG pairs from SDGs 1, 3, and 7, two open-weight LLMs labeled with identical fixed prompts agree on 83.6% of decisions but with only moderate chance-corrected agreement (Cohen's $\\kappa = 0.467$), and the 16.4% disagreement is concentrated in interpretable directions: permutation tests after FDR correction show that LLaMA-relevant documents are enriched for terms like health, care, and insurance in SDG 1 while Qwen-relevant documents are enriched for inequality, tax, and wealth, with parallel clinical-versus-molecular and systems-versus-electrochemistry splits in SDGs 3 and 7. When both models' relevant subsets are ranked against the same TF-IDF query over the shared disagreement pool, the top-20 compositions diverge sharply (19 vs. 1 for SDG 7 under centroid ranking), and a logistic regression trained only on TF-IDF features can predict which model labeled a document with mean AUCs of 0.759 (SDG 1), 0.762 (SDG 3), and 0.746 (SDG 7). The paper explicitly does not claim one model is right; it claims that filtering disagreement propagates into retrieval divergence and is learnable from surface features.","pith_inferences":["The same structured-disagreement phenomenon should be expected in other value-laden thematic searches, such as systematic-review screening or grant-portfolio mapping, where relevance is multidimensional and no gold labels exist; the paper's methods transfer directly to those settings.","If disagreement is learnable from lexical features, then sampling only the disagreement set for human review is a potentially far more efficient way to build ground-truth labels than random sampling, since those are exactly the cases where model output is contested.","A natural next test the paper does not run is to collect expert judgments on a sample of disagreement cases; such data could reveal whether one model's lexical preference aligns better with expert SDG contribution judgments, turning 'bias' from a retrieval-divergence descriptor into a correctness claim.","The observed split between clinical and molecular terms in SDG 3 suggests the models may be optimizing for different implicit definitions of 'contribution' — one favoring direct patient-level impact and the other favoring mechanistic knowledge; reconciling these definitions is a scientific question, not just a modeling artifact."],"forward_implications":["In a retrieval pipeline that uses a single LLM as a relevance filter before ranking, the set of documents surfaced depends on which LLM is chosen, even when the prompt and ranking logic are held fixed.","Disagreement has interpretable direction (e.g., clinical vs. molecular in health, systems vs. materials in energy), so audits of LLM-filtered corpora can target the specific term clusters each model favors.","Because a simple TF-IDF classifier can predict which model labeled a document with AUC above 0.74, disagreement sets can be flagged automatically for adjudication or ensemble filtering without access to human ground truth.","In thematic search tasks like SDG mapping, agreement statistics alone are misleading: high raw agreement (up to 89.6%) can coincide with low kappa and with substantial divergence in what is retrievable.","Retrieval divergence under shared scoring shows that filtering, not ranking, is where model choice exerts its main influence on top-k outcomes in borderline cases."],"supporting_citations":[{"why":"Supplies the Elsevier SDG-aligned Boolean queries that define the retrieval corpus.","marker":"[4]"},{"why":"Establishes that document-level LLM relevance judgments are inconsistent and motivates treating LLM labels as provisional signals.","marker":"[7]"},{"why":"Prior work whose LLM-based SDG filtering approach this study extends.","marker":"[8]"},{"why":"Shows that disagreement in subjective annotation tasks is systematic, not random.","marker":"[10]"},{"why":"Shows divergent LLM predictions can fill judgment gaps, supporting the disagreement-as-signal framing.","marker":"[12]"},{"why":"The LLaMA model whose relevance labels are analyzed.","marker":"[14]"},{"why":"Argues for studying disagreement beyond majority vote in subjective annotations.","marker":"[15]"},{"why":"The Qwen model whose relevance labels are analyzed.","marker":"[17]"}],"fun_headline_variants":["LLM disagreement skews SDG retrieval systematically","Two LLMs split on 16% of SDG abstracts, shifting rankings","Which LLM filters? SDG search results diverge predictably","Systematic LLM disagreement alters SDG document rankings","LLM labeling gaps ripple into SDG search outcomes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The models must actually be receiving and following the same instructions: LLaMA prompts were wrapped with its chat-template utility while Qwen prompts were passed as raw message dictionaries, and every label was parsed automatically with no human check, so part of the 'model-specific' disagreement could in principle come from prompt-wrapping or parsing artifacts.","fun_headline_variants_meta":{"raw":{"variants":["LLM disagreement skews SDG retrieval systematically","Two LLMs split on 16% of SDG abstracts, shifting rankings","Which LLM filters? SDG search results diverge predictably","Systematic LLM disagreement alters SDG document rankings","LLM labeling gaps ripple into SDG search outcomes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000573,"raw_usage":{"total_tokens":2744,"prompt_tokens":1017,"completion_tokens":1727,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":1644}},"tokens_in":633,"tokens_out":1727,"duration_ms":13198,"temperature":1.0,"reasoning_tokens":1644,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:37:09.559531+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the labeling pipeline invoking both models through the same chat-template mechanism and have a human review a random sample of parsed labels; if the disagreement rate drops markedly, the lexical clusters blur, and the classifier AUC falls toward 0.5, the systematic patterns were artifacts of invocation rather than model-specific relevance criteria.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Elsevier SDG-aligned Boolean queries that define the retrieval corpus."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that document-level LLM relevance judgments are inconsistent and motivates treating LLM labels as provisional signals."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The LLaMA model whose relevance labels are analyzed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Qwen model whose relevance labels are analyzed."}],"review_version":1}