{"id":"196d39b9-ffca-4589-a014-3e2f0dc56d6e","arxiv_id":"2608.10385","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Task-oriented personas shift LLM relevance judgment strictness and local system ranks, but global system rankings stay stable for high-capacity models while small models amplify instability.","lead":"This paper tests whether giving AI relevance judges different 'personas', such as domain expert, strict fact-checker, or deliberately contrastive reader, changes their ratings and the resulting ranking of search systems. It finds the effects are mostly small and localized for high-capacity models, but smaller models become unstable, so persona prompting can be used as a stress test for AI-based evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reusing a single 80-token GPT-4o summary per document across all personas may strip the evidence, intent, and credibility cues that the assessor roles are designed to respond to, so the reported structured sensitivity could be an artifact of summarization rather than persona conditioning.","rationale":"The reader's CONDITIONAL verdict was driven primarily by the unvalidated summary assumption, and my read converges on the same point as the single most load-bearing threat. The paper is internally consistent and the empirical design is mostly careful (fixed UMBRELA prompt, temperature 0, bootstrap CIs for model-level sensitivity, public code), but the summary-reuse design is upstream of every result: judgment-level kappa, human win rates, Kendall's tau/RBO, and system-type sensitivity all derive from judgments on the same 80-token GPT-4o summaries. If summarization eliminates the cues that personas respond to, the central 'structured sensitivity' finding is an artifact of the document representation, not a property of persona-conditioned LLM judging. This is testable with a paired full-document comparison on a sample; until then, CONDITIONAL remains the right verdict. I do not see a more fundamental internal inconsistency or a reason to reject; the concern is about external validity of the probe.","tokens_in":20808,"tokens_out":4511,"duration_ms":52222,"concrete_test":"Run a paired summary-versus-full-document study on a stratified sample: all 54 DL20 queries plus 54 randomly sampled RAG24 queries, with GPT-4o and LLaMA-3.1-8B as judges and at least UMBRELA, Evidence, GAP, and one Orthogonal persona. Judge each query-document pair from both the existing 80-token summary and the full document, then compute the Section 4 metrics (weighted kappa vs UMBRELA, human win rates, Kendall's tau/RBO, mean absolute rank displacement by system type) and bootstrap 95% confidence intervals on the summary-minus-full difference. If the difference is outside the CI for any core pattern—for example, Evidence/GAP displacement is materially larger with full documents, or the set of most-sensitive system types changes—the reported structured-sensitivity conclusion is not robust to the summarization design; if all differences are within CI, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition for the central claim is that the judging setup actually exposes persona-induced variation in real relevance assessment. Section 3.4 fixes each document to an approximately 80-token GPT-4o summary and reuses that summary across all assessor personas, persona sources, and judge models. The cited support [37] validates summary-based judging for a standard UMBRELA-style configuration, but it does not establish that summaries preserve the cues that distinguish the five roles used here. The Evidence-Verification role is defined as penalizing speculative or unsubstantiated claims and favoring source credibility; the Orthogonal role depends on alternative interpretations of query intent; the Domain-Expert role relies on specialist vocabulary and domain-specific criteria. An 80-token abstractive summary produced by GPT-4o plausibly removes or normalizes precisely these cues (hedging, source attributions, domain terminology, alternative framings), homogenizing the input across roles and models. If so, the measured 'structured, model-dependent sensitivity'—small displacements for strong models, role- and capacity-dependent kappa—understates or re-shapes what full-document persona-conditioned judging would reveal. The concern is external validity, not internal consistency: the paper's own methods are coherent, but the probe's fidelity to full-document evaluation is unvalidated in exactly the setting the paper claims to diagnose.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using persona conditioning as a diagnostic probe for LLM-based IR evaluation. It instantiates five assessor roles (Query-Aligned, Domain-Expert, Orthogonal, Evidence-Verification, and GAP) drawn from two persona sources (PersonaHub and Nemotron-Personas-USA), compares them against a standard UMBRELA baseline, and measures judgment-level agreement, system-ranking stability, and local rank displacement across six LLM backbones on TREC DL20 and RAG24. The main empirical claims are that persona conditioning produces structured, model-dependent sensitivity rather than uniform instability: judgments usually remain close to the UMBRELA baseline, global system rankings stay stable for high-capacity models, and sensitivity concentrates on particular retrieval system types. The paper concludes that persona-conditioned judging can act as a controlled stress test for LLM-based evaluation pipelines.","tokens_in":21136,"tokens_out":5947,"duration_ms":61555,"significance":"If the empirical findings hold, the paper provides a practical and interpretable method for exposing assessor framing effects in LLM-based IR evaluation, which is a timely topic given the rapid adoption of LLM judges. The study has notable methodological strengths: it ships public code, uses temperature 0, fixes the UMBRELA prompt, reuses fixed summaries to isolate persona variation, and computes 95% bootstrap confidence intervals for model-level sensitivity estimates. It also makes falsifiable predictions about the role of model capacity and assessor role, and it tests two persona sources. These strengths make the paper useful as a systematic sensitivity analysis. However, the central claims depend on an unvalidated summary-based judging setup, and several key sensitivity claims lack statistical support (persona-level displacement in Table 5, system-type concentration in Table 7). These issues do not invalidate the design internally but require additional work to make the diagnostic claims convincing.","major_comments":[{"comment":"The summary-based judging design reuses a single approximately 80-token GPT-4o summary per document across all persona conditions, with the validity of this choice referenced to the authors' prior work [37]. That prior work establishes that concise summaries preserve judgment behavior and system-level stability for a standard UMBRELA-style configuration, but it does not establish that 80-token abstractive summaries preserve the cues that the Evidence-Verification, Orthogonal, and Domain-Expert personas are explicitly designed to respond to (e.g., source credibility, hedging, domain-specific terminology, and alternative framings, as defined in Section 3.2.1). Because the paper's central claim is that persona conditioning reveals structured sensitivity in LLM-based evaluation, the fidelity of the probe for persona-conditioned judging is an unvalidated input. I request a validation subset (for example, one dataset judged on full documents or substantially longer passages under at least a few persona/model combinations) or a clear delimitation of all conclusions to summary-based judging.","section":"Section 3.4"},{"comment":"The conclusion that sensitivity concentrates on particular retrieval system types (neural ranking/reranking systems on DL20, retrieval-augmented/generation-oriented pipelines on RAG24) rests on a small set of 'representative systems' with no statistical comparison to the distribution of rank displacement across all systems. The reported Mean|Δr| ranges (3.69–4.19 on DL20 and 4.31–8.75 on RAG24) have no confidence intervals and are not tested against a null distribution. Under the bootstrap procedure already used for Table 6, the authors should report whether these systems' displacements are significantly larger than the system-level average, or temper the system-type conclusion to 'illustrative examples' rather than a claim of concentration.","section":"Section 4.4.2, Table 7"},{"comment":"Persona-level mean absolute rank displacement is reported without confidence intervals, despite the same bootstrap resampling approach being available that was used for model-level sensitivity in Table 6. The claim that 'USPersona Orthogonal yields the highest mean absolute rank displacement on both datasets' may be within sampling error; for example, on DL20 USPersona Orthogonal (2.31) is close to Evidence (2.21) and GAP (2.11). Please add confidence intervals or a significance test for all persona-level values in Table 5, or state explicitly that these differences are descriptive and not tested for statistical significance.","section":"Section 4.4.1, Table 5"}],"minor_comments":[{"comment":"For Orthogonal persona retrieval, the paper says 'top three candidate personas' are retrieved for inspection, but it does not specify whether this means the three with lowest cosine similarity, how ties are broken, or whether a dissimilarity threshold is enforced. Please clarify the retrieval procedure for the Orthogonal role.","section":"Section 3.2.2"},{"comment":"The summary length is described as 'approximately 80-token'; please state whether the prompt enforces a hard token limit and report the actual distribution of summary lengths across documents.","section":"Section 3.4"},{"comment":"Several RBO values are notably higher than the corresponding UMBRELA baseline (e.g., RAG24 LLaMA-3.1-70B PersonaHub Query RBO 0.991 vs. UMBRELA 0.639). Please explain whether these values reflect genuine top-rank agreement or artifacts such as tied NDCG scores or small numbers of systems.","section":"Table 4"},{"comment":"The sentence 'only 7 out of 472 system–persona pairs show consistent directional behavior' should define the denominator and the consistency criterion (e.g., all six models moving in the same direction with a nonzero rank shift).","section":"Section 4.4.2"},{"comment":"The paper says 'we use instruction-tuned conversational variants of the open-weight models [39]' but reference [39] is the InstructGPT paper; please specify the exact model identifiers (e.g., meta-llama/Meta-Llama-3.1-8B-Instruct) in the experimental setup.","section":"Section 3.1"},{"comment":"The Domain-Expert role is described as emphasizing 'domain-specific relevance criteria,' but the taxonomy in Table 1 includes a 'General Knowledge and Reasoning' domain. Please clarify how domain-specific criteria are operationalized for such a broad domain.","section":"Section 3.2.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a substantial number of self-citations (e.g., [36], [37] are from the same group, and several persona-related works [6-8,12,13] also originate from this group). This is not necessarily problematic, but the editor may wish to verify that reference [37] actually provides the claimed support for reusing summaries across persona conditions, since the referee report's major comment turns on that point. The paper fits the scope of CIKM well and is likely of interest to the IR evaluation community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a well-designed empirical study that gives LLM-based IR evaluation a practical diagnostic, but the 80-token summary reuse is a real external validity gap that should be addressed before the central claims are taken as general findings about assessor sensitivity.\n\nWhat is actually new: it systematically varies assessor roles (Query, Domain, Orthogonal, Evidence, GAP) and persona sources (abstract PersonaHub vs skill-grounded USPersona) across six LLM backbones and two datasets, and examines effects at three levels: judgment agreement, system-ranking stability, and local rank displacement. The persona-source comparison and the local sensitivity analysis by system type are genuinely new. The experimental design is clean: fixed UMBRELA prompt, temperature 0, fixed summaries, bootstrap CIs for model-level sensitivity. The finding that persona effects are structured rather than uniform—mostly local shifts, concentrated on particular systems, modulated by model capacity—is a useful contribution. Code is public.\n\nSoft spots: the main one is the single 80-token GPT-4o summary per document reused across all personas. Prior work [37] validates summary-based judging for a standard UMBRELA setup, but it does not establish that such short summaries preserve the specific cues (hedging, source attributions, domain terminology, alternative framings) that the Evidence-Verification, Domain-Expert, and Orthogonal roles are designed to respond to. The paper frames its contribution as a probe for assessor sensitivity, so this is not a fatal internal flaw, but it is a genuine external validity gap. If summaries normalize away the cues that personas react to, the reported sensitivity is an artifact of the summarization pipeline. A small full-document validation subset would address this.\n\nSmaller issues: Table 5 reports persona-level displacement without confidence intervals; Table 7's system-type conclusion rests on a handful of representative runs without significance tests; the win-rate analysis is descriptive. The paper largely acknowledges these limitations, so they are not fatal.\n\nOverall: a serious empirical contribution that deserves referee time. The authors should be asked to either validate the summary-preservation assumption for persona-conditioned judging or narrow their claims to describe summary-based judging pipelines. I would bring it to a reading group and cite it if I were working on LLM-based evaluation.","headline":"Solid, well-designed empirical study of persona-conditioned LLM judging with a real but addressable external validity gap around summary-based inputs; worth a careful review.","tokens_in":21577,"tokens_out":2905,"would_cite":true,"duration_ms":28914,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Giving an LLM relevance judge a different persona acts as a controlled diagnostic probe: verdicts shift in localized ways while global system rankings hold for capable models, and sensitivity concentrates on particular system types.","keywords":["Information Retrieval Evaluation","Large Language Models","LLM-as-a-Judge","Relevance Judgment","Persona Conditioning","Ranking Stability","Assessor Sensitivity"],"falsifier":"Re-run the persona protocol on DL20 with full documents instead of the reused 80-token summaries: if persona-conditioned kappa spreads and mean rank displacements shrink or vanish, the reported sensitivity is an artifact of summary judging. A complementary check: hand the same five persona instructions to human assessors; if the structured sensitivity pattern disappears, it is a property of LLM judges rather than of the assessment task itself.","tokens_in":1938,"feed_emoji":"🎭","tokens_out":2648,"duration_ms":151894,"temperature":0.7,"pith_summary":"LLM-based relevance judges are becoming a low-cost stand-in for human assessors in information retrieval evaluation, but their verdicts depend on prompt framing. This paper asks whether that dependence is a fatal flaw or a measurable, structured property, and argues for the latter: switching the judge's persona produces local shifts in strictness, evidential threshold, and interpretation emphasis rather than widespread relevance reversals. The evidence comes from five task-oriented assessor roles applied across six LLM backbones and two TREC datasets, with a standard UMBRELA judge as the fixed baseline and one reused 80-token summary per document so that only the assessor perspective varies. High-capacity models preserve global system rankings under persona change, smaller models amplify instability, and sensitivity concentrates on particular retrieval-system types, namely neural ranking/reranking runs on DL20 and RAG-oriented pipelines on RAG24. If the paper is right, persona-conditioned judging is a controlled diagnostic stress test that evaluation pipelines can run to find systems whose measured quality depends on who is doing the judging.","feed_headline":"Change an LLM judge's persona and only some search rankings move","feed_subtitle":"Giving LLM judges different assessor personas is a stress test that flags search systems sensitive to framing.","key_machinery":"The load-bearing mechanism is persona conditioning itself: prefixing the fixed UMBRELA relevance-judging prompt with \"You are acting as {persona}.\" Five assessor roles perturb only the assessor perspective: Query-Aligned (query-specific intent interpretation), Domain-Expert (a nine-domain taxonomy with per-query domain assignment at inter-annotator agreement $\\kappa = 0.86$), Orthogonal (a deliberately dissimilar interpretation), Evidence-Verification (factual correctness and source credibility), and the Global Assessor Persona (a professional search-quality rater). Each document is summarized once into an approximately 80-token GPT-4o summary and reused across all conditions, so observed differences are attributed to the persona rather than to document context. Sensitivity is read out at three levels: quadratic-weighted Cohen's kappa for judgment agreement, Kendall's tau and Rank-Biased Overlap ($\\phi = 0.9$) for system-ranking agreement with human-derived rankings, and mean absolute rank displacement $\\text{Sensitivity}(s) = \\frac{1}{|P|}\\sum_{p\\in P}|\\Delta r(s,p)|$ relative to UMBRELA for localized system movement.","core_discovery":"Persona conditioning produces structured, model-dependent sensitivity rather than uniform evaluation instability. Across six backbones (GPT-4o, GPT-4o-mini, LLaMA-3.1-70B, LLaMA-3.1-8B, Qwen-2.5-72B, Qwen-2.5-7B) and two datasets (TREC DL20, RAG24), persona-conditioned labels generally remain close to the UMBRELA baseline, with differences appearing as localized shifts in assessment strictness, evidential threshold, or interpretation emphasis instead of widespread relevance inversions. At the system level, Kendall's tau between persona-derived and human-derived rankings stays high for high-capacity models, while smaller models produce substantially larger rank displacement (up to a mean of 5.19 on DL20 for LLaMA-3.1-8B). Local rank-displacement analysis shows that sensitivity is concentrated on particular systems and system types: transformer-based neural ranking and reranking runs on DL20 and retrieval-augmented and generation-oriented pipelines on RAG24. Persona source, abstract PersonaHub profiles versus skill-grounded USPersona profiles, has a secondary effect relative to assessor role and model capacity; the contrastive USPersona Orthogonal role induces the largest mean rank displacement on both datasets, positioning persona-conditioned judging as a stress test rather than an alternative labeling strategy.","pith_inferences":["The same probe plausibly transfers to other LLM-as-a-judge settings, such as summarization, dialogue, or code generation, where a parallel claim would be that persona sensitivity concentrates on particular output types and architectures.","If the findings hold, evaluation practice would likely move toward reporting persona-robustness alongside agreement, for example publishing rank-displacement intervals or a small persona battery next to each NDCG table so consumers can see which system comparisons depend on framing.","The concentration of sensitivity in neural rerankers and RAG pipelines hints at a mechanism the paper does not test: these systems may produce outputs that share surface characteristics with LLM-preferred text, so certain personas change how much an LLM judge rewards those cues, a hypothesis a paraphrase or cue-injection experiment could isolate.","Because persona source mattered little while role mattered a lot, the effective variable may be the semantic position of the role instruction in prompt space rather than the persona's descriptive content, testable by measuring sensitivity against instruction embeddings."],"forward_implications":["Evaluation pipelines can run persona-conditioned judging as a stress test: systems whose ranks move sharply under contrasting assessor roles are flagged as evaluator-sensitive, while a stable global ranking indicates the evaluation is insensitive to framing.","Smaller LLM judges should not be used for persona-conditioned evaluation: LLaMA-3.1-8B and Qwen-2.5-7B convert persona instructions into broad judgment instability rather than controlled perspective shifts.","The contrastive, skill-grounded Orthogonal persona is the most effective probe, producing the largest mean rank displacement on both DL20 (2.31) and RAG24 (2.66), while Domain perspectives consistently induce among the smaller shifts.","Evaluation conclusions about transformer-based neural ranking and reranking systems on DL20 and about RAG-oriented pipelines on RAG24 carry extra uncertainty, because these are the system types where assessor framing moves ranks most.","Persona source is a second-order factor: abstract PersonaHub and skill-grounded USPersona profiles produce broadly similar sensitivity patterns, with differences confined to specific role-model combinations."],"supporting_citations":[{"why":"Supplies the fixed UMBRELA relevance-judging prompt that every persona run modifies by prefixing an assessor instruction; it is the baseline all comparisons are made against.","marker":"[50]"},{"why":"Provides the summary-based judging design, one approximately 80-token GPT-4o summary per document reused across all conditions, which makes the multi-persona experiment feasible and is the paper's key unvalidated premise.","marker":"[37]"},{"why":"PersonaHub is one of the two persona sources, supplying abstract task-oriented profiles for the Query, Domain, and Orthogonal roles.","marker":"[24]"},{"why":"The Nemotron-Personas-USA collection supplies the skill-grounded USPersona profiles used as the second persona source to test source effects.","marker":"[34]"},{"why":"Defines the TREC Deep Learning 2020 dataset (queries, graded labels, submitted runs) used for the first experimental setting.","marker":"[16]"},{"why":"Defines the TREC RAG 2024 dataset used for the second, more interpretive experimental setting where sensitivity is larger.","marker":"[38]"},{"why":"Underpins the Global Assessor Persona (GAP) by following prior work on global assessor impersonation and searcher preferences.","marker":"[47]"},{"why":"Grounds the framing that assessor variation is a structured component of evaluation rather than annotation noise, the perspective this paper extends to LLM judges.","marker":"[51]"}],"fun_headline_variants":["LLM judge personas shift rankings, but only for some models","Persona-conditioned LLM assessors: targeted sensitivity, not chaos","Small models amplify persona-induced rank instability in LLM judges","Personas expose which search systems are fragile to LLM judge framing","Structured not uniform: LLM assessor personas cause localized shifts"],"cache_read_input_tokens":23808,"weakest_assumption_plain":"The paper assumes that judging from a single reused 80-token summary of each document, rather than the full text, preserves how much a judge's persona changes verdicts and rankings; that premise is carried over from prior summarization work and is not revalidated under persona conditioning.","fun_headline_variants_meta":{"raw":{"variants":["LLM judge personas shift rankings, but only for some models","Persona-conditioned LLM assessors: targeted sensitivity, not chaos","Small models amplify persona-induced rank instability in LLM judges","Personas expose which search systems are fragile to LLM judge framing","Structured not uniform: LLM assessor personas cause localized shifts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000333,"raw_usage":{"total_tokens":1918,"prompt_tokens":1080,"completion_tokens":838,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":696,"completion_tokens_details":{"reasoning_tokens":749}},"tokens_in":696,"tokens_out":838,"duration_ms":8802,"temperature":1.0,"reasoning_tokens":749,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:46:55.028874+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the persona protocol on DL20 with full documents instead of the reused 80-token summaries: if persona-conditioned kappa spreads and mean rank displacements shrink or vanish, the reported sensitivity is an artifact of summary judging. A complementary check: hand the same five persona instructions to human assessors; if the structured sensitivity pattern disappears, it is a property of LLM judges rather than of the assessment task itself.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the summary-based judging design, one approximately 80-token GPT-4o summary per document reused across all conditions, which makes the multi-persona experiment feasible and is the paper's key unvalidated premise."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Nemotron-Personas-USA collection supplies the skill-grounded USPersona profiles used as the second persona source to test source effects."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the TREC Deep Learning 2020 dataset (queries, graded labels, submitted runs) used for the first experimental setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the TREC RAG 2024 dataset used for the second, more interpretive experimental setting where sensitivity is larger."}],"review_version":1}