Pith. sign in

REVIEW 4 major objections 4 minor 38 references

Training Documents Reranker with Search Rubrics for Deep Research Agent

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A reranker trained on query-specific search rubrics selects document sets that better support deep research agents, outperforming the strongest baseline by 2.6 points and generalizing to RAG.

desk verdict Contributions are real, but the evidence is thinner than the abstract implies: a worthwhile system paper with a WebWalkerQA contamination risk and no variance reporting. read the letter →

arxiv 2608.03527 v1 pith:QKSWZCPA submitted 2026-08-04 cs.IR cs.AIcs.CL

Wenhan Liu , Yu Lu , Qiaolin Xia , Hui Xu , Tong Zhao , Jian Xi , Yutao Zhu , Haijin Liang
show 3 more authors
Haibo Shi Hao Wang Zhicheng Dou
This is my paper
classification cs.IRcs.AIcs.CL
keywords deepresearchagentsdocumentrerankingsearchrubricsset-levelrelevancereinforcementlearningretrieval-augmentedgenerationLLMjudgesetselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the right supervision signal for a reranker serving a deep research agent is not single-document relevance but explicit, query-specific rubrics describing what a good document set must jointly cover. To test this, the authors build RubricRanker, trained in two stages: supervised fine-tuning on silver document-set labels produced by a strong LLM guided by hierarchical rubrics, then reinforcement learning with a rubric-weighted reward over set-level and document-level scores. On four deep research benchmarks it outperforms the strongest prior reranker by 2.6 average points, and it transfers to five RAG benchmarks with about a 2-point exact-match gain. If the claim holds, rerankers for agentic search should be optimized for set-level properties—diversity, conciseness, consistency, authority—rather than relevance alone.

What carries the argument

Hierarchical search rubrics: a two-level rubric structure that starts from fixed meta rubrics (set-level Relevance, Conciseness, Consistency; document-level Source Authority, Timeliness) and expands them per query with query-specific descriptions and 1–5 importance weights, using GPT-5.1 with a synthesized reference answer as the information-need blueprint. These rubrics do two jobs: they prompt a teacher LLM to emit silver document-set labels for SFT, and they are turned into a scalar reward via rubric-weighted aggregation (set-level LLM judge scores plus averaged document-level scores) used in GRPO reinforcement learning. RubricRanker is a Qwen3-8B model fine-tuned this way; at inference i

What would settle it

Replace GPT-5.1 as teacher, reward judge, and rubric generator with a different, independently calibrated judge (e.g., human rubric annotations on a sample of document sets, or a smaller open-source judge), retrain RubricRanker, and check whether the reported gains on deep research and RAG benchmarks survive; if the gains disappear or reverse, the central claim that rubric-based training teaches set-level quality would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that document-set quality for agent queries can be captured by hierarchical search rubrics—set-level rubrics for relevance (coverage of key aspects), conciseness (low redundancy), and consistency (compatible facts), plus document-level rubrics for source authority and timeliness—and that training a reranker against such rubrics yields document selections that better support downstream LLM generation than relevance-ranking or answer-preference training alone. RubricRanker is trained without needing rubrics at inference: it takes a query and candidate list and directly outputs a ranked subset. The experimental claim is an average score of 60.1 on four deep research benchma

Load-bearing premise

The load-bearing premise is that GPT-5.1's rubric-guided labels and rubric-based rewards faithfully measure the true quality of a selected document set; if those LLM judgments are biased or noisy, the training signal passes that bias into the reranker, and the LLM-judged evaluation could hide the problem.

Editorial extensions

If this is right

  • Rerankers trained on rubric-guided set selection generalize across deep research and RAG tasks, suggesting set-level criteria transfer rather than overfitting to one benchmark family.
  • Equipping a deep research agent with RubricRanker reduces the number of search actions needed (9–17% fewer calls on evaluated sets), implying better evidence per step and lower latency.
  • Rubric-based rewards can be computed for queries without verifiable answers, so the training recipe applies to open-ended domains where relevance labels are insufficient.
  • Generation-oriented rerankers trained on closed-form RAG tasks improve little on open-ended deep research benchmarks, whereas rubric-based training improves both, pointing to a gap in current training signal design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One implication the authors leave implicit: the same rubric construction could be turned into a reusable reward model, which the limitations section names as future work; if that works, rubric-based training need not pay GPT-5.1 evaluation costs at scale.
  • A testable extension would apply the rubric-generation step to domain-specific agent queries (e.g., legal or financial research) where the authority dimension carries more weight, and measure whether RubricRanker's advantage grows in high-stakes domains.
  • Because the evaluation itself uses LLM judges for deep research answers, an independent check with human-annotated answer quality would tell whether the 2.6-point gain reflects real evidence quality or shared rubric-format bias between training and evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes RubricRanker, a document-set reranker for deep research agents and RAG. Search-oriented rubrics are constructed by prompting GPT-5.1 to expand a fixed meta-rubric framework (set-level relevance, conciseness, consistency; document-level source authority and timeliness) into query-specific rubrics with weights. A two-stage training pipeline is used: rubrics-guided SFT with GPT-5.1 silver labels, followed by GRPO RL with a rubric-based reward computed by a GPT-5.1 judge. At inference, RubricRanker selects and ranks a subset of retrieved documents without access to rubrics. Experiments report an average gain of 2.6 points over the strongest baseline on four deep research benchmarks and about 2 EM points over Rank4Gen on five RAG benchmarks, plus ablations and a search-call analysis.

Significance. If the reported gains are robust, the paper makes a useful contribution by moving reranking supervision from single-document relevance to explicit set-level criteria—relevance, conciseness, consistency, authority, and timeliness—and by showing that such rubrics can be distilled into a small reranker. The RAG exact-match results provide an external, non-LLM-judge validation that partially supports the method. The authors also include ablations isolating the SFT and RL stages, a sensitivity analysis over the number of reranked documents, and a search-call efficiency analysis, and they release code. These are strengths. However, the deep-research claim rests on small samples, no variance reporting, a likely train/test overlap in the largest-gain benchmark, and an output-budget asymmetry between RubricRanker and vanilla baselines; these issues need to be resolved before the central claim can be accepted.

major comments (4)
  1. [§4.1.2 / Table 4 / Table 1] WebWalkerQA contamination risk is load-bearing. Training sub-queries are extracted from WebWalker-Silver (§4.1.2, Table 4), and evaluation uses WebWalkerQA from the same dataset family (§5.1). No deduplication between the 200 sampled WebWalkerQA test queries and the WebWalker-Silver training trajectories is reported. WebWalkerQA contributes the largest single gain in Table 1 (58.0 vs. 52.0, +6.0), while the other three benchmarks show gains of +1.9, +0.1, and +0.9 over the strongest baselines. The paper should report overlap at the question, sub-query, and URL levels, and should present deep-research averages with WebWalkerQA excluded. Without this, the headline 2.6-point improvement cannot be interpreted as evidence of generalization.
  2. [§5.1 / Figures 12–13] The comparison is not budget-matched. Section 5.1 states that vanilla rerankers 'output a full ranking' and the top five documents are used, while RubricRanker's inference prompt (Figure 12, and Figure 13 for RAG) allows selection of 'at most 10' documents. If RubricRanker returns more documents on average than the baselines, the observed gains could partly reflect a larger evidence budget rather than better set selection, especially on EM-based RAG benchmarks. Please report the average/median number of selected documents per query for each method and add a controlled condition in which all rerankers return exactly five documents, or match budgets otherwise.
  3. [§5.1 / Tables 1–2] The central 'consistent gains' claim lacks statistical support. Deep-research test sets are only 100 or 200 queries, no seeds are reported for sampling, and no variance, confidence intervals, or significance tests are given. Removing WebWalkerQA, the average deep-research gain over the strongest baseline drops to roughly +1.0 point (HealthBench +1.9, DRB +0.1, ResearchQA +0.9), which is within plausible noise for these sample sizes. The authors should report multiple evaluation runs with different samples, or bootstrap confidence intervals/significance tests, particularly for the non-WebWalkerQA benchmarks where the improvements are small.
  4. [§4.2.1 / §4.2.2 / Eq. (1)–(3)] There is a self-referential supervision concern that should be addressed. The rubric generator, the SFT teacher, and the RL reward judge are all GPT-5.1, and the deep-research evaluation also uses LLM judges (GPT-4.1, Gemini 2.5 Flash, GPT-4.1-mini). This creates a risk that RubricRanker learns to exploit the preferences of LLM judges rather than intrinsic document-set quality. The RAG exact-match results are reassuring, but they do not validate the deep-research claim. Please add at least one of: (i) an independent human-judged or expert-judged evaluation on a subset of deep-research queries; (ii) an analysis of agreement between the GPT-5.1 teacher/judge and a different model family; or (iii) a direct test of whether the selected sets improve factuality or answer completeness under an objective metric.
minor comments (4)
  1. [Abstract / §1] Typographical issue: 'may not form asetthat' is missing a space; similar spacing errors occur elsewhere in the text.
  2. [§5.5] The text says RubricRanker 'consistently leads to fewer search calls' and 'significantly reduces' them, but no variance or significance test is reported for the search-call counts. Please soften the wording or add error bars.
  3. [§4.1.1] The meta-rubric framework is asserted as a sufficient decomposition of document-set quality, but this assumption is not tested. A brief validation (e.g., expert or LLM agreement that the generated rubrics cover the information need) would strengthen the methodology.
  4. [Appendix A] For DRB, the evaluation mixes 50 English and 50 Chinese questions with a macro average. Please clarify whether the reported DRB score is comparable across languages and whether the sampled 100 questions are stratified by language.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; self-referential LLM supervision is a limitation, not a formal circular loop.

full rationale

RubricRanker's training uses GPT-5.1 for rubric generation, silver labels, and RL rewards, which creates a self-referential supervision pipeline. However, the paper's central claims are not reduced to that loop by construction. The deep research benchmarks are evaluated by different LLM judges (GPT-4.1, Gemini 2.5 Flash, GPT-4.1-mini) using official rubric protocols, not by the same GPT-5.1 reward model, so the evaluation is not identical to the training objective. More importantly, the RAG benchmarks are scored with exact match (EM), an external, objective metric, and RubricRanker improves there as well, providing independent evidence that the method generalizes. The paper explicitly acknowledges the LLM-dependent evaluation in the Limitations section and calls for more objective metrics. The WebWalkerQA/WebWalker-Silver overlap is a potential data contamination risk rather than a circularity, since no exact equation or fitted parameter reduces the prediction to the training input. No load-bearing self-citation, uniqueness theorem, or ansatz smuggled via citation is present. Therefore, while the self-referential LLM supervision is a legitimate methodological limitation, it does not amount to circularity under the strict criteria used here.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central method rests on LLM-generated rubrics and LLM judges. There are no fitted numeric parameters in the classical sense, but per-query rubric weights are arbitrary assignments by an LLM. The meta-rubric framework is an author-proposed taxonomy with no external validation, and the reliability of the LLM reward signal is a load-bearing assumption.

free parameters (2)
  • Per-query rubric weights (sw_i, dw_j) = Set by GPT-5.1, integers 1-5
    The weights in the reward aggregation (Eq. 1) determine the relative importance of each rubric and directly shape the RL training signal. They are assigned by the LLM rubric generator, not learned from data or derived from first principles.
  • Maximum number of selected documents = 10
    The reranker is trained and evaluated to output at most 10 documents (prompts in Figures 10-13). This is a fixed design choice that affects both training and inference behavior.
assumptions (3)
  • domain assumption GPT-5.1 rubric generation produces query-specific rubrics that accurately operationalize document-set quality
    Section 4.1.4 relies on GPT-5.1 to expand meta-rubrics into query-specific rubrics. If these rubrics are inaccurate or miss key needs, all downstream supervision is compromised.
  • domain assumption The LLM judge scores used as rewards are reliable and consistent enough for RL
    Section 4.2.2 (Eq. 1-3) uses GPT-5.1 judge scores as rewards. Noise or bias in these scores directly affects the RL optimization and the quality of the trained reranker.
  • ad hoc to paper The meta-rubric framework (Relevance, Conciseness, Consistency, Source Authority, Timeliness) is a sufficient and non-overlapping decomposition of document-set quality
    Section 4.1.1 introduces this fixed framework. There is no independent evidence that these dimensions are complete or mutually exclusive, yet the entire training signal is built on them.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training Documents Reranker with Search Rubrics for Deep Research Agent." pith.science (2026). https://pith.science/paper/QKSWZCPA

@misc{pith2026260803527,
  author       = {Pith},
  title        = {Pith review of: Training Documents Reranker with Search Rubrics for Deep Research Agent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QKSWZCPA}},
  note         = {Machine review of arXiv:2608.03527}
}
abstract

Retrieval systems help deep research agents generate high-quality answers by providing relevant documents. However, existing retrievers typically select documents through relevance matching, while individually well-matched top-$k$ documents may not form a \textit{set} that satisfies the complex information needs of an agent query (\eg, diverse, concise and authoritative documents). In this paper, we propose search-oriented rubrics that \textit{explicitly} define the requirements that high-quality document sets should satisfy for each agent query. Our search rubrics are organized into a hierarchical structure and synthesized using a powerful LLM. Based on these search rubrics, we further train a document reranker \textbf{RubricRanker} to select a high-quality subset from retrieved documents. We design a two-stage training framework that consists of rubrics-guided supervised fine-tuning and rubric-based reinforcement learning. Extensive experiments demonstrate that RubricRanker outperforms the strongest baseline by 2.6 points on four deep research benchmarks and generalizes well to five RAG benchmarks.

Figures

Figures reproduced from arXiv: 2608.03527 by the authors.

Figure 1
Figure 1. Document-set selection based on single￾document relevance matching cannot satisfy the com￾plex information needs of agent queries, such as cover￾age of diverse aspects, conciseness, and authority. rectly affects the agent’s ability to gather sufficient evidence and generate a reliable final response. Existing document retrievers (Xiao et al., 2024; Wang et al., 2022) and rerankers (Zhuang et al., 2023; Nogueira et a… view at source ↗
Figure 2
Figure 2. An overview of our framework, which constructs query-specific search rubrics and uses them to train a [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance of RubricRanker when reranking [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (13 more)
Figure 5
Figure 5. Figure 5: The prompt for generating an answer to an agent sub-query. The input consists of the query and query [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: The prompt for generating query-specific rubrics for an agent sub-query (Part I). The input consists of the [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: The prompt for generating query-specific rubrics for an agent sub-query (Part II). The input consists of the [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: The prompt for generating query-specific rubrics for a user question in the RAG scenario (Part I). The [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: The prompt for generating query-specific rubrics for a user question in the RAG scenario (Part II). The [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: The prompt for constructing silver labels for an agent sub-query in the deep research scenario. The input [PITH_FULL_IMAGE:figures/full_fig_p021_10.png]
Figure 11
Figure 11. Figure 11: The prompt for constructing silver labels for a user question in the RAG scenario. The input consists of [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: The RubricRanker prompt for an agent sub-query in the deep research scenario. The input consists of the [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: The RubricRanker prompt for a user question in the RAG scenario. The input consists of the user [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: The prompt for set-level reward scoring for an agent sub-query in the deep research scenario. The input [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: The prompt for set-level reward scoring for a user question in the RAG scenario. The input consists of [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]
Figure 16
Figure 16. Figure 16: The prompt for document-level reward scoring for an agent sub-query in the deep research scenario. The [PITH_FULL_IMAGE:figures/full_fig_p027_16.png]
Figure 17
Figure 17. Figure 17: The prompt for document-level reward scoring for a user question in the RAG scenario. The input [PITH_FULL_IMAGE:figures/full_fig_p028_17.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 35 canonical work pages

  1. [2]

    motion dazzle human perception algorithms heuristics

    the query intent - The answer should contain key information from the retrieved supporting documents required to address the query, especially the different aspects explicitly mentioned in the query or strongly implied by the relevant part of the query intent. While covering the key information, keep the answer as concise as possible and stay at an approp...

    work page 2000

  2. [3]

    Note that the query intent may contain irrelevant, broader, or overly expanded thoughts

    A reference answer. Note that the query intent may contain irrelevant, broader, or overly expanded thoughts. So use the query as the primary anchor. The query intent is only for clarifying the real goal, scope, constraints, and expected answer shape of this query. Stay tightly focused on the query itself. The answer often contains key information across d...

    work page

  3. [4]

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren

    Deep research: A systematic survey.CoRR, abs/2512.02038. Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large language models as re-ranking agents. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 202...

    arXiv 2023

  4. [5]

    To what extent and how can the motion dazzle effect give significant insight into the general algorithms or heuristics that underlie general perception in humans?

    focuses on complex web and website-level question answering. Solving its questions requires agents to search, browse, and connect information distributed across web pages, rather than relying on a single retrieved passage. We evaluate answers with GPT-4.1 as the LLM judge under a unified evaluation pipeline, using a sampled test set of 200 queries. DeepRe...

    work page 2025

  5. [7]

    The query intent, which clarifies the true information need behind the query

    work page

  6. [13]

    set_level_rubrics

    Timeliness Explanation: This meta rubric evaluates whether a single document is relevant to the query, provides evidence for information needed to answer it, and satisfies the temporal requirements of the query. It should only be triggered when the query explicitly contains a time constraint or clearly requires the latest / most up-to-date information Cou...

    work page

  7. [14]

    Your job: - First analyze the query, the query intent, and the reference answer

    A reference answer. Your job: - First analyze the query, the query intent, and the reference answer. - Then, generate query-specific rubrics under each meta rubric. If a meta rubric (such as Source Authority and Timeliness) is not needed for this query, output an empty list [] for that meta rubric. - Each query-specific rubric must have an integer weight ...

    work page

  8. [15]

    Each distinct paragraph in the reference answer typically corresponds to a need aspect that the document set should support with evidence

    Relevance Explanation: Whether the document set covers the information needs required to answer the query, and in particular whether it covers the key information mentioned across the different paragraphs of the reference answer. Each distinct paragraph in the reference answer typically corresponds to a need aspect that the document set should support wit...

    work page

Show all 38 references
  1. [16]

    Count limit: Generate exactly one query-specific rubric under this meta rubric

    Conciseness Explanation: Whether the documents in the set have low redundancy with each other (i.e., different documents do not repeat the same information heavily) and the set contains few documents irrelevant to the query. Count limit: Generate exactly one query-specific rub...

  2. [17]

    Consistency Explanation: Whether the key facts, claims, and conclusions used to answer the query across the document set are mutually compatible rather than contradictory. A high-quality set should not contain serious unresolved conflicts on important points; if multiple viewp...

  3. [18]

    It should only be triggered when the query clearly belongs to a domain where source authority is critical, such as medicine, law, finance, policy

    Source Authority Explanation: This meta rubric evaluates whether a single document is relevant to the query, provides evidence for information needed to answer it, and comes from an authoritative and trustworthy website. It should only be triggered when the query clearly belon...

  4. [19]

    set_level_rubrics

    Timeliness Explanation: This meta rubric evaluates whether a single document is relevant to the query, provides evidence for information needed to answer it, and satisfies the temporal requirements of the query. It should only be triggered when the query explicitly contains a ...

  5. [20]

    The query intent, which is the agent's reasoning process before issuing the query and may clarify why this query was issued and what information it is trying to find. The query intent may also contain broader, exploratory, or irrelevant thoughts; stay anchored on the query its...

  6. [22]

    Your job:

    A list of {num_docs} retrieved web documents (with title, URL, search snippet and full text), each prefixed by a numerical identifier like [1], [2], .... Your job:

  7. [23]

    Read the query and the relevant part of the query intent to infer the actual information need

  8. [25]

    Select at most {top_n} documents from the retrieved web documents so that the document set satisfies the Set-level Rubrics and Doc-level Rubrics as much as possible, especially high- weight rubrics

  9. [26]

    Output format (STRICT): Return ONLY the selected document identifiers in order, each wrapped in square brackets and separated by a single space

    After selection, rank the selected documents from MOST to LEAST useful for answering the query. Output format (STRICT): Return ONLY the selected document identifiers in order, each wrapped in square brackets and separated by a single space. No other text, no reasoning, no expl...

  10. [27]

    These are split into: - **Set-level rubrics** (Relevance / Conciseness / Consistency): judge the whole selected set jointly

    The hierarchical rubrics used to judge document-set quality. These are split into: - **Set-level rubrics** (Relevance / Conciseness / Consistency): judge the whole selected set jointly. * **Relevance is the primary objective**. The document set you select should satisfy these ...

  11. [28]

    Your job:

    A list of {num_docs} retrieved passages, each prefixed by a numerical identifier like [1], [2], .... Your job:

  12. [29]

    Read the user question to infer the actual information need

  13. [30]

    Read the rubrics, which evaluate the quality of a document set

  14. [31]

    The selected set should cover the important information needs, avoid unnecessary redundancy, and prefer documents that satisfy doc-level rubrics (if provided)

    Select at most {top_n} documents from the retrieved web documents so that the document set satisfies the Set-level Rubrics and Doc-level Rubrics as much as possible, especially high- weight rubrics. The selected set should cover the important information needs, avoid unnecessa...

  15. [32]

    Output format (STRICT): Return ONLY the selected document identifiers in order, each wrapped in square brackets and separated by a single space

    After selection, rank the selected documents from MOST to LEAST useful for answering the user question. Output format (STRICT): Return ONLY the selected document identifiers in order, each wrapped in square brackets and separated by a single space. No other text, no reasoning,...

  16. [33]

    Relevance (set-level, primary objective): the selected set should jointly cover the key information needed to satisfy the query, in light of the query intent

  17. [34]

    Conciseness (set-level): avoid redundant documents that repeat information already covered by another selected document

  18. [35]

    Consistency (set-level): prefer sets whose documents agree on facts; avoid mixing in documents that contradict the rest without good reason

  19. [36]

    For general, everyday, entertainment, or low-stakes informational queries, ignore this rubric

    Source Authority (doc-level): only consider this when the query clearly belongs to a high- stakes or expertise-sensitive domain such as medicine, law, finance, policy, government regulation, or science. For general, everyday, entertainment, or low-stakes informational queries,...

  20. [37]

    For queries with no temporal requirement, ignore this rubric

    Timeliness (doc-level): only consider this when the query explicitly contains a time constraint (a specific year, date, or time range) or clearly requires the latest / most up- to- date information. For queries with no temporal requirement, ignore this rubric. When it does app...

  21. [38]

    Do not output any other text or explanation

    [2] [7] [11] [1]. Do not output any other text or explanation. I will provide you with {num} documents, each indicated by a numerical identifier []. Select and rank the documents based on their joint usefulness for the search query: {question}. [QUERY INTENT] {query_intent} Se...

  22. [39]

    Relevance (set-level, primary objective): the selected set should jointly cover the key information needed to answer the user question

  23. [40]

    Conciseness (set-level): avoid redundant passages that repeat information already covered by another selected passage

  24. [41]

    Consistency (set-level): prefer sets whose passages agree on facts; avoid mixing in passages that contradict the rest without good reason

  25. [42]

    For general, everyday, entertainment, or low-stakes informational questions, ignore this rubric

    Source Authority (doc-level): only consider this when the question clearly belongs to a high-stakes or expertise-sensitive domain such as medicine, law, finance, policy, government regulation, or science. For general, everyday, entertainment, or low-stakes informational questi...

  26. [43]

    For questions with no temporal requirement, ignore this rubric

    Timeliness (doc-level): only consider this when the question explicitly contains a time constraint (a specific year, date, or time range) or clearly requires the latest / most up- to- date information. For questions with no temporal requirement, ignore this rubric. When it doe...

  27. [44]

    Do not output any other text or explanation

    [2] [7] [11] [1]. Do not output any other text or explanation. I will provide you with {num} passages, each indicated by a numerical identifier []. Select and rank the passages based on their joint usefulness for the user question: {question}. [PASSAGES] {context} User Questio...

  28. [2024]

    CoRR, abs/2412.14574

    Sliding windows are not the end: Exploring full ranking with long-context large language models. CoRR, abs/2412.14574. Wenhan Liu, Yutao Zhu, Zhicheng Dou, and Yujia Zhou. 2026c. Demorank: Selecting effective demonstra- tions for large language models in ranking task.ACM Trans...

    2023 arXiv

  29. [2025]

    Yun He, Wenzhe Li, Hejia Zhang, Songlin Li, Karishma Mandyam, Sopan Khosla, Yuanhao Xiong, Nan- shu Wang, Xiaoliang Peng, Beibin Li, Shengjie Bi, Shishir G

    Rubrics as rewards: Reinforcement learn- ing beyond verifiable domains.arXiv preprint arXiv:2507.17746. Yun He, Wenzhe Li, Hejia Zhang, Songlin Li, Karishma Mandyam, Sopan Khosla, Yuanhao Xiong, Nan- shu Wang, Xiaoliang Peng, Beibin Li, Shengjie Bi, Shishir G. Patil, Qi Qi, Sh...

    2025 arXiv

  30. [2026]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang

    Sumrank: Aligning summarization mod- els for long-document listwise reranking.CoRR, abs/2603.24204. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2023. Retrieval- augmented generation for large...

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.