GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluation

Adri\`a de Gispert; Christopher Davis; Ionut-Teodor Sorodoc; Leonardo F. R. Ribeiro; Rexhina Blloshmi

REVIEW 4 major objections 6 minor 31 references

GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read State-of-the-art RAG models ground only 60% of answers on relevant evidence and rarely deflect when evidence is insufficient.

desk verdict A solid, usable RAG benchmark with real per-passage human grounding labels, but the headline numbers rest on a relevance-label protocol that needs tightening before they can be trusted at face value. read the letter →

arxiv 2506.07671 v1 pith:SOHMXLUJ submitted 2025-06-09 cs.CL cs.AI

Ionut-Teodor Sorodoc , Leonardo F. R. Ribeiro , Rexhina Blloshmi , Christopher Davis , Adrià de Gispert This is my paper

classification cs.CLcs.AI

keywords retrieval-augmentedgenerationgroundingannotationsfactualityevaluationdeflectionattributionhumanannotationLLMbenchmarkrelevance-aware

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces GaRAGe, a benchmark for retrieval-augmented generation (RAG) that labels each retrieved passage as relevant or not, and asks whether LLMs can restrict their answers to the relevant evidence. The central claim is that current state-of-the-art LLMs, given a mixture of relevant and irrelevant passages, tend to summarise everything rather than ground responses strictly on relevant sources, scoring at most 60% on the new Relevance-Aware Factuality Score. When no relevant grounding exists, the models rarely say so, deflecting at a true positive rate of at most 31%. Because RAG systems are widely deployed on private and web documents, the ability to ignore irrelevant evidence and to refuse answers is what determines whether such systems hallucinate. The paper argues that this failure is measurable and currently large.

What carries the argument

The load-bearing mechanism is the benchmark's per-passage relevance annotation combined with the Relevance-Aware Factuality (RAF) metric. Each of over 35,000 grounding passages is labelled by professional annotators as ANSWER-THE-QUESTION, RELATED-INFORMATION, OUTDATED, UNKNOWN, or IRRELEVANT; human-written long-form answers carry citation markers to the relevant passages. RAF is the percentage of model answers that are both eligible and supported sentence-by-sentence by the relevant-labelled passages only. This makes the metric able to separate a correct summary of everything retrieved from a correct answer grounded strictly in relevant evidence, which is the distinction the paper argues matters for real RAG use.

What would settle it

Re-annotate a random sample of 500 grounding passages from GaRAGe with an independent team using the same taxonomy (ANSWER-THE-QUESTION, RELATED-INFORMATION, OUTDATED, UNKNOWN, IRRELEVANT) and measure inter-annotator agreement. If agreement is low (for example, Cohen's kappa below 0.6), the relevance labels are not reproducible enough to serve as ground truth; if high, the benchmark's core measurement is stable.

Watch

Extended reading notes

Core claim

The paper's central discovery is a measurable gap in RAG answer generation: across eleven proprietary and open-weights models, none reliably identifies the relevant subset of retrieved passages. Using human annotations that label each grounding passage as answering the question, related but not answering, outdated, unknown, or irrelevant, the benchmark defines the Relevance-Aware Factuality Score (RAF), which requires an answer to be both eligible (addresses the user request) and supported only by the relevant passages. The best models reach roughly 60% RAF, about 8 to 10 points below their unadjusted factuality scores, showing that models use irrelevant or outdated passages as summarisers. On the 427 questions whose grounding is insufficient, the best true positive deflection rate is 31.1% (GPT-4o), meaning models usually attempt an answer instead of deflecting; false positives on sufficient-grounding questions stay under 3.5%. Attribution F1 against human-cited sources tops out at 58.9%. Performance drops further on fast-changing questions and on questions grounded in sparser private knowledge bases.

Load-bearing premise

The results stand on the accuracy of the human relevance labels; if annotators disagree about which passages are relevant, the RAF and deflection scores lose their meaning as ground truth.

Editorial extensions

If this is right

Current LLM RAG systems should not be assumed to filter evidence; benchmark scores indicate that retrieved irrelevant content leaks into final answers.
Deflection is a rare behaviour: on insufficient-grounding questions, even the best model deflects less than a third of the time, so hallucination risk is high in low-quality retrieval settings.
Enriching grounding with relevant passages helps: RAF scores improve as the fraction of relevant passages increases, showing that retrieval quality directly caps answer quality.
Time-sensitive questions and private-domain questions are systematically harder, with drops of about 10 points in RAF.
Attribution fidelity to the right sources is modest (F1 at most 58.9%), so citation markers in RAG outputs should be audited rather than trusted.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable consequence the authors leave implicit: if models were trained or prompted to first classify each passage's relevance before composing an answer, RAF should rise; the benchmark's labels make this a direct, measurable intervention.
The benchmark's label distribution (31.2% answer-the-question, 26.6% irrelevant, 7.8% outdated) suggests retrieval pipelines are a major bottleneck; improving retrieval precision may matter as much as generation changes.
Because the paper uses a single LLM judge (GPT-4o), the gap between reported scores and human judgments is an open question; a human re-scoring of the same model outputs would test whether the reported deficits are real or partly an artifact of the judge.
The deflection subset (427 questions) offers a ready test bed for abstention methods; one could measure whether adding an explicit no-sufficient-information instruction closes the gap toward the 100% true positive ceiling.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

GaRAGe is worth taking seriously. The dataset fills a real niche: per-passage human grounding labels over mixed private and web sources, human-written long-form answers with citations, and a deflection subset. The RAF metric is a natural extension of FACTS grounding, and Table 1's feature comparison is accurate enough. The authors also release data and prompts, which makes the benchmark usable rather than aspirational.

The central empirical story is believable. Across all evaluated models, restricting factuality to relevant passages drops scores relative to unadjusted factuality, and deflection true-positive rates top out at 31%. That pattern is consistent and not an artifact of any single modeling choice; it supports the paper's claim that current RAG systems act more like summarizers than relevance filters.

The soft spots are real but not fatal. The relevance taxonomy is not pinned down tightly enough for the headline metrics. Section 2.3 says annotators first decide whether a passage is relevant, then assign ANSWER-THE-QUESTION, RELATED-INFORMATION, OUTDATED, or UNKNOWN. Appendix A.2 treats OUTDATED and UNKNOWN as labels alongside RELEVANT, and the answer-writing guidelines permit using them if they are annotated relevant. The paper never states exactly which classes uRAF/RAF treat as "relevant" when the judge context is built. If OUTDATED/UNKNOWN are excluded, answers that legitimately incorporated them get penalized; if included, the metric is not measuring strictly relevant grounding. This ambiguity could shift the 60% and 31% numbers, though it probably would not erase the qualitative gap. The absence of inter-annotator agreement is a second soft spot: the N=300 validation checked that answers are grounded, not that the relevance labels themselves are reproducible, and the five-class taxonomy is exactly where annotator disagreement would concentrate. Finally, all metrics come from one GPT-4o judge, single run. The authors acknowledge this in the Limitations, but for a benchmark whose value is measuring small differences between models, variance matters. That said, these are revision items, not reasons to reject.

Who gets value from this: anyone building or evaluating RAG systems, especially work on grounding, deflection, and attribution. I would cite it and bring it to a reading group. It deserves a serious referee. I would send it out with a request for inter-annotator agreement statistics, an explicit relevance-class mapping for RAF/uRAF, and either multiple judges or at least variance estimates on the headline numbers.

Referee Report

4 major / 6 minor

Summary. The paper introduces GaRAGe, a retrieval-augmented generation (RAG) benchmark containing 2,366 questions with human-curated long-form answers, citation markers, and relevance annotations for over 35K retrieved passages drawn from both Web sources and private knowledge bases. The authors define the Relevance-Aware Factuality (RAF) metric, which scores responses only against human-annotated relevant passages, and evaluate 11 LLMs, reporting that the best model reaches 60% RAF, that the deflection true positive rate is at most 31%, and that attribution F1 is at most 58.9%. They also analyze performance across question dynamism, source type, and grounding quality. The dataset and evaluation prompts are released.

Significance. If the relevance annotations are reliable, GaRAGe is a distinctive and valuable benchmark resource: it combines long-form answers, mixed private/Web grounding, per-passage relevance labels, and citation-level annotations, enabling fine-grained measurement of grounding selection and deflection behavior. The construction pipeline is careful and multi-stage, and the authors report a human validation of 300 answers in which 97% of 2,340 claims are grounded in at least one passage. The release of the dataset and prompts is a concrete strength that supports community adoption. However, the headline quantitative claims rest on annotation reproducibility and a single-judge evaluation protocol, both of which are currently under-supported; the benchmark's potential is clear, but the empirical conclusions need stronger evidence before they can be taken as authoritative.

major comments (4)

[2.3 / 3 / Appendix A.2] The relevance taxonomy is applied inconsistently in the definition of the metrics, which directly affects the headline RAF numbers. In §2.3, OUTDATED and UNKNOWN are assigned only after a passage is identified as relevant, so under the annotation scheme they are subclasses of RELEVANT; Appendix A.2 lists them as separate labels, and the answer guidelines explicitly allow including OUTDATED or UNKNOWN information 'as long as they are annotated as relevant.' Yet §4.1 interprets the drop from Factuality to RAF as models using 'irrelevant or outdated grounding,' implying that OUTDATED passages are excluded from the 'relevant' context used for uRAF and RAF. The paper never states which labels are included in the judge's context, so the reader cannot tell whether the 60% RAF figure is computed against ANSWER-THE-QUESTION + RELATED-INFORMATION only, or also against OUTDATED and UNKNOWN. Please specify the exact context construction for each metric and report the sensitivity of the results to including/excluding OUTDATED and UNKNOWN.
[2.3] No inter-annotator agreement is reported for the grounding relevance labels or for the 427 deflection labels. The N=300 validation described in §2.3 checks whether the human-written answers are grounded in at least one passage (97% of claims), but it does not measure whether the relevance labels themselves are reproducible. Since the RAF and deflection metrics are computed from these labels, a low agreement would directly undermine the reported model rankings and the 'at most 60%' / 'at most 31%' claims. Please add a double-annotation study on a representative sample (e.g., 200–300 passages/questions), reporting agreement (e.g., Cohen's kappa or Krippendorff's alpha) for the five-class relevance taxonomy and for the deflection decision.
[3 / Limitations] The evaluation uses GPT-4o as the sole judge with one run per model. The Limitations section acknowledges the single-judge and single-run protocol, but the abstract and §4 present the 60% RAF and 31% deflection numbers as definitive measurements of LLM ability. A single judge can carry systematic bias (e.g., favoring one model family), and a single run ignores sampling variance, so the exact ordering and absolute values in Tables 2–4 are not established to the precision that the paper's claims imply. Please add a human evaluation of judge decisions on a subsample, or run at least one additional independent judge and report agreement; at a minimum, report variance across repeated runs for one or two models.
[2.2 / 3 / Appendix B.3] The construction of the {context_document} input to the factuality judge is not specified. It is not stated whether the context includes all annotated passages (including IRRELEVANT), only the top-K retrieved passages, or only the human-annotated relevant subset, and the value of K is never given. Because the factuality and uRAF/RAF scores are computed over this context, different choices can materially change the reported numbers. Please specify the exact context-inclusion rule for each metric and release the resolved prompt template used in the released evaluation code.

minor comments (6)

[Conclusion] The phrase 'Reference-Aware Factuality (RAF) Score' is a typo; the correct term elsewhere is 'Relevance-Aware Factuality Score.'
[Table 1] The meaning of the symbols '✓', '·', and blank cells is not defined in the caption; please add a legend.
[4.1] The statement that models 'act as a summariser' is a direct consequence of the metric definition (uRAF is stricter than Factuality by construction), not an independently observed behavioral phenomenon; please rephrase it as a measurement result rather than a behavioral characterization.
[2.2] The value of the top-K parameter is not reported; please provide it or point to the dataset documentation where it is specified.
[2.3] The criteria for labeling a question as 'requiring a deflection' are described only qualitatively; please add concrete examples and a decision rule to make this annotation step reproducible.
[Appendix B.4] The deflection judge prompt labels the two categories 'missing' and 'attempted'; 'missing' is a somewhat unusual term for a proper deflection response, and the prompt could be clearer that 'missing' includes explicit 'I do not know' statements.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GaRAGe reports empirical measurements against independently human-annotated labels, not derivations from fitted inputs.

full rationale

GaRAGe does not present a fitted derivation or a first-principles result; its claims are empirical measurements. The headline quantities (RAF at most 60%, deflection true-positive rate at most 31%) are computed from model outputs plus human-annotated grounding labels, and the annotations are produced independently of the models being evaluated. The RAF metric is by definition factuality judged only against the human-relevant context, so the observation that RAF is lower than unadjusted factuality is an intended contrast between support in all retrieved passages and support in relevant ones; this is a measurement design, not a disguised identity. Likewise, the deflection subset is defined by human judgments that the grounding is insufficient, and model behavior on that subset is measured, not fitted. No parameter is fitted to a subset and then reported as a prediction, and no load-bearing claim rests on a self-citation; the only self-citation (Gabburo et al., 2024, sharing an author) concerns question-complexity dimensions and is not load-bearing. The missing inter-annotator agreement and the unresolved treatment of OUTDATED and UNKNOWN labels in the relevant context are validity and robustness concerns, not circularity. Under the rule that a circular step must reduce an equation or result to its own inputs, no such step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark is an empirical resource, so no fitted parameters are needed to make a derivation work. The hand-chosen design parameters (top-K, judge temperature, thresholds) are listed as free parameters. The axioms are the domain assumptions about annotation quality, judge reliability, and the realism of the grounding sources. No new entities such as particles or forces are introduced.

free parameters (3)

Top-K retrieved grounding passages
The number K of grounding passages selected per question is not reported, though the paper states 'the final grounding consists of the top K documents ranked by their cross-encoder scores'. This K affects all downstream scores but is not specified.
GPT-4o judge temperature = 0.2
Section 3 states 'gpt-4o-2024-11-20 with a temperature of 0.2'. The temperature is a hand-chosen value that can affect the stability of the LLM judge outputs.
Grounding quality thresholds = 33% and 66%
Section 4.2 defines low, medium, and high grounding quality using pass_r thresholds of 33% and 66%. These cut points are chosen by the authors and shape the analysis of grounding quality.

assumptions (4)

domain assumption Professional annotator relevance labels are accurate and consistent enough to serve as ground truth.
Section 2.3 relies on annotators to label passages as ANSWER-THE-QUESTION, RELATED-INFORMATION, OUTDATED, UNKNOWN, or IRRELEVANT, and to write answers from relevant passages. No inter-annotator agreement is reported, so the ground truth validity is assumed.
domain assumption GPT-4o is a reliable judge for eligibility, factuality, and deflection.
Section 3 uses GPT-4o as the sole judge for all metrics. The paper acknowledges in the Limitations that results may be biased towards GPT models, meaning the scores presuppose that GPT-4o's assessments agree with human judgement.
domain assumption The private KBs and Web mix reflect realistic RAG use cases.
Section 2.2 selects Enron emails, Arxiv abstracts, AWS DevOps guides, and SEC filings as private sources. The paper asserts this reflects real-world RAG, but this is a design assumption, not externally validated.
domain assumption The STS filter and cross-encoder reranker do not systematically bias the grounding mix.
Section 2.2 uses a SentenceTransformer STS classifier and a BGE cross-encoder to filter sub-queries and rerank passages. If these tools systematically drop or demote relevant passages, the annotated relevance distribution could be skewed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluation." pith.science (2026). https://pith.science/paper/SOHMXLUJ

@misc{pith2026250607671,
  author       = {Pith},
  title        = {Pith review of: GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SOHMXLUJ}},
  note         = {Machine review of arXiv:2506.07671}
}

read the original abstract

We present GaRAGe, a large RAG benchmark with human-curated long-form answers and annotations of each grounding passage, allowing a fine-grained evaluation of whether LLMs can identify relevant grounding when generating RAG answers. Our benchmark contains 2366 questions of diverse complexity, dynamism, and topics, and includes over 35K annotated passages retrieved from both private document sets and the Web, to reflect real-world RAG use cases. This makes it an ideal test bed to evaluate an LLM's ability to identify only the relevant information necessary to compose a response, or provide a deflective response when there is insufficient information. Evaluations of multiple state-of-the-art LLMs on GaRAGe show that the models tend to over-summarise rather than (a) ground their answers strictly on the annotated relevant passages (reaching at most a Relevance-Aware Factuality Score of 60%), or (b) deflect when no relevant grounding is available (reaching at most 31% true positive rate in deflections). The F1 in attribution to relevant sources is at most 58.9%, and we show that performance is particularly reduced when answering time-sensitive questions and when having to draw knowledge from sparser private grounding sources.

Figures

Figures reproduced from arXiv: 2506.07671 by the authors.

**Figure 1.** Example of GaRAGe Q&A datapoint with a complex question and human curated long answers. and attributable sources. For RAG applications to be successful at satisfying these needs, LLMs must simultaneously exhibit strong performance at two core competencies: query generation and answer generation. For query generation, LLMs must be able to determine when to invoke the available external sources and generate the right… view at source ↗

**Figure 2.** Pipeline for constructing GaRAGe contains mainly 3 steps: (i) complex question generation through a [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Question statistics. We report the percentage of question types for time-sensitivity, complexity, popularity and category dimensions. In total our dataset contains 2366 examples. Private 13.4% Web 86.6% Grounding Source Answer 31.2% Related 26.4% 7.8% Outdated Unknown 8.0% Irrelevant 26.6% Grounding Quality [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Grounding statistics. We report the percentage of grounding sources and grounding quality. question, as long as they are annotated as relevant. Annotators extract claims from questions and add citation markers to ground each answer claim in the supporting evidence.10 …

**Figure 5.** Figure 5: Performance of different models measured by RAF score on 1) different temporal question dynamism on [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Performance of different models measured by [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 29 canonical work pages

[1]

**Decompose the response into individual sentences.**

work page
[2]

Provide a supporting excerpt from the context

**For each sentence, assign one of the following labels:** * **‘supported‘**: The sentence is entailed by the given context. Provide a supporting excerpt from the context. The supporting except must *fully* entail the sentence. If you need to cite multiple supporting excepts, simply concatenate them. * **‘unsupported‘**: The sentence is not entailed by th...

work page
[3]

**For each label, provide a short rationale explaining your decision.** The rationale should be separate from the excerpt

work page
[4]

grounding_quality

**Be very strict with your ‘supported‘ and ‘contradictory‘ decisions.** Unless you can find straightforward, indisputable evidence excerpts *in the context* that a sentence is ‘supported‘ or ‘contradictory‘, consider it ‘unsupported‘. You should not employ world knowledge unless it is truly trivial. **Input Format:** The input will consist of two parts, c...

work page
[5]

Use its actual name

Not use pronouns to refer to something in the question. Use its actual name. The number of tasks can range from 1 to 3, depending on the complexity of the question. Simpler questions may only require one or two tasks, while more complex ones might need up to three. Present your list of research tasks in the following format: <research_tasks>

work page
[6]

How do fractional derivatives enhance the NLS model in ocean engineering?

The questions should NOT have more than 20 words, they must be succinct. Don’t make the generated questions too long! Example of generated elaborate questions: what age did ferdinand magelan discovered the philippines? how many grammy awards were won by the song plan b until 62nd grammy? how many feet is the place with the lowest elevation in vermont? whe...

work page 1984
[7]

The main topic or subject of the question

work page
[8]

Any specific details, dates, or names mentioned

work page

Show all 31 references

[9]

The type of information needed (e.g., factual, comparative, historical)
[10]

Each task should:

Any implicit sub-questions within the main question Based on your analysis, create a list of research tasks. Each task should:
[11]

Be clear, specific, and concise
[12]

Focus on one aspect of the research
[13]

Be actionable and help in gathering relevant information
[14]

Be ordered logically, if there’s a natural sequence to the research
[16]

[First research task]
[17]

[Second research task]
[18]

[Third research task] ... </research_tasks> If a subsequent task specifically refers to an entity or other information that a preceding task can provide, mark that entity or item with a note in parentheses that indicates the task number that provides it. Format: (from task N) ...
[19]

Identify the name of
[20]

of that person (from task 1)

Find the ... of that person (from task 1). It is important to use the correct format. If entity X was identified in task 1 and used in task 2, say X (from task 1). Creating Search Queries You are an AI research assistant tasked with understanding and researching a given task. ...
[21]

Carefully read and analyze the given task and the results of related tasks
[22]

Consider what information you need to gather to fully address the task

Think about the key aspects of the task that require further research. Consider what information you need to gather to fully address the task
[23]

Each query should focus on a different aspect of the task or seek different types of information

Based on your analysis, generate three Web search queries that will help you research the task effectively. Each query should focus on a different aspect of the task or seek different types of information
[24]

You may use the current year in your Web search queries where appropriate

Today’s date is date . You may use the current year in your Web search queries where appropriate
[25]

First, provide a brief explanation of your understanding of the task and how you’ve incorporated information from related tasks

Format your output as follows: a. First, provide a brief explanation of your understanding of the task and how you’ve incorporated information from related tasks. Write this explanation inside <task_analysis> tags. b. Then, list your three Web search queries, each wrapped in <...
[26]

Look for common themes or connections between the sub-answers
[27]

how many days did Thurgood Marshall serve as a Supreme Court justice?

Based on the high level topics, generate complex questions which needs reasoning to obtain answers (e.g., "how many days did Thurgood Marshall serve as a Supreme Court justice?", "how many Oscar awards did Meryl Streep win?")
[28]

Avoid to create simple questions that can be answered by just composing and summarizing the answers
[29]

who acted in Ang Lee’s latest movie?

The questions must require chaining multiple pieces of information to compose the answer (e.g., "who acted in Ang Lee’s latest movie?")
[30]

Generate a list of elaborate questions using an unordered list
[2024]

InFindings of the Association for Computational Linguistics: ACL 2024, pages 14636–14650, Bangkok, Thailand

Measuring retrieval complexity in question answering systems. InFindings of the Association for Computational Linguistics: ACL 2024, pages 14636–14650, Bangkok, Thailand. Association for Computational Linguistics. Gauthier Guinet, Behrooz Omidvar-Tehrani, Anoop Deoras, and Lau...

2024 arXiv
[9474]

The model cannot find sufficient information to answer the question

Curran Associates, Inc. 10 Yi Liu, Lianzhe Huang, Shicheng Li, Sishuo Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023. Recall: A benchmark for llms robustness against external counterfactual knowledge.arXiv preprint arXiv:2311.08147. Yiqi Liu, Nafise Moosavi, and Chen...

2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

**Decompose the response into individual sentences.**

work page

[2] [2]

Provide a supporting excerpt from the context

**For each sentence, assign one of the following labels:** * **‘supported‘**: The sentence is entailed by the given context. Provide a supporting excerpt from the context. The supporting except must *fully* entail the sentence. If you need to cite multiple supporting excepts, simply concatenate them. * **‘unsupported‘**: The sentence is not entailed by th...

work page

[3] [3]

**For each label, provide a short rationale explaining your decision.** The rationale should be separate from the excerpt

work page

[4] [4]

grounding_quality

**Be very strict with your ‘supported‘ and ‘contradictory‘ decisions.** Unless you can find straightforward, indisputable evidence excerpts *in the context* that a sentence is ‘supported‘ or ‘contradictory‘, consider it ‘unsupported‘. You should not employ world knowledge unless it is truly trivial. **Input Format:** The input will consist of two parts, c...

work page

[5] [5]

Use its actual name

Not use pronouns to refer to something in the question. Use its actual name. The number of tasks can range from 1 to 3, depending on the complexity of the question. Simpler questions may only require one or two tasks, while more complex ones might need up to three. Present your list of research tasks in the following format: <research_tasks>

work page

[6] [6]

How do fractional derivatives enhance the NLS model in ocean engineering?

The questions should NOT have more than 20 words, they must be succinct. Don’t make the generated questions too long! Example of generated elaborate questions: what age did ferdinand magelan discovered the philippines? how many grammy awards were won by the song plan b until 62nd grammy? how many feet is the place with the lowest elevation in vermont? whe...

work page 1984

[7] [7]

The main topic or subject of the question

work page

[8] [8]

Any specific details, dates, or names mentioned

work page