NeoQA: Evidence-based Question Answering with Generated News Events

Iryna Gurevych; Leonardo F. R. Ribeiro; Markus Dreyer; Max Glockner; Xiang Jiang

REVIEW 4 major objections 4 minor 1 cited by

NeoQA: Evidence-based Question Answering with Generated News Events

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that evidence-based QA benchmarks go stale because LLMs absorb news during pretraining, and that its all-fictional benchmark, NeoQA, neutralizes that advantage — only to reveal that models shortcut past missing evidence…

desk verdict NeoQA is a genuinely useful contamination-resistant RAG benchmark with a solid empirical study, though its abstract overstates leakage resistance and the insufficiency labels rest on imperfect NLI verification. read the letter →

arxiv 2505.05949 v1 pith:2PYV7EC6 submitted 2025-05-09 cs.CL

Max Glockner , Xiang Jiang , Leonardo F. R. Ribeiro , Iryna Gurevych , Markus Dreyer This is my paper · ORCID

classification cs.CL

keywords retrieval-augmentedgenerationevidence-basedquestionansweringfictionaltimelinesshortcutreasoningunanswerablequestionsparametricknowledgeinterferencebenchmarkstalenessADTScore

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NeoQA is a benchmark built entirely from fictional news events, designed so that no language model can answer its questions from pretrained memory, only from the evidence documents it is given. The paper's central claim is that this controlled setting exposes a specific failure mode: when key evidence is missing, LLMs often answer anyway, as though the evidence were complete, instead of deflecting. Experiments on seven LLMs show that models handle answerable multi-hop questions well but frequently fail on insufficient-evidence cases, especially when the question's bridge entity is missing from the evidence, and that they struggle to detect subtle false-premise and over-specific questions. The paper also demonstrates empirically that older real-world RAG benchmarks can be solved without retrieval at all, because newer models have internalized the relevant news. If the benchmark works as intended, it gives RAG evaluation a stable, contamination-resistant platform for measuring whether a model can honestly say "I do not know."

What carries the argument

The load-bearing mechanism is the fictional timeline: ten sequentially generated events, each with an outline of 20-30 atomic outline items, a knowledge base of fictional named entities checked against Wikipedia, and news articles generated from selected subsets of outline items under four reporting profiles. Questions — multi-hop with a shared bridge entity, time-span, false premise, and uncertain specificity — are grounded in exactly the outline items needed to answer them, so the dataset can automatically pair any question with evidence that is sufficient, insufficient, or distracting. Two requirements enforce this: the selected outline items must be necessary and sufficient for the answer (Requirement 1), and news articles must convey all selected outline facts while excluding unselected ones (Requirement 2). The evaluation metric, ADTScore, is the harmonic mean $$\text{ADTScore} = \frac{2\, \mathrm{acc}_a \, \mathrm{acc}_u}{\mathrm{acc}_a + \mathrm{acc}_u}$$ of accuracy on answerable instances and accuracy on unanswerable instances requiring deflection, which penalizes a model that answers well but refuses to say "I do not know."

What would settle it

Take a random sample of the insufficient-evidence instances and have annotators judge whether the news articles do in fact state the information the pipeline claims to have omitted; if a substantial share of instances contain the supposedly missing facts, the deflection labels are wrong and the shortcut-reasoning finding would be an artifact of mislabeled evidence rather than a model behavior.

Watch

Extended reading notes

Core claim

NeoQA establishes that when questions and evidence are both drawn from fictional timelines, so that parametric knowledge cannot help, LLMs frequently resort to shortcut reasoning: instead of verifying that the evidence supports the question, they answer as if nothing were missing. When the evidence omitted the answer information, models tended to pick a misleading distractor (52.9%-77.9% of errors), and when it omitted only the bridge-entity information, models answered as if the evidence were sufficient in 69.7%-90.7% of errors. The paper also reports a significant negative association between accuracy on answerable multi-hop questions and correct deflection on subtle variants (false premise, uncertain specificity, and bridge-entity omission), suggesting that the same reasoning strength that produces correct answers can drive over-confidence when evidence is incomplete. The companion RealTimeQA experiment shows that a model's accuracy on dated news quizzes rises with question age, confirming that benchmarks based on real events lose their ability to measure evidence-based reasoning as models absorb those events during pretraining.

Load-bearing premise

The entire controlled-evidence design rests on the claim that news articles contain every fact from the selected outline items and nothing from the non-selected ones; if that mapping is wrong, the labels "sufficient" and "insufficient" are wrong, and every deflection result collapses.

Editorial extensions

If this is right

Real-world RAG benchmarks are vulnerable to parametric knowledge leakage: accuracy that supposedly measures retrieval can instead measure memorization, so evaluations on dated events need either fictional worlds or frequent regeneration.
A model can score well on answerable questions while failing on unanswerable ones, so RAG evaluation should report balanced measures like ADTScore rather than raw answer accuracy.
When evidence is incomplete, the most dangerous case is not missing everything but missing the bridge entity while the answer is present: models then answer confidently on the basis of partial support, which is precisely the behavior that undermines trust in deployed systems.
Adding irrelevant documents degrades performance most sharply within the first 20 documents, indicating that retrieval noise, not context length alone, drives much of the accuracy loss in long-context RAG.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because NeoQA's entities are fictional, the benchmark can double as a contamination detector: if a future model answers NeoQA questions above chance without evidence, that would signal leakage of the benchmark itself rather than genuine world knowledge.
The benchmark's controlled evidence design could be transplanted to other evaluation settings where parametric interference distorts measurement, such as tool-use planning, long-context summarization, or counterfactual reasoning tasks.
The near-random no-evidence accuracy on time-span questions but above-chance accuracy on multi-hop questions hints that some question phrasings leak statistical cues from the generating model; a systematic probe of how much signal resides in the question rather than the evidence could sharpen the benchmark further.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

NeoQA is worth your time: it builds a contamination-resistant RAG benchmark on fully fictional timelines with recurring entities, and it is the first to combine controlled evidence sufficiency with false-premise and uncertain-specificity variants. The construction pipeline is elaborate and transparent—sequential event outlines, Wikipedia cross-checks for entity names, NLI filtering, human annotation, and a conservative instance-selection strategy. The experiments cover seven open-weight models plus GPT-4 Turbo probes, with careful attention to prediction patterns for missing answer vs. missing bridge information. The central finding—models often answer as if evidence were sufficient when the bridge entity is missing—is plausible and consistent with prior work on shortcut reasoning. That result alone makes the dataset useful.

Where I'd push back: the abstract claims no prior evidence can exist in pretraining data, but the paper's own GPT-4 Turbo probe shows 53.6% accuracy on multi-hop questions with no evidence, which the authors attribute to token-distribution leakage. That overclaim should be qualified. More importantly, the controlled 'insufficient evidence' labels depend on Requirement 2—news articles must exclude non-selected outline items—verified only by a T5 NLI model that agrees on 92.2% of unselected pairs and leaves 7.3% unlabeled. The conservative filter drops the unknown cases, which helps, but an article that paraphrases the omitted fact in a way the NLI misses would make the supposedly insufficient evidence sufficient, weakening the shortcut-reasoning interpretation for those instances. The 350-instance human annotation checks reference answers, not the sufficiency labels. This is a real soft spot, but not a fatal one; the direction of the findings is unlikely to reverse entirely.

Net: this is a strong benchmark paper, better executed than most, with an honest limitations section that acknowledges the leakage issue. It deserves a serious referee and likely publication after the abstract is aligned with the body and the evidence-label verification is discussed more carefully. I'd cite it for future RAG evaluation.

Referee Report

4 major / 4 minor

Summary. The paper introduces NeoQA, a fully synthetic question-answering benchmark built from fictional news timelines, with the stated goal of preventing LLMs from relying on parametric knowledge during RAG evaluation. The dataset links each question to atomic outline items and independently generated news articles, allowing automatic construction of instances with sufficient evidence, insufficient evidence, false-premise questions, and uncertain-specificity questions. The authors evaluate seven Phi and Qwen models, introduce the ADTScore metric as a harmonic mean of answerable and deflection accuracy, and report that models frequently answer as if evidence were sufficient when key information is missing, and that they struggle to detect subtle mismatches in false-premise and uncertain-specificity questions. The paper also reports quality checks including LLM filtering, T5 NLI verification, and human annotation of 350 instances.

Significance. If the quality-control assumptions hold, NeoQA is a valuable contribution: it addresses the well-known problem of benchmark staleness in RAG evaluation, provides controlled evidence conditions, and makes a concrete attack on shortcut reasoning. The experimental protocol is extensive and transparent: filtering steps are documented, NLI and human checks are reported, statistical significance is tested, and the dataset is released. The main significance, however, rests on the correctness of the sufficiency/insufficiency labels for evidence sets, and on the claim that parametric knowledge cannot help. Those two load-bearing points are only partially verified, and the paper itself provides evidence of token-distribution leakage (Section 5). With additional validation or appropriately qualified claims, the benchmark would be a solid contribution.

major comments (4)

[Section 4.3 and Appendix E] Requirement 2 is only partially verified, and the instance construction does not fully enforce it. The T5 NLI check (Table 8) agrees with the expected label on only 92.2% of unselected outline-item/article pairs and leaves 7.3% of them unlabeled. The conservative filtering described in Appendix E checks, for each insufficient-evidence instance, only the single intentionally omitted outline item; it does not require that all other non-selected outline items be NLI-verified as absent from the retained articles. A news article that entails the answer fact through a different outline item, or through a paraphrase that the NLI model misses, makes a supposedly insufficient evidence set sufficient. Since the central shortcut-reasoning result (Figure 7, Table 14) is defined against these sufficiency labels, this leakage would conflate correct evidence-based answering with shortcut reasoning. I recommend either stronger post-hoc filtering over all non-selected items or a sensitivity analysis that excludes instances with any NLI-unknown unselected items.
[Section 5, GPT-4 Turbo no-evidence probe] The no-evidence accuracy of 53.6% on multi-hop questions, far above the 25% random baseline, demonstrates token-distribution leakage from the synthetic generation process and directly undercuts the claim that NeoQA prevents LLMs from leveraging pretraining knowledge. The paper acknowledges this and hypothesizes that the dataset was sampled from the same token distribution, but this leakage has the same consequence for the controlled experiments as parametric knowledge: a model can produce an "as-if-sufficient" answer on an insufficient-evidence instance without doing evidence-based reasoning. The paper should either quantify and remove leakage-prone instances, or substantially weaken the claim that no prior evidence exists in training data, and should discuss how leakage affects the deflection results.
[Appendix D.3] The human annotation validates the reference answer, not the sufficiency or insufficiency labels for the constructed evidence sets. For unanswerable questions, annotators are shown only the relevant outline items rather than the full news-article evidence sets, so the reported 94% agreement (Table 9) does not confirm that the full evidence sets are actually insufficient. Given that the entire controlled-evidence experiment depends on the correctness of these labels, the paper needs an additional human or automated validation at the level of complete instances, or an explicit statement that label validity is assumed rather than verified.
[Appendix D.1] Requirement 1 is only checked for sufficiency, not necessity. The LLM self-answer filter removes questions that cannot be answered from the selected outline items, but it does not check the instruction that no other outline item can answer the question. If an unselected outline item also determines the answer, then removing the selected evidence items may still leave sufficient evidence, again invalidating the insufficient-evidence labels. This is a separate gap from the NLI check on news articles and should be tested directly.

minor comments (4)

[Figure 2] The figure caption contains typos: "Anwserable" and "Unanwserable" should be "Answerable" and "Unanswerable".
[Appendix C] The subsection numbering is inconsistent: "C.3 News Articles" is immediately followed by "C.4 Generation", which appears to be a continuation of C.3 rather than a separate subsection.
[Table 8 and Appendix D.2] The sentence "In most cases the outline items where predicted as 'unknown' rather than 'entailed'" should read "were predicted".
[Figure 7] The panel labels and color coding in Figure 7 are hard to parse in printed form; adding explicit row and column headers (e.g., "No Answer" and "No Bridge") directly above each diagram would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: NeoQA's evidence labels and shortcut-reasoning findings are constructed from independently specified outline-item links and verified by external checks, not by fitting or self-citation.

full rationale

The paper builds NeoQA by generating fictional timelines, knowledge bases, news articles, and question-answer pairs, then links each question to explicitly selected outline items. Sufficient/insufficient evidence instances are constructed by including or omitting articles that contain those specific outline items (Section 4 and Appendix E), so the gold answerability labels are not derived from the evaluated models' outputs. The main finding, that models answer as if evidence were sufficient when bridge-entity information is missing, is an empirical measurement against these independently fixed labels. The quality filters (GPT-4o self-answerability in Appendix D.1, T5 NLI in Appendix D.2, and human annotation in Appendix D.3) are verification steps, not fitted parameters renamed as predictions; the paper transparently reports NLI disagreement and handles it conservatively. Self-citations, such as Glockner et al. (2024) and Pratapa et al. (2023), occur only as peripheral related-work references and do not carry any load-bearing argument or uniqueness claim. There is no equation or construction step in which a prediction reduces to its own input by definition, and no fitted quantity is relabeled as a benchmark conclusion.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no physical or theoretical entities. The fictional people and organizations are dataset content, not postulated mechanisms. The axioms listed are the load-bearing assumptions on which the benchmark's validity rests.

free parameters (1)

none
No numerical constants are fitted to data. Design choices such as the number of timelines, events per timeline, answer options, and NLI thresholds are stated design decisions rather than fitted parameters.

assumptions (4)

domain assumption News articles faithfully include all selected outline items and exclude non-selected ones (Requirement 2, Section 4.3).
The construction of sufficient and insufficient evidence instances depends on this mapping; verified only partially by an NLI model (92.2% agreement, 7.3% unlabeled).
domain assumption Fictional named entities and timelines do not overlap with real-world knowledge (Sections 4.1 and 5).
The out-of-training guarantee relies on Wikipedia checks and mutually exclusive summaries, but Section 5 reports above-chance no-evidence accuracy, indicating partial leakage.
domain assumption The LLM used for generation can produce unambiguous questions and valid answers, and filtering with the same model removes defective questions (Sections 4.3 and D.1).
Answerability labels are generated and filtered by GPT-4o; human annotation on only 350 instances limits independent verification.
domain assumption Timelines obey real-world physical laws and commonsense so that models can reason with common sense (Sections 1 and 4.1).
This is a stated design goal, not formally verified; inconsistencies in timelines are acknowledged as out of scope in the Limitations section.

how reviews work

0 comments

Cite this review

Pith. "Pith review of NeoQA: Evidence-based Question Answering with Generated News Events." pith.science (2026). https://pith.science/paper/2PYV7EC6

@misc{pith2026250505949,
  author       = {Pith},
  title        = {Pith review of: NeoQA: Evidence-based Question Answering with Generated News Events},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2PYV7EC6}},
  note         = {Machine review of arXiv:2505.05949}
}

read the original abstract

Evaluating Retrieval-Augmented Generation (RAG) in large language models (LLMs) is challenging because benchmarks can quickly become stale. Questions initially requiring retrieval may become answerable from pretraining knowledge as newer models incorporate more recent information during pretraining, making it difficult to distinguish evidence-based reasoning from recall. We introduce NeoQA (News Events for Out-of-training Question Answering), a benchmark designed to address this issue. To construct NeoQA, we generated timelines and knowledge bases of fictional news events and entities along with news articles and Q\&A pairs to prevent LLMs from leveraging pretraining knowledge, ensuring that no prior evidence exists in their training data. We propose our dataset as a new platform for evaluating evidence-based question answering, as it requires LLMs to generate responses exclusively from retrieved evidence and only when sufficient evidence is available. NeoQA enables controlled evaluation across various evidence scenarios, including cases with missing or misleading details. Our findings indicate that LLMs struggle to distinguish subtle mismatches between questions and evidence, and suffer from short-cut reasoning when key information required to answer a question is missing from the evidence, underscoring key limitations in evidence-based reasoning.

Figures

Figures reproduced from arXiv: 2505.05949 by the authors.

**Figure 1.** Left: NEOQA features LLM-generated questions and documents about events from a fictional timeline, ensuring that LLMs can only answer by reasoning over the documents. Right: Real-world RAG datasets become ineffective for newer LLMs that have internalized knowledge of recent events, rendering the provided evidence documents redundant. over multiple evidence documents is critical to producing verifiable answers gr… view at source ↗

**Figure 2.** An extract of a timeline from NEOQA with six out of ten events (summarized for visualization) with highlighted fictional named entities. Answering a multi-hop question requires combining information from two events. The model should deflect when only partial (insufficient) information is available or when subtle permutations make the question unanswerable (e.g., false premise questions). interference from LLMs with … view at source ↗

**Figure 3.** GPT-4 Turbo accuracy on RealTimeQA ques [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗

Figures from the paper (16 more)

**Figure 4.** Figure 4: Events are generated sequentially based on a [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Model deflection ratio in multi-hop questions [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Performance over all instances (left), answer [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: Multi-hop question predictions change after removing key information containing the answer ( [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

**Figure 8.** Figure 8: The outline with all outline items from the first event with resolved named entities via [PITH_FULL_IMAGE:figures/full_fig_p029_8.png]

**Figure 9.** Figure 9: A news article from NEOQA. The LLM used the profile of ConservativeNews to select the outline sentences and write the news article. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_9.png]

**Figure 10.** Figure 10: An example of a multi-hop question using three news articles as evidence: the first two articles must [PITH_FULL_IMAGE:figures/full_fig_p031_10.png]

**Figure 11.** Figure 11: Token distribution of the news articles. [PITH_FULL_IMAGE:figures/full_fig_p032_11.png]

**Figure 12.** Figure 12: Example of an annotation HIT for a multiple-choice question. [PITH_FULL_IMAGE:figures/full_fig_p033_12.png]

**Figure 13.** Figure 13: Example of an annotation HIT for a false premise and uncertain specificity question. [PITH_FULL_IMAGE:figures/full_fig_p034_13.png]

**Figure 14.** Figure 14: Distribution of the number of provided news [PITH_FULL_IMAGE:figures/full_fig_p035_14.png]

**Figure 15.** Figure 15: Distribution of the number of provided news [PITH_FULL_IMAGE:figures/full_fig_p035_15.png]

**Figure 16.** Figure 16: Ratio correct deflections (green) and incorrect predictions (blue and red) for multi-hop questions with [PITH_FULL_IMAGE:figures/full_fig_p036_16.png]

**Figure 17.** Figure 17: Error categories on time-span questions with [PITH_FULL_IMAGE:figures/full_fig_p036_17.png]

**Figure 18.** Figure 18: Performance per LLM and question type by the number of added irrelevant documents. [PITH_FULL_IMAGE:figures/full_fig_p037_18.png]

**Figure 19.** Figure 19: Change in prediction between the multi-hop [PITH_FULL_IMAGE:figures/full_fig_p037_19.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Natural Context Drift Undermines the Natural Language Understanding of Large Language Models
cs.CL 2025-09 conditional novelty 6.0 of 10

QA accuracy of open-weight LLMs drops as Wikipedia passages semantically drift from training-time content, while human accuracy stays flat.

Reference graph

Works this paper leans on

292 extracted references · 77 canonical work pages · cited by 1 Pith paper

[1]

Outline Item Selection:Given one or two events, the LLM selects a subset of two outline items as the basis for the question
[2]

InFirst Conference on Language Modeling

Dated Data: Tracing Knowledge Cutoffs in Large Language Models. InFirst Conference on Language Modeling. Yejin Choi. 2022. The Curious Case of Commonsense Intelligence.Daedalus, 151(2):139–155. Elizabeth Clark, Yangfeng Ji, and Noah A. Smith. 2018. Neural Text Generation in Stories Using Entity Rep- resentations as Context. InProceedings of the 2018 Confe...

arXiv 2022
[3]

Each question requires two distinct pieces of in- formation (or more, if additional outline items are added) for sufficient grounding

Distractor Generation:The LLM generates plausible but incorrect distractor options for the question, along with justifications explain- ing their plausibility and incorrectness. Each question requires two distinct pieces of in- formation (or more, if additional outline items are added) for sufficient grounding. Multiple questions are generated for all n+1...

2018
[4]

early June

ConservativeNews (System Prompt H.26) Generating a news article follows these steps: 19 Step 1: Select outline itemsGiven the outlines of all past events, and the current event outline, the LLM selects four subsets of outline item ids from the current event outline, which will be used to generate four different news articles. We use a temperature of t= 0....

2022
[5]

Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran

A Survey on Event-Based News Narrative Extraction.ACM Computing Surveys, 55(14s):1–39. Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which Linguist In- vented the Lightbulb? Presupposition Verification for Question-Answering. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the ...

arXiv 2021
[6]

Generate exactly {{NUM_STORYITEMS}} distinct story items
[7]

What shapes your bias?

Background Summarization of Event Time- lines. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8111–8136, Singapore. Association for Com- putational Linguistics. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the li...

arXiv 2023
[8]

Do not merge multiple pieces of information into one storyitem

Make storyitems as atomic as possible, communicating only a single piece of relevant information per item. Do not merge multiple pieces of information into one storyitem

Show all 292 references

[9]

If necessary, additional outline items from the selected events may be included

Question Writing:Using the selected out- line items, the full event history, and named entities, the LLM drafts a question and an- swer pair that (a) can be answered using the selected outline items, (b) requires both out- line items for a complete answer, and (c) has a 18 uni...
[10]

The outline must logically follow chronological events described in the history

Maintain consistency with the provided <history> that discusses past events fictional events. The outline must logically follow chronological events described in the history
[11]

SensationalNews (System Prompt H.23)
[12]

ObjectiveNews (System Prompt H.24)
[13]

ProgressiveNews (System Prompt H.25)
[14]

If a date is provided: Use the provided date
[15]

Without irrelevant evidence:We do not add additional irrelevant documents
[16]

Noisy retrieval:Includes all evidence docu- ments up to the question date
[17]

Each set simulates sufficient and insufficient ev- idence and includes unanswerable questions

Controlled ablation:Varies the number of irrelevant documents. Each set simulates sufficient and insufficient ev- idence and includes unanswerable questions. To distinguish between sufficient and insufficient evi- dence, we assume news articles accurately report all relevant o...
[18]

For sufficient evidence:An outline item re- quired to answer the question is considered included in the news article only if it is among the selected outline items for the news article and the NLI model predicts it as entailed by the article (excluding cases in which the LLM p...
[19]

gpt-4-turbo

For insufficient evidence:For each inten- tionally omitted outline item that renders the evidence insufficient for answering the ques- tion, we consider the outline item excluded from a news article only if it is not among the selected outline items for the news article and th...
[20]

A set of sufficient news articles exists
[21]

The set of sufficient news articles includes exactly two required news articles
[22]

Unan- swerable

80 irrelevant news articles of to the same or previous events of the question exist. For each question, we generate sufficient-evidence instances with the two relevant (and sufficient) news articles and additional irrelevant news arti- cles in increments of 0, 20, 40, 60, and ...

2023
[23]

28% Question 2:What is the name of the center- piece installation created by the individual who adapted her creative process to align with new guidelines, emphasizing sustainable materials and environmental testing?
[24]

Rebirth in Motion

“Rebirth in Motion” (correct)
[25]

Resonance of Memories

“Resonance of Memories”
[26]

Industrial Bloom

“Industrial Bloom” Question 3:What specific issue, mentioned by a clinic administrator in Larnwick, could be 25 Answerable Unanswerable Model Prompt ADTScore Multi H. Time S. Multi H. Time S. False P. Uncertain S. prompt-1 0.169 0.891 0.109 0.092 0.365 0.006 0.013 prompt-2 0.1...
[27]

Challenges in recruiting independent data pri- vacy experts for community feedback ses- sions
[28]

(correct)

Resource shortages caused by delays in iden- tifying hotspots during past norovirus out- breaks. (correct)
[29]

Mixed public opinions about the app’s privacy safeguards in Misterine City
[30]

perfect harmony of earthy and floral notes,

Concerns about the app’s encryption protocols being insufficient to prevent cyberattacks. 26 F.5 Analysis over Varying Numbers of Irrelevant Documents Figure 18 shows the performance for each LLM and each question category as the number of irrele- vant documents increases from...

2024
[31]

The event must be realistic but must not reference any existing real-world or known fictional named entities

Create an entirely fictional event based on the given genre, event summary, and history of previous events. The event must be realistic but must not reference any existing real-world or known fictional named entities
[32]

When creating names, use unique combinations unlikely to match real named entities

Invent new named entities as needed, ensuring they don’t exist in the real world or in existing works of fiction. When creating names, use unique combinations unlikely to match real named entities
[33]

Each statement must discuss only one fact or sub-event, structured sequentially in a logical temporal order when applicable

Construct the outline using short, concise, factual, and objective statements. Each statement must discuss only one fact or sub-event, structured sequentially in a logical temporal order when applicable
[34]

Ensure all statements form a coherent outline
[35]

Output each statement within a <storyitem> tag
[37]

Include a mix of main events, reactions, consequences, and contextual information

Ensure logical progression, with each statement following chronologically when applicable. Include a mix of main events, reactions, consequences, and contextual information
[39]

Ensure the story sounds realistic without explicitly stating it’s fictional
[41]

Ensure that any mention of these entities is consistent with the information you have about the named entity

Incorporate some or all of the provided named entities in your outline. Ensure that any mention of these entities is consistent with the information you have about the named entity. You may introduce additional fictional entities as needed, but they must not conflict with the ...
[42]

When referencing any named entities from the provided inputs, maintain consistency in their descriptions and roles within the story
[43]

year-month-day

If no date is provided: Generate a complete date for the event, including the year. The date should be formatted as "year-month-day" (e.g., "2024-12-03" or "2025-06-13"). This date should be consistent with the timeline established in the <history>

2024
[45]

The outline can include quotes from the named entities where applicable
[46]

Do not repeat the information from the previous events from the <history>
[47]

DO NOT refer to the named entities using the ID

Refer to all named entities (the new named entities and the known named entities) by their full "name" property. DO NOT refer to the named entities using the ID
[48]

DO NOT use pronouns to refer to a named entity from the previous story item

Make sure that you refer to all named entities within each storyitem per full name at least once. DO NOT use pronouns to refer to a named entity from the previous story item
[49]

groundbreaking

Think about the content that is appropriate for the event summary given the genre, provided history: Think about which dimensions align with all of those, and sound like a realistic event. Your output should be formatted as follows: <scratchpad>[Your thoughts go here]</scratch...
[50]

Read the provided fictional event summary of the genre {{GENRE}}: <event_summary> {{EVENT_SUMMARY}} </event_summary>
[51]

These are the fictional known entities: <known_entities> <LOCATIONS> {{LOCATIONS_XML}} </LOCATIONS> <PERSONS> {{PERSONS_XML}} </PERSONS> <ORGANIZATIONS> {{ORGANIZATIONS_XML}} </ORGANIZATIONS> <PRODUCTS> {{PRODUCTS_XML}} </PRODUCTS> <ARTS> {{ARTS_XML}} </ARTS> <EVENTS> {{EVENTS...
[52]

Review the outline of previous events that have occurred in this fictional world: <history> {{HISTORY_XML}} </history> 40
[53]

Review the date and outline of the event: Date: {{DATE}} <outline> {{OUTLINE}} </outline>
[54]

The enhanced outline must discuss the same events as before and must not extend the events that happened in the outline

Your task is to enrich this outline with additional details. The enhanced outline must discuss the same events as before and must not extend the events that happened in the outline. It must only provide supplementary details about these events in the outline
[55]

Examine each sentence in the provided outline

Follow these rules for enrichment: a. Examine each sentence in the provided outline. b. For each sentence, identify information that is unspecific or can be elaborated with more detail. c. Consider the outline to be all information that is provided to a reporter about this fic...
[56]

groundbreaking

Additional guidelines: - Do not modify the existing sentences. Only add new sentences for supplementary details. - Ensure added sentences focus exclusively on new, specific information without repeating existing content. - Maintain consistency with the original outline in all ...
[57]

- Treat each new sentence you have created as a separate <storyitem>

Present your enriched outline in the following format: - Make sure that the sentences with the additional specific details are listed as separate <storyitem> and placed at the correct position within the outline. - Treat each new sentence you have created as a separate <storyi...
[58]

A list of fictional entities: <entities> <LOCATIONS> {{LOCATIONS_XML}} </LOCATIONS> <PERSONS> {{PERSONS_XML}} </PERSONS> <ORGANIZATIONS> {{ORGANIZATIONS_XML}} </ORGANIZATIONS> <PRODUCTS> {{PRODUCTS_XML}} </PRODUCTS> <ARTS> {{ARTS_XML}} </ARTS> <EVENTS> {{EVENTS_XML}} </EVENTS>...
[59]

A history of fictional events involving these entities: <history> {{HISTORY_XML}} </history>
[60]

The date of the next fictional event: {{DATE}}
[61]

An outline describing the next fictional event: <outline> {{OUTLINE}} </outline> Follow these steps to complete your task: 42
[62]

Look for any in- consistencies or contradictions

Carefully compare the events described in the outline with the events from the history. Look for any in- consistencies or contradictions
[63]

Ensure they are consistent

Compare the named entities described in the outline with the list of provided named entities. Ensure they are consistent
[64]

Note that changes to known named entities are acceptable if they are reasonably discussed within the out- line
[65]

Make minimal changes to the outline to resolve the issues

If you find any inconsistencies or contradictions: a. Make minimal changes to the outline to resolve the issues. b. Ensure your changes maintain the original structure and flow of the outline as much as possible. c. Double-check after fixing the inconsistencies to ensure they ...
[66]

On the 27th birthday of Carla Short

If the outline is consistent with the provided list of entities and previous events (i.e., not contradictory), output the outline without any changes. Present your results in the following format: <results> <scratchpad> (Your reasoning process, including any inconsistencies fo...
[67]

Consider all named entities

Identification of named entities: Carefully read through the OUTLINE and identify all named entities (lo- cations, persons, organizations, products, art, events, buildings, or miscellaneous) that are mentioned by name. Consider all named entities. Carefully follow the definiti...
[68]

For each named entity you identify, check if it already exists in the <entities>

Remove entities that are already known and only keep new named entities: a. For each named entity you identify, check if it already exists in the <entities>. If <entities> is empty, consider all 44 named entities as new. b. Only keep new named entities which cannot be found in...
[69]

Review your preliminary list of new named entities

Verification and refinement of new named entities: a. Review your preliminary list of new named entities. b. Double-check each named entity against the existing XML structures to ensure it is truly new and not already present. c. For each new named entity, determine its full n...
[70]

Categorize each verified new named entity as either a location, person, organization, product, art, event, building, or miscellaneous

Formatting and outputting the final list: a. Categorize each verified new named entity as either a location, person, organization, product, art, event, building, or miscellaneous. b. Format your output for each new named entity using the appropriate XML tag based on its catego...
[71]

organization

Double-check miscellaneous a. If you classified any new entity as miscellaneous: - Compare the entity and how it is used in context with all other named entity types - Check if any of the other entity types fits this named entity (it does not need to fit 100- If any of these e...
[72]

The new name must be fictional, but it must sound realistic

For each entity in the lists, create a new name that is different from the original but maintains a similar style or feel. The new name must be fictional, but it must sound realistic. Avoid names that are clearly fictional
[73]

This in- cludes names of people, places, organizations, products, artworks, buildings, events, or characters from books, movies, or other media

Ensure that the new names are not associated with any known real-world or fictional entities. This in- cludes names of people, places, organizations, products, artworks, buildings, events, or characters from books, movies, or other media
[74]

- For organizations: Keep a professional or institutional tone similar to the original

When creating new names: - For locations: Maintain a geographical feel appropriate to the original name’s region. - For organizations: Keep a professional or institutional tone similar to the original. - For persons: Preserve the cultural or ethnic flavor of the original name ...
[75]

Avoid using common words, phrases, or combinations that might accidentally reference existing enti- ties
[76]

For each renamed entity, provide both the new name and the old name
[77]

Provide your complete list of renamed entities in the specified XML format

Output your results in the following XML format: <results> <location> <name>[New Location Name]</name> <old_name>[Original Location Name]</old_name> </location> <organization> <name>[New Organization Name]</name> <old_name>[Original Organization Name]</old_name> </organization...
[78]

Only update the names of entities that have been changed

Make minimal changes to the outline. Only update the names of entities that have been changed
[79]

Apply changes on each sentence individually
[80]

Output each updated sentence as a separate <storyitem>
[81]

If a sentence does not contain any entities that need to be changed, output it as is
[82]

Ensure that you maintain the original structure and content of the OUTLINE, changing only the necessary entity names
[83]

Output format: Place all your outputs in a root node <results>

Always use the full name as defined by the "name" property of the entities. Output format: Place all your outputs in a root node <results>. Do not output any content outside of this root node. Each sentence should be in its own <storyitem> tag. Your output should look like thi...
[84]

The date for this fictional event outline is: <date>{{DATE}}</date>
[85]

Here is the outline you need to analyze: <outline> {{OUTLINE}} </outline>
[86]

Here is the list of known (fictional) named entities: <entities> {{LOCATIONS_XML}} {{PERSONS_XML}} {{ORGANIZATIONS_XML}} {{PRODUCTS_XML}} {{ARTS_XML}} {{EVENTS_XML}} {{BUILDINGS_XML}} {{MISCELLANEOUSS_XML}} {{ADJUSTED_LOCATIONS_XML}} {{ADJUSTED_PERSONS_XML}} {{ADJUSTED_ORGANIZ...
[87]

For each named entity, check if it is explicitly referred to by name in the outline

Your task is to go through all of the named entities in the provided list. For each named entity, check if it is explicitly referred to by name in the outline. If a named entity is explicitly mentioned by name, include it in your results
[88]

For each entity you identify, list them using the following format: <[entity_type]><id>[id of the entity]</id><name>[full name of the entity]</name></[entity_type]>
[89]

Present your final output within a single <results> root node, structured as follows: <results> <entities> (List all identified entities here as described in step 5) </entities> </results> 49
[90]

If you cannot find that the entity is mentioned explicitly by name in the outline, do not include it in your results

Double-check your work: For each entity that you have identified, make sure to find the sentence within the outline that explicitly refers to this identified entity by name. If you cannot find that the entity is mentioned explicitly by name in the outline, do not include it in...
[91]

Do not include any XML tags within your scratchpad

You can use a <scratchpad> for your thinking process. Do not include any XML tags within your scratchpad
[92]

Remember, only include entities that are explicitly mentioned by name in the outline

Make sure that you have not missed any named entity from the outline that was also provided to you in the list of named entities. Remember, only include entities that are explicitly mentioned by name in the outline. Do not infer or assume the presence of entities that are not ...
[93]

Each entity will have an associated ID and a name

Carefully review the list of entities provided in the <entities> section. Each entity will have an associated ID and a name
[94]

Search the outline for all occurrences of each entity in the list
[95]

full name

For each occurrence found, replace it with the format: {full name|ID} Where "full name" is the full name as provided via the "name" property of the entity
[96]

Maintain the original structure and formatting of the outline, only changing the entities as described
[97]

After processing all entities, review the entire outline to ensure all occurrences have been properly marked and no entities were missed
[98]

Important points to remember: - Be thorough in your search for entities, including variations or partial mentions

Output the processed outline, maintaining its original structure but with all entity occurrences replaced as instructed. Important points to remember: - Be thorough in your search for entities, including variations or partial mentions. - Always use the ID and name as provided ...
[99]

Review the existing entities (if provided): <existing_entities> {{LOCATIONS_XML}} {{PERSONS_XML}} {{ORGANIZATIONS_XML}} {{PRODUCTS_XML}} {{ARTS_XML}} {{EVENTS_XML}} {{BUILDINGS_XML}} {{MISCELLANEOUSS_XML}} </existing_entities>
[100]

Review the names of new entities to be created: <new_entity_names> {{USED_NEW-LOCATIONS_XML}} {{USED_NEW-PERSONS_XML}} {{USED_NEW-ORGANIZATIONS_XML}} {{USED_NEW-PRODUCTS_XML}} {{USED_NEW-ARTS_XML}} {{USED_NEW-EVENTS_XML}} {{USED_NEW-BUILDINGS_XML}} {{USED_NEW-MISCELLANEOUSS_XM...
[101]

- Maintain realism and coherence with the provided outline and other entities

Carefully read the provided outline: Date: {{DATE}} <outline> {{OUTLINE}} </outline> Based on the information provided, create detailed descriptions for each new entity following these guide- lines: General instructions: - Ensure all created entities are entirely fictional and...
[102]

For each new location: 51 - Use the provided name and ID - Determine an appropriate type (city, village, country, region, etc.) - Write a concise single-sentence description that: * Does not refer to any other entities * Provides only background information, not related to the...
[103]

For each new person: - Use the provided name and ID - Create fictional details for: date_of_birth, gender, profession, nationality, education - Write a concise single-sentence description that: * Focuses on background, personality, and motivations * Does not refer to any other...
[104]

For each new organization: - Use the provided name and ID - Determine an appropriate type (company, non-profit, educational institution, government agency, etc.) - Write a concise single-sentence description that: * Does not refer to any other entities * Provides only backgrou...
[105]

For each new product: - Use the provided name and ID - Determine an appropriate type (consumer good, software, service, etc.) - Write a concise single-sentence description that: * Does not refer to any other entities * Provides only background information, not related to the e...
[106]

For each new art piece: - Use the provided name and ID - Determine an appropriate type (painting, sculpture, novel, film, etc.) - Write a concise single-sentence description that: * Does not refer to any other entities * Provides only background information, not related to the...
[107]

For each new building: - Use the provided name and ID - Determine an appropriate type (residential, commercial, public, etc.) - Write a concise single-sentence description that: * Does not refer to any other entities * Provides only background information, not related to the e...
[108]

For each new event: - Use the provided name and ID - Determine an appropriate type (historical, cultural, sporting, etc.) - Write a concise single-sentence description that: * Does not refer to any other entities * Provides only background information, not related to the event...
[109]

For each new miscellaneous entity: - Use the provided name and ID - Determine an appropriate type (concept, theory, phenomenon, etc.) - Write a concise single-sentence description that: * Does not refer to any other entities * Provides only background information, not related ...
[110]

First, read the outline of the fictional event: <outline> {{OUTLINE}} </outline>
[111]

Next, review the list of fictional entities: <entities> {{USED_LOCATION_XML}} {{USED_PERSON_XML}} {{USED_ORGANIZATION_XML}} {{USED_PRODUCT_XML}} {{USED_ART_XML}} {{USED_EVENT_XML}} {{USED_BUILDING_XML}} {{USED_MISCELLANEOUS_XML}} </entities>
[112]

id", "created_at

For each entity in the list, follow these steps: a. Identify the entity’s role in the outline. Create an update sentence describing how the entity was affected by or involved in the events described in the outline. b. Review all properties of the entity EXCEPT for "id", "creat...
[113]

Output your results for each entity in the following format: <results> <[entity_type]> <entity_id>[Insert entity id here]</entity_id> <update>[Insert your update sentence here]</update> (List any properties that were changed, with their new values, using the format:) <[propert...
[114]

Do not invent new details or events

Important reminders: - Stick to the information provided in the outline and entities list. Do not invent new details or events. - Ensure all updates and changes are consistent with the outline and with each other. - Be concise in your updates, focusing only on relevant changes...
[115]

First, you will be given known entities in this fictional world. These will be provided in the following format: <known_entities> {{LOCATIONS_XML}} {{PERSONS_XML}} {{ORGANIZATIONS_XML}} {{PRODUCTS_XML}} {{ARTS_XML}} {{EVENTS_XML}} {{BUILDINGS_XML}} {{MISCELLANEOUSS_XML}} </kno...
[117]

These new summaries should be consistent with the existing story and represent plausible continuations or developments of the original narrative

Your task is to create {{NUMBER_SUMMARIES}} new summaries that describe future fictional events following the last event from the <history>. These new summaries should be consistent with the existing story and represent plausible continuations or developments of the original narrative
[118]

A summary text (a single concise sentence) b

For each summary, create: a. A summary text (a single concise sentence) b. The date on which this fictional next event happens
[119]

Ensure your continuations are consistent with these expected followup events

Before starting, check if the history of events indicates specific dates for followup events. Ensure your continuations are consistent with these expected followup events. All your summaries must either concern this event, or happen before this event
[120]

Make sure that each summary you generate focuses on at least one of the main named entities from the history of events
[121]

Ensure all summaries are fictional and not based on real events or real people

Follow these guidelines when creating your summaries: a. Ensure all summaries are fictional and not based on real events or real people. b. Make the summaries sound realistic and plausible as follow-up stories to the previous outlines. c. Think about plausible next events base...
[122]

Output format: 55 - Enclose each summary in <summary> tags. - Each summary must have two child properties: <text>[The generated summary]</text> <date>[The date for the next event]</date> - Before each summary, explain your thought process in <thought_process> tags. Make sure t...
[123]

I met {Boris Bowman|PERSON-1} yesterday

Special instructions: - The history and entities have special formatting. They sometimes include statements like {[PHRASE]|[ID]}. Read it as [PHRASE], while the [ID] specifies the ID of the linked entity. - Example: "I met {Boris Bowman|PERSON-1} yesterday." Read as: "I met Bo...
[124]

- Ensure that each summary explores a different aspect or potential next step of the fictional situation presented in the HISTORY

Final reminders: - Repeat this process for all {{NUMBER_SUMMARIES}} summaries. - Ensure that each summary explores a different aspect or potential next step of the fictional situation presented in the HISTORY . - Each summary text MUST BE ONLY A SINGLE concise sentence. - The ...
[125]

First, you will be given known entities in this fictional world: <known_entities> {{LOCATIONS_XML}} {{PERSONS_XML}} {{ORGANIZATIONS_XML}} {{PRODUCTS_XML}} {{ARTS_XML}} {{EVENTS_XML}} {{BUILDINGS_XML}} {{MISCELLANEOUSS_XML}} </known_entities>
[126]

Next, you will be provided with the history of events that have already occurred in this fictional world: <history> {{HISTORY_XML}} </history>
[127]

You will be given the following plausible summary of how the fictional event evolves: <continuation_summary> <text>{{CONTINUATION_TEXT}}</text> <date>{{CONTINUATION_DATE}}</date> </continuation_summary>
[128]

Each of these continuation summaries must make subtle changes to the continuation summary such that they are contradictory alternatives to one another

Your task is to create {{NUM_ALTERNATIVES}} contradictory alternative summaries of how the fic- tional event can progress based on the provided continuation summary. Each of these continuation summaries must make subtle changes to the continuation summary such that they are co...
[129]

When changing the provided continuation summary, maintain these key properties: a) The central topic and involved main entity b) The stance (whether this is a positive, neutral or negative story evolvement) c) The impact (whether this is a high impact, medium impact or low imp...
[130]

- Write each continuation summary as a single concise sentence in an objective tone

Guidelines for creating alternative summaries: - Ensure each new continuation summary is consistent with the existing story and represents a plausible continuation or development of the original narrative. - Write each continuation summary as a single concise sentence in an ob...
[131]

- Before the summaries, explain your thought process in <thought_process> tags

Output format: - Enclose your entire response in <results> tags. - Before the summaries, explain your thought process in <thought_process> tags. Make sure to identify all known followup events based on the provided history first, and verify that your continuations are consiste...
[132]

I met {Boris Bowman|PERSON-1} yesterday

Special instructions: - The history and entities have special formatting. They sometimes include statements like {[PHRASE]|[ID]}. Read it as [PHRASE], while the [ID] specifies the ID of the linked entity. - Example: "I met {Boris Bowman|PERSON-1} yesterday." Read as: "I met Bo...
[133]

Begin your response with the <results> tag and end it with the </results> tag

Here’s an example of the desired output format: <results> <thought_process> (Your explanation of the thought process, including identification of known followup events and verification of consistency) </thought_process> <summaries> <summary> <text>[First alternative summary]</...
[134]

six weeks ago

Sentences mentioning relative time differences (e.g., "six weeks ago", "in five days", "after two hours"). Try to determine the specific date or time they refer to based on the context
[135]

10:00 am

Sentences mentioning a specific date or time (e.g., "10:00 am", "March 22nd")
[136]

Your task is to select tuples of NUM_SENTENCES sentences from which you can derive two specific points in time and compute a meaningful time span (duration)

Sentences containing specific details unique to the current event. Your task is to select tuples of NUM_SENTENCES sentences from which you can derive two specific points in time and compute a meaningful time span (duration). Follow these guidelines: - The duration should be co...
[137]

Sentences 2 and 5: Mention specific dates, can calculate exact duration
[138]

Sentences 1 and 7: Relative time reference and specific time, need to infer exact start point
[139]

Decision: Include pairs 1 and 3 in the output

Sentences 3 and 8: Both mention specific times on the same day, can calculate duration. Decision: Include pairs 1 and 3 in the output. </scratchpad> <time-span> <ids>2, 5</ids> <explanation>Sentence 2 mentions the event starting on July 1st, while sentence 5 states it ended on...
[140]

six weeks ago

Identify the absolute date or time within each of the selected sentences if possible. If a sentence doesn’t provide additional time information, use the event date as time information. Remember that the dates of individual sub-events within each event may differ from the event...
[141]

Think about the different options of absolute points of time and which time span between them would be most challenging to compute, ensuring that all selected sentences are required to compute the duration
[142]

This question can only be solved if one derives the absolute points in time from each of the selected sentences

Draft a specific question that asks for the time duration. This question can only be solved if one derives the absolute points in time from each of the selected sentences. The question must always ask about the duration between two points in time. Prefer time durations that di...
[143]

Only use the information from the selected sentences and make the question very specific to avoid ambiguities

Formulate a precise and correct answer to the question, which must be an absolute duration. Only use the information from the selected sentences and make the question very specific to avoid ambiguities. In the scratchpad, include a derivation of the answer using the absolute d...
[144]

- Make sure that ALL of the selected sentences are needed to answer the question

Ensure the validity of the questions: - Ensure that the question can ONLY be answered when ALL of the selected sentences are present. - Make sure that ALL of the selected sentences are needed to answer the question. - Make sure that no other information from the <complete_outl...
[145]

Try to replace explicitly mentioned dates by anchoring them into some content of the <complete_outline> instead

Avoid absolute dates or times in your question if possible: - If you needed to make an assumption about when something happened, try to explain it relative to other existing events or information from the <complete_outline>. Try to replace explicitly mentioned dates by anchori...
[146]

If you must integrate additional information beyond what is specified within the selected sentences: - Identify the unique information you must add to the question within the provided <complete_outline> and select the sentence communicating this unique information - Explain wh...
[147]

Carefully examine the response and compare it to the task instructions
[148]

Check if the response meets all criteria outlined in the task instructions
[149]

If you find any problems or discrepancies, describe them in detail using the <scratchpad> tags
[150]

If corrections are needed, make them and explain your changes within the <scratchpad> tags
[151]

Provide the complete and corrected response, even if you didn’t change anything. Use the following output format for your response: <scratchpad> (Outline your reasoning here, including any problems found and explanations for corrections made) </scratchpad> <results> <qa> <ques...
[154]

If possible, incorporate specific values or details from the outline to increase believability
[157]

Ensure distractors are distinct from each other and the correct answer
[158]

Make sure that the distractor is not by accident a valid answer based on different information from the outline
[159]

Make sure all distractor options are plausible. Present your {{NUM_DISTRACTORS}} distractor options in the following format: <distractors> <distractor> <answer>[The incorrect answer]</answer> <explanation>[A brief explanation why it is incorrect]</explanation> <distractor-sent...
[160]

Fictional events to analyze: <events> {{OUTLINES}} </events>
[161]

The bridge entity name for this task: <bridge_entity> {{BRIDGE_ENTITY_NAME}} </bridge_entity>
[162]

Known named entities: <known-named-entities> {{KNOWN_PREV_NAMED_ENTITIES}} </known-named-entities>
[163]

List of sentence IDs that include the bridge entity: <possible_sentence_ids> {{POSSIBLE_SENTENCE_IDS}} </possible_sentence_ids> When selecting sentences, follow these guidelines:
[164]

If two events are provided, ensure that each tuple covers both events (i.e., one sentence ID stems from the first event, and the other sentence ID stems from the second event)
[165]

If only one event is provided, the selected sentence IDs can stem from the same event
[166]

Look for diverse combinations of sentences across the selected events
[167]

Only consider sentence IDs from the provided list of possible sentence IDs
[168]

Ensure that ALL selected sentence IDs include the bridge named entity
[169]

Make sure that the specific information needed to identify the bridge entity cannot be known from any other sentence, nor from the knowledge base entries from the known named entities
[170]

Before providing your final answer, use a <scratchpad> section to analyze the events and think through po- tential sentence combinations

Verify that the information from the known named entities is not sufficient to replace the detailed information found in the selected sentences. Before providing your final answer, use a <scratchpad> section to analyze the events and think through po- tential sentence combinat...
[171]

Identify sentences that contain specific information about the bridge entity
[172]

Look for connections between sentences that could form the basis of a multi-hop question
[173]

Evaluate whether the selected sentences provide unique information not available in other sentences or known named entities. 63
[174]

Consider how the sentences could be used to create a challenging and interesting question. For each suitable combination of sentences you find, output your selection using this format: <selection> <ids>[IDs of the selected sentences (comma separated)]</ids> <explanation>[Expla...
[175]

This information should be as specific as possible, to avoid any ambiguities or overlap with other information from other sentences

Identify information about the bridge entity in both sentences that is unique to these two sentences and cannot be found based on any other information from the provided storyline outline. This information should be as specific as possible, to avoid any ambiguities or overlap ...
[176]

Make sure that the correct answer is concise and factual

To generate a question-answer pair, ask for specific information about the bridge entity from one sen- tence while describing the bridge entity with information from the other sentence. Make sure that the correct answer is concise and factual. The answer should focus on very s...
[177]

Your question must not rely on other information that is only communicated in different sentences

Make sure that the information from the selected sentences is sufficient to answer the question with cer- tainty. Your question must not rely on other information that is only communicated in different sentences. If it is important to include additional information that is not...
[178]

Paraphrase the bridge named entity using the unique information from one of the selected sentences

Do not mention the bridge named entity explicitly in your questions. Paraphrase the bridge named entity using the unique information from one of the selected sentences
[179]

Make sure that all selected sentences must be considered to answer the question

Ensure that the question can ONLY be answered when having access to the information from BOTH sen- tences. Make sure that all selected sentences must be considered to answer the question. Avoid using the bridge entity itself as the answer
[180]

Ensure that the information that is required from each sentence is unique within this sentence: It can nei- ther be inferred nor extracted from any other sentence within the provided storyline outline. If the question can be answered based on the other information from the sto...
[181]

- You can only assume that the bridge entity is identical across all sentences

Verify that your question does not assume any relations that are not clear from the selected sentences. - You can only assume that the bridge entity is identical across all sentences. Other information may not refer to the identical entity. For example, a group of people in on...
[182]

artist" if the <selected_sentences> do not introduce this person as

Compare each specific detail in the question with the selected sentences. Make sure that each detail can with certainty be inferred from the selected sentences. If not, omit or generalize the specific details that cannot be inferred from the selected sentences. - Only focus on...
[183]

Ensure each distractor is clearly incorrect when compared to the correct answer
[184]

Use information from the fictional event outline to make distractors sound plausible
[185]

If possible, incorporate specific values or details from the outline to increase believability. 66
[186]

Align distractors with non-answer text from the outline to make them more challenging
[187]

Vary the type and structure of distractors to avoid patterns
[188]

Ensure distractors are distinct from each other and the correct answer. Present your distractor options in the following format: <distractors> <distractor> <answer>[The incorrect answer]</answer> <explanation>[A brief explanation why it is incorrect]</explanation> <distractor-...
[190]

Create a question that contradicts this key information while keeping other details intact
[191]

Ensure the false premise is mutually exclusive with the original information
[192]

in a 60-year-old building

Make the questions challenging, with false premises that are easy to miss but mutually exclusive to the evidence sentences and context. For example: - If you change a name, change the lastname only - If you refer to a person or place, rather than changing the name, refer to a ...
[194]

Consider the additional context when creating false premises:

Only include ONE false premise for in each question. Consider the additional context when creating false premises:
[195]

Avoid creating questions that can be validly answered using information from the context
[196]

67 Generate multiple false-premise questions if possible, each based on different key information from the se- lected sentences

Ensure that the false premise remains inconsistent with both the selected sentences and the context. 67 Generate multiple false-premise questions if possible, each based on different key information from the se- lected sentences. The changed information must be contradictory t...
[197]

Identify key information in one of the two selected sentences
[198]

- The added details must be POSSIBLE based on the provided sentences and the provided list of all outlines

Modify the question by making it more specific. - The added details must be POSSIBLE based on the provided sentences and the provided list of all outlines. - The added details must be UNVERIFIED based on the provided sentences and the provided list of all outlines. - The added...
[199]

Keep the question as similar as possible to the original question, asking for the same information but changing small details that contradict the two sentences
[200]

Consider the additional context when creating the new question:

Only include ONE specific unverified detail to the question. Consider the additional context when creating the new question:
[201]

Make sure that the details you add cannot be confirmed by any of the sentences
[202]

a criminal of the ring,

Make sure that the details you add cannot be refuted by any of the sentences Generate multiple such questions if possible, each based on different key information from the selected sen- tences. 68 Only output the questions for which you are certain that: - The details you add ...
[203]

It is not afraid to push boundaries and be daring in its reporting

**Boldness** – SensationalNews embraces the power of bold, attention-grabbing headlines and stories. It is not afraid to push boundaries and be daring in its reporting
[204]

**Emotional Engagement** – The newspaper believes in connecting with its readers emotionally, stirring curiosity, excitement, and often shock
[205]

**Entertainment Focus** – While reporting on serious issues, SensationalNews aims to entertain as much as inform, blending elements of storytelling with journalism
[206]

**Instant Impact** – The publication values creating immediate reactions from its readers, with a preference for stories that generate quick buzz and wide readership
[207]

Reporting Style:

**Provocation** – SensationalNews sees value in challenging the status quo and providing thought-provoking commentary that sparks conversation. Reporting Style:
[208]

**Dramatic Headlines** – Stories are often framed with sensational headlines that pull readers in with their emotional intensity and exaggerated flair
[209]

**Exaggerated Descriptions** – Reportage often includes vivid and overstated language to enhance the emotional impact of the story
[210]

**Heavy Use of Visuals** – Eye-catching images, graphics, and infographics are a staple of the paper, with emphasis on impactful visuals that complement the story’s drama
[211]

**Personalized Narratives** – Stories are often told from a personal angle, involving individual experiences that amplify the human element
[212]

**Speculative** – Rather than just focusing on the facts, stories often venture into speculation, offering bold predictions or unverified claims to keep the audience intrigued
[213]

Perspective on Common Issues:

**Use of Hyperbole** – The reporting often utilizes hyperbolic language to make events appear larger-than-life and more significant. Perspective on Common Issues:
[214]

It prefers to highlight stories about financial collapses, corporate corruption, or the 69 shocking impact of economic disparity

**Economics** – SensationalNews is generally critical of traditional economic systems, often focusing on scandals involving the rich and powerful. It prefers to highlight stories about financial collapses, corporate corruption, or the 69 shocking impact of economic disparity
[215]

It tends to adopt a skeptical, sometimes alarmist stance on foreign relations

**Foreign Policy** – SensationalNews is dramatic in its portrayal of international events, often emphasizing conflicts, scandals, or conspiracies involving world leaders and governments. It tends to adopt a skeptical, sometimes alarmist stance on foreign relations
[216]

It tends to amplify sensational aspects of social movements, such as protests, controversies, or public figures involved in scandals

**Social Topics** – The newspaper highlights polarizing social issues, often focusing on divisive cultural debates. It tends to amplify sensational aspects of social movements, such as protests, controversies, or public figures involved in scandals
[217]

**Environment** – SensationalNews might portray environmental issues in a dramatic light, focusing on disasters, environmental collapses, or highly controversial claims about climate change, often exaggerating the urgency or apocalyptic aspects of the problem
[218]

Preferred Topics:

**Technology** – The paper covers the darker side of technology, emphasizing security breaches, data privacy violations, and the dangers of technological advancement rather than celebrating progress. Preferred Topics:
[219]

**Celebrity Scandals** – SensationalNews thrives on high-profile stories involving celebrities, with a focus on personal drama, breakups, and tabloid-like revelations
[220]

**Crime and Scandals** – Reports on criminal activities, particularly those involving famous individuals or shocking details, dominate the coverage
[221]

**Political Confrontations** – The paper frequently covers political scandals, corruption, and rivalries, focusing on the drama and intrigue surrounding political figures
[222]

**Natural Disasters** – The publication has a keen interest in reporting on natural disasters, often dramatizing the scale and devastation of events to maintain reader engagement
[223]

Things They Like:

**Conspiracy Theories** – SensationalNews is known for reporting on conspiracy theories, often promoting speculative narratives that stir curiosity and fuel widespread discussions. Things They Like:
[224]

**Conflict and Controversy** – SensationalNews enjoys highlighting dramatic confrontations, whether in politics, entertainment, or social issues, preferring stories with high emotional stakes
[225]

**Shocking Revelations** – The newspaper thrives on uncovering secrets, hidden truths, or surprising twists that keep readers on the edge of their seats
[226]

**Daring Individuals** – People who challenge norms or disrupt established systems are portrayed positively, especially if they are seen as bold or rebellious
[227]

**Unpredictable Events** – The paper enjoys reporting on events that are unpredictable and out of the ordinary, especially if they provide an opportunity for dramatic storytelling
[228]

Things They Dislike:

**Misinformation and Sensational Claims** – SensationalNews tends to embrace bold, unverified claims or takes stories with sensational twists, appealing to readers who enjoy speculation. Things They Dislike:
[229]

**Bureaucracy and Red Tape** – SensationalNews dislikes bureaucratic systems and slow, cautious approaches to news reporting, preferring quick and dramatic action over formalities
[230]

**Censorship and Control** – The newspaper is critical of any form of censorship and dislikes anything that restricts freedom of speech or access to sensational content
[231]

**Mediocre News** – Stories that are deemed ’boring’ or lacking in dramatic flair are typically downplayed or not covered at all
[232]

**Overly Technical Reporting** – SensationalNews tends to avoid overly complex, fact-heavy reports that lack emotional appeal or dramatic tension, preferring stories that are easily digestible and engaging
[233]

ObjectiveNews

**Conservative, Mainstream Views** – The publication frequently criticizes mainstream perspectives, especially if they are perceived as dull, traditional, or not engaging enough for its audience. This profile should help to fully simulate the voice, character, and editorial st...
[234]

- Integrity: Upholding the truth and reporting facts as accurately as possible, without distortion or exaggeration

**Core Values:** - Objectivity: Commitment to presenting information in a neutral, impartial manner, free from personal bias or opinion. - Integrity: Upholding the truth and reporting facts as accurately as possible, without distortion or exaggeration. - Accountability: Holdin...
[235]

- Clear and Concise: Information is presented in a straightforward manner, avoiding sensationalism and overly complex language

**Reporting Style:** 70 - Fact-based Analysis: Reporting is rooted in verified data, evidence, and credible sources, with minimal use of speculation or conjecture. - Clear and Concise: Information is presented in a straightforward manner, avoiding sensationalism and overly com...
[236]

Cautions against overly partisan economic rhetoric

**Perspective on Common Issues:** - **Economics:** Advocates for policies that promote sustainable economic growth, with a focus on fairness, equity, and long-term stability. Cautions against overly partisan economic rhetoric. - **Foreign Policy:** Supports diplomacy and peace...
[237]

- **Public Health:** Reports on healthcare policies, advancements in medical research, and public health crises

**Preferred Topics:** - **Political Integrity:** Stories examining government transparency, accountability, and the ethical behavior of political figures. - **Public Health:** Reports on healthcare policies, advancements in medical research, and public health crises. - **Educa...
[238]

- **Data-Driven Reporting:** Stories that use credible statistics and research to inform the narrative

**Things They Like:** - **Diverse Opinions:** A variety of perspectives in opinion pieces, as long as they are well-supported by facts and logic. - **Data-Driven Reporting:** Stories that use credible statistics and research to inform the narrative. - **Positive Social Change:...
[239]

ObjectiveNews

**Things They Dislike:** - **Sensationalism:** Coverage that distorts or exaggerates facts to provoke emotional reactions, rather than providing balanced, fact-based information. - **Partisan Bias:** Reporting that favors one political party or ideology over another, especiall...
[240]

- Environmental Sustainability: Promoting policies and actions aimed at protecting the environment and combating 71 climate change

**Core Values:** - Equality and Social Justice: Advocating for equal rights and opportunities for all individuals, regardless of race, gender, sexual orientation, or socioeconomic status. - Environmental Sustainability: Promoting policies and actions aimed at protecting the en...
[241]

- Investigative Journalism: Prioritizing deep dives into complex issues, uncovering hidden truths, and revealing systemic injustices

**Reporting Style:** - Fact-Based Analysis: Ensuring that all reporting is supported by solid evidence and provides a thorough understanding of the issues at hand. - Investigative Journalism: Prioritizing deep dives into complex issues, uncovering hidden truths, and revealing ...
[242]

Critical of corporate greed and neoliberal economic policies

**Perspective on Common Issues:** - Economics: Advocating for wealth redistribution, progressive taxation, universal healthcare, and a living wage. Critical of corporate greed and neoliberal economic policies. - Foreign Policy: Supporting human rights, international diplomacy,...
[243]

- Climate Change and Environmental Advocacy: Articles on environmental issues, sustainability practices, and climate action policies

**Preferred Topics:** - Social Inequality and Justice Reform: In-depth coverage on issues related to racial justice, criminal justice reform, gender equality, and LGBTQ+ rights. - Climate Change and Environmental Advocacy: Articles on environmental issues, sustainability pract...
[244]

- Progressive Legislation: Celebrating successful progressive policies, especially those that promote equality, environmental protection, and economic reform

**Things They Like:** - Grassroots Movements: Supporting and covering local activism, protests, and grassroots initiatives that aim to create social change. - Progressive Legislation: Celebrating successful progressive policies, especially those that promote equality, environm...
[245]

ConservativeNews,

**Things They Dislike:** - Corporate Influence in Politics: Opposing the influence of large corporations and wealthy donors in politics, which they believe undermines democracy and equality. - Inequality and Economic Exploitation: Criticizing wealth inequality, the concentrati...
[246]

**Traditional Family Values**: Emphasizes the importance of family structures, promoting policies that align with the preservation of traditional family roles
[247]

**Limited Government**: Advocates for a smaller government with reduced taxes and fewer regulations, favoring individual freedom and local control
[248]

**Patriotism and National Pride**: Supports strong national defense and respects the heritage, history, and symbols of the nation
[249]

**Free Market Economy**: Promotes capitalist principles, emphasizing deregulation, entrepreneurship, and minimal government interference in business
[250]

**Respect for Law and Order**: Stands for a strict interpretation of the law, emphasizing the importance of personal responsibility and strong criminal justice systems
[251]

Reporting Style:

**Religious Freedom**: Upholds the belief that religious expression should be protected, often promoting Christianity as an integral part of cultural identity. Reporting Style:
[252]

**Fact-based Analysis**: Prioritizes logical, evidence-driven reporting that appeals to rationality and often relies on data, research, and expert opinions
[253]

**Concise and Direct**: Articles tend to be clear, direct, and to the point, often eschewing unnecessary detail for efficiency in communication
[254]

**Opinionated Commentary**: Features strong editorial perspectives, often weaving political opinion into news coverage, particularly on contentious issues
[255]

**Investigative Reporting**: Tends to focus on exposing government overreach, corruption, and liberal biases, with a focus on transparency and accountability
[256]

us vs. them

**Emotive Storytelling**: Occasionally uses emotional appeal to underline stories, especially related to cultural or national pride, portraying a clear "us vs. them" narrative. Perspective on Common Issues:
[257]

**Economics**: Advocates for tax cuts, deregulation, and economic policies that favor businesses, with a focus on reducing government spending and promoting job creation
[258]

**Foreign Policy**: Strongly favors national sovereignty, supports a robust military, and tends to oppose international agreements or organizations that may undermine the country’s interests
[259]

**Social Issues**: Often critical of progressive movements, especially when it comes to issues like LGBTQ+ rights, abortion, and social justice activism, preferring policies that protect traditional institutions
[260]

**Immigration**: Advocates for strict border control and immigration laws, emphasizing the need for national security and the protection of American workers
[261]

Preferred Topics:

**Environmental Policy**: Generally skeptical of climate change policies that impose significant regulations on industries, favoring market-based solutions to environmental concerns. Preferred Topics:
[262]

**Political Conservatism**: Covers topics like conservative victories, prominent conservative figures, and conservative solutions to political issues
[263]

**National Security and Military**: Focuses on defense policy, military readiness, and law enforcement, with an emphasis on strengthening national security
[264]

**Economic Policy and Market Trends**: Regularly covers free market economics, tax policy, and analysis of financial markets, businesses, and job creation
[265]

**Cultural and Religious Traditions**: Often features discussions on maintaining cultural and religious traditions, with particular attention to Christian values
[266]

Things They Like:

**Second Amendment Rights**: Covers gun rights, self-defense, and legal battles surrounding the Second Amendment. Things They Like:
[267]

**National Pride and Patriotism**: Strong support for events like Independence Day, military recognition, and other symbols of American identity
[268]

**Entrepreneurship and Small Business**: Celebrates success stories in business, advocating for policies that benefit entrepreneurs and small businesses
[269]

**Traditional Institutions**: Upholds the value of marriage, family, and community as the cornerstone of a strong society
[270]

**Pro-Growth Policies**: Enthusiastically supports tax cuts, deregulation, and policies that stimulate economic growth and job creation
[271]

Things They Dislike:

**Strong Borders and National Security**: Portrays the need for secure borders, immigration reform, and a well-funded military in a positive light. Things They Dislike:
[272]

**Liberal Social Movements**: Criticizes progressive movements on issues like social justice, gender equality, and racial justice, viewing them as threats to traditional values. 73
[273]

**Big Government and High Taxes**: Strongly opposes government overreach, excessive taxes, and regulation, particularly when it comes to economic matters
[274]

**Globalism**: Opposes international agreements or organizations that diminish national sovereignty or place restrictions on the country’s ability to act independently
[275]

**Political Correctness**: Critical of what it sees as an overemphasis on political correctness in media, culture, and public discourse, arguing it stifles free speech
[276]

**Anti-Religious Sentiments**: Disapproves of secularism or any movements that seek to diminish the role of religion in public life. H.27 Sentence Selection Prompt for News Article Generation You are an AI assistant tasked with selecting subsets of sentences from a fictional c...
[277]

Review the optional history of past events: <history> {{HISTORY}} </history>
[278]

Examine the current event: <current_event> {{CURRENT_EVENT}} </current_event>
[279]

- Identify the most important or newsworthy sentences

Analyze the current event: - Read and understand the content thoroughly. - Identify the most important or newsworthy sentences. - Consider how different combinations of sentences could create coherent and interesting news stories. - Think about which content aligns well with w...
[280]

- Ensure each subset tells a slightly different aspect or perspective of the event

Create {{NUM_SUBSETS}} distinct subsets of sentences, following these guidelines: - Select groups of sentence IDs that, when combined, would create compelling news stories. - Ensure each subset tells a slightly different aspect or perspective of the event. - Include enough con...
[281]

Output your results in the following format: <results> <subset>[Comma-separated list of sentence IDs]</subset> </results> Repeat the <subset> element {{NUM_SUBSETS}} times, each containing a unique selection of sentence IDs
[282]

Your goal is to provide versatile options for creating varied news stories from the same event

Remember to create distinct subsets that capture different aspects of the story. Your goal is to provide versatile options for creating varied news stories from the same event. Before providing your final answer, use a <scratchpad> to think through your selection process and e...
[283]

First, review the background information about the fictional entities involved in the event. This informa- tion is provided in XML format for various categories: <entities> <LOCATIONS> {{LOCATIONS_XML}} </LOCATIONS> 74 <PERSONS> {{PERSONS_XML}} </PERSONS> <ORGANIZATIONS> {{ORG...
[284]

Next, carefully read the event information provided with links to the provided entities as background in- formation: <event_info> {{EVENT_INFO}} </event_info>
[285]

Consider the profile of the newspaper you’re writing for
[286]

Include ALL the information provided in the event_info section

Write a news article about this fictional event following these guidelines: a. Include ALL the information provided in the event_info section. b. Ensure your article aligns with the newspaper’s profile. c. Maintain a professional tone that aligns with your newspaper’s profile....
[287]

Present your news article in the following format: <result> <scratchpad> (Plan your approach here) </scratchpad> <headline> (Write a headline here) </headline> <article> <paragraph> <text>[First paragraph text]</text> </paragraph> <paragraph> <text>[Second paragraph text]</tex...
[288]

Remember, your goal is to create a realistic and engaging news article based on the provided fictional event information while adhering to the newspaper’s profile

After writing the article, double-check that you’ve included all the information from the event_info sec- tion. Remember, your goal is to create a realistic and engaging news article based on the provided fictional event information while adhering to the newspaper’s profile. G...
[289]

First, review the ground truth information about the fictional named entities: <entities> <LOCATIONS> {{LOCATIONS_XML}} </LOCATIONS> <PERSONS> {{PERSONS_XML}} </PERSONS> <ORGANIZATIONS> {{ORGANIZATIONS_XML}} </ORGANIZATIONS> <PRODUCTS> {{PRODUCTS_XML}} </PRODUCTS> <ARTS> {{ART...
[290]

Next, review the ground truth information about the fictional event: <event_info> {{EVENT_INFO}} </event_info>
[292]

Only revise the content of the article paragraphs

To revise the news article, follow these steps. Only revise the content of the article paragraphs. Do not revise the article title: a) Analyze the style of the original news article. Pay attention to tone, vocabulary, and sentence structure. Any changes you make should maintai...
[293]

H.30 Missing Information Adding Prompt in News Article Generation You are an AI assistant tasked with improving a news article about a fictional event

Output the revised news article in the following format: <result> <scratchpad> (Plan your approach here, outlining the main changes you intend to make) </scratchpad> <headline> (Write a revised headline that accurately reflects the content of the article) </headline> <article>...
[294]

First, review the list of fictional named entities as background information: <entities> <LOCATIONS> {{LOCATIONS_XML}} </LOCATIONS> <PERSONS> {{PERSONS_XML}} </PERSONS> <ORGANIZATIONS> {{ORGANIZATIONS_XML}} </ORGANIZATIONS> 77 <PRODUCTS> {{PRODUCTS_XML}} </PRODUCTS> <ARTS> {{A...
[295]

Next, review the ground truth outline of the fictional event, including all details that must be communi- cated in the news article: <event_info> {{EVENT_INFO}} </event_info>
[296]

Now, read the generated news article that needs to be revised: <news-article> {{CURRENT_NEWS_ARTICLE_XML}} </news-article>
[297]

Only revise the content of the article paragraphs

To revise the news article, follow these steps. Only revise the content of the article paragraphs. Do not revise the article title: a) Analyze the style of the original news article. Pay attention to tone, vocabulary, and sentence structure. Any changes you make should maintai...
[298]

H.31 Named Entity Resolution Prompt in News Article Generation You are an AI assistant tasked with processing news passages by identifying and marking named entities

Output the revised news article in the following format: <result> <scratchpad> (Plan your approach here, outlining the main changes you intend to make) </scratchpad> <headline> (Write a revised headline that accurately reflects the content of the article) </headline> <article>...
[299]

Each entity will have an associated ID

Carefully review the list of entities provided in the <entities> section. Each entity will have an associated ID
[300]

Search the passages for all occurrences of each entity in the list
[2019]

In Plain Sight: Media Bias Through the Lens of Factual Reporting. InProceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6343–6349, Hong Kong, Ch...

2019
[2021]

InProceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 7052–7063, Online and Punta Cana, Dominican Republic

Entity-Based Knowledge Conflicts in Ques- tion Answering. InProceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 7052–7063, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Kelvin Luu, Daniel Kha...

2021
[2023]

InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 7349–7360, Singapore

A Lightweight Method to Generate Unan- swerable Questions in English. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 7349–7360, Singapore. Association for Computational Linguistics. Max Glockner, Ieva Stali¯unait˙e, James Thorne, Gisela Vallejo,...

2023
[2024]

InProceedings of the AAAI Conference on Artificial Intelligence, vol- ume 38, pages 17754–17762

Benchmarking large language models in retrieval-augmented generation. InProceedings of the AAAI Conference on Artificial Intelligence, vol- ume 38, pages 17754–17762. Jifan Chen and Greg Durrett. 2019. Understanding Dataset Design Choices for Multi-hop Reasoning. InProceedings...

2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Outline Item Selection:Given one or two events, the LLM selects a subset of two outline items as the basis for the question

[2] [2]

InFirst Conference on Language Modeling

Dated Data: Tracing Knowledge Cutoffs in Large Language Models. InFirst Conference on Language Modeling. Yejin Choi. 2022. The Curious Case of Commonsense Intelligence.Daedalus, 151(2):139–155. Elizabeth Clark, Yangfeng Ji, and Noah A. Smith. 2018. Neural Text Generation in Stories Using Entity Rep- resentations as Context. InProceedings of the 2018 Confe...

arXiv 2022

[3] [3]

Each question requires two distinct pieces of in- formation (or more, if additional outline items are added) for sufficient grounding

Distractor Generation:The LLM generates plausible but incorrect distractor options for the question, along with justifications explain- ing their plausibility and incorrectness. Each question requires two distinct pieces of in- formation (or more, if additional outline items are added) for sufficient grounding. Multiple questions are generated for all n+1...

2018

[4] [4]

early June

ConservativeNews (System Prompt H.26) Generating a news article follows these steps: 19 Step 1: Select outline itemsGiven the outlines of all past events, and the current event outline, the LLM selects four subsets of outline item ids from the current event outline, which will be used to generate four different news articles. We use a temperature of t= 0....

2022

[5] [5]

Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran

A Survey on Event-Based News Narrative Extraction.ACM Computing Surveys, 55(14s):1–39. Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which Linguist In- vented the Lightbulb? Presupposition Verification for Question-Answering. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the ...

arXiv 2021

[6] [6]

Generate exactly {{NUM_STORYITEMS}} distinct story items

[7] [7]

What shapes your bias?

Background Summarization of Event Time- lines. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8111–8136, Singapore. Association for Com- putational Linguistics. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the li...

arXiv 2023

[8] [8]

Do not merge multiple pieces of information into one storyitem

Make storyitems as atomic as possible, communicating only a single piece of relevant information per item. Do not merge multiple pieces of information into one storyitem