{"id":"5ab3bd52-d808-4e3a-a5fd-0f37071fd02a","arxiv_id":"2412.16694","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"DragonVerseQA is a 3,200-pair question-answering dataset for House of the Dragon and Game of Thrones episodes, built from summaries, reviews, and wiki data to support long-form narrative QA.","lead":"A new question-answering dataset for the Game of Thrones and House of the Dragon shows combines episode summaries, user reviews, and structured data. The authors say it enables longer, more context-rich answers than existing QA datasets, but the paper's evidence for that comparison is not convincing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset-quality evidence is invalid: BLEU/ROUGE/EM scores are computed without reference answers and use inconsistent scales across tables, so the 'new benchmark' claim is unsupported.","rationale":"The reader's weakest assumption was that the automated pipeline generates accurate, relevant, unbiased QA pairs without per-item verification. I agree that this is a central risk, but the more immediate and decisive problem is that the paper's reported validation metrics are internally incoherent: the scores cannot be computing what the paper claims, and the scale inconsistencies suggest the numbers are not trustworthy. That alone is sufficient to reject the 'new benchmark' claim, independent of whether the pipeline actually works. In good faith, I also note that the paper attempts some quality assurance (manual review, automated format checks, spam/bias filtering), but the manual review is small and lacks agreement metrics, and the automated checks do not address answer correctness. The novelty claim is also contradicted by the paper's own citations, which weakens the central claim further. Because the reader already reached REJECT with moderate confidence, my concern does not change the verdict; it strengthens it. I would keep the verdict as REJECT but emphasize that the evaluation methodology, rather than only the unverified pipeline, is the load-bearing flaw.","tokens_in":13452,"tokens_out":3608,"duration_ms":32167,"concrete_test":"Independently sample 100 QA pairs from the released prototype, have three annotators verify whether each answer is fully supported by the corresponding episode summary/review context and whether the question is answerable from that context; compute the proportion of unsupported or off-context answers and the inter-annotator agreement (Cohen's kappa). If more than 5% of answers are unsupported or kappa is below 0.6, the pipeline-quality assumption fails and the 'high-quality benchmark' claim collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that DragonVerseQA 'sets a new benchmark' depends on demonstrated dataset quality, but the empirical evidence is not coherent. Section V.A reports F1, BLEU, and ROUGE-L for DragonVerseQA and baselines (Table VI) even though the generated QA pairs have no gold reference answers; BLEU and ROUGE require references, and F1 against what is undefined. The ablation tables are internally inconsistent: Tables IX–XI report the same baseline as BLEU 85.2, ROUGE 88.1, Exact Match 78.4, while Table XII reports the same hyperparameters (learning rate 1e-4, batch size 32) as BLEU 0.55, ROUGE 0.60, EM 0.70—a 100x scale mismatch with no explanation. The only human validation is a manual review of 200 of 3,200 pairs (Section V.B) with no inter-annotator agreement, no error analysis, and no per-item grounding check; the 'Context Relevance' score in Table VI is similarly reported without agreement or rubric details. Consequently, the paper's quality claim rests on the unverified assumption that the GPT-3 summarization plus BLANC answer-span plus BERT-HLSQG question-generation pipeline produces accurate, relevant, grounded pairs. This assumption is particularly fragile because the pipeline is zero/few-shot and fine-tuned on ELI5, a very different domain. Additionally, the paper's own Related Works section cites TVQA, MovieQA, DramaQA, NarrativeQA, FairytaleQA, and the authors' GameofthronesQA, directly contradicting the 'first QA dataset for TV narratives' novelty claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DragonVerseQA, a question-answering dataset for the TV series 'House of the Dragon' and 'Game of Thrones', assembled from episode summaries, user reviews, and structured WikiData, with QA pairs generated by an automated pipeline (GPT-3 zero-shot summarization, BLANC answer-span prediction, and BERT-HLSQG question generation fine-tuned on ELI5). The paper claims that this is the first QA dataset capturing the complexities of serialized TV narratives and that it constitutes a new benchmark for OTT content. It reports comparisons with SQuAD 2.0, TriviaQA, and Natural Questions, plus manual and automated validation and ablation studies of the question-generation component.","tokens_in":13892,"tokens_out":3905,"duration_ms":36350,"significance":"If the dataset were rigorously validated and the novelty claim were accurate, DragonVerseQA could be a useful resource for narrative QA, particularly because it integrates multiple context sources and includes a knowledge-graph visualization and a semi-supervised spam/bias filtering procedure. The paper is transparent about some limitations and makes a prototype publicly available. However, the evidence presented does not currently support the central claims: the main evaluation metrics lack gold references, the ablation results are internally inconsistent, the manual validation covers only a small fraction of the data without agreement or error analysis, and the 'first QA dataset for TV series narratives' claim is contradicted by the paper's own related-work discussion. The significance of the contribution is therefore not established as submitted.","major_comments":[{"comment":"The reported F1, BLEU, and ROUGE-L scores for DragonVerseQA are not meaningful without a defined gold-reference answer for each generated QA pair. BLEU and ROUGE require reference texts, and it is unclear what the F1 score is computed against. Because the answers were produced by the same pipeline that created the dataset, these numbers largely measure the generator's self-consistency rather than dataset quality. This is load-bearing because the 'new benchmark' claim depends on demonstrating the quality of the dataset, and the current evaluation does not do so.","section":"Section V.A, Table VI"},{"comment":"The ablation tables use inconsistent scales for the same baseline configuration. Tables IX-XI report the baseline as BLEU 85.2, ROUGE 88.1, and Exact Match 78.4, while Table XII reports the same hyperparameters (learning rate 1e-4, batch size 32) as BLEU 0.55, ROUGE 0.60, and Exact Match 0.70. No explanation is given for this roughly 100x scale difference, so the relative conclusions of the ablation study are not interpretable.","section":"Section V.C, Tables IX-XII"},{"comment":"The manual validation covers only 200 of 3,200 QA pairs, with no inter-annotator agreement score, no error analysis, and no per-item check that answers are grounded in the source episode summaries. The 'Context Relevance' score in Table VI is likewise reported without a rubric, annotation instructions, or annotator agreement. Given that the remainder of the dataset is generated automatically, this is insufficient evidence for the claim that the dataset is high-quality and contextually accurate.","section":"Section V.B, Table VII"},{"comment":"The claim that DragonVerseQA is 'the first QA dataset to capture the intricate complexities of TV series narratives' is directly contradicted by the paper's own related-work section, which cites TVQA, MovieQA, DramaQA, NarrativeQA, and the authors' earlier GameofthronesQA. The novelty claim needs to be substantially revised or removed.","section":"Section I.B vs. Section II"},{"comment":"The automated generation pipeline uses zero-shot GPT-3 summarization and a question-generation model fine-tuned only on ELI5, a domain very different from serialized TV narratives. The paper provides no factuality verification of the generated answers against the original episode summaries or reviews. Because the dataset quality claim rests entirely on this pipeline operating reliably across 3,200 pairs, this is a load-bearing correctness risk that needs to be addressed with per-item grounding checks or a substantially larger human evaluation.","section":"Section III.D-E"}],"minor_comments":[{"comment":"There is a typo: 'Game of the Thrones' should be 'Game of Thrones'.","section":"Section IV.A"},{"comment":"The text says the same procedure is used for extracting user reviews and the algorithm is shown as 'Collect Episode Summaries'; the algorithm labels should be clarified so that the review-collection pseudocode is distinct.","section":"Section III.A, Algorithm 2"},{"comment":"The ELI5 answer length is listed as 'around 40 words', but DragonVerseQA's average answer length is reported as 31 words in Table V; this comparison needs a consistent definition of answer length.","section":"Table IV"},{"comment":"The PDF rendering of Equation (3) is broken across lines, and the text should explicitly define the window size and the role of the decay hyperparameter q before the equation.","section":"Equation (3)"},{"comment":"Several citations do not match their context: reference [10] is attributed to 'Ramakrishna et al.' but the listed work is by Hebbar et al.; reference [24] is cited as the source of the nlgeval library but the title refers to a knowledge-graph QA generation paper. Please verify and correct the bibliography.","section":"References"}],"recommendation":"reject","confidential_remarks":"The central claims are not supported by the evidence as submitted: the headline metrics are circular, the ablation tables are inconsistent, the human validation is too thin, and the novelty claim is contradicted by prior TVQA and MovieQA work. A resubmission would need a fundamentally new evaluation design with per-item human validation, corrected and consistent metrics, and a carefully repositioned contribution statement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: DragonVerseQA is a modest, plausibly useful resource for narrative QA in a single franchise, but the paper's empirical claims don't hold up. The evaluation metrics are undefined for the generated QA pairs, the ablation tables have a 100x scale mismatch, and the \"first TV-series QA dataset\" claim contradicts the paper's own related work.\n\nWhat's actually new: the dataset artifact itself — 3,200 QA pairs covering House of the Dragon S1 and Game of Thrones S8, combining episode summaries, filtered user reviews, and WikiData, plus a small knowledge graph. The pipeline is described in enough detail to reproduce, and the manual review of 200 pairs, while small, gives a sanity check that the generated questions are mostly relevant. The authors also include a limitations section that names the genre-specificity and reliance on pretrained filters.\n\nWhere it falls apart: Section V. Table VI reports F1, BLEU, and ROUGE-L for DragonVerseQA against NQ, TriviaQA, and SQuAD 2.0, but the DragonVerseQA answers are generated by the same pipeline with no gold reference, so BLEU/ROUGE have nothing to compare against. The stress-test note is right. Then the ablation tables: Tables IX–XI list the same baseline as BLEU 85.2, ROUGE 88.1, EM 78.4, while Table XII gives BLEU 0.55, ROUGE 0.60, EM 0.70 for the same learning rate (1e-4) and batch size (32). That is not a minor typo; it means the metrics were computed inconsistently, and the ablation conclusions don't follow. The \"Context Relevance\" score of 4.8 comes from two MTurk annotators with no reported agreement. And the novelty claim — \"first QA dataset to capture the intricate complexities of TV series narratives\" — is contradicted by MovieQA, TVQA, DramaQA, NarrativeQA, and the authors' own GameofthronesQA, all cited in the paper.\n\nIs the central argument dead? The dataset might still be useful, but the paper as written doesn't establish it. The evaluation would need to be redone with held-out human references, the ablation units reconciled, and the novelty claim softened to a domain-specific extension. I wouldn't cite this version, but I'd send it to review with a strong request for revision, because the underlying resource has some value and the flaws are fixable in principle.\n\nRecommendation: send to peer review, but expect heavy revisions. The authors should be told to fix the metrics or remove them.","headline":"A potentially useful but unvalidated dataset paper whose evaluation metrics are internally inconsistent and whose 'first TV-QA' claim is contradicted by its own references.","tokens_in":14322,"tokens_out":2102,"would_cite":false,"duration_ms":17784,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DragonVerseQA is a new dataset that brings long-form question answering to serialized TV narratives.","keywords":["question answering dataset","long-form QA","narrative understanding","TV series","context-aware","knowledge graph","data filtering","benchmark"],"falsifier":"Take a random sample of QA pairs that were not part of the 200-pair manual review and have two experts familiar with the series check whether each answer is grounded in its episode's source summaries and actually answers the question; if a meaningful fraction (say more than 10 percent) are unsupported, off-context, or hallucinated, the claim that DragonVerseQA is a high-quality context-aware benchmark would fail.","tokens_in":13206,"feed_emoji":"🐉","tokens_out":10014,"duration_ms":77461,"temperature":0.7,"pith_summary":"This paper proposes DragonVerseQA, a question-answering dataset built for serialized TV narratives, centered on House of the Dragon and Game of Thrones. The authors argue that established fact-based QA benchmarks, which draw short answers mainly from encyclopedic articles, cannot capture character dynamics, plot progression, or audience interpretation. DragonVerseQA instead assembles a multi-source context from episode summaries, filtered user reviews, and structured character data, then generates long-form question-answer pairs from that context. If correct, it gives conversational AI and narrative-analysis research a domain-specific resource with substantially richer context and longer answers than existing open-domain long-form QA datasets.","feed_headline":"DragonVerseQA adds 3,200 long-form QA pairs for TV stories","feed_subtitle":"Combines episode summaries, filtered reviews, and character data to answer narrative questions with depth.","key_machinery":"The load-bearing machinery is the context-aware question-answer generation pipeline. Each source chunk is first compressed by zero-shot abstractive summarization; then the BLANC block-attention model predicts answer spans, assigning soft context labels that decay with distance from the span; then an answer-aware question generator built on BERT-HLSQG, fine-tuned on ELI5, creates questions with [HL] highlight tokens bracketing the predicted span so duplicate mentions in long summaries do not mislead generation. A final automated stage screens each QA pair for spam and bias before it is added to the dataset.","core_discovery":"The paper's central claim is that DragonVerseQA is the first QA dataset designed for the narrative complexity of serialized TV series, demonstrated on House of the Dragon Season 1 and Game of Thrones Season 8. It contains 3,200 long-form QA pairs, with questions averaging 15 words and answers averaging 31 words, each grounded in a multi-dimensional context built from episode summaries, filtered user reviews, and structured character metadata. The pairs are produced by a three-stage pipeline: zero-shot abstractive summarization of source chunks, answer-span prediction with the BLANC block-attention model, and answer-aware question generation using a BERT-HLSQG model fine-tuned on the ELI5 long-form QA dataset, followed by automated spam and bias filtering. The paper reports F1 of 85.6, BLEU of 0.45, ROUGE-L of 0.58, and a human-rated Context Relevance of 4.8 out of 5, and it presents a knowledge graph linking episodes, characters, and reviews to support narrative analysis.","pith_inferences":["Editorial inference: if the automated pipeline generalizes, the same three-stage recipe could be reapplied to other serialized genres, turning DragonVerseQA into a template for domain-specific dataset construction.","Editorial inference: the main unverified risk is that only 200 of the 3,200 QA pairs were manually reviewed; a larger independent audit would test whether the reported Context Relevance score holds across the full dataset.","Editorial inference: because answers are generated from summarized source chunks rather than raw episode text, the dataset may carry summarization artifacts; comparing a sample of answers against the original episode summaries would reveal how much context is preserved."],"forward_implications":["QA systems evaluated on DragonVerseQA are tested on long, context-dependent answers rather than single factoid spans, which is closer to how people discuss serialized fiction.","The combination of episode summaries, reviews, and character metadata in one resource supports narrative understanding, sentiment analysis, and relation extraction without stitching together separate datasets.","The reported evaluation scores position DragonVerseQA as a stronger narrative-depth benchmark than established fact-based QA datasets, provided the quality check holds.","The knowledge graph linking episodes, characters, and reviews offers a structured view of narrative and audience response that can feed recommendation and personalization systems."],"supporting_citations":[{"why":"Supplies the answer-aware question generation model, fine-tuned on long-form QA data, that produces the dataset's questions.","marker":"[2]"},{"why":"Supplies the block-attention answer span predictor that identifies answer phrases and resolves duplicate-span ambiguity in long episode summaries.","marker":"[21]"},{"why":"Provides the long-form QA training data on which the question generation model is fine-tuned, setting the expected answer style.","marker":"[3]"},{"why":"Is the context-specific QA benchmark that DragonVerseQA compares against on answer length and narrative depth.","marker":"[4]"},{"why":"Is the open-domain web-sourced benchmark used as a comparison baseline.","marker":"[5]"},{"why":"Is the real-user open-domain benchmark used as a comparison baseline.","marker":"[6]"},{"why":"Is the prior narrative QA dataset that DragonVerseQA extends from movies to serialized TV storytelling.","marker":"[7]"},{"why":"Is the prior TV-series QA dataset against which the claim of being first for serialized narrative complexity is positioned.","marker":"[8]"},{"why":"Supplies a pre-trained spam-detection model whose labels feed the semi-supervised review filtering pipeline.","marker":"[19]"},{"why":"Provides sentiment polarity scoring used to filter extreme or biased reviews before they enter the context.","marker":"[20]"}],"fun_headline_variants":["DragonVerseQA: 3,200 long-form QA pairs for GoT and HotD","Long-form QA for TV narratives: DragonVerseQA's 3,200 pairs","DragonVerseQA brings long-form answers to GoT and House of the Dragon QA","DragonVerseQA: 3,200 context-rich QA pairs for GoT and HotD narratives"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quality of the 3,200 QA pairs rests on the automated generation pipeline working reliably across every episode, even though only 200 of those pairs were manually reviewed by human experts.","fun_headline_variants_meta":{"raw":{"variants":["DragonVerseQA: 3,200 long-form QA pairs for GoT and HotD","Long-form QA for TV narratives: DragonVerseQA's 3,200 pairs","DragonVerseQA brings long-form answers to GoT and House of the Dragon QA","DragonVerseQA: 3,200 context-rich QA pairs for GoT and HotD narratives"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001057,"raw_usage":{"total_tokens":4494,"prompt_tokens":1065,"completion_tokens":3429,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":3346}},"tokens_in":681,"tokens_out":3429,"duration_ms":20129,"temperature":1.0,"reasoning_tokens":3346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:20:51.679639+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of QA pairs that were not part of the 200-pair manual review and have two experts familiar with the series check whether each answer is grounded in its episode's source summaries and actually answers the question; if a meaningful fraction (say more than 10 percent) are unsupported, off-context, or hallucinated, the claim that DragonVerseQA is a high-quality context-aware benchmark would fail.","supporting_citations":[{"cited_title":"”Natural questions: a benchmark for question answering research.” Transactions of the Association for Computational Linguistics 7 (2019): 453-466","cited_arxiv_id":null,"evidence_quote":"Is the real-user open-domain benchmark used as a comparison baseline."},{"cited_title":"”Movieqa: Understanding stories in movies through question-answering.” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp","cited_arxiv_id":null,"evidence_quote":"Is the prior narrative QA dataset that DragonVerseQA extends from movies to serialized TV storytelling."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a pre-trained spam-detection model whose labels feed the semi-supervised review filtering pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the answer-aware question generation model, fine-tuned on long-form QA data, that produces the dataset's questions."},{"cited_title":"Context-Aware Answer Extraction in Question Answering","cited_arxiv_id":"2011.02687","evidence_quote":"Supplies the block-attention answer span predictor that identifies answer phrases and resolves duplicate-span ambiguity in long episode summaries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides sentiment polarity scoring used to filter extreme or biased reviews before they enter the context."}],"review_version":1}