Pith. sign in

REVIEW 4 major objections 5 minor 38 references

Can LLMs Ask Good Questions?

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Default-prompt LLMs, asked to write questions from a paragraph, consistently prefer descriptive questions that demand long answers and spread attention evenly across the context, unlike human-authored QA questions.

desk verdict Useful first behavioral profile of LLM question generation, but the headline answer-length result rests on comparing gold-span lengths to self-rated generated answers. read the letter →

arxiv 2501.03491 v2 pith:TRTBOJ77 submitted 2025-01-07 cs.CL cs.AI

Yueheng Zhang , Xiaoyuan Liu , Yiyou Sun , Atheer Alharbi , Hend Alzahrani , Tianneng Shi , Basel Alomair , Dawn Song This is my paper · ORCID
classification cs.CLcs.AI
keywords questiongenerationLLMevaluationcharacteristicscontextcoveragepositionalbiasanswerabilityasjudge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks what kinds of questions large language models write on their own, when prompted only to generate self-contained questions from a paragraph of Wikipedia text. Comparing 4,096 questions from four LLMs (Llama-3.3, DeepSeek-V3, Claude-3.7, and GPT-4o) with human questions from HotpotQA and TriviaQA across six dimensions, it finds a consistent profile: LLMs favor descriptive and fact-figure questions, keep question length near a stable 15-24 words with low variance, spread their attention evenly across the context rather than concentrating on its opening, and produce questions that need substantially longer answers than human questions even after answer compression. The authors argue these are default-prompt preferences rather than artifacts of one model, because the pattern holds across all four models and three prompt phrasings. A sympathetic reader would care because these defaults determine what synthetic question datasets look like before any prompt engineering, and therefore shape downstream uses such as retrieval-augmented generation (RAG) evaluation and hallucination testing.

What carries the argument

The evaluation protocol itself is the machinery: a pipeline that (1) maps questions into ten inductively derived types by LLM classification refined by human review, (2) measures word-level and sentence-level context coverage and buckets the context into ten positional regions, and (3) uses GPT-4o as judge to rate answers generated with and without context on a 0-5 scale, plus an answer-shortening step that repeatedly asks for a minimum-length answer and keeps the shortest version that preserves the rating. The shortening step is what lets the paper distinguish 'the question demands a long answer' from 'the model writes long answers.'

What would settle it

Take the same 256 WikiText paragraphs, place the only answer-bearing sentence at the paragraph's end, and ask the four LLMs to generate questions; if their context-coverage curves still peak away from the start and their questions stay descriptive, the balanced-focus and descriptive-type findings reflect generation preferences, whereas a shift to front-loaded factoid questions would show the findings are artifacts of where facts sit in the source text.

Watch

Extended reading notes

Core claim

The central claim is that LLM-generated questions have a distinct, model-independent signature: they are disproportionately descriptive/characterization questions (27-45% across models vs 1.5-3% in human datasets), they request specific facts and figures, their question lengths are tightly clustered, their relevant context spans are spread across the full paragraph rather than front-loaded, and the answers they require remain several times longer than human golden answers even when an LLM judge certifies a compressed version as equally good. The paper interprets the evenly distributed context focus as the inverse of the positional bias reported in QA: when generating, not answering, LLMs do not anchor on the beginning of the paragraph. It further reports that about one quarter of LLM questions are not answerable without the context, a higher rate than HotpotQA's, which it frames as useful for stress-testing retrieval and hallucination.

Load-bearing premise

The comparison assumes HotpotQA and TriviaQA are fair human-authored baselines for questions LLMs generate from a single WikiText paragraph, even though the human datasets were built differently: HotpotQA asked workers to write multi-hop questions from multiple contexts, and TriviaQA began with existing questions and later attached evidence.

Editorial extensions

If this is right

  • Synthetic question sets built with default prompts will be dominated by descriptive questions whose reference answers are multi-sentence, so downstream QA and RAG benchmarks should expect longer target answers.
  • LLM-generated questions can serve as retrieval stress tests because they draw evenly from the full context, unlike human questions that favor the opening.
  • The roughly one-in-four questions that cannot be answered without context give hallucination detectors a natural source of negative probes.
  • Prompt rewording does not change the preference profile, so applications that need factoid questions must constrain the prompt explicitly.
  • The consistency across two open-source and two proprietary models suggests the pattern is a general property of current LLM question generation, not a quirk of one model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves open is whether the descriptive preference shifts when LLMs are fine-tuned on factoid QA pairs; the authors only vary prompt wording, not model training.
  • The balanced context focus may be a byproduct of choosing descriptive question types that synthesize several sentences, rather than an independent positional habit; comparing coverage by question type would separate the two.
  • If these defaults hold in specialized domains, synthetic question generators in medicine or finance will need type and length constraints to avoid producing uniformly long descriptive questions.
  • The TriviaQA comparison should be read cautiously: because those questions were written before evidence was attached, their front-loaded focus may reflect evidence selection rather than human question-asking behavior.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies question generation (QG) by four LLMs (Llama-3.3, DeepSeek-V3, Claude-3.7, GPT-4o) on a shared set of 256 WikiText paragraphs, yielding 1,024 questions per model. It compares these against human-authored questions from HotpotQA and TriviaQA along six dimensions: question type, question length, context coverage, answerability, uncommonness, and required answer length. The central claim, stated in the Abstract, is that LLM-generated questions tend to demand longer descriptive answers and exhibit more evenly distributed context focus, in contrast to human questions and to the positional bias reported in QA tasks. The methodology uses GPT-4o both as question generator (for the GPT-4o condition) and as judge for classification, answerability, uncommonness, and answer-length shortening. The paper includes prompt-variation checks in Appendix A, a small human-alignment validation for the answer-rating task, and public code at a GitHub repository.

Significance. If the central claim survives the measurement concerns below, this is a useful descriptive contribution: it is among the first systematic characterizations of LLM preferences in context-based question generation, and the proposed six-dimension evaluation pipeline could be reused for QG benchmarks and for tasks such as RAG evaluation and hallucination detection. The paper's strengths include the public code, the inclusion of two open and two closed models, the prompt-robustness check in Appendix A, and the explicit human-correlation check (Pearson 0.76 on 300 answer-rating annotations). The findings are plausible and the direction is consistent across the reported distributions, but the headline comparisons currently rest on asymmetric measurements and on an unvalidated automated-judging stack.

major comments (4)
  1. [§3.3, Table 4] The 'Shortened Answer Length' comparison in Table 4 is asymmetric and directly supports the Abstract's claim that LLM questions 'demand longer descriptive answers.' For human datasets, the reported values are gold-span lengths (TriviaQA 2.0±1.8; HotpotQA 2.2±1.8), i.e., the curated answer spans chosen by dataset annotators. For LLM questions, the values are lengths of free-text answers generated by GPT-4o and then compressed by GPT-4o, with GPT-4o itself judging whether the compressed version preserves the original rating. These measure different constructs: the human number reflects annotation style, while the LLM number reflects a model's self-consistent compression with no validation that the compressed length is the minimum information required by the question. To support the headline claim, the same generation-and-shortening pipeline should be applied to the human questions (with their gold answers or generated answers), or an independent human-annotation protocol should measure minimum necessary answer length for both human and LLM questions.
  2. [§3.1, §4, Figure 2] The comparison treats HotpotQA and TriviaQA as fair human-authored baselines, but the two datasets were built under workflows that differ from each other and from the LLM setup. HotpotQA is context-first but multi-hop, with crowdworkers generating questions from multiple evidence paragraphs; TriviaQA is question-first, with evidence identified later; the LLM experiments use a single WikiText paragraph and a prompt that explicitly asks for 'self-contained' questions. These workflow differences are acknowledged in §3.1, but they are not controlled in the comparisons in Table 1 and Figure 2. In particular, the claim of 'more evenly distributed context focus' versus humans is based on HotpotQA only, since TriviaQA is excluded from the coverage analysis; the human beginning-of-context bias could reflect the annotation interface rather than an intrinsic human preference. The paper should either add a human QG condition with the same single-paragraph context-first protocol or temper the cross-dataset claim.
  3. [§4, Tables 1–4] The paper reports percentage distributions, means, and standard deviations without confidence intervals or significance tests, yet the text uses comparative language such as 'significantly longer answers' and 'consistent patterns.' For example, in Table 4 the shortened answer lengths are 7.3±12.6 (Llama), 13.7±19.1 (DeepSeek), 7.5±11.8 (Claude), and 10.4±15.7 (GPT-4o); without uncertainty intervals or effect sizes, the ordering and the claim of consistency across models are not quantitatively supported. The sample size (1,024 questions per model) is large enough that even small differences may be statistically significant, but the report should provide bootstrap confidence intervals or paired significance tests, especially for the central comparisons between human and LLM questions.
  4. [§3.2, §3.3, Figure 3] The experimental design uses GPT-4o as judge for question-type classification (ten categories), context-coverage sentence selection, answerability ratings, uncommonness ratings, and answer-length sufficiency. The only human validation reported is the Pearson 0.76 correlation on 300 answer-rating annotations, and that validation is not extended to the other five metrics. Since GPT-4o is also the generator for the GPT-4o condition, the self-judging loop is a circularity risk for all of the headline measurements, not only for answerability. The paper should validate at least the question-type and context-coverage classifiers against human labels on a similar scale, and it should report agreement separately for questions generated by each model rather than as a single pooled correlation.
minor comments (5)
  1. [Table 3] Table 3 should be reformatted: the shared context statistics for the LLM conditions appear only in the HotpotQA and Llama rows, and the sentence/word counts should be reported once for the common WikiText subsample rather than repeated sparsely across rows.
  2. [Table 5] The 'Others' row for GPT-4o shows 0.1% under v3 in Appendix A but 0.0% in the main Table 1; the discrepancy should be reconciled or explained.
  3. [References and typos] There are several minor typographical issues: 'CC-BY-AS-3.0' in Section 4 should likely be 'CC-BY-SA 3.0', and the HotpotQA reference in the bibliography has 'V olume' with a stray space. The reference list should be proofread.
  4. [Figure 2] Figure 2 would benefit from explicit axis labels and a statement of the sample size underlying each region bucket; the text mentions 'ten region buckets' but the figure as described does not show how the buckets are defined or whether they are normalized by context length.
  5. [Section 3.3] The sentence 'Our approach significantly reduces answer length' uses 'significantly' without a statistical test; this should be reworded to avoid implying a formal significance result, which is also consistent with the major comment on missing uncertainty quantification.

Circularity Check

1 steps flagged · score 6.0 of 10

The 'longer descriptive answers' result is partially an artifact of comparing self-rated LLM compression to gold spans, rather than a fully independent comparison.

  1. fitted input called prediction [Section 3.3 (Required answer length) and Table 4]
    "If a shorter version of the answer receives the same rating as the original, we treat the shorter one as sufficient."

    The human rows in Table 4 are dataset gold-answer spans (TriviaQA 2.0 +/- 1.8; HotpotQA 2.2 +/- 1.8), while LLM rows are answers GPT-4o itself compresses and re-rates until the same rating holds. The claim that LLM questions 'still require significantly longer answers' thus compares two different constructs: a curated minimal span vs. a self-consistent compression by the same model that generated the question and answer. The rating rubric rewards detail ('Fully accurate and detailed'), so the LLM-side 'required length' is partly set by the judge's preference. The paper never runs the generation-and-shortening procedure on human questions, so the headline direction is partly built into the metric.

full rationale

This is an empirical measurement paper, and most of its six metrics are operationalized in the same way for human and LLM questions, so they are not circular. There is no load-bearing self-citation: the cited evaluation methods (Ragas, star-rating, positional-bias work) are external, and the paper never imports a uniqueness theorem from its own authors. The question-type classification was built from inductive coding and then applied with the same LLM judge to both sides; the context-coverage comparison uses the same LLM judge for human and generated questions; the answerability rating is externally calibrated against 300 manual annotations with Pearson 0.76. The one place where the claim reduces toward its own construction is the required-answer-length comparison. For LLM questions, 'required length' is defined as the length of an answer generated by GPT-4o and compressed only while GPT-4o's own rating is unchanged; for human questions, it is the dataset's gold-answer span. The paper's central claim that LLM questions 'demand longer descriptive answers' therefore depends on an asymmetric metric whose LLM side is partly set by the same model's rubric. This is a partial, localized circularity, not a wholesale derivation-from-input; the balanced context-focus claim and the question-type distribution are independent findings.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are postulated. The paper's invented artifact is its ten-category question taxonomy, which is a classification instrument rather than an entity; it is captured above as a free parameter and as an axiom instead of as an invented entity.

free parameters (5)
  • Context coverage region buckets = 10
    Figure 2 divides each context into ten equal buckets to report positional focus; the balanced-focus finding is a function of this bin count.
  • Answer shortening word limits = 1, 2, 3, 4, 8
    Section 3.3 defines required answer length as the smallest word limit that preserves the LLM-judge rating; these hand-chosen limits set the scale of the metric.
  • Question-type taxonomy size = 10
    The ten categories in Section 3.2 were developed by inductive coding on a mix of human and LLM questions; all type-distribution results depend on this hand-built taxonomy.
  • Sampling temperature = 0
    Section 4 runs all models with temperature set to 0, which reduces output diversity and may narrow the range of question types; the preference results are conditional on this setting.
  • Context and question sample sizes = 256 contexts, 4 questions each
    Section 4 samples 256 WikiText paragraphs and requests N=4 questions each, giving 1,024 questions per model; all distributional results reflect this sample, and the paper does not report power or confidence intervals.
assumptions (4)
  • domain assumption HotpotQA and TriviaQA provide a fair human-authored baseline for comparison with LLM-generated questions from WikiText paragraphs.
    Section 3.1 and Section 4 compare against these datasets despite different construction workflows; if the workflows are not comparable, observed differences could be artifacts.
  • domain assumption GPT-4o's ratings, question-type classifications, and context-coverage selections reflect human judgments.
    Section 4 validates the rating step on 300 human annotations with Pearson correlation 0.76, but the classification and coverage steps are not validated against human labels.
  • domain assumption The ten question-type categories are exhaustive and close to orthogonal.
    Section 3.2 defines the categories by inductive coding and manual refinement; the paper reports over 99.9 percent classification success, which supports but does not prove the taxonomy is unbiased.
  • domain assumption The context-coverage sentence selection by the LLM accurately identifies which sentences a question depends on.
    Section 3.2 uses a prompt to have the LLM output the minimal set of relevant sentences; there is no human check of these selections in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can LLMs Ask Good Questions?." pith.science (2026). https://pith.science/paper/TRTBOJ77

@misc{pith2026250103491,
  author       = {Pith},
  title        = {Pith review of: Can LLMs Ask Good Questions?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TRTBOJ77}},
  note         = {Machine review of arXiv:2501.03491}
}
read the original abstract

We evaluate questions generated by large language models (LLMs) from context, comparing them to human-authored questions across six dimensions: question type, question length, context coverage, answerability, uncommonness, and required answer length. Our study spans two open-source and two proprietary state-of-the-art models. Results reveal that LLM-generated questions tend to demand longer descriptive answers and exhibit more evenly distributed context focus, in contrast to the positional bias often seen in QA tasks. These findings provide insights into the distinctive characteristics of LLM-generated questions and inform future work on question quality and downstream applications.

Figures

Figures reproduced from arXiv: 2501.03491 by the authors.

Figure 1
Figure 1. Investigation on the characteristics of LLM-generated questions. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Context coverage region analysis. Ten region buckets are created and access frequency is [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Answer ratings distribution with and w/o context. No zero-rating answers are found. HQA [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Shortened answer length distribution vs. original answer length distribution across questions [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Shortened Answer Length Distribution vs. Original Answer Length Distribution across [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 11 canonical work pages

  1. [1]

    Cosmopedia, 2024

    Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Cosmopedia, 2024. URL https://huggingface.co/datasets/HuggingFaceTB/cosmopedia

    2024

  2. [2]

    Yllias Chali and Sadid A. Hasan. Towards Topic-to-Question Generation . Computational Linguistics, 41 0 (1): 0 1--20, 03 2015. ISSN 0891-2017. doi:10.1162/COLI_a_00206. URL https://doi.org/10.1162/COLI\_a\_00206

    doi:10.1162/coli_a_00206 2015

  3. [3]

    Under the surface: Tracking the artifactuality of llm-generated data, 2024

    Debarati Das, Karin De Langis, Anna Martin, Jaehyung Kim, Minhwa Lee, Zae Myung Kim, Shirley Hayati, Risako Owan, Bin Hu, Ritik Parkar, Ryan Koo, Jonginn Park, Aahan Tyagi, Libby Ferland, Sanjali Roy, Vincent Liu, and Dongyeop Kang. Under the surface: Tracking the artifactuality of llm-generated data, 2024

    work page 2024

  4. [4]

    Deepseek-v3 technical report, 2024

    DeepSeek-AI. Deepseek-v3 technical report, 2024. URL https://arxiv.org/abs/2412.19437

    arXiv 2024

  5. [5]

    Question generation for question answering

    Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. Question generation for question answering. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 866--874, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi:10.18653/v1/D17-...

    doi:10.18653/v1/d17-1090 2017

  6. [6]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  7. [7]

    Tinystories: How small can language models be and still speak coherent english?, 2023

    Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023. URL https://arxiv.org/abs/2305.07759

    arXiv 2023

  8. [8]

    Ragas: Automated evaluation of retrieval augmented generation

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217, 2023

    arXiv 2023

Show all 38 references
  1. [9]

    Gptscore: Evaluate as you desire

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166, 2023

    2023 arXiv

  2. [10]

    Qgeval: A benchmark for question generation evaluation, 2024

    Weiping Fu, Bifan Wei, Jianxiang Hu, Zhongmin Cai, and Jun Liu. Qgeval: A benchmark for question generation evaluation, 2024. URL https://arxiv.org/abs/2406.05707

    2024 arXiv

  3. [11]

    Exploring quality criteria and evaluation methods in automated question generation: A comprehensive survey

    Guher Gorgun and Okan Bulut. Exploring quality criteria and evaluation methods in automated question generation: A comprehensive survey. Education and Information Technologies, pages 1573--7608, 2024. doi:10.1007/s10639-024-12771-3

    2024 doi

  4. [12]

    A survey on neural question generation: Methods, applications, and prospects, 2024

    Shasha Guo, Lizi Liao, Cuiping Li, and Tat-Seng Chua. A survey on neural question generation: Methods, applications, and prospects, 2024

    2024

  5. [13]

    T oxi G en: A large-scale machine-generated dataset for adversarial and implicit hate speech detection

    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. T oxi G en: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings...

    2022 doi

  6. [14]

    Automatic generation of short answer questions for reading comprehension assessment

    YAN HUANG and LIANZHEN HE. Automatic generation of short answer questions for reading comprehension assessment. Natural Language Engineering, 22 0 (3): 0 457–489, 2016. doi:10.1017/S1351324915000455

    2016 doi

  7. [15]

    Beavertails: Towards improved safety alignment of LLM via a human-preference dataset

    Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. In Thirty-seventh Conference on Neural Information Processing Syst...

    2023

  8. [16]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, July 2017. ...

    2017

  9. [17]

    Hagrid: A human-llm collaborative dataset for generative information-seeking with attribution, 2023

    Ehsan Kamalloo, Aref Jafari, Xinyu Zhang, Nandan Thakur, and Jimmy Lin. Hagrid: A human-llm collaborative dataset for generative information-seeking with attribution, 2023. URL https://arxiv.org/abs/2307.16883

    2023 arXiv

  10. [18]

    Difficulty-controllable multi-hop question generation from knowledge graphs

    Vishwajeet Kumar, Yuncheng Hua, Ganesh Ramakrishnan, Guilin Qi, Lianli Gao, and Yuan-Fang Li. Difficulty-controllable multi-hop question generation from knowledge graphs. In Chiara Ghidini, Olaf Hartig, Maria Maleshkova, Vojt e ch Sv \'a tek, Isabel Cruz, Aidan Hogan, Jie Song...

    2019

  11. [19]

    Natural questions: a benchmark for question answering research

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computa...

    2019

  12. [20]

    Visual question generation as dual task of visual question answering

    Yikang Li, Nan Duan, Bolei Zhou, Xiao Chu, Wanli Ouyang, Xiaogang Wang, and Ming Zhou. Visual question generation as dual task of visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018

    2018

  13. [21]

    ROUGE : A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE : A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74--81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013

    2004

  14. [22]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

    2016

  15. [23]

    Generating natural questions about an image

    Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. Generating natural questions about an image. arXiv preprint arXiv:1603.06059, 2016

    2016 arXiv

  16. [24]

    Recent advances in neural question generation, 2019

    Liangming Pan, Wenqiang Lei, Tat-Seng Chua, and Min-Yen Kan. Recent advances in neural question generation, 2019. URL https://arxiv.org/abs/1905.08949

    2019 arXiv

  17. [25]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311–318, USA, 2002. Association for Computational ...

    2002

  18. [26]

    Squad: 100,000+ questions for machine comprehension of text, 2016

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. URL https://arxiv.org/abs/1606.05250

    2016 arXiv

  19. [27]

    Khapra, and Sachindra Joshi

    Sathish Reddy, Dinesh Raghu, Mitesh M. Khapra, and Sachindra Joshi. Generating natural language question-answer pairs from a knowledge graph using a RNN based question generation model. In Mirella Lapata, Phil Blunsom, and Alexander Koller, editors, Proceedings of the 15th Con...

    2017

  20. [28]

    Where is the answer? investigating positional bias in language model knowledge extraction, 2024

    Kuniaki Saito, Kihyuk Sohn, Chen-Yu Lee, and Yoshitaka Ushiku. Where is the answer? investigating positional bias in language model knowledge extraction, 2024. URL https://arxiv.org/abs/2402.12170

    2024 arXiv

  21. [29]

    Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus, 2016

    Iulian Vlad Serban, Alberto García-Durán, Caglar Gulcehre, Sungjin Ahn, Sarath Chandar, Aaron Courville, and Yoshua Bengio. Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus, 2016. URL https://arxiv.org/abs/1603.06807

    2016 arXiv

  22. [30]

    Evaluating open-qa evaluation

    Cunxiang Wang, Sirui Cheng, Qipeng Guo, Yuanhao Yue, Bowen Ding, Zhikun Xu, Yidong Wang, Xiangkun Hu, Zheng Zhang, and Yue Zhang. Evaluating open-qa evaluation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  23. [31]

    Is chatgpt a good nlg evaluator? a preliminary study

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048, 2023 a

    2023 arXiv

  24. [32]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023 b . URL https://arxiv.org/abs/2212.10560

    2023 arXiv

  25. [33]

    Knowledge-based visual question generation

    Jiayuan Xie, Wenhao Fang, Yi Cai, Qingbao Huang, and Qing Li. Knowledge-based visual question generation. IEEE Transactions on Circuits and Systems for Video Technology, 32 0 (11): 0 7547--7558, 2022. doi:10.1109/TCSVT.2022.3189242

    2022

  26. [34]

    Autoqgs: Auto-prompt for low-resource knowledge-based question generation from sparql

    Guanming Xiong, Junwei Bao, Wen Zhao, Youzheng Wu, and Xiaodong He. Autoqgs: Auto-prompt for low-resource knowledge-based question generation from sparql. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM '22, page 2250–2259, N...

    2022

  27. [35]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018

    2018 arXiv

  28. [36]

    Dynosaur: A dynamic growth paradigm for instruction-tuning data curation, 2023

    Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei Chang. Dynosaur: A dynamic growth paradigm for instruction-tuning data curation, 2023. URL https://arxiv.org/abs/2305.14327

    2023 arXiv

  29. [37]

    Towards a unified multi-dimensional evaluator for text generation

    Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards a unified multi-dimensional evaluator for text generation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empi...

    2022 doi

  30. [38]

    Toolqa: A dataset for llm question answering with external tools

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for llm question answering with external tools. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages...

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.