{"id":"a98924ee-3732-488a-a1f3-cb6ba94d71c4","arxiv_id":"2501.03491","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LLM-generated questions mostly ask for descriptions and long answers, and they use the source text more evenly than human-authored questions.","lead":"This paper compares questions written by four large language models with questions written by humans, measuring six properties such as type, length, and how much of the source text each question uses. It finds that LLMs tend to ask descriptive questions that need long answers, while spreading their attention more evenly across the text than human question writers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Answer-length comparison is asymmetric: human 'required length' is gold-span length, while LLM length is LLM-generated and self-rated, so 'LLMs demand longer answers' may be a measurement artifact.","rationale":"The reader's weakest assumption correctly flags that HotpotQA and TriviaQA are not perfectly matched human baselines. My stress-test sharpens one specific, load-bearing consequence of that mismatch: the required-answer-length metric is not applied symmetrically. Human questions are represented by gold answer spans designed for extractive QA, while LLM questions are represented by model-generated answers that the same model shortens and validates. This asymmetry directly affects the 'longer answers' half of the central claim, independent of other dataset-workflow differences. The paper does provide useful evidence elsewhere: the question-type classification shows LLMs favor descriptive categories, the two-prompt robustness check in Appendix A is reassuring, and the code is released. These support a conditional reading rather than rejection. A focused re-measurement of answer length on human questions would settle whether the headline survives. Until that check is run, the conditional verdict remains appropriate, but the concern is concrete and testable, not a global skepticism of the paper's contribution.","tokens_in":12748,"tokens_out":5655,"duration_ms":55894,"concrete_test":"Take a random sample (e.g., 300) of HotpotQA and TriviaQA questions with their provided contexts, run the exact Section 3.3 pipeline: GPT-4o generates a short answer with context, applies the 1/2/3/4/8-word limits, rates equivalence, and reports the minimal same-rating length. Compare this distribution to the LLM-generated shortened lengths in Table 4. As an additional check, have human annotators write the shortest complete answer to the same questions. If the human minimal lengths are statistically indistinguishable from LLM lengths, the 'longer answers' component of the headline is an artifact; if human lengths remain near 2 words, the claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing comparison in Table 4 is 'Shortened Answer Length.' For human datasets, the reported value is the golden-answer length (TriviaQA 2.0 +/- 1.8; HotpotQA 2.2 +/- 1.8), i.e., the curated span chosen by dataset annotators. For LLM questions, the value is the length of a GPT-4o-generated free-text answer after GPT-4o is asked to shorten it, with GPT-4o itself rating whether the shortened version maintains the same quality (Section 3.3). These measure different constructs: the human number captures annotation style, not the minimum information demanded by the question; the LLM number captures a model's self-consistent compression, with no validation that a same-rating short answer is necessary or sufficient. The central claim that LLM-generated questions 'demand longer descriptive answers' therefore rests on comparing a gold-span length to a generated-and-self-validated length. The paper's Section 5 acknowledges limited human annotation, but the asymmetry is not corrected. If the same generation-and-shortening pipeline were applied to human questions, their measured 'required answer length' could rise to LLM levels and the headline difference could disappear.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies question generation (QG) by four LLMs (Llama-3.3, DeepSeek-V3, Claude-3.7, GPT-4o) on a shared set of 256 WikiText paragraphs, yielding 1,024 questions per model. It compares these against human-authored questions from HotpotQA and TriviaQA along six dimensions: question type, question length, context coverage, answerability, uncommonness, and required answer length. The central claim, stated in the Abstract, is that LLM-generated questions tend to demand longer descriptive answers and exhibit more evenly distributed context focus, in contrast to human questions and to the positional bias reported in QA tasks. The methodology uses GPT-4o both as question generator (for the GPT-4o condition) and as judge for classification, answerability, uncommonness, and answer-length shortening. The paper includes prompt-variation checks in Appendix A, a small human-alignment validation for the answer-rating task, and public code at a GitHub repository.","tokens_in":12974,"tokens_out":4834,"duration_ms":46246,"significance":"If the central claim survives the measurement concerns below, this is a useful descriptive contribution: it is among the first systematic characterizations of LLM preferences in context-based question generation, and the proposed six-dimension evaluation pipeline could be reused for QG benchmarks and for tasks such as RAG evaluation and hallucination detection. The paper's strengths include the public code, the inclusion of two open and two closed models, the prompt-robustness check in Appendix A, and the explicit human-correlation check (Pearson 0.76 on 300 answer-rating annotations). The findings are plausible and the direction is consistent across the reported distributions, but the headline comparisons currently rest on asymmetric measurements and on an unvalidated automated-judging stack.","major_comments":[{"comment":"The 'Shortened Answer Length' comparison in Table 4 is asymmetric and directly supports the Abstract's claim that LLM questions 'demand longer descriptive answers.' For human datasets, the reported values are gold-span lengths (TriviaQA 2.0±1.8; HotpotQA 2.2±1.8), i.e., the curated answer spans chosen by dataset annotators. For LLM questions, the values are lengths of free-text answers generated by GPT-4o and then compressed by GPT-4o, with GPT-4o itself judging whether the compressed version preserves the original rating. These measure different constructs: the human number reflects annotation style, while the LLM number reflects a model's self-consistent compression with no validation that the compressed length is the minimum information required by the question. To support the headline claim, the same generation-and-shortening pipeline should be applied to the human questions (with their gold answers or generated answers), or an independent human-annotation protocol should measure minimum necessary answer length for both human and LLM questions.","section":"§3.3, Table 4"},{"comment":"The comparison treats HotpotQA and TriviaQA as fair human-authored baselines, but the two datasets were built under workflows that differ from each other and from the LLM setup. HotpotQA is context-first but multi-hop, with crowdworkers generating questions from multiple evidence paragraphs; TriviaQA is question-first, with evidence identified later; the LLM experiments use a single WikiText paragraph and a prompt that explicitly asks for 'self-contained' questions. These workflow differences are acknowledged in §3.1, but they are not controlled in the comparisons in Table 1 and Figure 2. In particular, the claim of 'more evenly distributed context focus' versus humans is based on HotpotQA only, since TriviaQA is excluded from the coverage analysis; the human beginning-of-context bias could reflect the annotation interface rather than an intrinsic human preference. The paper should either add a human QG condition with the same single-paragraph context-first protocol or temper the cross-dataset claim.","section":"§3.1, §4, Figure 2"},{"comment":"The paper reports percentage distributions, means, and standard deviations without confidence intervals or significance tests, yet the text uses comparative language such as 'significantly longer answers' and 'consistent patterns.' For example, in Table 4 the shortened answer lengths are 7.3±12.6 (Llama), 13.7±19.1 (DeepSeek), 7.5±11.8 (Claude), and 10.4±15.7 (GPT-4o); without uncertainty intervals or effect sizes, the ordering and the claim of consistency across models are not quantitatively supported. The sample size (1,024 questions per model) is large enough that even small differences may be statistically significant, but the report should provide bootstrap confidence intervals or paired significance tests, especially for the central comparisons between human and LLM questions.","section":"§4, Tables 1–4"},{"comment":"The experimental design uses GPT-4o as judge for question-type classification (ten categories), context-coverage sentence selection, answerability ratings, uncommonness ratings, and answer-length sufficiency. The only human validation reported is the Pearson 0.76 correlation on 300 answer-rating annotations, and that validation is not extended to the other five metrics. Since GPT-4o is also the generator for the GPT-4o condition, the self-judging loop is a circularity risk for all of the headline measurements, not only for answerability. The paper should validate at least the question-type and context-coverage classifiers against human labels on a similar scale, and it should report agreement separately for questions generated by each model rather than as a single pooled correlation.","section":"§3.2, §3.3, Figure 3"}],"minor_comments":[{"comment":"Table 3 should be reformatted: the shared context statistics for the LLM conditions appear only in the HotpotQA and Llama rows, and the sentence/word counts should be reported once for the common WikiText subsample rather than repeated sparsely across rows.","section":"Table 3"},{"comment":"The 'Others' row for GPT-4o shows 0.1% under v3 in Appendix A but 0.0% in the main Table 1; the discrepancy should be reconciled or explained.","section":"Table 5"},{"comment":"There are several minor typographical issues: 'CC-BY-AS-3.0' in Section 4 should likely be 'CC-BY-SA 3.0', and the HotpotQA reference in the bibliography has 'V olume' with a stray space. The reference list should be proofread.","section":"References and typos"},{"comment":"Figure 2 would benefit from explicit axis labels and a statement of the sample size underlying each region bucket; the text mentions 'ten region buckets' but the figure as described does not show how the buckets are defined or whether they are normalized by context length.","section":"Figure 2"},{"comment":"The sentence 'Our approach significantly reduces answer length' uses 'significantly' without a statistical test; this should be reworded to avoid implying a formal significance result, which is also consistent with the major comment on missing uncertainty quantification.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable descriptive study with a clear and testable central claim, but the headline 'LLMs demand longer descriptive answers' currently rests on an asymmetric measurement that could be an artifact of comparing gold spans to self-compressed model outputs. The missing statistical tests and the limited human validation of the automated metrics are fixable within the manuscript's scope, so I do not recommend rejection. If the authors add the human-side shortening experiment or a human-validated minimum-answer-length protocol, plus confidence intervals and classifier validation, the contribution would be suitable for publication. The novelty is moderate for a journal venue, but the release of code and the prompt-robustness checks add value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read if you care about LLM-as-judge or QG evaluation. The genuinely new thing is the six-dimension behavioral comparison across four models with prompt-variation checks and public code. The question-type result is real: LLMs default to descriptive/characterization questions far more than HotpotQA or TriviaQA authors do, and the distribution is stable across prompt paraphrases. The context-focus observation also holds up: LLM-generated questions spread attention over the whole paragraph, while HotpotQA questions cluster at the start, consistent with dataset construction.\n\nThe soft spot is the answer-length claim. Table 4 compares human golden-answer length (2.0/2.2 words) against an LLM-generated free-text answer that is shortened and judged by the same model. Those measure different constructs. The human number reflects annotation style; the LLM number reflects self-consistent compression, with no evidence the shortened answer is minimal or necessary. Running the same pipeline on human questions could easily erase the headline difference. The paper admits limited human annotation but does not address this asymmetry.\n\nThe percentage tables lack confidence intervals or significance tests, and the question-type taxonomy is LLM-derived with light human review. Those are standard fixes. The LLM-as-judge loop for answerability is partly mitigated by a 300-sample human correlation of 0.76, which is decent but not decisive.\n\nOverall: the descriptive-question preference and even context focus are likely real and useful for prompt engineering and synthetic data design. The 'longer answers' part is not established. With revisions — same pipeline on human questions, significance tests, more human validation of the taxonomy — this becomes a solid empirical contribution.\n\nIt deserves a serious referee, not a desk reject. I'd send it out and ask for those specific changes.","headline":"Useful first behavioral profile of LLM question generation, but the headline answer-length result rests on comparing gold-span lengths to self-rated generated answers.","tokens_in":13521,"tokens_out":2552,"would_cite":true,"duration_ms":23313,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Default-prompt LLMs, asked to write questions from a paragraph, consistently prefer descriptive questions that demand long answers and spread attention evenly across the context, unlike human-authored QA questions.","keywords":["question generation","LLM evaluation","question characteristics","context coverage","positional bias","answerability","LLM as judge"],"falsifier":"Take the same 256 WikiText paragraphs, place the only answer-bearing sentence at the paragraph's end, and ask the four LLMs to generate questions; if their context-coverage curves still peak away from the start and their questions stay descriptive, the balanced-focus and descriptive-type findings reflect generation preferences, whereas a shift to front-loaded factoid questions would show the findings are artifacts of where facts sit in the source text.","tokens_in":12553,"feed_emoji":"❓","tokens_out":6513,"duration_ms":56672,"temperature":0.7,"pith_summary":"This paper asks what kinds of questions large language models write on their own, when prompted only to generate self-contained questions from a paragraph of Wikipedia text. Comparing 4,096 questions from four LLMs (Llama-3.3, DeepSeek-V3, Claude-3.7, and GPT-4o) with human questions from HotpotQA and TriviaQA across six dimensions, it finds a consistent profile: LLMs favor descriptive and fact-figure questions, keep question length near a stable 15-24 words with low variance, spread their attention evenly across the context rather than concentrating on its opening, and produce questions that need substantially longer answers than human questions even after answer compression. The authors argue these are default-prompt preferences rather than artifacts of one model, because the pattern holds across all four models and three prompt phrasings. A sympathetic reader would care because these defaults determine what synthetic question datasets look like before any prompt engineering, and therefore shape downstream uses such as retrieval-augmented generation (RAG) evaluation and hallucination testing.","feed_headline":"LLMs default to descriptive questions that need long answers","feed_subtitle":"Across four models, generated questions spread focus across the whole context, unlike human QA questions.","key_machinery":"The evaluation protocol itself is the machinery: a pipeline that (1) maps questions into ten inductively derived types by LLM classification refined by human review, (2) measures word-level and sentence-level context coverage and buckets the context into ten positional regions, and (3) uses GPT-4o as judge to rate answers generated with and without context on a 0-5 scale, plus an answer-shortening step that repeatedly asks for a minimum-length answer and keeps the shortest version that preserves the rating. The shortening step is what lets the paper distinguish 'the question demands a long answer' from 'the model writes long answers.'","core_discovery":"The central claim is that LLM-generated questions have a distinct, model-independent signature: they are disproportionately descriptive/characterization questions (27-45% across models vs 1.5-3% in human datasets), they request specific facts and figures, their question lengths are tightly clustered, their relevant context spans are spread across the full paragraph rather than front-loaded, and the answers they require remain several times longer than human golden answers even when an LLM judge certifies a compressed version as equally good. The paper interprets the evenly distributed context focus as the inverse of the positional bias reported in QA: when generating, not answering, LLMs do not anchor on the beginning of the paragraph. It further reports that about one quarter of LLM questions are not answerable without the context, a higher rate than HotpotQA's, which it frames as useful for stress-testing retrieval and hallucination.","pith_inferences":["A testable extension the paper leaves open is whether the descriptive preference shifts when LLMs are fine-tuned on factoid QA pairs; the authors only vary prompt wording, not model training.","The balanced context focus may be a byproduct of choosing descriptive question types that synthesize several sentences, rather than an independent positional habit; comparing coverage by question type would separate the two.","If these defaults hold in specialized domains, synthetic question generators in medicine or finance will need type and length constraints to avoid producing uniformly long descriptive questions.","The TriviaQA comparison should be read cautiously: because those questions were written before evidence was attached, their front-loaded focus may reflect evidence selection rather than human question-asking behavior."],"forward_implications":["Synthetic question sets built with default prompts will be dominated by descriptive questions whose reference answers are multi-sentence, so downstream QA and RAG benchmarks should expect longer target answers.","LLM-generated questions can serve as retrieval stress tests because they draw evenly from the full context, unlike human questions that favor the opening.","The roughly one-in-four questions that cannot be answered without context give hallucination detectors a natural source of negative probes.","Prompt rewording does not change the preference profile, so applications that need factoid questions must constrain the prompt explicitly.","The consistency across two open-source and two proprietary models suggests the pattern is a general property of current LLM question generation, not a quirk of one model."],"supporting_citations":[{"why":"Provides the HotpotQA human questions, the context-first baseline LLM questions are compared against.","marker":"[Yang et al., 2018]"},{"why":"Provides TriviaQA human trivia questions, the question-first baseline used for type and length comparison.","marker":"[Joshi et al., 2017]"},{"why":"Documents the positional bias in QA that the paper's evenly distributed context focus is contrasted with.","marker":"[Saito et al., 2024]"},{"why":"Supplies the star-rating LLM-as-judge scheme used for answerability and uncommonness ratings.","marker":"[Wang et al., 2023a]"},{"why":"Supplies the sentence-level coverage prompt and RAG-style evaluation ideas that this paper extends to word level.","marker":"[Es et al., 2023]"},{"why":"Supplies the WikiText paragraphs that serve as generation contexts for all LLM questions.","marker":"[Merity et al., 2016]"},{"why":"Defines Llama-3.3-70B, one of the two open-source models whose generated questions are analyzed.","marker":"[Dubey et al., 2024]"},{"why":"Defines DeepSeek-V3, the other open-source model used in the generation experiments.","marker":"[DeepSeek-AI, 2024]"}],"fun_headline_variants":["LLM questions are descriptive, spread thin, need long answers","LLMs write questions that demand long answers and full context","LLM-generated questions: more descriptive, less positional bias","Why LLM questions spread focus: no front-loading, long answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes HotpotQA and TriviaQA are fair human-authored baselines for questions LLMs generate from a single WikiText paragraph, even though the human datasets were built differently: HotpotQA asked workers to write multi-hop questions from multiple contexts, and TriviaQA began with existing questions and later attached evidence.","fun_headline_variants_meta":{"raw":{"variants":["LLM questions are descriptive, spread thin, need long answers","LLMs write questions that demand long answers and full context","LLM-generated questions: more descriptive, less positional bias","Why LLM questions spread focus: no front-loading, long answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1172,"prompt_tokens":795,"completion_tokens":377,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":411,"completion_tokens_details":{"reasoning_tokens":306}},"tokens_in":411,"tokens_out":377,"duration_ms":4055,"temperature":1.0,"reasoning_tokens":306,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:52:19.581220+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 256 WikiText paragraphs, place the only answer-bearing sentence at the paragraph's end, and ask the four LLMs to generate questions; if their context-coverage curves still peak away from the start and their questions stay descriptive, the balanced-focus and descriptive-type findings reflect generation preferences, whereas a shift to front-loaded factoid questions would show the findings are artifacts of where facts sit in the source text.","supporting_citations":[],"review_version":1}