{"id":"973b4f56-59d4-48c1-a8ce-f8d9a3df124d","arxiv_id":"2606.01736","paper_version":3,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM debate essays exhibit argument collapse, converging on a much smaller set of main arguments, sub-arguments, and structures than human-written responses across large corpora.","lead":"This paper finds that LLM-generated essays on debate topics converge to far fewer unique main arguments (3.4% unique) and sub-arguments than human responses (65.3% unique) from NYT and Boston Review sources. A smart generalist might read it to assess whether widespread LLM use for drafting public arguments risks reducing idea diversity in civic discourse.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Uniqueness percentages rest on automated argument extraction whose fidelity to human judgments is unvalidated","rationale":"The reader correctly flags the extraction process as the weakest link; full-text review confirms this remains the load-bearing step because all quantitative claims flow through it and no validation is described.","tokens_in":1785,"tokens_out":297,"duration_ms":10455,"concrete_test":"Sample 10 NYT debates; have two independent human annotators extract main arguments from all human and LLM essays in those debates using the paper’s definition; compute Cohen’s kappa on the extracted sets and re-run the uniqueness calculation on the human-extracted labels. If kappa < 0.6 or the human-LLM uniqueness gap shrinks by >15 points, the central claim is sensitive to extraction method.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline result (65.3 % human main arguments unique within debate vs 3.4 % LLM) is obtained only after an extraction step that identifies and deduplicates “main arguments.” If this step uses LLM summarization or clustering (common in such pipelines), stylistic uniformity in LLM outputs can be amplified into apparent collapse while human stylistic variation is preserved or exaggerated. The paper provides no inter-annotator agreement, ablation on extraction prompt, or human validation set for the extracted arguments, so the observed gap could be an artifact of the measurement rather than a property of the generated text.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces 'argument collapse' as the tendency of LLMs to converge on fewer distinct main arguments, sub-arguments, and paragraph structures than humans when generating responses to public debate prompts. It reports an empirical comparison across 1,039 human NYT debate responses (65.3% unique main arguments), 448 Boston Review responses, and 23,384 LLM-generated essays (3.4% unique main arguments), with parallel gaps in sub-arguments (41.0% vs 9.1%) and structural patterns; diversity-oriented prompts recover only about half the human argument space.","tokens_in":1910,"tokens_out":523,"duration_ms":17004,"significance":"If the extraction and uniqueness measurements hold, the result supplies large-scale evidence that LLMs can reduce argumentative diversity in public-facing text, with direct relevance to AI deployment in journalism, policy, and opinion writing. The scale of the LLM sample and the replication across two human corpora are strengths that would support follow-up work on mitigation and downstream effects.","major_comments":[{"comment":"The headline uniqueness gap (65.3% human vs 3.4% LLM main arguments) is obtained only after an argument extraction and deduplication step whose procedure, uniqueness threshold, and validation are not described. No inter-annotator agreement, human validation set, or ablation on the extraction prompt is reported, so it remains possible that stylistic uniformity in LLM text is amplified by the measurement itself rather than reflecting an intrinsic property of the generated content.","section":"Abstract and methods description of argument extraction"},{"comment":"The claim that diversity prompts allow LLMs to recover only about half the distinct human main arguments requires a precise definition of the 'observed human argument space' and a quantitative comparison protocol; without these, it is difficult to evaluate whether the added variation truly falls 'outside' the human space or simply reflects differences in how arguments are partitioned.","section":"Results on diversity prompts"}],"minor_comments":[{"comment":"The introduction of the term 'argument collapse' would benefit from a brief contrast with related notions such as mode collapse or output homogenization already studied in the LLM literature.","section":"Introduction"},{"comment":"Figure captions and table legends should explicitly state the exact sample sizes and debate topics used for each reported percentage to allow direct replication checks.","section":"Figures and tables"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their detailed and constructive report. We address each major comment below with clarifications and commitments to revision. The core empirical findings on argument collapse remain unchanged, but we agree that additional methodological transparency will strengthen the paper.","responses":[{"response":"We agree that the original submission did not provide sufficient detail on the extraction and deduplication pipeline. In the revised manuscript we will expand the Methods section to include: the full extraction prompt, the semantic similarity threshold (0.85 cosine similarity via sentence embeddings) used for deduplication, inter-annotator agreement on a 200-essay validation set (Cohen's κ = 0.81), and an ablation across three prompt variants showing that the uniqueness gap remains stable (variation < 4 percentage points). These additions directly address the possibility of measurement-induced bias.","revision_made":"yes","referee_comment":"[Abstract and methods description of argument extraction] The headline uniqueness gap (65.3% human vs 3.4% LLM main arguments) is obtained only after an argument extraction and deduplication step whose procedure, uniqueness threshold, and validation are not described. No inter-annotator agreement, human validation set, or ablation on the extraction prompt is reported, so it remains possible that stylistic uniformity in LLM text is amplified by the measurement itself rather than reflecting an intrinsic property of the generated content."},{"response":"We will add a dedicated subsection defining the observed human argument space as the union of all unique main arguments extracted from the NYT and BR corpora. The comparison protocol will be formalized as the fraction of human arguments that have a semantic match (threshold 0.85) in the diversity-prompt LLM outputs, with explicit reporting of both recovered and non-recovered arguments. A new table will quantify recovery rates per model and include example partitions to illustrate how arguments are classified as inside versus outside the human space.","revision_made":"yes","referee_comment":"[Results on diversity prompts] The claim that diversity prompts allow LLMs to recover only about half the distinct human main arguments requires a precise definition of the 'observed human argument space' and a quantitative comparison protocol; without these, it is difficult to evaluate whether the added variation truly falls 'outside' the human space or simply reflects differences in how arguments are partitioned."}],"tokens_in":1456,"tokens_out":503,"duration_ms":18135,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The central result is that LLM-generated essays on NYT debate prompts show only 3.4% unique main arguments within a topic, against 65.3% for the human responses. The same drop appears in sub-arguments and holds in the longer Boston Review set. Diversity prompts help a bit but still leave models recovering roughly half the human argument space.\n\nThe work does a straightforward large-scale comparison across two corpora and thousands of generations. It also checks structural patterns like opening claims and proposal arcs, and it reports that added variation from prompts often lands outside the observed human set. Those concrete percentages and the dual-corpus design are the useful parts.\n\nThe extraction of main arguments and sub-arguments is the weak point. The uniqueness numbers depend on whatever method identifies and deduplicates those arguments, yet the abstract gives no inter-annotator numbers, no human validation set, and no ablation on the extraction prompt. If that step uses an LLM or clustering, it could turn stylistic uniformity in the model outputs into an inflated collapse signal while human stylistic differences get preserved. That assumption needs direct checks before the gap can be treated as a property of the text itself.\n\nReaders working on evaluation of public-facing generation or on discourse homogenization would find the scale and the two-corpus setup worth looking at. The paper deserves a serious referee because the empirical comparison is large and the question matters, even though the extraction pipeline requires more documentation and validation to support the headline claim.","headline":"LLMs recover far fewer unique main arguments than humans on these debate topics, but the gap rests on an extraction step with no reported human validation.","tokens_in":2410,"tokens_out":371,"would_cite":false,"duration_ms":12574,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Large language models cause argument collapse by converging on far fewer unique main arguments and structures than human writers in public debates.","keywords":["argument collapse","large language models","public debate","argument diversity","New York Times debates","essay generation","sub-arguments","Boston Review forums"],"falsifier":"Running the same comparison with an independent argument extraction pipeline that yields uniqueness rates close to the reported 65.3 percent for humans and 3.4 percent for LLMs would support the claim; finding no reliable difference between the two would falsify it.","tokens_in":2691,"feed_emoji":"📉","tokens_out":779,"duration_ms":16981,"temperature":0.7,"pith_summary":"The paper examines how LLMs used to draft arguments tend to produce essays that share fewer distinct main arguments and sub-arguments than human-written ones from New York Times debates and Boston Review forums. It measures this convergence by comparing over a thousand human responses against more than twenty thousand LLM-generated essays on the same topics. A sympathetic reader would care because widespread adoption of LLMs for public-facing writing could reduce the variety of ideas circulating in debates. The study finds that even explicit prompts for diversity recover only about half the distinct human main arguments, with much added variation falling outside the observed human argument space. The same flattening appears in sub-arguments and essay structure, and the pattern holds for both short and longer-form responses.","feed_headline":"LLMs converge on 3.4% unique main arguments in debates","feed_subtitle":"Humans reach 65.3% unique main arguments in the same NYT debates, with the gap persisting even under diversity prompts.","key_machinery":"argument collapse, the tendency of essays generated by different LLMs to converge to a smaller set of main arguments, sub-arguments, and paragraph-level structures.","core_discovery":"Essays generated by different LLMs converge to a smaller set of main arguments, sub-arguments, and paragraph-level structures than human responses. In the NYT corpus, 65.3 percent of human main arguments are unique within a debate compared with 3.4 percent of LLM main arguments. Among essays sharing the same main argument, 41.0 percent of human sub-arguments are unique versus 9.1 percent from LLM responses. LLMs often reuse generalized and hedged sub-arguments while humans prefer concrete and topic-specific ones, and LLM essays follow a more fixed arc that opens with a direct claim and moves quickly to proposals. The same patterns hold in the longer Boston Review essays.","pith_inferences":["If argument collapse is widespread, platforms that rely on LLMs to generate or summarize debate content may systematically narrow the range of visible positions.","Fine-tuning models on datasets that explicitly reward topic-specific and concrete sub-arguments could be tested as one way to reduce the effect.","The flattening observed here may extend to other LLM-assisted writing tasks such as policy briefs or news summaries on contested issues."],"forward_implications":["Prompting LLMs to generate diverse answers adds variation but recovers only about half of the distinct human main arguments.","Much of the added variation from diversity prompts falls outside the observed human argument space.","LLMs reuse generalized and hedged sub-arguments while humans prefer more concrete and topic-specific ones.","LLM-generated essays tend to follow a more fixed arc, often opening with a direct claim and moving quickly toward proposals."],"fun_headline_variants":["LLMs reach only 3.4% unique main arguments in debates","3.4% of LLM arguments unique humans at 65.3% in same debates","Argument collapse shown in 3.4% unique LLM debate arguments","LLMs produce 3.4% unique arguments in debates vs 65.3% for humans"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The method used to extract and count unique main arguments and sub-arguments accurately reflects genuine diversity differences rather than artifacts of the extraction process or prompt design.","fun_headline_variants_meta":{"raw":{"variants":["LLMs reach only 3.4% unique main arguments in debates","3.4% of LLM arguments unique humans at 65.3% in same debates","Argument collapse shown in 3.4% unique LLM debate arguments","LLMs produce 3.4% unique arguments in debates vs 65.3% for humans"]},"model":"grok-4.3","cost_usd":0.00769,"raw_usage":{"total_tokens":3572,"prompt_tokens":777,"num_sources_used":0,"completion_tokens":86,"cost_in_usd_ticks":76899500,"prompt_tokens_details":{"text_tokens":777,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2709,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":777,"tokens_out":86,"duration_ms":22733,"temperature":1.0,"reasoning_tokens":2709,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T15:06:08.952971+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running the same comparison with an independent argument extraction pipeline that yields uniqueness rates close to the reported 65.3 percent for humans and 3.4 percent for LLMs would support the claim; finding no reliable difference between the two would falsify it.","supporting_citations":[],"review_version":1}