{"id":"14174392-43f5-45d6-a493-7adcb35e69dc","arxiv_id":"2506.06930","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DiscoSum pairs news articles with cross-platform human summaries and shows that beam search guided by a discourse labeler produces summaries that better match a target sentence structure.","lead":"This paper introduces DiscoSum, a collection of over 20,000 news articles paired with more than 100,000 social media and newsletter posts, plus a five-label discourse schema and a beam search method that generates summaries matching a desired sentence structure. If validated, it gives summarization researchers a new benchmark and a simple decoding trick for controlling how a news story is retold across platforms.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Automatic structural scores are circular: beam search optimizes the same labeler C(·) used to compute Match Score, so the 0.72 result is partly by construction; human de-circularization is too small and depends on an unvalidated random-label schema.","rationale":"Good-faith reading: the paper's contribution is a controllable summarization method plus a dataset. For the central claim to hold, the automatic and human structural evaluations must measure something real and not be satisfiable by the generator's own reward. The weak point is Sections 5.2/5.4 combined with Algorithm 1: the same labeler C(·) both selects sentences (score←C(s′,t_i)) and computes the automatic Match Score (L=Labeler(S)). Because the algorithm is explicitly maximizing C's agreement with T, high automatic Match Score is expected even if C is a poor or arbitrary discourse model; it only shows the optimizer can game its scorer. I therefore disagree slightly with the reader's emphasis: the more immediate problem is not only whether C is cross-culturally faithful, but that the headline metric is definitionally coupled to the optimization target. The human evaluation is the intended de-circularization, but it is small and reuses the same five-label taxonomy. If that taxonomy is not independently validated (κ=0.615, two journalists), human structural Match Score is not strong evidence of 'narrative fidelity.' The random target sequences in Section 5.1 further make the task one of arbitrary constraint satisfaction. The paper does provide a useful new dataset and transparent methods, and the beam search may still be a sound controllable-generation technique; the issue is that this version does not yet demonstrate it. Hence the conditional verdict remains appropriate, still pending independent evaluation with a held-out labeler or larger human study and code/data release.","tokens_in":18686,"tokens_out":6372,"duration_ms":68028,"concrete_test":"Ask two fresh annotators (or a second labeler trained on the existing 500 human-labeled sentences) to label a held-out set of 500 generated summaries from vanilla and beam search, without access to the target label sequences. Recompute Match Score, Levenshtein distance, and LCS with these independent labels and report annotator agreement. If the beam-search advantage over vanilla shrinks or reverses, the reported 0.72 automatic Match Score is largely a reward-hacking artifact of optimizing C. A cheaper auxiliary check: apply C to the candidate pool of a vanilla generation and re-rank sentences by predicted label match to T; if re-ranking alone reproduces most of the structural gain, no new decoding method is needed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 5.4, the headline result is that the beam-search variant achieves automatic Match Score 0.72 versus 0.21 for vanilla LLaMa-3-8B. But Algorithm 1 (line 8) scores each candidate summary with the same classifier used to define the automatic structural metrics in Section 5.2: L = Labeler(s1,...,sn) and Match Score compares L to the target T. The beam search selects sentences specifically because C predicts the requested label, so the automatic Match Score is substantially an artifact of optimizing the evaluator; it does not independently establish that the generated summaries have recognizable discourse organization. The two checks that could break the circularity are weak. The human structural evaluation in Table 3 uses the same five-label schema whose validity is supported only by two journalists and κ=0.615 (Section 4.1); if the schema is arbitrary, human agreement measures conformity to an arbitrary taxonomy. Second, Section 5.1 states that target label sequences were randomly generated for each experiment, so the task evaluated is matching a random label string rather than producing the structure a reader or editor would regard as natural. The supporting claim that structure is enforced 'without sacrificing factual alignment' is also mixed: in Table 3, beam search's ROUGE-L (42.98) is below vanilla (47.18), and its AlignScore (0.3890) is only marginally above several proprietary baselines (0.3882–0.3888). Thus the central empirical claim rests on a partly circular automatic metric, a small human sample (100 summaries per model, two annotators), and an unvalidated schema.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DiscoSum, a dataset of 20,811 news articles matched with multiple social-media and newsletter summaries, together with a five-label discourse schema automatically induced from LLM-generated labels and k-means clustering. The authors propose a sentence-level beam search decoding method, DiscoSum, that uses a trained discourse labeler C(·) to select sentences matching a user-specified target label sequence T. They report automatic and human evaluations claiming that the beam search variant of LLaMa-3-8B achieves the best structural alignment (automatic Match Score 0.72, human Match Score 0.55) while maintaining competitive factual alignment (AlignScore 0.3890). The paper also provides baseline comparisons with proprietary and open-source models.","tokens_in":19057,"tokens_out":8053,"duration_ms":79204,"significance":"DiscoSum is a potentially valuable resource for studying cross-platform news summarization, and the sentence-level beam search method offers a practical test-time approach to controllable structure. The human evaluation and human validation of the schema provide a partial de-circularization of the evaluation. However, the automatic structural metrics are computed with the same labeler used as the decoding reward, the target label sequences are randomly generated, and the dataset statistics contain internal inconsistencies. If these issues are resolved, the dataset and method could support future work in structure-aware summarization.","major_comments":[{"comment":"The automatic structural metrics (Match Score, Levenshtein distance, LCS) are computed using the same labeler C(·) that serves as the beam search reward in Algorithm 1 (line 8, score←C(s′, ti)). Consequently, the beam search's automatic Match Score of 0.72 is substantially an artifact of optimizing the evaluator; it does not independently establish that the generated summaries have recognizable discourse organization. The paper should de-circularize the evaluation by using a separately trained or independently validated labeler for automatic scoring, or by making the human structural scores the primary evidence. In addition, the human evaluation should be expanded (more annotators, more summaries) and inter-annotator agreement should be reported.","section":"§5.2 and Algorithm 1, line 8"},{"comment":"The abstract and Section 1 claim \"20k news articles ... multiply paired with over 100k human-written summaries,\" but Table 2 reports only 45,195 matched article-summary pairs, while Table 1's post counts sum to 103,788. The split description states a 70%/20%/10% split into \"14k/4k/2k article-summary pairs,\" which sums to 20k, not 45k. Please clarify whether the split is at the article level or the pair level, and reconcile these counts, as the current numbers make the dataset statistics unreliable.","section":"§3.2, Tables 1 and 2"},{"comment":"For each generation, the authors \"randomly generate a list of structural tags\" as the target label sequence. This means the evaluation measures how well the model matches an arbitrary label string rather than a discourse structure a reader or editor would consider natural. The paper's claim that beam search \"consistently aligns more closely with the designated discourse label sequences\" is thus about matching random sequences. Please evaluate on target sequences derived from the actual human-written summaries in the dataset (e.g., label sequences produced by applying the labeler to reference summaries) and report those results, to demonstrate practical relevance.","section":"§5.1"},{"comment":"The claim that beam search \"maintains competitive performance in surface-level metrics\" and \"achieves the highest AlignScore (0.3890)\" is not well-supported. The beam search's ROUGE-L (42.98) is lower than vanilla LLaMa-3-8B (47.18), and the AlignScore advantage over proprietary models (0.3882–0.3888) is within 0.001, likely not statistically significant. Please report confidence intervals or significance tests, and discuss the tradeoff between structural alignment and content overlap more carefully.","section":"§5.4, Table 3"}],"minor_comments":[{"comment":"The notation \"score←C(s′, ti)\" is ambiguous because C is defined as a sentence-level classifier; please clarify whether it is applied to the newly generated candidate sentence c or to the last sentence of s′, and then compared with ti.","section":"Algorithm 1, line 8"},{"comment":"The training labels for the discourse labeler are not described. It is unclear whether the classifier is trained on the LLM-generated labels, the cluster assignments from k-means, or the manual annotations. Please specify the annotation protocol for the training data.","section":"§4.2"},{"comment":"The human preference evaluation reports MRR but no inter-annotator agreement or statistical significance. Also, if the same two annotators who validated the schema also performed the preference rankings, that could introduce bias; please clarify the annotator identities.","section":"§5.5"},{"comment":"The text says \"we ask annotators to evaluate 100 summaries for each model,\" but Table 3 lacks human structural scores for O1; please clarify which models were included in the human evaluation and why O1 was excluded.","section":"§5.2"},{"comment":"The schema validity is supported by κ=0.615 between two journalists, which is moderate agreement. The paper should discuss the limitations of this validation, especially because the same schema underpins both the decoding reward and the evaluation metrics.","section":"§4.1"},{"comment":"There are minor typos in the prompts, e.g., \"{dis course_la bels}\" appears in the definitions prompt; these should be fixed for clarity.","section":"Appendix D.4"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a useful dataset and a promising method, but the circularity of the automatic evaluation is a load-bearing issue that must be addressed. The dataset statistics also need to be reconciled. I would encourage the authors to release the dataset and code to facilitate replication and to consider evaluating on reference-derived target sequences in addition to random ones. The paper fits the journal's scope but requires substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The DiscoSum dataset is the reason to read this paper. A one-to-many collection of professionally written news summaries across four platforms, with 20k articles and over 100k summaries, is genuinely new and likely useful for anyone working on platform-adaptive summarization or studying how journalistic structure shifts across media. The schema induction via LLM labels plus clustering is a reasonable approach, and the authors deserve credit for explicitly acknowledging limitations and validating with journalist annotators, even if that validation is thin.\n\nThe method, however, is mostly FUDGE-style test-time alignment applied at the sentence level to a discourse labeler. That is fine as a baseline, but not a major algorithmic novelty. The bigger problem is that the automatic structural metrics are circular: the beam search selects sentences using the same labeler that computes Match Score, Levenshtein, and LCS. So the 0.72 Match Score is partly by construction. The human structural evaluation partially breaks the loop, but 100 summaries per model with two annotators is small, and the schema itself rests on only two journalists and κ=0.615. If the schema is arbitrary, human agreement measures conformity to an arbitrary taxonomy. The paper also has internal inconsistencies: the abstract says over 100k summaries, Table 1 counts 103,788 posts, but Table 2 reports only 45,195 matched pairs. The split counts (14k/4k/2k article-summary pairs) do not clearly align with the 20,811 articles or the matched-pair count. And the claim that structure is enforced without sacrificing factual alignment is shaky: ROUGE-L drops from 47.18 to 42.98 for beam search, though AlignScore does improve slightly.\n\nThese are addressable issues, not fatal ones. The authors should release the data and code, redo the automatic structural evaluation with a held-out labeler or human-annotated gold labels, clarify the counts, and report the schema validation more carefully. As it stands, the dataset remains a solid contribution, but the empirical claims about the method need de-circularization before they can be taken at face value.\n\nThis paper deserves serious peer review. It introduces a valuable resource and a clearly defined task, and the flaws are fixable. I would recommend accepting it for review with a strong request for major revisions and dataset release.","headline":"The dataset is the real contribution; the decoding method is a recycled idea with a circular evaluation problem that needs fixing before the empirical claims can be trusted.","tokens_in":19569,"tokens_out":1424,"would_cite":true,"duration_ms":18673,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Discourse-guided beam search can enforce a target summary structure without sacrificing factual alignment.","keywords":["news summarization","discourse structure","discourse-aware summarization","beam search","controllable generation","social media summaries","summarization dataset","discourse schema"],"falsifier":"Take summaries from outlets or platforms absent from DiscoSum, have independent annotators label each sentence with the five discourse roles, and measure agreement with the schema; if agreement is near chance or well below the reported inter-annotator $\\kappa=0.615$, the labeler is not a trustworthy structural oracle and both the beam-search reward and the structural metrics lose their intended meaning.","tokens_in":18511,"feed_emoji":"📰","tokens_out":10002,"duration_ms":90265,"temperature":0.7,"pith_summary":"DiscoSum claims that news summarization can be made structure-aware by decoding with a sentence-level beam search scored by a discourse labeler. The paper introduces a dataset of 20k news articles paired with more than 100k human-written summaries across Facebook, Instagram, Twitter, and newsletters, plus a five-label discourse schema (Introductory Elements, Contextual Details, Event Narration, Source Attribution, Engagement Directive). On both automatic and human evaluations, the beam-search variant of a fine-tuned language model matches target discourse label sequences better than vanilla or edit-based baselines, while staying factually aligned with the source. The central payoff is that a user-supplied label sequence, mirroring how newsrooms already write for different platforms, can be enforced at generation time without sacrificing content accuracy.","feed_headline":"Beam search with discourse labels matches target summary structure","feed_subtitle":"Five discourse labels plus sentence-level beam search keep summaries structurally on-target and factually aligned.","key_machinery":"The mechanism that carries the argument is a sentence-level discourse labeler $C(\\cdot)$ trained on the DiscoSum train split, used as a reward inside a beam search: at each position $i$, the generation model proposes $k$ candidate sentences and the decoder keeps the one maximizing $C(s', t_i)$ against the target label $t_i$. The five-label schema is itself induced automatically by asking an LLM to label each summary sentence, embedding those labels with SBERT, and clustering with k-means. This same labeler is later used to compute the structural evaluation metrics, so it is simultaneously the oracle for generation and the ruler for measuring success.","core_discovery":"The paper's central claim is that structural fidelity and factual fidelity can be achieved together in news summarization. Its sentence-level beam search generates several candidate sentences for each position in a user-supplied discourse-label sequence, keeps the candidate whose predicted label matches the target, and thereby produces summaries that align with the designated structure while remaining factually consistent with the source. On automatic evaluation the beam-search variant of a fine-tuned 8-billion-parameter language model reaches a Match Score of 0.72 and the highest AlignScore (a factual-consistency metric) among the compared systems at 0.3890; human annotators give it a structural Match Score of 0.55 and rank it first with a mean reciprocal rank of 0.71. The authors interpret this as evidence that decoding-time guidance by a discourse labeler is a viable alternative to post-hoc editing or fine-tuning for structure-aware summarization.","pith_inferences":["The authors leave the prediction of the target label sequence from the source article for future work; one testable extension is to learn a platform-specific editor model that outputs the label sequence, turning DiscoSum into a full end-to-end structure-aware summarization system.","Because the same labeler is used to score candidates during beam search and to evaluate the final summaries, the reported Match Scores partly measure self-consistency; an independent annotation study on final outputs would separate genuine structural control from metric circularity.","The dependence on LLM-generated candidates means the beam search inherits whatever stylistic diversity the base model's sampling has; a low-diversity sampling distribution could make all $k$ candidates near-identical and reduce the beam to a single path.","The five-label schema was validated by only two journalists, so cross-cultural generalizability is untested; a natural next step is to check whether the same clusters emerge when applying the schema-induction procedure to non-Western outlets."],"forward_implications":["If the result holds, newsrooms can supply a platform template (for example, LinkedIn's professional framing versus Instagram's engagement-first order) as a label sequence and the decoder will shape the summary accordingly.","Because the beam search improves AlignScore as well as structural Match Score, structure-aware decoding appears to help rather than hurt factual consistency relative to vanilla generation.","Larger beams monotonically improve structural alignment (rising LCS, falling Levenshtein distance), suggesting that computational cost can be traded for structural fidelity.","The DiscoSum dataset's one-to-many article-summary pairings enable future work on predicting a target structure from the source article and platform rather than requiring the user to supply the label sequence.","The approach may transfer to other structured summarization settings, such as legal, medical, or dialogue summarization, wherever a discourse labeler can be trained."],"supporting_citations":[{"why":"Supplies the Wayback Machine scraping and LLM-based HTML cleaning pipeline that builds the news article side of DiscoSum.","marker":"(Spangher et al., 2024a)"},{"why":"Validates the LLM pairwise comparison method used to match articles to summaries across outlets.","marker":"(Spangher et al., 2024b)"},{"why":"Provides the sentence-level discourse classification approach used to train the labeler $C(\\cdot)$.","marker":"(Spangher et al., 2021, 2022a)"},{"why":"Supplies the prompt-based LLM labeling and clustering procedure used to induce the five-label discourse schema.","marker":"(Pham et al., 2024)"},{"why":"Provides the SBERT embeddings used both to find candidate article-summary matches and to embed labels for k-means clustering.","marker":"(Reimers, 2019)"},{"why":"Establishes the future-discriminator test-time alignment paradigm that the sentence-level beam search operationalizes.","marker":"(Yang and Klein, 2021)"},{"why":"Motivates the existence of canonical news discourse structures that a target label sequence can represent.","marker":"(Van Dijk, 1988)"}],"fun_headline_variants":["Discourse labels steer beam search to summary structure","Beam search with discourse labels matches target structure","DiscoSum: discourse-aware beam search for news summaries","Structure-aware beam search hits summary targets in news"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatically induced five-label discourse schema and the labeler trained on it faithfully capture how news summaries are organized across platforms and cultures, rather than imposing a dataset-specific taxonomy.","fun_headline_variants_meta":{"raw":{"variants":["Discourse labels steer beam search to summary structure","Beam search with discourse labels matches target structure","DiscoSum: discourse-aware beam search for news summaries","Structure-aware beam search hits summary targets in news"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000568,"raw_usage":{"total_tokens":2643,"prompt_tokens":852,"completion_tokens":1791,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":468,"completion_tokens_details":{"reasoning_tokens":1742}},"tokens_in":468,"tokens_out":1791,"duration_ms":15421,"temperature":1.0,"reasoning_tokens":1742,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:45:48.200193+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take summaries from outlets or platforms absent from DiscoSum, have independent annotators label each sentence with the five discourse roles, and measure agreement with the schema; if agreement is near chance or well below the reported inter-annotator $\\kappa=0.615$, the labeler is not a trustworthy structural oracle and both the beam-search reward and the structural metrics lose their intended meaning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the sentence-level discourse classification approach used to train the labeler $C(\\cdot)$."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the prompt-based LLM labeling and clustering procedure used to induce the five-label discourse schema."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the existence of canonical news discourse structures that a target label sequence can represent."}],"review_version":1}