{"id":"59708583-a420-4705-a6a5-a44a76be1979","arxiv_id":"2606.19591","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"A BART-based hierarchical approach with golden-summary-driven document shortening achieves ROUGE2-F1 of 0.2468 on the VLSP 2022 Vietnamese multi-document summarization task and releases additional training data.","lead":"The paper describes a hierarchical BART model for Vietnamese multi-document abstractive summarization that shortens each document using the golden summary to guide condensation before aggregation. A smart generalist might read it to see how NLP techniques are adapted for low-resource languages and data scarcity.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Golden-summary-driven shortening requires references at training and risks leakage into the hierarchical pipeline","rationale":"The reader's weakest assumption correctly isolates the single assumption whose violation would most directly undermine the reported score. Because the abstract presents the golden-summary strategy as the key novelty and no counter-evidence (e.g., ablation without references) is supplied in the provided text, the concern remains load-bearing even after acknowledging the external-data contribution.","tokens_in":1627,"tokens_out":317,"duration_ms":13124,"concrete_test":"Re-implement the shortening stage using only source-document signals (e.g., TF-IDF sentence ranking or fixed-length truncation) instead of golden-summary guidance; retrain the full pipeline on the same VLSP training split plus external data and re-evaluate ROUGE2-F1 on the public test set. A drop exceeding 0.02 relative to 0.2468 indicates the headline result depends on reference leakage.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (ROUGE2-F1 = 0.2468) depends on a hierarchical pipeline whose first stage shortens each document using the golden summary to enforce correlation with the final summary. This construction is only feasible when reference summaries exist at training time; at inference only raw documents are available. If the shortening step injects reference-derived signals into the condensed inputs seen by the second-stage BART aggregator, the reported score reflects a training regime unavailable in standard multi-document settings and may not be reproducible without the same leakage.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes a hierarchical BART-based approach for Vietnamese abstractive multi-document summarization. It introduces a novel strategy to shorten documents driven by the golden summary to ensure high correlation between the condensation and aggregation stages. The method achieves a ROUGE2-F1 score of 0.2468 on the VLSP 2022 public test set and releases additional data from external sources for the community.","tokens_in":1731,"tokens_out":451,"duration_ms":20780,"significance":"If the reported performance holds without reference leakage in the pipeline, this work provides a practical method and additional resources for Vietnamese MDS, which is an under-resourced area. The hierarchical approach is standard, but the specific shortening strategy could be a contribution if properly validated. However, the absence of baselines, ablations, and detailed training information makes it difficult to assess the significance of the result.","major_comments":[{"comment":"The document-shortening strategy driven by the golden summary (described in the hierarchical approach section) requires reference summaries at training time. This risks introducing leakage into the condensed documents used for the second-stage BART aggregator, as the shortening enforces correlation using information unavailable at inference. The central ROUGE2-F1 claim of 0.2468 depends on this construction, which may not generalize to standard settings without references.","section":"Proposed Method"},{"comment":"No baselines, ablation studies, error analysis, or comparisons to other methods are reported despite the claim of effectiveness for the hierarchical strategy. Without these, the ROUGE2-F1 score of 0.2468 cannot be contextualized and the contribution of the proposed shortening method remains unverifiable.","section":"Experiments"}],"minor_comments":[{"comment":"The abstract states that external sources are used for extra data but provides no details on collection, filtering, or how the additional data is integrated into training.","section":null}],"recommendation":"major_revision","confidential_remarks":"This appears to be a short technical report focused on a shared task; it may have limited novelty for a full journal article and the citation pattern is minimal."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on our technical report. We address each major comment below and commit to revisions that clarify the method and strengthen the experimental section.","responses":[{"response":"We acknowledge the concern regarding potential reference leakage. The golden-summary-driven shortening is used only at training time to construct condensed documents that exhibit high overlap with the target summaries, thereby providing stronger supervision signals for the second-stage aggregator. At inference, document shortening is performed in a reference-free manner using the first-stage BART model itself to select and condense content. We agree that the original manuscript does not sufficiently distinguish these phases or validate the inference procedure. In revision we will expand the method section with an explicit description of the reference-free inference pipeline and report additional ROUGE scores obtained under that setting to demonstrate generalization.","revision_made":"yes","referee_comment":"[Proposed Method] The document-shortening strategy driven by the golden summary (described in the hierarchical approach section) requires reference summaries at training time. This risks introducing leakage into the condensed documents used for the second-stage BART aggregator, as the shortening enforces correlation using information unavailable at inference. The central ROUGE2-F1 claim of 0.2468 depends on this construction, which may not generalize to standard settings without references."},{"response":"We agree that the lack of baselines and ablations limits the ability to assess the contribution of the shortening strategy. The revised manuscript will include (i) a baseline hierarchical BART system without the proposed shortening, (ii) ablation results isolating the effect of golden-summary-driven shortening, and (iii) an error analysis of generated summaries. Where possible we will also compare against other published Vietnamese summarization approaches.","revision_made":"yes","referee_comment":"[Experiments] No baselines, ablation studies, error analysis, or comparisons to other methods are reported despite the claim of effectiveness for the hierarchical strategy. Without these, the ROUGE2-F1 score of 0.2468 cannot be contextualized and the contribution of the proposed shortening method remains unverifiable."}],"tokens_in":1289,"tokens_out":449,"duration_ms":27484,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper applies a hierarchical BART pipeline to Vietnamese multi-document abstractive summarization. Each document is first shortened using the golden summary as guidance, then the shortened versions are aggregated into a final summary. They report ROUGE-2 F1 of 0.2468 on the VLSP public test set and release additional scraped data to increase training volume.\n\nWhat is actually new is the language application plus the data release. Vietnamese MDS has limited prior resources, so extra training examples and a concrete score on the shared task test set give the community something to start from. The shortening tactic is presented as a simple way to maintain correlation across stages.\n\nThe work is straightforward and the data contribution is useful on its own. Releasing the extra corpus is a clear positive that others working on low-resource summarization can use directly.\n\nThe soft spots are more central. No baselines, no ablations, and no training details appear in the description, so the single reported score cannot be judged against standard hierarchical BART or other Vietnamese systems. The shortening method itself depends on golden summaries being available during the first stage. That information will not exist at inference, which creates a mismatch and possible leakage of reference signals into the condensed inputs. Without explicit handling of this gap or experiments showing the effect, the result is hard to trust as a general method. The abstract also gives no error analysis or comparison to simpler pipelines.\n\nThis is mainly for researchers already focused on Vietnamese NLP or the VLSP shared task who want the extra data and a quick baseline number. A broader reader gets little beyond the observation that hierarchical BART can be run on this language with some extra data. The experimental gaps are large enough that the paper does not yet support strong claims about the strategy's effectiveness.\n\nI would not send it for peer review in this form. It needs the missing comparisons, implementation details, and a direct test of whether the shortening step works without reference guidance before it can be evaluated properly.","headline":"The golden-summary shortening step creates a training-inference gap that undercuts the reported ROUGE score and leaves the work too thin on details for serious evaluation.","tokens_in":2177,"tokens_out":477,"would_cite":false,"duration_ms":17792,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A BART-based hierarchical approach using golden-summary-driven document shortening improves Vietnamese multi-document abstractive summarization.","keywords":["Vietnamese","multi-document summarization","abstractive summarization","BART","hierarchical strategy","VLSP","document shortening"],"falsifier":"A controlled experiment where the model is trained and tested using only automatically generated or no-reference shortening and the ROUGE2-F1 score is compared to the reported value.","tokens_in":2525,"feed_emoji":"📄","tokens_out":423,"duration_ms":34465,"temperature":0.7,"pith_summary":"The paper proposes a method for Vietnamese multi-document abstractive summarization by first shortening individual documents using information from the reference summary, then aggregating and summarizing the condensed versions with a BART model. This hierarchical strategy aims to maintain high correlation between the condensation and final summarization stages. The approach also incorporates additional data from external sources to increase training data volume. A sympathetic reader would care because multi-document summarization in Vietnamese is a challenging task with limited resources, and this method demonstrates a way to achieve fluent outputs.","feed_headline":"Golden-summary shortening enables Vietnamese multi-doc summarization","feed_subtitle":"Hierarchical BART condenses documents using reference info before final aggregation on VLSP data.","key_machinery":"The hierarchical approach of condensing each document before aggregation and summarization, with the condensation step guided by the golden summary to ensure relevance.","core_discovery":"The authors establish that their BART-based model with a hierarchical strategy, where documents are shortened in a manner driven by the golden summary, reaches a ROUGE2-F1 score of 0.2468 on the VLSP public test set while generating fluent and concise summaries. They further show that augmenting the dataset with external sources significantly increases the available training data for this task.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Golden summaries shorten Vietnamese docs for BART hierarchy","BART hierarchy reaches 0.2468 ROUGE2-F1 on VLSP Vietnamese data","Hierarchical BART condenses Vietnamese documents with golden info","Vietnamese multi-doc summarization uses BART and reference shortening"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The document shortening strategy assumes that golden reference summaries are available at training time to select which content to keep.","fun_headline_variants_meta":{"raw":{"variants":["Golden summaries shorten Vietnamese docs for BART hierarchy","BART hierarchy reaches 0.2468 ROUGE2-F1 on VLSP Vietnamese data","Hierarchical BART condenses Vietnamese documents with golden info","Vietnamese multi-doc summarization uses BART and reference shortening"]},"model":"grok-4.3","cost_usd":0.004319,"raw_usage":{"total_tokens":2124,"prompt_tokens":578,"num_sources_used":0,"completion_tokens":72,"cost_in_usd_ticks":43187000,"prompt_tokens_details":{"text_tokens":578,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1474,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":578,"tokens_out":72,"duration_ms":11551,"temperature":1.0,"reasoning_tokens":1474,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T20:37:15.841476+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A controlled experiment where the model is trained and tested using only automatically generated or no-reference shortening and the ROUGE2-F1 score is compared to the reported value.","supporting_citations":[],"review_version":1}