{"id":"799626cc-9b8d-427b-a596-9819569f8de4","arxiv_id":"2505.06862","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"On arXiv and BigPatent documents longer than 20,000 tokens, the SPIN 3 variant, which splits documents into 4,096-token chunks and selects the best chunk summary by ROUGE-L, improves ROUGE scores over a truncated BIGBIRD-PEGASUS baseline, though the evaluation setup is not fully controlled.","lead":"This paper proposes SPIN, a split-then-join method that fine-tunes BIGBIRD-PEGASUS on chunks of very long documents and then joins or selects per-chunk summaries. It reports ROUGE gains over a truncated baseline on arXiv and BigPatent documents longer than 20,000 tokens. The result offers a lightweight alternative to truncation, but the evaluation lacks a train/test split and error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No held-out evaluation is reported, so Table 3's claimed SPIN 3 gains could reflect memorization rather than generalization; the central comparison is not yet supported.","rationale":"The reader's stated weakest_assumption emphasizes cross-chunk independence, which is a legitimate transfer concern, but the more immediate and falsifiable problem is that Table 3 is the only evidence for the central claim and it is not attached to any described evaluation protocol. Section 4.1 filters documents, Section 4.2 describes training on those same datasets, and no held-out split, number of seeds, or error bars are reported. With a 576M-parameter model and small filtered sets, in-sample evaluation can produce high ROUGE through memorization, and the SPIN 3 source-overlap selection rule may amplify this effect. This is a correctness risk in the evidence, not a disagreement with common consensus. The method may still be salvageable, but the paper as written does not demonstrate the claimed superiority. I therefore keep the reader's reject verdict; a clean held-out rerun is the decisive check.","tokens_in":7380,"tokens_out":7763,"duration_ms":81487,"concrete_test":"Rerun the Table 3 pipeline on the >20k-token filtered subsets with a documented 80/10/10 split, training SPIN 3 and the truncated BIGBIRD-PEGASUS baseline on identical folds with at least three seeds, and report mean±std ROUGE on the held-out folds. If SPIN 3's held-out ROUGE-1 margin over baseline is within one standard deviation or reverses, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the Section 5 assertion that 'SPIN 3 is better than SPIN 1, SPIN 2 and BIGBIRD-PEGASUS,' supported only by Table 3. For that claim to hold, Table 3 must measure performance on documents the model did not train on. The manuscript never describes a train/test split: Section 4.1 filters arXiv/BigPatent to documents over 20,000 tokens, and Section 4.2 says the model was fine-tuned on those datasets, with no held-out set, seed count, or error bar. With only 3,533 arXiv and 6,589 BigPatent documents and a 576M-parameter model, scoring on the training documents can reflect memorization rather than summarization ability. The SPIN 3 inference selection rule (choose the chunk summary with the highest ROUGE-L against its source chunk) may also favor chunks whose generated outputs are most extractive, which can interact with memorization to inflate ROUGE. The baseline's training data and truncation procedure are under-specified, so the comparison in Table 3 is not independently reproducible. This is a correctness risk in the evidence for the central claim, not a mere stylistic issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes SPIN (Split-then-Join), an approach for abstractive summarization of very long documents whose length exceeds the 4,096-token input limit of BIGBIRD-PEGASUS. Documents are split into 4,096-token parts in training; three variants pair document parts with either split summary parts selected by ROUGE-L (SPIN 1) or with the full target summary (SPIN 2 and SPIN 3). At inference, SPIN 1 and SPIN 2 concatenate per-part summaries, while SPIN 3 selects the per-part summary with the highest ROUGE-L against its source chunk. The authors filter arXiv and BigPatent to documents over 20,000 tokens, fine-tune BIGBIRD-PEGASUS on the filtered sets, and report ROUGE-1/2/L scores in Table 3, claiming that SPIN 3 outperforms SPIN 1, SPIN 2, and the BIGBIRD-PEGASUS truncation baseline.","tokens_in":7565,"tokens_out":3078,"duration_ms":32644,"significance":"If the reported gains were obtained on held-out documents, the paper would offer a remarkably simple recipe for low-resource very-long-document summarization: fine-tune on chunked data and select the best chunk summary by ROUGE-L. The main strengths are the clearly stated splitting scheme, the public source-code link, and the use of two standard long-document datasets. However, the significance is conditional because the experimental protocol does not establish generalization: the manuscript never describes a train/test split, and the ROUGE-L selection rule is part of both the training-pair construction and the inference-time choice, while ROUGE is also the evaluation metric. As submitted, the central empirical claim is not supported.","major_comments":[{"comment":"No train/test split is described anywhere in the manuscript. Section 4.1 filters arXiv and BigPatent to documents over 20,000 tokens, and Section 4.2 states that the model was fine-tuned on those datasets, but there is no statement of a held-out evaluation set, seed count, or error bars. With only 3,533 arXiv and 6,589 BigPatent documents and a 576M-parameter model, scores computed on the training documents could reflect memorization rather than summarization ability. The central claim in Section 5 that SPIN 3 is better than the baselines therefore lacks the required evidence. The authors must specify the exact split, report results on the held-out portion, and provide variance estimates.","section":"Section 4, Table 3"},{"comment":"ROUGE-L is embedded twice in the pipeline: Algorithm 1 uses ROUGE-L recall to pair document parts with summary parts in training, and SPIN 3 uses ROUGE-L at inference to select the final summary, while the evaluation in Table 3 is also based on ROUGE. This does not force the reported numbers by construction, but it means the inference-time selection criterion is the same metric family used for evaluation, which can inflate scores by selecting the most extractive or most memorized chunk summary rather than the most informative one. The authors should evaluate with a metric or protocol that is not used in selection, and should report abstractiveness or overlap statistics to show the selected summaries are not simply extractive fragments.","section":"Section 3.2, Algorithm 1, Eq. (1), Table 3"},{"comment":"The description of SPIN 2 and SPIN 3 is inconsistent with Algorithm 1. Section 3.1.2 says that in SPIN 2 and SPIN 3 the target summary is not split and each document part is paired with the full target summary, but Algorithm 1 always splits the target summary into `n_parts` pieces and pairs each document part with a split summary part using ROUGE-L. As written, the algorithm cannot reproduce SPIN 2 or SPIN 3. Furthermore, Algorithm 1 uses undefined variables: `K` in line 5, `nparts` in line 9, and `lk` is computed but not used. The manuscript must provide a precise, self-contained description of each variant so the experiments are reproducible.","section":"Section 3.1.2, Section 3.2, Algorithm 1"},{"comment":"The explanation for SPIN 2's poor performance contradicts the method definition. Section 5 says 'by splitting summaries with a fixed length equal to the number of document parts make the document-summary pair become unrelevant,' but Section 3.1.2 states that SPIN 2 does not split the target summary. This contradiction must be resolved, and the actual training procedure for the baseline ('standard training') is also under-specified: the manuscript does not state how truncation was applied, how many epochs were run, what learning rate and optimizer were used, or whether the baseline received the same augmented data. Without this information, the comparison in Table 3 is not independently reproducible or verifiable.","section":"Section 5, Table 3"}],"minor_comments":[{"comment":"There are several typos and formatting errors, including 'V ery' in the title, 'reasearch', 'it's', and 'respectivelly'. These should be corrected.","section":"Title and Abstract"},{"comment":"The caption reads 'Result comparison on dataset BigPatent' but the table reports results for both arXiv and BigPatent. The caption should name both datasets.","section":"Table 3 caption"},{"comment":"The statement that 'only arXiv and BigPatent contain enough samples' among summarization datasets is unsupported; a citation or a brief survey of dataset statistics is needed.","section":"Section 4.1"},{"comment":"The inference procedure is described only in prose; a pseudocode block for the generation and selection steps of SPIN 1, SPIN 2, and SPIN 3 would improve clarity and reproducibility.","section":"Section 3.2"},{"comment":"Some references lack page ranges or are formatted inconsistently (for example, the entry for Nenkova 2011 and the Radford et al. entry). The reference list should be checked against the venue's style.","section":"References"}],"recommendation":"reject","confidential_remarks":"The central empirical claim is not supported because the manuscript reports no evidence that the evaluated documents were held out from training. Even if the authors intend to add such a split, the algorithmic description of SPIN 2/3 is also internally inconsistent, so the experiments as reported are neither valid nor reproducible. I recommend reject rather than major revision, because the required fix is not a local correction but a new, properly controlled experimental study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has a real idea buried under an unsupported evaluation. SPIN 3, selecting the chunk summary with highest ROUGE-L against its source chunk, is a simple but sensible extension of DANCER; the paper cites DANCER and frames itself as an adaptation. The empirical comparison of three splitting strategies on very long (>20k token) documents is new as far as I know. The motivation is practical: BIGBIRD-PEGASUS caps at 4,096 tokens and truncation is a poor default.\n\nWhat the paper does well: the split-then-augment idea is a reasonable low-resource answer to domain shift, and the author is honest about building on DANCER. The code link is a plus.\n\nThe soft spots are load-bearing. The manuscript never says there is a held-out test set. Section 4.2 says the model was fine-tuned on the filtered arXiv and BigPatent documents; Table 3 then reports scores. With only a few thousand documents and a 576M-parameter model, those numbers could be memorization. No seeds, no error bars, no significance tests. The baseline (truncated BIGBIRD-PEGASUS) is not described in enough detail to know whether it was fine-tuned on the same data, with what hyperparameters, or on the same documents. That makes the central 'SPIN 3 is better' claim not independently verifiable.\n\nThere is also a circularity concern: ROUGE-L is used to pair summary parts to document parts (Algorithm 1), to select the final summary in SPIN 3, and ROUGE is the evaluation metric. That does not force the result, but it can inflate scores in ways the paper doesn't discuss. SPIN 1's ROUGE-1 drops to 11.7 from the 39.6 baseline, which suggests the part-pairing or training objective may be broken; the paper doesn't explain this. The pseudo-code has undefined variables (K, nparts), and the prose is full of typos.\n\nAs it stands, the numbers in Table 3 cannot be trusted. The paper is a draft, not a paper. For whom is it useful? Someone working on long-document summarization with BIGBIRD-PEGASUS might take the SPIN 3 selection rule as a trick to test. But the lack of a train/test split is a foundational problem, not a revision-level concern. I would not send this to peer review until the author reruns with held-out documents, multiple seeds, and ablations that separate augmentation from chunk selection. I'd tell the author to resubmit when that exists.","headline":"A sensible SPIN 3 selection rule is buried under an evaluation with no held-out split, so Table 3's gains are not yet evidence.","tokens_in":8170,"tokens_out":3158,"would_cite":false,"duration_ms":30346,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that splitting a very long document into 4,096-token chunks, summarizing each chunk, and selecting the chunk summary with the highest ROUGE-L score yields better whole-document summaries than truncating the document to…","keywords":["abstractive summarization","long documents","low-resource","BIGBIRD-PEGASUS","SPIN","ROUGE-L","document splitting","data augmentation"],"falsifier":"Take a set of very long documents whose gold summaries require combining facts from widely separated chunks, apply SPIN 3, and compare with the truncated baseline; if ROUGE scores on that set fall below truncation, the independence assumption fails.","tokens_in":7062,"feed_emoji":"📄","tokens_out":7167,"duration_ms":62708,"temperature":0.7,"pith_summary":"The paper tries to establish that the usual fix for very long documents—cutting them off at the model's 4,096-token limit—throws away information needed for a good summary. It proposes a split-then-join procedure, called SPIN, that divides a document into 4,096-token chunks, summarizes each chunk, and then joins or selects those chunk summaries. On documents longer than 20,000 tokens from two large summarization corpora, the variant that keeps the single chunk summary with the highest ROUGE-L score, SPIN 3, outperforms both the truncated BIGBIRD-PEGASUS baseline and the other SPIN variants. A sympathetic reader would care because the result suggests that important summary content is often not in the document's opening section, and that chunk-level selection can recover it without a model that processes the whole document at once.","feed_headline":"Chunk, summarize, and pick beats truncation for long documents","feed_subtitle":"Selecting the best chunk summary lifted ROUGE-1 over the truncation baseline on paper and patent datasets.","key_machinery":"The load-bearing mechanism is the split-then-join pipeline. During training, a document longer than 4,096 tokens is cut into chunks of at most 4,096 tokens, and the target summary is either split into the same number of parts or kept whole; each chunk is paired with the summary or summary part that gives the highest ROUGE-L recall, a longest-common-subsequence based overlap measure, producing many more training pairs from a small set. At inference, each chunk gets its own summary, and the variants differ only in how the chunk summaries are combined: SPIN 1 and SPIN 2 concatenate them, while SPIN 3 computes ROUGE-L between each chunk and its generated summary and keeps only the best one as the final summary. This selection rule is what carries the reported improvement.","core_discovery":"The central claim, stated the way the author would state it, is that SPIN 3 is better than SPIN 1, SPIN 2, and BIGBIRD-PEGASUS for very long documents. In the paper's experiments on documents with more than 20,000 tokens, SPIN 3 reaches ROUGE-1 of 41.7 on scientific papers and 35.6 on patents, against 39.6 and 23.0 for the truncated BIGBIRD-PEGASUS baseline. The reason offered is that the most important information is not always at the head of a document; it can sit in the middle or near the end, so choosing the best chunk-level summary recovers content that truncation discards. The paper also reports that SPIN 1, which splits the target summary into parts matched to chunks, and SPIN 2, which pairs every chunk with the full summary but concatenates outputs, both perform worse than SPIN 3.","pith_inferences":["The paper does not test whether chunks are truly independent; a natural extension would be to build summaries that require evidence from multiple chunks and measure whether SPIN 3 still beats truncation.","Selecting by ROUGE-L against the chunk may implicitly favor summaries that stay lexically close to the chunk, which could matter for abstractive quality even if ROUGE improves.","The same chunk-and-select recipe could be applied to other long-input encoders, not just the BIGBIRD-PEGASUS architecture, to see whether the gain transfers."],"forward_implications":["If SPIN 3 is right, truncation is not a harmless preprocessing step for very long documents; discarding everything past 4,096 tokens can cost several ROUGE points.","The data-augmentation effect of splitting document-summary pairs means a low-resource long-document setting can be trained from fewer original examples.","Chunk summaries can be produced independently and in parallel, so inference cost scales with the number of chunks rather than with full-document attention.","The choice of selection metric matters: picking by ROUGE-L outperforms concatenating all chunk summaries, which is the paper's direct comparison between SPIN 3 and SPIN 2."],"supporting_citations":[{"why":"Supplies the BIGBIRD-PEGASUS model and its 4,096-token input limit, the baseline that SPIN is compared against.","marker":"Zaheer et al., 2021"},{"why":"Provides the PEGASUS pretraining objective that BIGBIRD-PEGASUS builds on.","marker":"Zhang et al., 2020"},{"why":"Supplies the divide-and-conquer strategy of splitting document and summary that SPIN adapts.","marker":"Gidiotis and Tsoumakas, 2020"},{"why":"Provides the scientific-paper summarization dataset used for the very-long-document subset.","marker":"Cohan et al., 2018"},{"why":"Provides the patent summarization dataset used for the very-long-document subset.","marker":"Sharma et al., 2019"},{"why":"Supplies the domain-shift and overfitting concern for low-resource transfer learning that motivates the splitting augmentation.","marker":"Chen and Shuai, 2021"},{"why":"Provides the pretrained BIGBIRD-PEGASUS weights used for fine-tuning.","marker":"Wolf et al., 2020"}],"fun_headline_variants":["Split-then-join beats truncation for very long docs","Chunk selection lifts summarization over truncation","For long docs, pick best chunk summary, not head","SPIN 3: split-then-join outperforms truncation","Long-document summarization: splitting beats cutting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a very long document can be cut into independent 4,096-token chunks, and that picking the single chunk summary with the highest ROUGE-L against its chunk also gives the best whole-document summary.","fun_headline_variants_meta":{"raw":{"variants":["Split-then-join beats truncation for very long docs","Chunk selection lifts summarization over truncation","For long docs, pick best chunk summary, not head","SPIN 3: split-then-join outperforms truncation","Long-document summarization: splitting beats cutting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1365,"prompt_tokens":940,"completion_tokens":425,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":344}},"tokens_in":556,"tokens_out":425,"duration_ms":4236,"temperature":1.0,"reasoning_tokens":344,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:30:39.117077+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of very long documents whose gold summaries require combining facts from widely separated chunks, apply SPIN 3, and compare with the truncated baseline; if ROUGE scores on that set fall below truncation, the independence assumption fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the divide-and-conquer strategy of splitting document and summary that SPIN adapts."}],"review_version":1}