{"id":"b6a1e0b1-db65-4fe2-ab58-0afa7e1f4453","arxiv_id":"2501.14673","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"A Mamba-based extractive summarizer with Poincare distance features posts mixed ROUGE results on a small Amazon split, but circular ROUGE-derived labels and unreported baselines undermine the claimed gains.","lead":"MPoincareSum applies a Mamba state space model, Poincare compression, and LoRA to pick sentences from Amazon reviews in a low-resource setting. Tested on a tiny 209-review split, it beats three heuristic or neural baselines on ROUGE-2 and ROUGE-L, but the evaluation is hampered by circular labels and missing details.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Table 3 contradicts the abstract's broad outperformance claim, and the reported margins are fragile due to tiny test sets, best-run selection, and a ROUGE-based labeling loop.","rationale":"The reader identifies the ROUGE-based labeling circularity as the weakest assumption, and that is a genuine concern. However, the more immediately decisive problem is that the paper's own results table runs against the abstract's claim: MPoincareSum loses on R1 to every baseline and wins on R2/RL by margins that are unquantified and plausibly within noise given 73 test reviews and best-run selection. This makes the central claim unsupported regardless of whether the labels are valid. The missing variance, the test-set selection, and the lack of baseline comparability are structural flaws that a re-evaluation could fix. I partially agree with the reader because the label issue is closely tied to the evaluation, but I would weight the empirical contradiction and fragility of the reported numbers as the primary load-bearing concern.","tokens_in":8599,"tokens_out":1766,"duration_ms":17597,"concrete_test":"Re-run the full pipeline on the same 73 test reviews with at least 10 random seeds, report mean and standard deviation for R1/R2/RL, and do not select the best run post hoc. If the mean R2/RL advantage over Copycat, MeanSum, and LexRank falls within one standard deviation of the baselines, the headline claim is not sustained. Additionally, regenerate training labels using a non-ROUGE procedure (e.g., human annotation or gold summary sentence alignment) and check whether the test-set ROUGE scores change materially; if they do, the original labels were the source of the apparent gain.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that MPoincareSum outperforms existing approaches in low-resource extractive summarization. The paper's own evidence does not support this: Table 3 shows MPoincareSum is worst on R1 (0.214 vs 0.269–0.372) and beats the baselines on R2 by only 0.012–0.021 and on RL by 0.017–0.040 over a test set of just 73 reviews. Section 4.2 says 'we selected the best-performing model' after repeating runs, which is test-set selection and inflates the reported scores. The abstract's claim is thus not merely overstated but contradicted by the table. A second load-bearing problem is the training-label pipeline: Section 4.1 states labels were generated from ROUGE and semantic similarity, and the model is then evaluated with ROUGE, so the reported ROUGE gains partially measure how well the model recovers the metric used to build its own labels. The label threshold and procedure are not described. Finally, Eq. 8 gives '1 + 2|a−b|² / ((1−|a|²)(1−|b|²))', which omits the arccosh in the standard Poincaré metric, making the 'Poincaré compression' a non-metric operation whose reported benefit is unvalidated. Without variance estimates, significance tests, or fixed evaluation protocols, 'outperforms' is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MPoincareSum, an extractive summarization method for low-resource Amazon review data. It encodes review-sentence pairs with the Mamba state space model, applies spectral clustering with a claimed Poincaré distance to produce features, trains a linear classifier with LoRA, and paraphrases selected sentences with BART. The experiments report ROUGE-1, ROUGE-2, and ROUGE-L scores on a dataset of 136 training and 73 test reviews and claim that MPoincareSum outperforms several existing approaches, including Copycat, MeanSum, LexRank, and FewSum.","tokens_in":8937,"tokens_out":4697,"duration_ms":40402,"significance":"If the central claim were established, the paper would offer a useful lightweight architecture combining state space models, spectral clustering, and parameter-efficient fine-tuning for low-resource extractive summarization. The combination of components is plausible and the topic is relevant. However, the paper does not provide code or data, and the current evaluation has load-bearing flaws: the abstract's outperformance claim is contradicted by the paper's own Table 3, the training labels are derived from the same ROUGE metric used for evaluation, and model selection is performed on the test set. As submitted, the evidence is not sufficient to support the stated claims.","major_comments":[{"comment":"The abstract's central claim that MPoincareSum 'outperforms several existing approaches' is not supported by Table 3. MPoincareSum has the lowest ROUGE-1 score (0.214 versus 0.269–0.372 among the listed baselines) and is below FewSum on ROUGE-L (0.200 versus 0.227). The only consistent statement supported by the table is that MPoincareSum beats Copycat, MeanSum, and LexRank on ROUGE-2 and ROUGE-L, with margins of 0.012–0.021 and 0.017–0.040, respectively. The authors should restate the claim to match the table and assess whether such small margins are meaningful on a 73-review test set.","section":"Abstract and §4.3/Table 3"},{"comment":"The training labels are generated using ROUGE scores and semantic similarity between each sentence and its corresponding review, and the model is then evaluated with ROUGE-1, ROUGE-2, and ROUGE-L against the same reference summaries. Section 4.1 does not specify the threshold, procedure, or validation of these labels. This creates a circularity: the reported ROUGE gains partly measure how well the model recovers the metric used to construct its own training signal. The labeling pipeline should be independently validated, or the evaluation should use a metric not derived from the labeling procedure.","section":"§4.1"},{"comment":"The sentence 'Each experiment was repeated multiple times, and we selected the best-performing model' indicates that model selection was performed on the test set. With only 73 test reviews, best-run selection on the test set inflates ROUGE scores and makes the reported point estimates incomparable to the literature baselines, for which no such selection is reported. The authors need a fixed evaluation protocol: select hyperparameters and checkpoints on a validation split, report the mean and standard deviation over runs, and provide significance tests or confidence intervals.","section":"§4.2"},{"comment":"Equation (8) states d_Poincare(a,b) = 1 + 2|a−b|² / ((1−|a|²)(1−|b|²)), which is not the standard Poincaré metric. The standard formula is arccosh(1 + 2|a−b|² / ((1−|a|²)(1−|b|²))). As written, the quantity is not a distance, and the paper does not verify that the Mamba embeddings lie in the unit ball. The purported benefit of 'Poincaré compression' is therefore not established by the reported experiments.","section":"Eq. (8)"},{"comment":"The ablation section repeatedly refers to 'Table 4', but no Table 4 appears in the manuscript. The claims that removing Poincaré compression, replacing Mamba with BERT, and switching from LoRA to full fine-tuning cause 'significant' ROUGE drops cannot be verified without the table, its exact numbers, and significance tests. This is a load-bearing omission because the ablation is used to justify the contribution of each proposed component.","section":"§7 Ablation Study"}],"minor_comments":[{"comment":"The section numbering is inconsistent: the Introduction is unnumbered, Related Work is numbered 5, Method is numbered 6, but the subsections are numbered 3.1–3.5. The numbering should be harmonized.","section":"Throughout"},{"comment":"The contributions list contains a typo: 'HiPPPO initialization' should read 'HiPPO initialization.'","section":"Contributions"},{"comment":"The reference list mixes citation styles (e.g., 'Gu, A., & Dao, T. (2023)' versus 'Arthur Bražinskas et al., 2020') and contains two entries for Bražinskas et al. 2020 under slightly different spellings. A consistent style should be used.","section":"References"},{"comment":"The symbol ⊕ in Eq. (3) is not defined; clarify whether the paraphrases of individual sentences are concatenated and how the final summary length is controlled.","section":"Eq. (3)"},{"comment":"The example in Table 1 is difficult to interpret because the intermediate extractive selection is not shown; it is unclear which sentences from the review were chosen and how BART paraphrased them into the displayed output.","section":"Table 1"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an unfinished draft, with missing Table 4 and inconsistent numbering. The rejection is driven by experimental methodology rather than by the novelty of the architecture. If the authors re-run the experiments with a proper validation split, report variance and significance tests, and either justify the ROUGE-derived labels or use independently validated labels, a revised submission could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on arXiv:2501.14673 (Ait Khayi). The proposed system—Mamba encoder, Poincaré-distance features via spectral clustering, LoRA fine-tuning, BART paraphrasing—is a genuine new combination for low-resource extractive summarization. That part is worth a look. The paper also correctly identifies that linear-scaling SSMs could be a better fit than transformers when data are scarce. Good instinct.\n\nBut the evidence is not there. The abstract says MPoincareSum \"outperforms several existing approaches,\" but Table 3 shows it with the lowest ROUGE-1 (0.214 vs 0.269–0.372) and losing to FewSum on ROUGE-L (0.200 vs 0.227). It only wins on ROUGE-2 by 0.021 over FewSum. On a 73-review test set, that's noise. The margins are within what you'd expect from random seed variation, and the authors admit to selecting the best run on the test set, which makes the numbers even less reliable. Second, the training labels are generated from ROUGE and semantic similarity to the reference summary; the model is then evaluated with ROUGE against the same references. That's a circular loop, and the reported ROUGE gains partly measure how well the model recovers the labeling rule. The label threshold and procedure are not described, so the targets are essentially unvalidated. Third, the ablation study is referenced but Table 4 is missing. So the claims that Poincaré compression and LoRA help are unsupported in the current manuscript. Fourth, Eq. 8 for the Poincaré distance omits the arcosh. What's implemented is a monotonic transformation, not the actual metric, which may still be usable for clustering, but the paper needs to say so and justify it.\n\nThe writing is rough throughout—typos, missing references (e.g., \"HiPPPO\"), and a self-referential footnote—but that's fixable. What's not fixable without new experiments is the evaluation.\n\nWho is this for? Someone curious about SSM-based summarization or low-resource methodology could read it as a description of a promising direction, not as a reliable result. It doesn't deserve a serious referee yet in my view; the load-bearing claims are contradicted by the paper's own data. I'd suggest the authors re-run with fixed splits, independent labels (or at least a held-out label validation), variance estimates, and the actual Poincaré formula. If they do that, the paper could be worth another look.\n\nMy honest recommendation: desk reject in current form, but keep an eye on a revised version.","headline":"Plausible architecture, but the paper's own numbers contradict its central claim and the evaluation is circular, so it needs major rework before it can be trusted.","tokens_in":9437,"tokens_out":3799,"would_cite":false,"duration_ms":30090,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Mamba-based encoder with Poincare-distance compression and LoRA tuning produces competitive extractive summaries in low-resource settings, beating Copycat, MeanSum, and LexRank on ROUGE-2 and ROUGE-L.","keywords":["extractive summarization","low-resource NLP","state space models","Mamba","Poincare compression","LoRA","ROUGE","spectral clustering"],"falsifier":"A reader could settle the claim by training MPoincareSum and Copycat on the same 136-review set and comparing them on a fresh, independently annotated 300-review test set across several random seeds; if the ROUGE-2 and ROUGE-L lead does not persist, the paper's central claim fails.","tokens_in":8385,"feed_emoji":"📝","tokens_out":10283,"duration_ms":82126,"temperature":0.7,"pith_summary":"The paper proposes MPoincareSum, a low-resource extractive summarization pipeline: a Mamba state space model encodes a review and each of its sentences, the concatenated embeddings are compressed by measuring Poincare distances to spectral-clustering centroids, a linear layer labels each sentence as relevant or not, and BART paraphrases the chosen sentences into the final summary. The central claim is that this combination outperforms several existing approaches, specifically Copycat, MeanSum, and LexRank, on ROUGE-2 and ROUGE-L when only 136 Amazon reviews are available for training. The reported scores are mixed: ROUGE-2 rises to 0.12 against 0.099 for the strongest compared system, FewSum, while ROUGE-1 falls to 0.214, below all compared baselines. A sympathetic reader would care because the result suggests state space models, hyperbolic-distance compression, and parameter-efficient tuning can carry a summarization system in settings where attention-based transformers are too expensive or too data-hungry.","feed_headline":"Mamba + Poincare compression wins on ROUGE-2 and ROUGE-L","feed_subtitle":"Trained on 136 Amazon reviews, the system beats Copycat, MeanSum, and LexRank on ROUGE-2 and ROUGE-L.","key_machinery":"The load-bearing mechanism is the interaction of three components. Mamba is a state space model that uses a selective scan to compress sequence information while keeping inference cost linear in sequence length; it supplies the sentence and review semantics. Poincare compression is the paper's dimensionality-reduction step: spectral clustering puts the concatenated Mamba embeddings into clusters, and each embedding is then represented by a vector of Poincare distances to the cluster centroids, which a batch-normalized linear layer uses to predict relevance. LoRA freezes the pretrained weights and adds low-rank trainable matrices, so most of the model stays fixed during low-resource training. These pieces are chained into a single sentence-relevance classifier, and the selected sentences are paraphrased by BART.","core_discovery":"On the paper's own terms, the discovery is that a state space model does not have to be used as a black-box encoder for extractive summarization: after Mamba produces representations for a review and a candidate sentence, projecting the concatenated vector into Poincare space and replacing it with distances to spectral-cluster centroids yields a compact feature set that a linear classifier can learn from with very little data. The ablations support this reading: removing the Poincare compression lowers ROUGE, swapping Mamba for a transformer encoder gives roughly similar results, and LoRA tuning beats full fine-tuning. The paper therefore claims that Mamba plus Poincare compression plus LoRA is a competitive lightweight architecture for low-resource extractive summarization, with the qualification that its ROUGE-1 score trails FewSum.","pith_inferences":["A next check the paper does not perform is whether the ROUGE/semantic-similarity labeling agrees with human judgments of salience; without that, the reported advantage should be read as an in-metric result rather than proof of better summaries.","The same architecture could be tested on longer-document low-resource corpora, where Mamba's linear-time inference would give a clearer practical edge than it does on 128-token reviews.","Because the Mamba-versus-BERT ablation is reported without variance, re-running it with multiple seeds would show whether the near-parity is stable or an artifact of selecting the best checkpoint."],"forward_implications":["State space encoders can substitute for attention-based encoders in low-resource extractive summarization without losing ROUGE-2 or ROUGE-L quality.","Poincare-distance compression of dense Mamba embeddings is a workable dimensionality-reduction step for sentence-relevance classification.","LoRA-style parameter-efficient tuning beats full fine-tuning when training data are very limited.","The matched ROUGE-1 gap to FewSum shows that fluency-oriented generation still has an edge on unigram overlap, so the claimed advantage is specific to ROUGE-2 and ROUGE-L.","The full pipeline runs within a single-GPU setting with 30 GB of RAM, which is realistic for low-resource practitioners."],"supporting_citations":[{"why":"Supplies the Mamba state space model that generates review and sentence semantics.","marker":"Gu et al., 2024"},{"why":"Supplies the spectral clustering method used inside the Poincare compression module.","marker":"Macgregor et al., 2023"},{"why":"Motivates embedding points in Poincare space for hierarchical text structure.","marker":"Nickel et al (2017)"},{"why":"Supplies the LoRA low-rank adaptation technique that reduces trainable parameters.","marker":"Hu et al., 2022"},{"why":"Supplies the BART model used to paraphrase the extracted relevant sentences.","marker":"Lewis et al., 2020"},{"why":"Provides the Amazon review dataset used for training and testing.","marker":"He and McAuley, 2016"},{"why":"Provides the Copycat and FewSum baselines that MPoincareSum is compared against.","marker":"Brazinskas et al., 2020"},{"why":"Provides the MeanSum baseline that MPoincareSum is compared against.","marker":"Chu and Liu, 2019"},{"why":"Provides the LexRank baseline that MPoincareSum is compared against.","marker":"Erkan and Radev, 2004"}],"fun_headline_variants":["Mamba + Poincare compression wins on ROUGE-2 and ROUGE-L","Poincare-mapped Mamba improves low-resource extractive summarization","State space model with Poincare features beats baselines on small data","Mamba+Poincare beats Copycat, MeanSum, LexRank on ROUGE-2/ROUGE-L"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation depends on the assumption that sentence labels built from ROUGE scores and semantic similarity to the reference summary are a trustworthy measure of what a good extractive summary should select; because ROUGE is also the test metric, the reported gains partly rest on that assumption.","fun_headline_variants_meta":{"raw":{"variants":["Mamba + Poincare compression wins on ROUGE-2 and ROUGE-L","Poincare-mapped Mamba improves low-resource extractive summarization","State space model with Poincare features beats baselines on small data","Mamba+Poincare beats Copycat, MeanSum, LexRank on ROUGE-2/ROUGE-L"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000829,"raw_usage":{"total_tokens":3577,"prompt_tokens":859,"completion_tokens":2718,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":2625}},"tokens_in":475,"tokens_out":2718,"duration_ms":18664,"temperature":1.0,"reasoning_tokens":2625,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:54:26.664985+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could settle the claim by training MPoincareSum and Copycat on the same 136-review set and comparing them on a fresh, independently annotated 300-review test set across several random seeds; if the ROUGE-2 and ROUGE-L lead does not persist, the paper's central claim fails.","supporting_citations":[],"review_version":1}