{"id":"67b93bcf-9b8c-42de-b977-fae17abe7806","arxiv_id":"2411.17181","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A word-pair Gaussian sentence similarity method combined with spectral clustering and TF-IDF is claimed to improve Bengali extractive summarization and to transfer to Hindi, Marathi, and Turkish.","lead":"Researchers propose WGSS, a sentence similarity method that matches words between sentences by nearest embedding distance and averages the similarities with a Gaussian kernel, then uses spectral clustering and TF-IDF to build Bengali extractive summaries. The paper reports large ROUGE gains over three baselines across four Bengali datasets and three other low-resource languages, plus a new 500-summary Bengali dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sigma=5e-11 in Eq. 3 forces every word-pair Gaussian similarity to underflow to zero, so the affinity matrix is degenerate and the reported summaries cannot be produced by the described algorithm.","rationale":"The reader's weakest_assumption is exactly the one I consider load-bearing: the numerical value of sigma in the Gaussian kernel. I independently checked the arithmetic: with sigma=5e-11, sigma^2=2.5e-21, so the kernel exp(-D^2/(2 sigma^2)) underflows for any D^2 above roughly 1e-19, far below any realistic FastText word-vector distance. This makes the affinity matrix all zeros, which would make spectral clustering degenerate and could not yield ROUGE-1 scores of 0.47–0.49. The reader and I agree on this point. I also considered whether other issues (uncontrolled summary length, weak baselines, lack of significance testing) could be the primary concern, but those are secondary: if the similarity computation is degenerate as described, the central claim fails regardless of evaluation protocol. The proposed one-line check—instrumenting the released code to print Dmsw and comparing the actual sigma used—would settle whether this is a typo in the paper or a genuine impossibility. Since the verdict REJECT was already based on this concern, my stress-test does not change the reader's verdict, hence UNCHANGED. I am not accusing the authors of fraud; the code may indeed contain a different sigma, but then the paper as written misrepresents the method, and the reported results cannot be attributed to the stated algorithm.","tokens_in":15582,"tokens_out":2391,"duration_ms":23659,"concrete_test":"Run the released WGSS implementation on the self-curated dataset and instrument Algorithm 1 to record the distribution of Dmsw values and the resulting sentence similarities. Compute max(Dmsw) for any sentence pair; if max(Dmsw)^2 > 1e-19 while sigma is set to 5e-11, then exp(-Dmsw^2/(2*5e-11^2)) underflows to 0.0 for that pair. Verify whether the code's sigma parameter equals 5e-11 as stated in Section 3.2 or differs; if it differs, confirm whether the reported ROUGE scores are reproducible only with the code's actual sigma, not with the paper's stated value.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends entirely on the sentence similarity computation in Eqs. 3–4. With sigma fixed at 5e-11, sigma^2 = 2.5e-21, so the exponent -D_i^2/(2*sigma^2) equals -D_i^2 / 5e-21. Any squared Euclidean distance D_i^2 above about 1e-19 produces an exponent below -745, and exp() underflows to 0.0 in IEEE double precision. FastText word vectors for different Bengali words have squared Euclidean distances on the order of 0.1 to 10, many orders of magnitude above this threshold. Therefore Wsim values are all exactly 0.0, the geometric mean Sim(X,Y) in Eq. 4 is 0.0, and every off-diagonal entry of the affinity matrix A in Eq. 5 is 0.0. Spectral clustering on this all-zero affinity matrix is degenerate (the Laplacian is zero, eigenvectors are arbitrary), and it cannot yield the meaningful clusters and TF-IDF picks that would produce ROUGE-1 scores around 0.47–0.49. The reported fine-tuning curve in Fig. 4 also shows ROUGE scores that are flat or near-zero for the smallest sigma values, which is consistent with underflow rather than a working similarity measure. If the released code uses a different sigma than the stated 5e-11, then the paper is internally inconsistent and the reported numbers do not follow from the described method; if the code actually uses 5e-11, the results are computationally impossible. Either way, the load-bearing numerical premise is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes WGSS, an unsupervised extractive summarization method for Bengali and other low-resource languages. Sentence similarity is computed by taking the geometric mean of Gaussian kernel similarities over word pairs, where each word is matched to its nearest counterpart in the other sentence via FastText embeddings (Eqs. 1-4). This similarity populates an affinity matrix (Eq. 5), spectral clustering groups sentences into topics, and the highest-TF-IDF sentence from each cluster is selected for the summary (Algorithms 1-2). The method is evaluated against BenSumm, SASbSC, and LexRank on four Bengali datasets using ROUGE, with additional experiments in Hindi, Marathi, and Turkish. The authors report an average improvement of 43.2% and release a new dataset of 250 articles with two human summaries each, along with code. The paper itself acknowledges limitations in Section 5, including reliance on pre-trained embeddings and the absence of word-order information.","tokens_in":15877,"tokens_out":3718,"duration_ms":33975,"significance":"If the reported results were reproducible, WGSS would be a simple, language-agnostic addition to extractive summarization for low-resource languages, and the released 500-summary Bengali dataset would be a useful community resource. The authors should be credited for making the implementation and data public and for testing across four languages. However, the central numerical claim is not reproducible as stated: the fixed value of sigma in Eq. (3) forces the Gaussian similarity to underflow to zero for realistic word-vector distances, so the affinity matrix degenerates. In addition, sigma is tuned on the evaluation sets themselves, which introduces circularity into the headline performance numbers. These issues bear directly on the paper's central claims, so the positive potential of the method and dataset does not translate into a publishable result in the current form.","major_comments":[{"comment":"The stated sigma value makes the proposed similarity computation degenerate. With sigma fixed at 5e-11, 2*sigma^2 is 5e-21, so the exponent -D_i^2/(2*sigma^2) is numerically below about -745 for any squared distance greater than roughly 3.7e-18. In IEEE double precision, exp() of such values underflows to zero. FastText word-vector distances between distinct words are many orders of magnitude larger (typically O(1) or more in squared Euclidean norm), so every Wsim term for distinct word pairs is exactly zero, and the geometric mean in Eq. (4) is zero for all non-identical sentence pairs. The affinity matrix in Eq. (5) therefore has zero off-diagonal entries, and spectral clustering on this matrix is degenerate. The ROUGE scores around 0.4-0.5 in Table 3 cannot follow from the described computation. The manuscript provides no rescaling or normalization that would make sigma = 5e-11 meaningful; if this value is a typo, the paper must state the intended value and recompute all results.","section":"Section 3.2, Eqs. (3)-(4), Algorithm 1"},{"comment":"The standard deviation sigma is selected by maximizing ROUGE-F1 on the evaluation datasets, including the Self-Curated dataset that is later reported as a test set in Table 3. Section 4.4.1 says that 63 values were 'experimented on' and that 5e-11 'was fixed as the value for sigma that gives the best similarity measurement,' with Fig. 4 showing ROUGE scores as a function of sigma. Tuning a hyperparameter on the same data that is used for final evaluation makes the reported improvements partly a product of test-set selection. An independent validation split, nested cross-validation, or a fixed sigma chosen without reference to the evaluation labels is required before the comparative claims can be accepted.","section":"Section 4.4.1 and Table 3"},{"comment":"The evaluation setup is internally inconsistent. Table 2 reports that TF-IDF ranking on the Self-Curated dataset with summary proportion 0.2 achieves ROUGE-1 0.50, ROUGE-2 0.40, and ROUGE-LCS 0.46. Table 3 reports the proposed WGSS on the same Self-Curated dataset, also with summary proportion 0.2, as achieving 0.47, 0.36, and 0.43. Since the TF-IDF ranking in Table 2 is the extraction strategy used inside WGSS, these two tables describe the same configuration and should not disagree. This discrepancy undermines confidence in the reproducibility of the reported results and must be reconciled.","section":"Table 2 versus Table 3"}],"minor_comments":[{"comment":"The description of ROUGE-1 says 'It calculates the total common characters between the summaries'; this should read 'common unigrams' rather than characters.","section":"Section 4.3, item 1"},{"comment":"The legend in Figs. 5 and 6 labels the proposed method as 'WSbSC' instead of 'WGSS'; please correct the label.","section":"Figures 5 and 6"},{"comment":"The 'Bengali (Average)' row lists ROUGE-2 as 0.38, but the mean of the four preceding Bengali ROUGE-2 values (0.36, 0.43, 0.34, 0.41) is 0.385, which rounds to 0.39; the averaging should be checked and reported consistently.","section":"Table 4"},{"comment":"The URL for the Turkish dataset is given as 'https://wwww.github.com/xtinge/turkish-extractive-summarization-dataset'; the extra 'w' and the use of 'github.com' rather than 'www.github.com' should be corrected.","section":"Footnote 13"},{"comment":"The x-axis labels of Fig. 4 are difficult to read and inconsistently formatted (e.g., 'E-12', 'E-1', '5E-1'); using a logarithmic axis with clear scientific notation would make the fine-tuning curve interpretable.","section":"Figure 4"}],"recommendation":"reject","confidential_remarks":"The numerical underflow issue is likely a typo in the manuscript, but as written the central algorithm cannot produce the reported results, and the sigma tuning on test data further weakens the empirical claims. If the authors can correct the sigma value and re-run the evaluation with an honest validation protocol, a revised submission could be considered; in its current form, the paper does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core sentence-similarity formula with sigma = 5e-11 cannot produce the reported results. Every off-diagonal affinity value underflows to zero, so the spectral clustering has nothing to work with. That is not a minor implementation detail; it is the load-bearing component of the method.\n\nWhat is genuinely new: the word-pair nearest-neighbor comparison (a symmetric Hausdorff-style distance) aggregated with the geometric mean of Gaussian kernels is a simple, reasonable alternative to word averaging. The authors correctly identify the failure mode of averaging and their Figure 1/3 example is apt. The new Bengali dataset of 250 articles with 500 expert summaries is a practical asset, and running the method on Hindi, Marathi, and Turkish is a sensible language-agnostic check. The code and data are public, which is good practice.\n\nThe soft spots are not small. With sigma fixed at 5e-11, sigma^2 = 2.5e-21, so the exponent in Eq. 3 is -D^2/(5e-21). FastText word vectors produce squared distances on the order of 0.1 to 10, so exp() underflows to zero. The affinity matrix A is then all zeros, the Laplacian is zero, and spectral clustering cannot produce meaningful clusters. The reported ROUGE-1 scores around 0.47–0.49 simply cannot follow from the described computation. Figure 4's near-zero tail at small sigma is consistent with underflow. If the released code uses a different sigma, then the paper is inconsistent with the code; if it uses 5e-11, the results are impossible. Either way, the central numerical premise is unsupported.\n\nThere are also evaluation issues. Sigma was chosen by maximizing ROUGE-F1 on the evaluation data (Section 4.4.1), which makes some of the results a product of tuning. Comparisons are not length-controlled and no significance tests are reported; the 43.2% headline hides a near-tie on BNLPC and relies on weak baselines. These would matter less if the core formula worked, but they need fixing regardless.\n\nBottom line: this paper is not publishable as submitted. But the idea is salvageable and the dataset has standalone value. My recommendation: reject, and invite the authors to fix sigma (or scale distances), rerun the experiments with fair baselines and significance tests, and resubmit. The dataset alone might be worth a short resource paper.","headline":"The word-pair similarity idea and the Bengali dataset are real, but the stated sigma makes the algorithm mathematically impossible as written; reject and invite a corrected resubmission.","tokens_in":16449,"tokens_out":2527,"would_cite":false,"duration_ms":21901,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A word-pair Gaussian similarity measure followed by spectral clustering and TF-IDF selection improves extractive summarization in Bengali and other low-resource languages.","keywords":["extractive text summarization","Bengali","sentence similarity","Gaussian kernel","word embeddings","spectral clustering","TF-IDF","low-resource languages"],"falsifier":"Take a sample of sentence pairs from the released dataset, compute the one-sided nearest-word distances $D_{\\mathrm{msw}}$ with the same FastText vectors, and evaluate $\\exp(-D_i^2/(2(5\\times10^{-11})^2))$. If almost every value rounds to zero, the affinity matrix is all zeros and spectral clustering has no signal; a second check is to count nonzero entries in the affinity matrix and verify that the resulting clusters are not trivial. That observation would settle whether the reported results can come from WGSS as written.","tokens_in":15332,"feed_emoji":"📝","tokens_out":10369,"duration_ms":80787,"temperature":0.7,"pith_summary":"The paper introduces WGSS, an extractive summarization method whose core is a new sentence similarity function. Instead of averaging the word vectors of a sentence into one vector, WGSS pairs every word in one sentence with its nearest word in the other sentence by Euclidean distance in a FastText embedding space, converts those distances into Gaussian similarities, and takes the geometric mean as the sentence similarity. These similarities build an affinity matrix, spectral clustering groups sentences into topic clusters, and the highest-TF-IDF sentence from each cluster is selected to form the summary. The authors report that this pipeline outperforms BenSumm, LexRank, and SASbSC by 43.2% on average ROUGE across four Bengali datasets, and that it transfers to Hindi, Marathi, and Turkish with only a tokenizer, stopword list, and embeddings. The paper also releases a new Bengali dataset of 250 news articles each with two expert-written summaries.","feed_headline":"Word-pair Gaussian similarity lifts Bengali summary scores by 43%","feed_subtitle":"Nearest-word-pair similarity plus spectral clustering beats three baselines and transfers to Hindi, Marathi, and Turkish.","key_machinery":"The mechanism that carries the argument is the two-sided nearest-word-pair distance set $D=\\{D_{\\mathrm{msw}}(x_i,Y)\\}\\cup\\{D_{\\mathrm{msw}}(y_j,X)\\}$, which the paper calls the Most Similar Word Distance. Each distance is mapped to a similarity by the Gaussian kernel $W_{\\mathrm{sim}}=\\exp(-D_i^2/(2\\sigma^2))$, and the geometric mean of those values gives the sentence similarity, exactly $Sim(X,Y)=\\exp(-\\sum_i D_i^2/(2n\\sigma^2))$. This object replaces the sentence-average vector used by the SASbSC baseline, and the affinity matrix it produces is what spectral clustering operates on. The second mechanism is the two-step extraction rule: group sentences with spectral clustering into $k=\\lceil Np\\rceil$ clusters, pick the maximum-TF-IDF sentence from each cluster, then restore document order.","core_discovery":"On the paper's own terms, the central discovery is that a sentence is better compared to another sentence through its individual word correspondences than through a single averaged vector. The proposed function is $Sim(X,Y)=\\exp\\left(-\\frac{1}{2n\\sigma^2}\\sum_{i=1}^{n}D_i^2\\right)$, where the $D_i$ come from the two-sided nearest-neighbor distances $D_{\\mathrm{msw}}(x,Y)=\\min_{y\\in Y}\\|x-y\\|$ and $D_{\\mathrm{msw}}(y,X)=\\min_{x\\in X}\\|y-x\\|$, and $\\sigma$ is set to $5\\times10^{-11}$. This symmetric word-pair distance set is meant to preserve local semantic links that averaging loses, and the geometric-mean Gaussian form is meant to damp outlier words. Feeding the resulting affinity matrix into spectral clustering with $k=\\lceil Np\\rceil$ clusters and choosing one sentence per cluster by TF-IDF yields summaries that the authors report are consistently better than the three comparison models on all four Bengali datasets and competitive on Hindi, Marathi, and Turkish.","pith_inferences":["Beyond the paper, the same word-pair similarity could be plugged into any graph-based NLP task that needs sentence-level affinity, such as keyphrase extraction or question-answer retrieval, since it only requires embeddings.","A testable extension the authors do not explore is making $\\sigma$ depend on the data, for example scaling it to the median nearest-word distance; that would test whether the fixed $5\\times10^{-11}$ value is essential or an artifact.","Because WGSS ignores word order, a natural follow-up is to add position-sensitive weighting or a small n-gram component and check whether ROUGE-LCS gains further; the paper itself concedes word order is a limitation.","If the reported numbers reproduce, the method's simplicity makes it a credible baseline for future Bengali summarization work; if the underflow issue is real, correcting the scaling would still leave the word-pair idea testable."],"forward_implications":["If WGSS works as claimed, unsupervised extractive summarization in Bengali needs no training data, only pretrained word vectors, a tokenizer, and a stopword list.","The approach should transfer to any language with FastText embeddings; the paper demonstrates this on Hindi, Marathi, and Turkish with scores close to Bengali.","Selecting one sentence per spectral cluster should reduce redundancy and improve topic coverage relative to ranking-only methods like LexRank.","The released 250-article, 500-summary Bengali dataset gives future work a common evaluation ground for extractive and abstractive summarization.","The reported 43.2% average ROUGE improvement implies that word-level matching, not additional training data, is the main source of the gain over the three baselines."],"supporting_citations":[{"why":"Supplies the FastText word vectors in 157 languages that WGSS uses for every word-pair distance and similarity computation.","marker":"[12]"},{"why":"Defines SASbSC, the word-averaging spectral clustering baseline WGSS is designed to outperform, and supplies the similarity-plus-spectral-clustering structure WGSS inherits.","marker":"[24]"},{"why":"Defines LexRank, the PageRank-based extractive baseline used as a comparison model.","marker":"[10]"},{"why":"Defines BenSumm, the TF-IDF-based extractive and abstractive baseline used for comparison, including the implementation used in the experiments.","marker":"[5]"},{"why":"Defines the ROUGE metrics used to evaluate all generated summaries against human references.","marker":"[17]"},{"why":"Provides the spectral clustering algorithm that groups the sentence-similarity graph into topic clusters.","marker":"[30]"},{"why":"Supplies the BNLPC dataset, one of the four Bengali evaluation corpora used in the comparison.","marker":"[14]"},{"why":"Provides the scale-space justification for using the Gaussian kernel as the word similarity function.","marker":"[3]"}],"fun_headline_variants":["Word-pair Gaussian similarity lifts Bengali summaries by 43%","New Gaussian similarity algorithm improves Bengali summarization by 43%","Word-pair Gaussian scoring beats baselines for Bengali summaries","Gaussian word-pair similarity: 43% better Bengali summaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing numerical premise is that with $\\sigma=5\\times10^{-11}$, the Gaussian factor $\\exp(-D_i^2/(2\\sigma^2))$ gives nonzero similarities for the FastText distances actually found in Bengali sentences; if real distances make these exponentials underflow to zero, the affinity matrix becomes degenerate and the claimed ROUGE gains cannot follow from the described computation.","fun_headline_variants_meta":{"raw":{"variants":["Word-pair Gaussian similarity lifts Bengali summaries by 43%","New Gaussian similarity algorithm improves Bengali summarization by 43%","Word-pair Gaussian scoring beats baselines for Bengali summaries","Gaussian word-pair similarity: 43% better Bengali summaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00101,"raw_usage":{"total_tokens":4341,"prompt_tokens":1092,"completion_tokens":3249,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":708,"completion_tokens_details":{"reasoning_tokens":3178}},"tokens_in":708,"tokens_out":3249,"duration_ms":21900,"temperature":1.0,"reasoning_tokens":3178,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:25:16.020826+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of sentence pairs from the released dataset, compute the one-sided nearest-word distances $D_{\\mathrm{msw}}$ with the same FastText vectors, and evaluate $\\exp(-D_i^2/(2(5\\times10^{-11})^2))$. If almost every value rounds to zero, the affinity matrix is all zeros and spectral clustering has no signal; a second check is to count nonzero entries in the affinity matrix and verify that the resulting clusters are not trivial. That observation would settle whether the reported results can come from WGSS as written.","supporting_citations":[{"cited_title":"Learning word vectors for 157 languages","cited_arxiv_id":null,"evidence_quote":"Supplies the FastText word vectors in 157 languages that WGSS uses for every word-pair distance and similarity computation."},{"cited_title":"Unsupervised Bengali text summarization using sentenc e embedding and spectral clustering","cited_arxiv_id":null,"evidence_quote":"Defines SASbSC, the word-averaging spectral clustering baseline WGSS is designed to outperform, and supplies the similarity-plus-spectral-clustering structure WGSS inherits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines LexRank, the PageRank-based extractive baseline used as a comparison model."},{"cited_title":"R., N AYEEM , M","cited_arxiv_id":null,"evidence_quote":"Defines BenSumm, the TF-IDF-based extractive and abstractive baseline used for comparison, including the implementation used in the experiments."},{"cited_title":"ROUGE: A package for automatic evaluation of summari es","cited_arxiv_id":null,"evidence_quote":"Defines the ROUGE metrics used to evaluate all generated summaries against human references."},{"cited_title":"A tutorial on spectral clustering","cited_arxiv_id":null,"evidence_quote":"Provides the spectral clustering algorithm that groups the sentence-similarity graph into topic clusters."},{"cited_title":"M., P ERVIN , S., AND BEGUM , Z","cited_arxiv_id":null,"evidence_quote":"Supplies the BNLPC dataset, one of the four Bengali evaluation corpora used in the comparison."},{"cited_title":"P., B AUDIN , M., AND DUDA , R","cited_arxiv_id":null,"evidence_quote":"Provides the scale-space justification for using the Gaussian kernel as the word similarity function."}],"review_version":1}