{"id":"d7c58da3-0b09-4df8-9561-a8d8aa2b9a48","arxiv_id":"1908.04200","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A collocation-based quality score, built from a corpus of high-impact CS papers, separates high- from low-impact test papers, but the measure is defined by the same conference-ranking standard it is asked to predict.","lead":"This paper proposes a score that rates scientific papers by how closely their two-word phrases match the language of top computer science conferences. The authors claim this content-only score can tell high-impact from low-impact papers without using metadata.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ADS may be measuring typicality of high-impact-venue language rather than paper quality, and the reported statistics are internally inconsistent.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: ADS assumes collocation log-likelihood in high-impact venues reflects quality, when it could instead reflect topic distribution or writing conventions. This is the central construct-validity threat, and the paper does not address it for the proposed measure while its own bag-of-words baseline experiments show that topic strongly affects content-based classification. The internal statistical inconsistencies in Section 6.1 and Table 5 are a second, independent reason to reject: even the basic evidence for the claimed separation is not reproducible from the manuscript. I agree with the reader's REJECT verdict and would not change it. The proposed measure is transparent and falsifiable, and the idea of using collocation typicality as a quality signal is worth investigating, but the current evaluation does not support the headline claim that automated assessment of scientific paper quality has been achieved. A revised version with an external quality criterion (e.g., accepted vs. rejected papers), topic controls, and corrected statistics could be reconsidered.","tokens_in":12175,"tokens_out":4947,"duration_ms":52919,"concrete_test":"Use accepted and rejected papers from a single top-tier conference (e.g., ICDM, for which the authors already have accepted papers; rejected papers would need to be obtained from the organizers as the paper itself suggests in Section 7). Compute ADS for both sets with the existing main corpus. If the score does not significantly separate accepted from rejected papers after controlling for topic (same venue, same year), the measure is not assessing quality but venue typicality. If the authors cannot obtain rejected papers, the alternative decisive check is a topic-matched test: match each high-impact paper to a low-impact paper from the same subfield (using keywords or TF-IDF clusters) and rerun the Section 6.1 t-test; if the difference collapses, the original result is a topic artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ADS, the average main-corpus log-likelihood of a paper's collocations, measures the overall quality of a scientific paper and can power a classifier. The load-bearing assumption is that collocation log-likelihood in the main corpus is a valid proxy for quality. This assumption is unvalidated, and the evaluation as designed cannot distinguish it from typicality. The main corpus is built exclusively from CORE A*/A venues; the high-impact test labels (SIGKDD/SIGIR/ICDM) are selected by the same CORE/Google Scholar ranking, and the low-impact labels come from WSEAS/EEEI. A paper's ADS is therefore the degree to which its collocations resemble the language of the positive class. High-impact venues are mainstream CS conferences; low-impact venues may use different topics and phraseology. That topic, not quality, drives content-based scores is demonstrated by the paper's own bag-of-words control: AUC rises from 0.57 to 0.74 when the held-out low-impact set is made topic-closer to training (Section 6.4). The proposed measure is never subjected to the same topic-matched test. Separately, the Section 6.1 statistics are internally inconsistent: text reports high-impact mean=898, SD=4066, n=2320 and t=5.21, while Table 5 reports mean=757, variance=131703 (SD≈363), n=2303 and t=32.45. Without reproducible statistics, the claimed significant separation cannot be verified. Thus the paper's central claim that ADS assesses quality rests on a plausible but untested equivalence between 'high-impact-venue language' and quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ADS, an average document score based on the mean log-likelihood of bigram collocations drawn from a main corpus of roughly 30,000 papers from CORE A*/A-ranked computer science conferences. The authors compare ADS on papers from high-impact conferences (SIGKDD, SIGIR, ICDM) against papers from low-impact venues (WSEAS, EEEI), report statistically significant differences via ANOVA and independent-samples t-tests, and build a logistic regression classifier that achieves AUC 0.85 when ADS is combined with readability and statistical features. They also compare against a bag-of-words SVM baseline, reporting AUC 0.57 (or 0.74 in a second setup), and conclude that the proposed measure can be used for automated assessment of scientific paper quality.","tokens_in":12424,"tokens_out":5671,"duration_ms":52346,"significance":"The idea of using corpus-derived collocation statistics for automatic paper quality assessment is novel, and the constructed main corpus (nearly 30,000 papers) is a potentially reusable resource. The authors state explicit hypotheses and report a variety of statistical and classification experiments, which is commendable. However, the central claim that ADS measures 'overall quality' is not established: the reference corpus and the test labels originate from the same venue-ranking system, no external quality ground truth is used, the headline statistics in Section 6.1 are internally inconsistent, and the comparison against the bag-of-words baseline is not controlled. If the validity issues were addressed, the measure might serve as a screening tool for domain-specific writing typicality, but as presented the evidence does not support the paper's claims.","major_comments":[{"comment":"The reported statistics for the normalized ADS are mutually inconsistent. The text states that high-impact papers have mean=898, SD=4066, and t(2337)=5.21, while Table 5 reports mean=757, variance=131703 (SD≈363), observations=2303, and t=32.45. The high-impact paper count is also given as 2,320 in the text but sums to 2,303 in Section 3 (SIGKDD 490, SIGIR 406, ICDM 1407). Because the primary evidence for the measure's discriminating power depends on these numbers, the main results cannot be verified without resolved reporting.","section":"Section 6.1, Table 5"},{"comment":"The measure is defined using log-likelihood weights estimated exclusively from papers in A*/A-ranked venues, and the test corpora are labeled high/low impact by the same CORE/Google Scholar ranking criteria. Under these conditions, ADS measures the degree to which a paper's collocations match the language of the positive class. The bag-of-words control in Section 6.4 shows that making the low-impact test set topically closer to the training set raises the AUC from 0.57 to 0.74, demonstrating that topic composition strongly affects content-based classification. No comparable topic-matched evaluation is performed for ADS, so the separation it observes may reflect conference-specific topics and writing conventions rather than paper quality.","section":"Sections 3 and 5.4"},{"comment":"The comparison between the proposed classifier and the bag-of-words baseline is not a controlled one. The classifiers in Section 6.3 are trained on an 80/20 split of the 4,822 test-corpus papers, whereas the bag-of-words model in Section 6.4 trains on 3,000 papers from the main high-impact corpus plus 1,718 low-impact papers, with a different test set. Because the training sets, test sets, and feature representations differ simultaneously, the claim that the proposed measure 'over-performed the baseline classifier' is not supported by the reported experiments.","section":"Section 6.4"},{"comment":"The paper never validates ADS against an independent measure of paper quality, such as citation counts, acceptance decisions, or expert judgments. The only target is the venue's impact ranking, which also determines the contents of the reference corpus. This circularity means the construct validity of 'quality' as measured by ADS remains unestablished; the conclusion claims the measure can 'predict the quality and impact of a scientific paper,' but the evidence only shows separation between preselected venue groups.","section":"Section 5.4 and Conclusions"}],"minor_comments":[{"comment":"The sentence 'In our work, explore collocations as a means of compensating for the limitations of readability metrics' is missing a verb; it should read 'we explore collocations.'","section":"Section 2"},{"comment":"The header 'AverageVariance' is missing a space; the table would also be clearer if it included standard deviations alongside the variances.","section":"Table 4"},{"comment":"The phrase 'the rate of spelling errors in scientific paper' is unclear; the intended meaning appears to be that the measure reflects correct and standard usage, but the wording is confusing.","section":"Section 5.4"},{"comment":"The dataset download link (https://goo.gl/1s4Jkx) is a shortened URL that may not be stable; a persistent repository identifier or DOI would be more appropriate.","section":"Section 3"}],"recommendation":"reject","confidential_remarks":"This manuscript reads like an early-stage preprint rather than a polished journal submission: the statistical reporting errors in Section 6.1 alone would require a full re-analysis, and the experimental protocol in Section 6.4 would need to be redesigned before the comparative claims can be evaluated. The corpus and the general corpus-linguistics approach could be of interest to the NLP and scientometrics communities, but the current version does not provide a valid test of the central quality-assessment claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nQuick take: the paper's ADS measure is a plausible and clearly described extension of collocation-frequency scoring to paper quality, and the released dataset (~30k papers, 9M collocations) is a genuine resource. But the central claim—that ADS measures overall paper quality—fails as submitted because the evaluation is set up so that 'high quality' and 'high-impact-venue language' are the same thing, and because key statistics in Section 6.1 don't reconcile.\n\nWhat is new and good. Applying average collocation log-likelihood (Anagnostou & Weir's idea, properly credited) to whole-paper quality prediction is new to me. The readability result—high-impact papers scoring as more readable, not less—is a useful counterpoint to older readability findings. And the authors had the right instinct to compare against a bag-of-words baseline; the held-out-conference design in Section 6.4 is more careful than most such baselines.\n\nSoft spots, in order of weight. First, circularity. The main corpus is A*/A CORE venues; the positive test set is A*/A venues (SIGKDD, SIGIR, ICDM); the negative set is WSEAS and EEEI. A paper's ADS is the average log-likelihood of its collocations in the main corpus, so the score measures how much the paper's language looks like the positive class. The paper demonstrates with the bag-of-words control that topic shifts move AUC from 0.57 to 0.74, but never runs that same topic-matched test on ADS. The load-bearing assumption—that collocation log-likelihood in high-impact venues is quality—is unvalidated. Second, the numbers don't add up. Section 6.1 text gives high-impact normalized mean=898, SD=4066, n=2320, t=5.21; Table 5 gives mean=757, variance=131703 (SD≈363), n=2303, t=32.45. Both cannot be right, and the count discrepancy (2320 vs 2303; 4822 vs 4839) appears again in Table 4 and Section 6.3. That makes the headline significance claim unverifiable as written. Third, the quality-measure classifier in Section 6.3 uses a random 80/20 split of the same conferences, so train and test share venue topics; the bag-of-words baseline is held-out and more honest, but then the 0.80/0.85 numbers are not directly comparable to 0.57/0.74.\n\nMinor: Equations (1) and (2) label ADSn 'normalized' but it divides by total words, not by collocation count—so it is density, not an average score. The text should say that explicitly.\n\nCitations are fine; the relation to Anagnostou-Weir and Louis-Nenkova is honestly drawn.\n\nBottom line: the dataset and the readability observation have value; the quality-measure claim does not survive as written. A revision with external validation (e.g., accepted vs. rejected manuscripts), topic-matched controls, and corrected statistics could change my mind. I'd accept the paper for peer review so the authors get a chance to fix those three things, but I'd expect heavy revision.","headline":"A useful collocation dataset and a plausible readability-related idea, but the central evaluation is circular and the reported statistics don't add up.","tokens_in":13012,"tokens_out":3790,"would_cite":false,"duration_ms":38773,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Averaging the log-likelihood of a paper's collocations, learned from ~30,000 top-tier CS papers, separates high- and low-impact papers and beats a bag-of-words baseline.","keywords":["collocation extraction","log-likelihood ratio","scientific paper quality","corpus linguistics","readability metrics","automated assessment","impact prediction"],"falsifier":"Take papers from the same subfield (e.g., data mining) published in both high- and low-impact venues, or accepted and rejected papers for the same conference, and compute the measure on each group while controlling for topic; if the scores do not separate the groups once topic is fixed, the measure is detecting topic rather than quality.","tokens_in":11878,"feed_emoji":"📊","tokens_out":8753,"duration_ms":72419,"temperature":0.7,"pith_summary":"The paper proposes a measure, ADS, that scores a scientific paper by the average log-likelihood of the bigram collocations it shares with a large corpus built from papers at top-ranked computer science conferences. The authors show that papers from high-impact venues receive significantly higher scores than papers from low-impact venues, and that a classifier using only this measure performs better than a standard bag-of-words baseline. If the measure is valid, it offers a way to assess paper quality from content alone, before publication and without citation or author metadata. The same approach could be carried to other fields by building a comparable gold-standard corpus.","feed_headline":"A collocation score from top venues ranks paper quality","feed_subtitle":"Average log-likelihood of a paper's bigrams, learned from 30,000 top-tier CS papers, separates high and low impact and beats a text…","key_machinery":"The central object is the collocation: a bigram that co-occurs more often than chance, scored by the log-likelihood ratio association measure. The method first builds a main corpus of papers from top-rated computer science conferences and extracts all bigram collocations, storing each collocation's log-likelihood score as a domain knowledge base. For a new paper, it looks up every collocation in that database and computes the average of the stored scores (ADS), optionally dividing by the paper's total word count to get ADSn. This transfers the statistical weight of accepted domain phrasing from the gold-standard corpus to the paper being assessed, providing a content-only quality signal.","core_discovery":"The central claim is that the average log-likelihood of a paper's collocations, where the log-likelihood weights come from a gold-standard corpus of nearly 30,000 papers from A*/A-ranked computer science conferences, captures the overall quality of the paper. The authors operationalize this as the Average Document Score (ADS) and its length-normalized variant ADSn. They report a significant difference between high-impact test papers (SIGKDD, SIGIR, ICDM) and low-impact test papers (WSEAS venues and EEEI), supported by ANOVA and independent-samples t-tests. A classifier built on the measure alone achieves an AUC of 0.80, above the bag-of-words baseline (0.57 and 0.74 in two splits), and the best classifier combines the measure with readability and basic document statistics to reach AUC 0.85.","pith_inferences":["The paper's own bag-of-words experiment shows that topic strongly affects classification, so a topic-controlled test (papers from the same subfield in high- and low-impact venues) would be needed to confirm the measure captures quality rather than subject area.","Because the raw ADS is an average over collocation types, it may be insensitive to the number of standard phrases a paper uses; the length-normalized ADSn may behave differently for verbose versus terse writing.","Papers from interdisciplinary or niche subfields could receive low scores if their collocations are underrepresented in the gold-standard corpus, so domain adaptation would likely be needed before using the measure broadly.","Extending from bigrams to trigrams or filtering out generic non-technical collocations, both suggested as future work in the paper, could alter the ranking and might strengthen the quality signal."],"forward_implications":["A paper can be scored before publication, using only its text, with no citation or author metadata.","The same method transfers to other scientific or technical fields by replacing the main corpus with papers from that field's top venues.","Combining the collocation score with readability metrics and basic document statistics yields the strongest classification, suggesting the signals are complementary.","The extracted collocation dataset from roughly 30,000 high-impact papers is a reusable resource for future research."],"supporting_citations":[{"why":"supplies the association measures, including the log-likelihood ratio, used to rank collocations.","marker":"[16]"},{"why":"previously used corpus-based collocation frequencies as a readability measure, which this method extends.","marker":"[1]"},{"why":"shows average collocation frequency as an indicator of semantic complexity, supporting the use of collocation scores as a content signal.","marker":"[2]"},{"why":"argues that domain-specific terminology matters for readability among expert readers, motivating a domain-specific collocation corpus.","marker":"[22]"}],"fun_headline_variants":["Bigram likelihood from top CS papers ranks paper quality","Collocation score from 30k elite papers separates high impact","Corpus-based collocation score predicts paper impact with AUC 0.80","Quality assessment using bigram likelihood from elite CS venues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measure assumes that how strongly a paper's phrases match the language of top-tier conferences reflects quality rather than topic or writing conventions; if high-impact venues simply favor certain subjects or phrasings, the score separates papers by topic, not by quality.","fun_headline_variants_meta":{"raw":{"variants":["Bigram likelihood from top CS papers ranks paper quality","Collocation score from 30k elite papers separates high impact","Corpus-based collocation score predicts paper impact with AUC 0.80","Quality assessment using bigram likelihood from elite CS venues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000482,"raw_usage":{"total_tokens":2392,"prompt_tokens":962,"completion_tokens":1430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":1360}},"tokens_in":578,"tokens_out":1430,"duration_ms":12443,"temperature":1.0,"reasoning_tokens":1360,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:47:59.167001+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take papers from the same subfield (e.g., data mining) published in both high- and low-impact venues, or accepted and rejected papers for the same conference, and compute the measure on each group while controlling for topic; if the scores do not separate the groups once topic is fixed, the measure is detecting topic rather than quality.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the association measures, including the log-likelihood ratio, used to rank collocations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"previously used corpus-based collocation frequencies as a readability measure, which this method extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"shows average collocation frequency as an indicator of semantic complexity, supporting the use of collocation scores as a content signal."},{"cited_title":"GOLD STANDARD","cited_arxiv_id":null,"evidence_quote":"argues that domain-specific terminology matters for readability among expert readers, motivating a domain-specific collocation corpus."}],"review_version":1}