Pith. sign in

REVIEW 4 major objections 4 minor 22 references

Assessing the Quality of Scientific Papers

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Averaging the log-likelihood of a paper's collocations, learned from ~30,000 top-tier CS papers, separates high- and low-impact papers and beats a bag-of-words baseline.

desk verdict A useful collocation dataset and a plausible readability-related idea, but the central evaluation is circular and the reported statistics don't add up. read the letter →

arxiv 1908.04200 v1 pith:IW6DZSKA submitted 2019-08-12 cs.IR cs.CLcs.DL

Roman Vainshtein , Gilad Katz , Bracha Shapira , Lior Rokach This is my paper · ORCID
classification cs.IRcs.CLcs.DL
keywords collocationextractionlog-likelihoodratioscientificpaperqualitycorpuslinguisticsreadabilitymetricsautomatedassessmentimpactprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a measure, ADS, that scores a scientific paper by the average log-likelihood of the bigram collocations it shares with a large corpus built from papers at top-ranked computer science conferences. The authors show that papers from high-impact venues receive significantly higher scores than papers from low-impact venues, and that a classifier using only this measure performs better than a standard bag-of-words baseline. If the measure is valid, it offers a way to assess paper quality from content alone, before publication and without citation or author metadata. The same approach could be carried to other fields by building a comparable gold-standard corpus.

What carries the argument

The central object is the collocation: a bigram that co-occurs more often than chance, scored by the log-likelihood ratio association measure. The method first builds a main corpus of papers from top-rated computer science conferences and extracts all bigram collocations, storing each collocation's log-likelihood score as a domain knowledge base. For a new paper, it looks up every collocation in that database and computes the average of the stored scores (ADS), optionally dividing by the paper's total word count to get ADSn. This transfers the statistical weight of accepted domain phrasing from the gold-standard corpus to the paper being assessed, providing a content-only quality signal.

What would settle it

Take papers from the same subfield (e.g., data mining) published in both high- and low-impact venues, or accepted and rejected papers for the same conference, and compute the measure on each group while controlling for topic; if the scores do not separate the groups once topic is fixed, the measure is detecting topic rather than quality.

Watch

Extended reading notes

Core claim

The central claim is that the average log-likelihood of a paper's collocations, where the log-likelihood weights come from a gold-standard corpus of nearly 30,000 papers from A*/A-ranked computer science conferences, captures the overall quality of the paper. The authors operationalize this as the Average Document Score (ADS) and its length-normalized variant ADSn. They report a significant difference between high-impact test papers (SIGKDD, SIGIR, ICDM) and low-impact test papers (WSEAS venues and EEEI), supported by ANOVA and independent-samples t-tests. A classifier built on the measure alone achieves an AUC of 0.80, above the bag-of-words baseline (0.57 and 0.74 in two splits), and the best classifier combines the measure with readability and basic document statistics to reach AUC 0.85.

Load-bearing premise

The measure assumes that how strongly a paper's phrases match the language of top-tier conferences reflects quality rather than topic or writing conventions; if high-impact venues simply favor certain subjects or phrasings, the score separates papers by topic, not by quality.

Editorial extensions

If this is right

  • A paper can be scored before publication, using only its text, with no citation or author metadata.
  • The same method transfers to other scientific or technical fields by replacing the main corpus with papers from that field's top venues.
  • Combining the collocation score with readability metrics and basic document statistics yields the strongest classification, suggesting the signals are complementary.
  • The extracted collocation dataset from roughly 30,000 high-impact papers is a reusable resource for future research.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own bag-of-words experiment shows that topic strongly affects classification, so a topic-controlled test (papers from the same subfield in high- and low-impact venues) would be needed to confirm the measure captures quality rather than subject area.
  • Because the raw ADS is an average over collocation types, it may be insensitive to the number of standard phrases a paper uses; the length-normalized ADSn may behave differently for verbose versus terse writing.
  • Papers from interdisciplinary or niche subfields could receive low scores if their collocations are underrepresented in the gold-standard corpus, so domain adaptation would likely be needed before using the measure broadly.
  • Extending from bigrams to trigrams or filtering out generic non-technical collocations, both suggested as future work in the paper, could alter the ranking and might strengthen the quality signal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes ADS, an average document score based on the mean log-likelihood of bigram collocations drawn from a main corpus of roughly 30,000 papers from CORE A*/A-ranked computer science conferences. The authors compare ADS on papers from high-impact conferences (SIGKDD, SIGIR, ICDM) against papers from low-impact venues (WSEAS, EEEI), report statistically significant differences via ANOVA and independent-samples t-tests, and build a logistic regression classifier that achieves AUC 0.85 when ADS is combined with readability and statistical features. They also compare against a bag-of-words SVM baseline, reporting AUC 0.57 (or 0.74 in a second setup), and conclude that the proposed measure can be used for automated assessment of scientific paper quality.

Significance. The idea of using corpus-derived collocation statistics for automatic paper quality assessment is novel, and the constructed main corpus (nearly 30,000 papers) is a potentially reusable resource. The authors state explicit hypotheses and report a variety of statistical and classification experiments, which is commendable. However, the central claim that ADS measures 'overall quality' is not established: the reference corpus and the test labels originate from the same venue-ranking system, no external quality ground truth is used, the headline statistics in Section 6.1 are internally inconsistent, and the comparison against the bag-of-words baseline is not controlled. If the validity issues were addressed, the measure might serve as a screening tool for domain-specific writing typicality, but as presented the evidence does not support the paper's claims.

major comments (4)
  1. [Section 6.1, Table 5] The reported statistics for the normalized ADS are mutually inconsistent. The text states that high-impact papers have mean=898, SD=4066, and t(2337)=5.21, while Table 5 reports mean=757, variance=131703 (SD≈363), observations=2303, and t=32.45. The high-impact paper count is also given as 2,320 in the text but sums to 2,303 in Section 3 (SIGKDD 490, SIGIR 406, ICDM 1407). Because the primary evidence for the measure's discriminating power depends on these numbers, the main results cannot be verified without resolved reporting.
  2. [Sections 3 and 5.4] The measure is defined using log-likelihood weights estimated exclusively from papers in A*/A-ranked venues, and the test corpora are labeled high/low impact by the same CORE/Google Scholar ranking criteria. Under these conditions, ADS measures the degree to which a paper's collocations match the language of the positive class. The bag-of-words control in Section 6.4 shows that making the low-impact test set topically closer to the training set raises the AUC from 0.57 to 0.74, demonstrating that topic composition strongly affects content-based classification. No comparable topic-matched evaluation is performed for ADS, so the separation it observes may reflect conference-specific topics and writing conventions rather than paper quality.
  3. [Section 6.4] The comparison between the proposed classifier and the bag-of-words baseline is not a controlled one. The classifiers in Section 6.3 are trained on an 80/20 split of the 4,822 test-corpus papers, whereas the bag-of-words model in Section 6.4 trains on 3,000 papers from the main high-impact corpus plus 1,718 low-impact papers, with a different test set. Because the training sets, test sets, and feature representations differ simultaneously, the claim that the proposed measure 'over-performed the baseline classifier' is not supported by the reported experiments.
  4. [Section 5.4 and Conclusions] The paper never validates ADS against an independent measure of paper quality, such as citation counts, acceptance decisions, or expert judgments. The only target is the venue's impact ranking, which also determines the contents of the reference corpus. This circularity means the construct validity of 'quality' as measured by ADS remains unestablished; the conclusion claims the measure can 'predict the quality and impact of a scientific paper,' but the evidence only shows separation between preselected venue groups.
minor comments (4)
  1. [Section 2] The sentence 'In our work, explore collocations as a means of compensating for the limitations of readability metrics' is missing a verb; it should read 'we explore collocations.'
  2. [Table 4] The header 'AverageVariance' is missing a space; the table would also be clearer if it included standard deviations alongside the variances.
  3. [Section 5.4] The phrase 'the rate of spelling errors in scientific paper' is unclear; the intended meaning appears to be that the measure reflects correct and standard usage, but the wording is confusing.
  4. [Section 3] The dataset download link (https://goo.gl/1s4Jkx) is a shortened URL that may not be stable; a persistent repository identifier or DOI would be more appropriate.

Circularity Check

1 steps flagged · score 6.0 of 10

ADS validation is construct-circular: main-corpus 'gold standard' and test labels come from the same CORE/Google Scholar venue ranking.

  1. self definitional [Section 3 (corpora and rankings), Section 5.4 (Eq. 1), Section 6.1 (hypothesis)]
    "The main corpus will serve as our "knowledge source" and as the "gold standard"... selection criteria for papers was that they had to be published in conferences with A* or A rankings... The impact ranking metrics that we used to select the conferences are based on the CORE Conference Rankings Portal and Google Scholar's citation metrics... Our hypothesis is that high impact corpus papers will obtain a higher score, and consequently, low impact corpus papers will obtain lower scores."

    ADS (Eq. 1) is the average of the main-corpus log-likelihood scores fi assigned to a test paper's collocations. The main corpus is selected from A*/A CORE-ranked conferences and is declared the "gold standard"; the test corpora are labeled high vs. low impact by the same CORE/Google Scholar ranking (Sec. 3, Table 1). Consequently the test "prediction" measures how much a paper's collocations resemble the language of A*/A-ranked venues. High-impact papers scoring higher is therefore a consequence of the shared selection criterion, not an independent validation that the score tracks paper quality. The paper's own note that the separation disappears if the main corpus is not high-impact (Sec.

full rationale

The central derivation of ADS is not mathematically circular in the narrow sense that fi is fitted to the test labels: no test label is used to estimate the collocation log-likelihoods. However, the validation is construct-circular: the gold-standard main corpus and the test labels are defined by the same CORE/Google Scholar conference-ranking criterion. The reported significant difference between high- and low-impact groups is therefore largely a check of internal consistency, not evidence that ADS measures an independent property of paper quality. No self-citation chain is involved. I also note that the Section 6.1 statistics are internally inconsistent (text reports high-impact mean=898, SD=4066, t=5.21; Table 5 reports mean=757, variance=131703, SD roughly 363, and t=32.45), which weakens the empirical claim but is a correctness issue rather than circularity. The bag-of-words control (Sec. 6.4) demonstrates a topic confound for content-based classification but is not applied to ADS, so it does not by itself establish circularity; the circularity lies in using the same venue-ranking criterion to build the gold standard and to label the test set.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the selection of a 'gold standard' high-impact corpus and on the assumption that collocation weights derived from that corpus are quality markers. These choices are domain assumptions, not independently validated facts. The collocation weights are fitted values in the sense that they are estimated from the training corpus and then used as the scoring function.

free parameters (2)
  • Collocation log-likelihood weights (f_i) = Computed from the main corpus of ~29,848 papers from 60 A*/A-ranked conferences (e.g., source code: 132,755.20)
    Each collocation's weight is estimated from the main corpus, which consists of the same category of venues used to label the test corpora as high or low impact. These weights, rather than any independent quality criterion, define the quality score.
  • Main corpus composition = 60 IEEE/ACM conferences selected from CORE A*/A rankings
    The manual choice of which conferences constitute 'high impact' determines all collocation scores, and no sensitivity analysis is provided for other corpus compositions.
assumptions (4)
  • domain assumption Papers published in CORE A*/A-ranked IEEE and ACM conferences are a valid gold standard of scientific paper quality.
    This assumption underlies both the main corpus and the high/low impact labels. The paper never validates this equivalence against independent quality measures such as expert judgment or citation impact.
  • ad hoc to paper The log-likelihood of a collocation in the main corpus is a valid index of the quality of any paper that uses that collocation.
    This equates collocation typicality in high-impact venues with quality, ignoring topic and stylistic conventions. The paper's own bag-of-words experiments show topic strongly affects classification.
  • domain assumption PDF-to-plain-text conversion preserves collocations accurately for all corpora.
    The preprocessing pipeline is not evaluated for conversion errors, which can vary by PDF format and paper quality.
  • domain assumption Bigram collocations alone are sufficient to represent domain-specific writing quality.
    Trigrams and longer collocations are not tested, and the paper lists their absence as a future direction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assessing the Quality of Scientific Papers." pith.science (2026). https://pith.science/paper/IW6DZSKA

@misc{pith2026190804200,
  author       = {Pith},
  title        = {Pith review of: Assessing the Quality of Scientific Papers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IW6DZSKA}},
  note         = {Machine review of arXiv:1908.04200}
}
read the original abstract

A multitude of factors are responsible for the overall quality of scientific papers, including readability, linguistic quality, fluency,semantic complexity, and of course domain-specific technical factors. These factors vary from one field of study to another. In this paper, we propose a measure and method for assessing the overall quality of the scientific papers in a particular field of study. We evaluate our method in the computer science domain, but it can be applied to other technical and scientific fields.Our method is based on the corpus linguistics technique. This technique enables the extraction of required information and knowledge associated with a specific domain. For this purpose, we have created a large corpus, consisting of papers from very high impact conferences. First, we analyze this corpus in order to extract rich domain-specific terminology and knowledge. Then we use the acquired knowledge to estimate the quality of scientific papers by applying our proposed measure. We examine our measure on high and low scientific impact test corpora. Our results show a significant difference in the measure scores of the high and low impact test corpora. Second, we develop a classifier based on our proposed measure and compare it to the baseline classifier. Our results show that the classifier based on our measure over-performed the baseline classifier. Based on the presented results the proposed measure and the technique can be used for automated assessment of scientific papers.

Figures

Figures reproduced from arXiv: 1908.04200 by the authors.

Figure 1
Figure 1. The entire process of knowledge acquisition and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages

  1. [1]

    N. K. Anagnostou and G. R. Weir. From corpus-based collocation frequencies to readability measure. In ICT in the Analysis, Teaching and Learning of Languages, Preprints of the ICTATLL Workshop 2006, pages 33–46, 2006

    work page 2006

  2. [2]

    N. K. Anagnostou and G. R. Weir. Average collocation frequency as an indicator of semantic complexity. In ICTATLL Workshop 2007 Preprints, pages 1–3, 2007

    work page 2007

  3. [3]

    Bailin and A

    A. Bailin and A. Grafstein. Towards a theory of readability. In Readability: Text and Context, pages 177–204. Springer, 2016

    work page 2016

  4. [4]

    J.-Y. Chang. The use of general and specialized corpora as reference sources for academic english writing: A case study. ReCALL, 26(02):243–259, 2014

    work page 2014

  5. [5]

    Coleman and T

    M. Coleman and T. L. Liau. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283, 1975

    work page 1975

  6. [6]

    Dolnicar and A

    S. Dolnicar and A. Chapple. The readability of articles in tourism journals.Annals of Tourism Research, 52:161–166, 2015

    work page 2015

  7. [7]

    Y. Dong, R. A. Johnson, and N. V. Chawla. Will this paper increase your h-index?: Scientific impact prediction. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining , pages 149–158. ACM, 2015

    work page 2015

  8. [8]

    R. F. Flesch et al. Art of readable writing. 1949

    work page 1949

Show all 22 references
  1. [9]

    Z.-M. Gao. Automatic extraction of english collocations and their chinese-english bilingual examples: A computational tool for bilingual lexicography. Concentric: Studies in Linguistics, 40(1):95–121, 2014

    2014

  2. [10]

    A. Gazni. Are the abstracts of high impact articles more readable? investigating the evidence from top research institutions in the world. Journal of Information Science, 37(3):273–281, 2011

    2011

  3. [11]

    R. Gunning. {The Technique of Clear Writing}. 1952

    1952

  4. [12]

    J. P. Kincaid, R. P. Fishburne Jr, R. L. Rogers, and B. S. Chissom. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, DTIC Document, 1975

    1975

  5. [13]

    G. Klare. The measurement of readability . Iowa State University Press, 1963

    1963

  6. [14]

    Louis and A

    A. Louis and A. Nenkova. A corpus of science journalism for analyzing writing quality. Dialogue and Discourse, 4(2):87–117, 2013

    2013

  7. [15]

    Louis and A

    A. Louis and A. Nenkova. What makes writing great? first experiments on article quality prediction in the science journalism domain. Transactions of the Association for Computational Linguistics , 1:341–352, 2013

    2013

  8. [16]

    C. D. Manning and H. Schütze. Foundations of statistical natural language pro- cessing, volume 999. MIT Press, 1999

    1999

  9. [17]

    G. H. Mc Laughlin. Smog grading-a new readability formula. Journal of reading, 12(8):639–646, 1969

    1969

  10. [18]

    McEnery and A

    T. McEnery and A. Hardie. Corpus linguistics: Method, theory and practice . Cam- bridge University Press, 2011

    2011

  11. [19]

    McEnery and A

    T. McEnery and A. Wilson. Corpus linguistics: An introduction . Edinburgh University Press, 2001

    2001

  12. [20]

    Pitler, A

    E. Pitler, A. Louis, and A. Nenkova. Automatic evaluation of linguistic quality in multi-document summarization. In Proceedings of the 48th annual meeting of the Association for Computational, pages 544–554. Association for Computational Linguistics, 2010

    2010

  13. [21]

    Senter and E

    R. Senter and E. A. Smith. Automated readability index. Technical report, DTIC Document, 1967

    1967

  14. [22]

    GOLD STANDARD

    G. R. Weir and N. K. Anagnostou. Optimising content clarity for human-machine systems. IFAC Proceedings Volumes, 40(16):314–319, 2007. A MAIN CORPUS SOURCES(THE CORPUS SERVES AS THE "GOLD STANDARD") Short Name, dates, venue Roman Vainshtein, Gilad Katz, Bracha Shapira, and Lio...

    2007

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.