Pith. sign in

REVIEW 3 major objections 7 minor 52 references

Is Peer-Reviewing Worth the Effort?

T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Citations in the first year after publication predict a paper's future impact better than the venue that published it.

desk verdict A clean, large-scale replication that early citations beat venue, but the forecasting claim needs out-of-sample validation before the DDI proposal carries weight. read the letter →

arxiv 2412.14351 v1 pith:VICMTZQ7 submitted 2024-12-18 cs.CL cs.AI

Kenneth Church , Raman Chandrasekar , John E. Ortega , Ibrahim Said Ahmad This is my paper · ORCID
classification cs.CLcs.AI
keywords peerreviewcitationpredictionearlycitationsvenueprestigeh-indeximpactfactorbibliometricsforecasting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether peer review is worth the effort by turning review quality into a forecasting problem: can we predict which papers will be highly cited later? The authors find, across three large publication corpora, that a paper's citation count in its first year after publication (early returns) is a much stronger predictor of its fourth-year citation percentile than the venue that published it. In their tables, papers with 20+ early citations have higher impact and h-index than every venue they examined, and papers in less selective venues with a few early citations outrank papers in more selective venues with none. The authors conclude that selecting papers by early citations would be more selective, more inclusive, and more stable than selecting by venue, and they sketch a 'Don't Do It' (DDI) alternative in which papers are posted on a preprint server and program committees focus on papers with impressive early citations plus nominations. The paper frames this as evidence that a simple citation-based rule could do much of the work peer review is currently asked to do, and it offers the DDI sketch mainly to start discussion.

What carries the argument

The central objects are 'early returns' and 'venue' as competing grouping variables, measured by three summary statistics: correlation $\rho$ between year-by-year citation counts, the h-index $h$, and impact factor $\mu$ (average citations). The paper's main mechanical move is to compare groups of papers conditioned on early citations (1+, 2+, 3+, 10+, 20+ citations in year one) against groups conditioned on venue, using fourth-year citations as the outcome, and then to fit an interpretable linear regression $percentile_{year+4} \sim venue + factor(\min(T, citations_{year+1}))$ so that each early-citation count receives its own coefficient. The h-index and impact factor are usually computed per author or per venue; here they are repurposed as evaluation metrics for the two grouping schemes, which is what lets the paper compare exclusivity, inclusivity, and stability in one analysis.

What would settle it

Find a corpus where year-one citation counts are decoupled from later impact: for example, a field with many 'sleeping beauties' (papers ignored for years that later become highly cited) or a venue where papers with zero early citations but strong reviewer endorsements consistently beat 20+ early-citation papers in fourth-year percentiles. Showing such a pattern in a large sample would refute the paper's claim that early returns dominate venue as a forecast.

Watch

Extended reading notes

Core claim

The paper's central claim is that early returns are more predictive than venue. Concretely, when papers are grouped by how many citations they received in the first year after publication, those groups have larger correlations with fourth-year citations ($\rho \approx 0.8$ between consecutive years, versus venue correlations mostly below $0.15$), higher h-index $h$ and impact $\mu$, and larger counts $N$ than grouping by venue. A regression predicting fourth-year citation percentiles from early citations plus venue shows the early-citation coefficients dominate: a paper with 10+ early citations is predicted to fall in the 75th percentile or better, while venue coefficients are small and unstable across years. The authors frame this as evidence that peer review, as currently practiced, is a costly way to achieve what a simple citation threshold already does.

Load-bearing premise

The load-bearing premise is stated in Section 2.2: reviews and other assessments of value should be leading indicators of future citations, meaning the paper equates 'important paper' with 'highly cited paper'; if citations track visibility, fashion, or manipulation rather than scholarly value, the comparison says nothing about whether peer review is worth the effort.

Editorial extensions

If this is right

  • If early returns really are more predictive than venue, readers, authors, and committees can rank candidate papers by first-year citations rather than by journal or conference prestige.
  • Conference programs that shift effort toward papers with early citations plus nominations could cut the number of full reviews while keeping or improving the quality of accepted papers, since the marginal value of a venue label is small.
  • The regression results imply that a paper with 10+ citations in year one is expected to land above the 75th percentile in year four, so early-citation thresholds provide a concrete and cheap triage rule.
  • Because the finding replicates across three large corpora and across 2016 and 2017 publication cohorts, the authors expect the advantage of early citations over venue to generalize across fields and time periods.
  • Venue effects, while statistically significant, are so small and unstable from year to year that they have little practical consequence for prioritization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not say this, but its logic implies that review scorecards could be benchmarked against early-citation forecasts: a review is useful exactly to the extent it improves prediction of later impact beyond what year-one citations already give.
  • A direct testable extension would be to run the same early-returns-versus-venue comparison on fields with long citation lags, where 'sleeping beauties' (papers ignored for years that later become highly cited) are more common; the authors note the phenomenon but do not quantify how much it erodes the rule.
  • If committees adopted the early-citation rule, author incentives would shift toward quick visibility and self-citation; the paper acknowledges that citations can be gamed but does not model how thresholds would change behavior.
  • One implicit consequence is that the 'Don't Do It' proposal treats the venue label as almost redundant for impact forecasting, which would reallocate prestige from editorial selection to community citation behavior; that change would take years to validate.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper frames the question of whether peer review identifies important papers as a forecasting problem. Using Semantic Scholar citation data for papers published in 2016-2019 in ACL Anthology, PubMed, and arXiv (with detailed analyses for 2016-2017), it compares two predictors of fourth-year citation standing: early citations (counts in the first year after publication) and publication venue. Evidence is presented in three forms: lagged correlation matrices of citation counts (Table 2) versus point-biserial correlations with venue dummies (Table 3, Figure 1); group summaries (h-index, impact factor mu, N, sigma) for papers grouped by early-citation thresholds versus by venue (Tables 4-5, Figure 2); and a regression model (Eq. 1) with a percentile outcome and factor terms for venue and early citations (Table 6, Figure 3). The authors conclude that early returns are more predictive and more robust than venue, propose posting papers on arXiv and using early citations plus nominations to triage reviewing (the DDI proposal in Section 4.2), and explicitly state the normative assumption that assessments of value should be leading indicators of future citations (Section 2.2).

Significance. If the forecasting claim is substantiated, the paper offers a broadly useful, policy-relevant result: a simple, transparent signal (early citations) that dominates venue labels for anticipating future citation standing, with implications for reading and reviewing effort at scale. The paper is strongest in its reproducibility and scale: data and code are posted on GitHub, samples include roughly one million papers per year for PubMed, and the main comparisons are replicated across three corpora and multiple publication years. The central descriptive finding is internally consistent and agrees with prior bibliometric literature (e.g., Abramo et al. 2019; Wang et al. 2013), and the normative assumption is stated transparently rather than hidden. However, two issues currently cap the significance. First, all reported evidence is in-sample; a prospective evaluation is needed before the word 'predictive' can support the DDI selection mechanism. Second, the evaluative conclusion inherits an untested assumption that citations measure value.

major comments (3)
  1. [§3.3, Eq. (1), Figure 3] The forecasting claim is only demonstrated in-sample, and this gap is load-bearing because the abstract frames the task as prediction and Section 4.2 proposes early citations as a selection mechanism. The regression described by Eq. (1) is fit once per publication year, and the boxplots in Figure 3 show fitted values for the same papers used to estimate the coefficients; no train/test split, out-of-sample R², rank correlation, or forecast-error metric is reported. The ANOVA invoked in Section 3.3 is stated without any test statistics, and Section 6 lists limitations about gaming, language coverage, and equity risks but does not acknowledge that the evaluation is entirely in-sample. This is not a circularity problem (early citations are lagged observations of the same citation stream), but it is a missing prospective test. Given the sample sizes, the shrinkage one would expect out of sample is probably modest, but the manuscript provides no way to assess it. I recommend fitting on 2016-2018 and evaluating on 2019 (or leave-one-year-out), reporting out-of-sample forecast accuracy for venue-only, early-citations-only, and combined models.
  2. [§3.1, Tables 2-3, Figure 1] The headline correlation comparison is not metric-comparable as presented. Table 2 reports Pearson correlations between two continuous citation counts (e.g., 0.80 for 2016 vs 2017 ACL papers), while Table 3 reports point-biserial correlations of individual venue indicator variables with future citation counts (e.g., 0.14 for ACL Conf). Comparing these magnitudes is apples-to-oranges: a continuous variable carries more information than a single binary dummy, so the gap in Figure 1 partly reflects variable coding rather than predictive merit. The regression in Eq. (1), which treats venue as a full factor and early citations as a factor, is the appropriate symmetric design; reporting the variance explained or multiple correlation of each factor from that model, together with the out-of-sample evaluation requested above, would put both predictors on a common footing.
  3. [§2.2, §4.2] The evaluative conclusion is conditional on an untested assumption, and this condition should be built into the conclusions and the DDI proposal. The paper states in Section 2.2 that 'we assume reviews and other assessments of value should be leading indicators of future citations,' and the recommendation to use early citations instead of reviewing inherits that assumption. If citations track visibility, author reputation, or trend rather than scholarly value, the demonstrated association advantage of early citations does not by itself establish that early citations should replace review. This is a scope concern rather than an internal inconsistency: the paper is transparent about the assumption, but the title question 'Is peer-reviewing worth the effort?' overreaches the evidence unless the conclusions read as conditional on the citation-impact definition. I suggest either explicitly conditioning the conclusions (e.g., 'for the purpose of predicting citation impact') or adding a small validation against an independent signal of value such as expert ratings, awards, or follow-on usage data.
minor comments (7)
  1. [Table 5] In the PubMed row for 2017, the sample size appears as '107,7437,' which is likely a typo for 1,077,437; please correct it.
  2. [§3.2] The text contains a few editing artifacts: 'Table 4 does this The main observation is...' is missing punctuation, and 'as evidence by the large σ' should read 'as evidenced by.'
  3. [§3.3] The percentile outcome is not fully defined: percentile within which reference set (per source, per year, pooled over sources)? Since the coefficients in Table 6 are interpreted as percentile changes, this should be specified precisely.
  4. [§3.2, Tables 4-5] The h-index comparisons between early-citation groups and venue groups conflate group size, since h grows with N; the µ comparisons are size-normalized, but the bullets in §3.2 and §4.1 that cite h should be read alongside N or a size-normalized variant such as h/N.
  5. [Figure 1 caption, §1.1] Semantic Scholar's venue field mixes journals, conferences, and non-peer-reviewed outlets, so venue correlations are an attenuated proxy for peer-review outcomes; a brief caveat near the headline figure would help readers connect the results to the title question.
  6. [§2.2] The motivation for the citation-as-value assumption cites only the authors' prior essays (Church 2005, 2020); citing the broader literature on citation-based research evaluation would strengthen the grounding.
  7. [Table 1] The header 'V enue Id' contains a stray space; Table 1 is illustrative only and would benefit from a sentence saying so.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the early-returns-vs-venue comparison is an empirical lagged association, not a derivation from its own inputs.

full rationale

The paper's central empirical claim is that early citation counts are more predictive of later citation counts than venue identity. This is not circular: the predictor 'early returns' is measured one year after publication, while the target is fourth-year citation counts, and the paper's Table 1 treats these as annual counts rather than as a cumulative total that includes the predictor by construction. The correlations in Table 2 and the venue correlations in Table 3 are empirical associations, and the regression in Eq. 1 fits coefficients for both early citations and venue on the same data; the relative sizes of those coefficients are not fixed by definition and could in principle have favored venue. Figure 3 and the ANOVA report in-sample fitted effects rather than out-of-sample forecast validation, which is a genuine weakness for a forecasting claim but is a validity limitation, not circularity. The normative premise in Section 2.2 that reviews should be leading indicators of future citations is explicitly introduced as an assumption and supported only by self-citations (Church, 2005, 2020); because it is an assumption rather than a derived result, it does not make the derivation circular. The DDI proposal in Section 4.2 would select on early citations and evaluate success by future citations, which has a self-referential policy flavor and is acknowledged in the Limitations as a risk of 'rich get richer,' but the paper's empirical comparison does not reduce to that proposal. No quoted step exhibits an equation that equals its own input, a fitted parameter renamed as a prediction, or a load-bearing uniqueness theorem imported from the authors' prior work. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The analysis rests on four stated or implicit assumptions about citation data and about using citations as ground truth for paper value. The only hand-chosen settings are the regression threshold T, the venue inclusion floor, and the early/future citation windows.

free parameters (3)
  • Threshold T for pmin transform = 10 (main regression), 30 (Figure 3)
    Chosen by hand to limit the number of factor levels in the regression; no model selection criterion is reported.
  • Venue inclusion threshold = 40 papers per venue
    Venues with fewer than 40 papers are assigned to misc; the cutoff is arbitrary and not justified by data.
  • Early and future citation windows = 1 year and 4 years after publication
    The definitions of early and future citations are chosen for convenience; the paper asserts robustness but does not report a sensitivity analysis over alternative windows.
assumptions (4)
  • ad hoc to paper Reviews and other assessments of value should be leading indicators of future citations.
    Section 2.2 states this assumption is controversial but provides an objective path forward; it is load-bearing for the title question because it equates review quality with citation prediction.
  • domain assumption Citation counts are a valid proxy for paper importance or impact.
    Throughout, future citations are used as ground truth for importance; the paper acknowledges in Section 2.2.3 and in the Limitations section that citations can be gamed.
  • domain assumption Semantic Scholar citation data and venue labels are sufficiently accurate.
    All tables rely on S2 data (Wade, 2022); no validation of S2 counts against other bibliographic sources is reported.
  • domain assumption The one-year and four-year citation windows can be fixed without loss of generality.
    Section 1.2 asserts results do not depend much on the details, but no alternative windows are tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is Peer-Reviewing Worth the Effort?." pith.science (2026). https://pith.science/paper/VICMTZQ7

@misc{pith2026241214351,
  author       = {Pith},
  title        = {Pith review of: Is Peer-Reviewing Worth the Effort?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VICMTZQ7}},
  note         = {Machine review of arXiv:2412.14351}
}
read the original abstract

How effective is peer-reviewing in identifying important papers? We treat this question as a forecasting task. Can we predict which papers will be highly cited in the future based on venue and "early returns" (citations soon after publication)? We show early returns are more predictive than venue. Finally, we end with constructive suggestions to address scaling challenges: (a) too many submissions and (b) too few qualified reviewers.

Figures

Figures reproduced from arXiv: 2412.14351 by the authors.

Figure 1
Figure 1. Early Returns (left) ≫ Venue (right), based on correlations (ρ) from Tables 2-3. Data is based on Semantic Scholar (S2) (Wade, 2022), where the venue field refers not only to conferences, but also to journals and more. How effective is peer-reviewing in identifying important papers? Since readers cannot afford to read everything, should they prioritize papers in top venues, or something else? Following Davletov et a… view at source ↗
Figure 2
Figure 2. Impact factor (µ) from [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Boxplots of predictions from regression model for ACL papers. The bars are so narrow that they are hard [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 38 canonical work pages

  1. [1]

    Giovanni Abramo, Ciriaco Andrea D’Angelo, and Giovanni Felici. 2019. Predicting publication long-term impact through a combination of early citations and journal impact factor. Journal of Informetrics, 13(1):32--49

    work page 2019

  2. [2]

    Ali Abrishami and Sadegh Aliakbary. 2019. Predicting citation counts based on deep neural network learning techniques. Journal of Informetrics, 13(2):485--499

    work page 2019

  3. [3]

    Xiaomei Bai, Fuli Zhang, and Ivan Lee. 2019. https://api.semanticscholar.org/CorpusID:86737716 Predicting the citations of scholarly paper . J. Informetrics, 13:407--418

    work page 2019

  4. [4]

    Joeran Beel and Bela Gipp. 2010. https://api.semanticscholar.org/CorpusID:7247536 Academic search engine spam and Google Scholar's resilience against it . Journal of Electronic Publishing, 13

    work page 2010

  5. [5]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. https://doi.org/10.1162/tacl_a_00051 Enriching word vectors with subword information . Transactions of the Association for Computational Linguistics, 5:135--146

    doi:10.1162/tacl_a_00051 2017

  6. [6]

    Lutz Bornmann, Robin Haunschild, and R \"u diger Mutz. 2021. Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanities and Social Sciences Communications, 8(1):1--15

    work page 2021

  7. [7]

    Lutz Bornmann, Loet Leydesdorff, and R \"u diger Mutz. 2012. https://api.semanticscholar.org/CorpusID:1106115 The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits . ArXiv, abs/1211.0381

    work page Pith review arXiv 2012

  8. [8]

    Lutz Bornmann, Loet Leydesdorff, and Jian Wang. 2014. How to improve the prediction based on citation impact percentiles for years shortly after the publication date? Journal of Informetrics, 8(1):175--180

    work page 2014

Show all 52 references
  1. [9]

    Anthony Chauvin, Philippe Ravaud, Gabriel Baron, Caroline Barnes, and Isabelle Boutron. 2015. The most important tasks for peer reviewers evaluating a randomized controlled trial are not congruent with the tasks most often requested by journal editors. BMC medicine, 13:1--10

    2015

  2. [10]

    Kunming Cheng, Zaijie Sun, Xiaojun Liu, Haiyang Wu, and Cheng Li. 2024. Generative artificial intelligence is infiltrating peer review process. Critical Care, 28(1):149

    2024

  3. [11]

    Kenneth Church. 2005. https://doi.org/10.1162/089120105775299131 Last words: Reviewing the reviewers . Computational Linguistics, 31(4):575--578

    2005 doi

  4. [12]

    Kenneth Church. 2017. Word2vec. Natural Language Engineering, 23(1):155--162

    2017

  5. [13]

    Kenneth Church. 2020. Emerging trends: Reviewing the reviewers (again). Natural Language Engineering, 26(2):245--257

    2020

  6. [14]

    Lawrence

    Corinna Cortes and Neil D. Lawrence. 2021. https://api.semanticscholar.org/CorpusID:237581155 Inconsistency in conference peer review: Revisiting the 2014 neurips experiment . ArXiv, abs/2109.09774

    2021 arXiv

  7. [15]

    Feruz Davletov, Ali Selman Aydin, and Ali Cakmak. 2014. https://doi.org/10.1145/2661829.2662066 High impact academic paper prediction using temporal and topological features . In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Ma...

    2014

  8. [16]

    Dennis R De Vries, Elizabeth A Marschall, and Roy A Stein. 2009. Exploring the peer review process: what is it, does it work, and can it be improved? Fisheries, 34(6):270--279

    2009

  9. [17]

    Joost de Winter. 2024. https://doi.org/10.1007/s11192-024-04939-y Can ChatGPT be used to predict citation counts, readership, and social media interaction? an exploration among 2222 scientific abstracts . Scientometrics, 129(4):2469–2487

    2024 doi

  10. [18]

    Gunther Eysenbach. 2006. Citation advantage of open access articles. PLoS biology, 4(5):e157

    2006

  11. [19]

    Eugene Garfield. 2006. https://api.semanticscholar.org/CorpusID:31183037 The history and meaning of the journal impact factor. JAMA, 295 1:90--3

    2006

  12. [20]

    Alexander Goldberg, Ivan Stelmakh, Kyunghyun Cho, Alice Oh, Alekh Agarwal, Danielle Belgrave, and Nihar B. Shah. 2023. https://api.semanticscholar.org/CorpusID:265220716 Peer reviews of peer reviews: A randomized controlled trial and other experiments . ArXiv, abs/2311.09497

    2023 arXiv

  13. [21]

    Jorge E Hirsch. 2005. An index to quantify an individual's scientific research output. Proceedings of the National academy of Sciences, 102(46):16569--16572

    2005

  14. [22]

    Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. 2020. https://api.semanticscholar.org/CorpusID:218487328 Open Graph Benchmark: D atasets for machine learning on graphs . ArXiv, abs/2005.00687

    2020 arXiv

  15. [23]

    Shengzhi Huang, Yong Huang, Yi Bu, Wei Lu, Jiajia Qian, and Dan Wang. 2022. https://doi.org/10.1016/j.ipm.2021.102799 Fine-grained citation count prediction via a transformer-based model with among-attention mechanism . Inf. Process. Manage., 59(2)

    2022

  16. [24]

    Jürgen Huber, Sabiou Inoua, Rudolf Kerschbamer, Christian König-Kersting, Stefan Palan, and Vernon L. Smith. 2022. https://doi.org/10.1073/pnas.2205779119 Nobel and novice: Author prominence affects peer review . Proceedings of the National Academy of Sciences, 119(41):e2205779119

    2022 doi

  17. [25]

    Ken Hyland. 2023. https://doi.org/10.1002/leap.1545 Enter the dragon: China and global academic publishing . Learned Publishing, 36(3):394--403

    2023 doi

  18. [26]

    Tom Jefferson, Philip Alderson, Elizabeth Wager, and Frank Davidoff. 2002. https://doi.org/10.1001/jama.287.21.2784 Effects of Editorial Peer Review: A Systematic Review . JAMA, 287(21):2784--2786

    2002 doi

  19. [27]

    Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications

    2018

  20. [28]

    Steve Lawrence. 2001. Free online availability substantially increases a paper's impact. Nature, 411(6837):521--521

    2001

  21. [29]

    Lee, Cassidy R

    Carole J. Lee, Cassidy R. Sugimoto, Guo Zhang, and Blaise Cronin. 2013. https://doi.org/10.1002/asi.22784 Bias in peer review . Journal of the American Society for Information Science and Technology, 64(1):2--17

    2013 doi

  22. [30]

    Benjamin List. 2017. https://api.semanticscholar.org/CorpusID:4447953 Crowd-based peer review can be good and fast . Nature, 546:9--9

    2017

  23. [31]

    Anqi Ma, Yu Liu, Xiujuan Xu, and Tao Dong. 2021. https://doi.org/10.1007/s11192-021-04033-7 A deep-learning based citation count prediction model with paper metadata semantic features . Scientometrics, 126(8):6803–6823

    2021 doi

  24. [32]

    Mark E. J. Newman. 2013. https://api.semanticscholar.org/CorpusID:14898987 Prediction of highly cited papers . Europhysics Letters, 105

    2013

  25. [33]

    Natsuo Onodera and Fuyuki Yoshikane. 2015. Factors affecting citation rates of research articles. Journal of the Association for Information Science and Technology, 66(4):739--764

    2015

  26. [34]

    Nataliia Pobiedina and Ryutaro Ichise. 2016. https://api.semanticscholar.org/CorpusID:16492250 Citation count prediction as a link prediction problem . Applied Intelligence, 44:252--268

    2016

  27. [35]

    Sidney Redner. 2005. https://api.semanticscholar.org/CorpusID:119554534 Citation statistics from 110 years of Physical Review . Physics Today, 58:49--54

    2005

  28. [36]

    Giuseppe Remuzzi. 2023. The ethics of peer review process. Updates in Surgery, 75(6):1391--1392

    2023

  29. [37]

    Sara Rockwell. 2006. Ethics of peer review: a guide for manuscript reviewers. URL: http://ori. dhhs. gov/education/products/yale/prethics. pdf.[Accessed on: 01-09-2013]

    2006

  30. [38]

    Anna Rogers and Isabelle Augenstein. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.112 What can we do to improve peer review in NLP ? In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1256--1262, Online. Association for Computational Linguistics

    2020 doi

  31. [39]

    Tony Ross-Hellauer. 2017. https://api.semanticscholar.org/CorpusID:1524207 What is open peer review? a systematic review . F1000Research, 6

    2017

  32. [40]

    Xuanmin Ruan, Yuanyang Zhu, Jiang Li, and Ying Cheng. 2020. Predicting the citation counts of individual papers via a BP neural network. Journal of Informetrics, 14(3):101039

    2020

  33. [41]

    David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. 2020. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International journal of forecasting, 36(3):1181--1191

    2020

  34. [42]

    Olivia M Smith, Kayla L Davis, Riley B Pizza, Robin Waterman, Kara C Dobson, Brianna Foster, Julie C Jarvey, Leonard N Jones, Wendy Leuenberger, Nan Nourn, et al. 2023. Peer review perpetuates barriers for historically excluded groups. Nature Ecology & Evolution, 7(4):512--523

    2023

  35. [43]

    Lawrence Souder. 2011. The ethics of scholarly peer review: a review of the literature. Learned Publishing, 24(1):55--72

    2011

  36. [44]

    Clara Stegehuis, Nelly Litvak, and Ludo Waltman. 2015. Predicting the long-term citation impact of recent publications. Journal of informetrics, 9(3):642--657

    2015

  37. [45]

    Iman Tahamtan, Askar Safipour Afshar, and Khadijeh Ahamdzadeh. 2016. Factors affecting number of citations: a comprehensive review of the literature. Scientometrics, 107:1195--1225

    2016

  38. [46]

    Anthony F. J. van Raan. 2004. https://api.semanticscholar.org/CorpusID:189864678 Sleeping beauties in science . Scientometrics, 59:467--472

    2004

  39. [47]

    Alex D Wade. 2022. The Semantic Scholar Academic Graph (S2AG) . Companion Proceedings of the Web Conference 2022

    2022

  40. [48]

    Dashun Wang, Chaoming Song, and Albert Ĺaszl \'o Barab \'a si. 2013. https://api.semanticscholar.org/CorpusID:260558492 Quantifying long-term scientific impact . Science, 342:127 -- 132

    2013

  41. [49]

    Pengwei Yan, Yangyang Kang, Zhuoren Jiang, Kaisong Song, Tianqianjin Lin, Changlong Sun, and Xiaozhong Liu. 2024. https://doi.org/10.1145/3626772.3657926 Modeling scholarly collaboration and temporal dynamics in citation networks for impact prediction . In Proceedings of the 4...

    2024

  42. [50]

    Rui Yan, Jie Tang, Xiaobing Liu, Dongdong Shan, and Xiaoming Li. 2011. https://doi.org/10.1145/2063576.2063757 Citation count prediction: learning to estimate future citations for literature . In Proceedings of the 20th ACM International Conference on Information and Knowledge...

    2011

  43. [51]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  44. [52]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.