Pith. sign in

REVIEW 4 major objections 4 minor 20 references

BIP! Ranker: A Software Library for Citation-Based Impact Indicators on Large-Scale Graphs

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BIP! Ranker claims a single open-source, distributed library can compute citation-based impact indicators across billions of citations in under about 80 minutes.

desk verdict Useful, shipping Spark library for citation indicators — the gap is real, but the paper claims correctness without a single numerical check. read the letter →

arxiv 2608.02004 v1 pith:YDVOWISQ submitted 2026-08-03 cs.DL cs.IR

Ilias Kanellos , Serafeim Chatzopoulos , Thanasis Vergoulis This is my paper
classification cs.DLcs.IR
keywords citation-basedimpactindicatorsdistributedgraphprocessingPageRankscientometricsopen-sourcesoftwarecitationnetworksfield-weighted
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces BIP! Ranker, an open distributed library whose purpose is to compute a diverse set of citation-based impact indicators — counting influence, popularity, early momentum, and field-relative performance — on citation graphs too large for single-machine tools. Its central claim is that one unified implementation, built on a distributed cluster engine, can process a snapshot of hundreds of millions of publications and billions of citation links in practical wall-clock times, producing both scores and percentile/field classes. A sympathetic reader would care because research evaluation and discovery currently lean on a single number like raw citation count, and reproducible, scalable alternatives have been scarce. The paper supports the claim with a large-scale run reporting citation count in about 24 minutes and iterative graph scores in about 72–79 minutes.

What carries the argument

The load-bearing mechanism is a Spark-on-distributed-storage pipeline built around a shared citation-graph representation and per-indicator scripts. For iterative indicators (PageRank, AttRank, ECM), the implementation checkpoints intermediate DataFrames during iteration to stop lineage growth, which is what lets the graph algorithms finish on a 385-million-node, 2.1-billion-edge network. Two identities let the families share computation paths: RAM is the single-hop case of ECM's chain-based propagation, and iCC is a time-restricted CC.

What would settle it

Take the released code and a citation graph small enough to solve exactly on one machine (a few thousand nodes), run both the distributed and single-node implementations of PageRank, AttRank, and ECM, and check whether all scores agree to within the configured convergence tolerance; any systematic drift would falsify the claimed equivalence.

Watch

Extended reading notes

Core claim

The paper's central assertion is that the multidimensional measurement of scientific impact can be put into production at the scale of major scholarly databases. BIP! Ranker implements the standard definitions of the selected indicators — Citation Count and PageRank for influence; RAM, ECM, and AttRank for popularity; iCC for early momentum; FWCI and its N-year variant for field-normalisation — and organizes them in one Spark-based pipeline that writes scores and percentile-based classes to distributed storage. The reported run on a 2.1-billion-citation network shows that even iterative, score-propagating algorithms complete in roughly one hour on a cluster, so the barrier to using complemen

Load-bearing premise

The load-bearing assumption is that the checkpointed distributed iteration converges to the same fixed points as the published equations for PageRank, AttRank, and ECM; the paper reports runtimes over a 2.1-billion-edge graph but does not show a reference-implementation cross-check, convergence diagnostics, or score-equality unit tests (Sections 3.2–3.3).

Editorial extensions

If this is right

  • Any research group with a citation-graph extract in the library's input format can reproduce these indicator scores and percentile classes without relying on proprietary precomputed metrics.
  • Literature-ranking services can present several complementary views of impact (long-term, current, momentum, field-relative) at a cost measured in tens of minutes on a 2.1-billion-citation graph.
  • Field-level and field-normalised classes become computable for the same large graph, enabling cross-disciplinary comparisons that raw counts cannot support.
  • Because the source code and a versioned archive are released, the indicators can be audited against their published definitions and re-run on fresh graph snapshots.
  • The checkpointing and partitioning choices form a reusable pattern for other iterative bibliometric algorithms operating at billion-edge scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of correctness would run the released code on a modest graph with a single-node reference implementation of PageRank, AttRank, and ECM and compare the distributed scores to the exact outputs; the paper itself reports performance but not this convergence check.
  • The library's input format appears generic enough that it could be pointed at other openly available scholarly graph exports with only mapping-layer changes, though the paper demonstrates results on one graph.
  • With reliable multidimensional indicators in hand, the next plausible step is to measure whether search and recommendation systems actually improve when they consume these signals instead of raw counts — an evaluation the paper does not attempt.
  • The reported runtimes imply that periodic recomputation over a refreshed scholarly graph is feasible on a moderate cluster, allowing impact scores to track a moving literature rather than freeze at a single snapshot.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces BIP! Ranker, an open-source Spark/PySpark library for computing citation-based impact indicators on large citation graphs. It implements Citation Count, PageRank, RAM, ECM, AttRank, iCC, and field-normalized variants (FWCI and N-year FWCI), plus percentile-based and field-level impact classes. The main empirical claim, reported in Section 3.3, is that on a 385M-publication, 2.1B-citation OpenAIRE Graph extract, running on a 32-worker cluster with 1,280 vCPUs, the library computed CC in about 24 minutes, PageRank in about 79 minutes, and AttRank in about 72 minutes. The artifact is publicly released on GitHub (v2.0.0) and archived on Zenodo, with a runnable demo and documentation.

Significance. If the central claims are correct, BIP! Ranker fills a genuine gap: it is a rare open, unified implementation of several complementary bibliometric indicators designed to operate at the scale of modern scholarly knowledge graphs. The paper's credibility is strengthened by the fact that the software is demonstrably shipped: the GitHub release, Zenodo DOI, run_demo.py, and detailed usage examples make the artifact independently inspectable. The reported operational adoption by OpenAIRE and bio.tools also indicates practical impact. However, the paper's scientific claim is not merely that the code runs, but that the Spark outputs are faithful to the indicator definitions in Section 3.2 and that the reported runtimes are representative. The current evidence for those two load-bearing points is thin, so the significance is conditional on validation that the manuscript does not yet provide.

major comments (4)
  1. [§3.3 and §6] The reproducibility claim in §6 that each indicator 'directly follows' the definitions in Section 3.2 is not supported by any numerical validation. The paper reports no comparison of the Spark outputs against a reference implementation on any graph, no unit tests on score equality, no convergence diagnostics for PageRank/AttRank, and no tolerance values against which the iterative results were checked. For a library whose value depends on computing the published indicators, this is load-bearing. I ask the authors to add a validation subsection: run the released code on small and medium citation graphs, compare against a single-node reference (e.g., NetworkX or a direct dense-matrix evaluation of Eqs. (1)–(6)), and report maximum absolute/relative differences and convergence thresholds. This would let the reader distinguish faithful implementation from approximation at the 2.1B-edge scale
  2. [§3.3] The scaling evidence consists of a single run on one cluster configuration. There is no runtime variance, no indication of how the 24/79/72-minute figures were measured (end-to-end including input parsing? pure compute? including checkpointing writes?), and no sensitivity to resource allocation or Spark partitioning. The claim that the library 'can process' 2.1B edges at those speeds would be considerably stronger with a small parameter study (e.g., edge-count scaling, worker-count scaling, or at least repeated runs). At minimum, the paper should specify the measurement methodology and report whether the cited times are reproducible.
  3. [§3.2, Eqs. (2), (4), (5)] The definitions in Section 3.2 leave several preprocessing and algorithmic conventions unspecified that materially affect the computed scores. For PageRank and AttRank: how are duplicate citation edges, self-loops, and dangling nodes (zero out-degree) handled in forming the column-normalized matrix P? What is the convergence error used in the experiments? For ECM: is the matrix power R^h accumulated by repeated multiplication, and is the truncated sum (4) exactly what the Spark job computes, or is some approximation used for the 2.1B-edge graph? These choices are not merely cosmetic: they can change the fixed points and hence the reported indicator scores. The authors should state the conventions explicitly and, ideally, test their sensitivity on a small graph.
  4. [§3.2, Eq. (5) and §3.3] AttRank is defined with parameters α, β, γ, ρ, a normalization constant c, and an attention score Att(i), but the paper gives no values or algorithm for these quantities. The runtimes in §3.3 are therefore associated with an underspecified configuration. The PageRank paragraph mentions default values 'chosen to perform well on citation networks' but does not give them. Since the paper claims reproducibility and that each indicator directly follows its definition, the authors should list the exact default parameters used in the reported experiments (including the α, β, γ, ρ, c, Att(i) construction, the PageRank damping factor, and the convergence threshold).
minor comments (4)
  1. [§3.3] Typo: 'Hadoop Y ARN' should be 'Hadoop YARN'. Also, the OpenAIRE Graph extract is described only as 'a citation network extracted from the OpenAIRE Graph'; the specific dataset version or snapshot date is not given, which is important for reproducibility since the OpenAIRE Graph is updated every six months.
  2. [§2] The Related Work states that BIP! Ranker has 'tested computation parameters and conventions,' but the paper does not report those tests. Either provide the tests or soften this claim so it does not anticipate the missing validation.
  3. [§3.2] In Eq. (3), RAM's decay uses t_c and t_j but it is not stated whether t_c is the current calendar year, the year of the latest citation in the dataset, or a user-supplied evaluation year. Please clarify, and likewise define the domain of t_j for papers published after t_c.
  4. [§4] The code examples show command lines for CC.py and PageRank.py, but it would help to give a single end-to-end example that also shows how the output scores are formatted and how to invoke the field-normalization script with a realistic mapping file, since that is part of the 'complete workflow' claimed in run_demo.py.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is a software-description with no fitted predictions; its self-citations are not load-bearing and the central implementational claim is publicly checkable.

full rationale

BIP! Ranker is a systems paper. Its central claim is that an open-source Spark library computes the specified citation-based indicators (CC, PageRank, RAM, ECM, AttRank, iCC, FWCI) at OpenAIRE scale. There is no derivation chain in which an input is defined in terms of an output, and no fitted parameter is relabeled as a prediction. The indicator definitions in Section 3.2 (Eqs. 1-8) are taken from the literature (PageRank [14], RAM/ECM [1], AttRank [2], FWCI [3]); the paper states that each implementation 'directly follows' these definitions, which is a checkable implementation claim rather than a circular derivation. The self-citations in the paper ([2], [4], [16]-[19]) document the authors' prior work, the AttRank definition, and the authors' own services that use the library, but none of these citations supplies the mathematical content of the implemented indicators in a way that makes the library's output equal to its input by construction. In particular, AttRank's defining equation is reproduced as Eq. 5 in this paper, so the load-bearing definition is present and externally checkable. The absence of a numerical cross-check against a reference implementation at the 2.1B-edge scale is a real validation gap and a legitimate correctness risk, but it is not circularity: no result is forced by a self-citation chain or by definition. The public GitHub release and Zenodo archive make the claim that the code 'directly follows' the definitions independently verifiable, which is precisely the kind of external checkpoint that keeps the circularity score at zero.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central scaling claim rests on standard graph-algorithm convergence assumptions, on trust in the input metadata (publication years, fields, citation edges), and on generalization from a single Spark cluster run. The indicator hyperparameters (PageRank alpha, RAM gamma, ECM lambda/H, AttRank weights, iCC N) are inherited from cited definitions and are not benchmarked here. No new theoretical entities are introduced; the label 'iCC (Incubation Citation Count)' is a new name for a time-restricted citation count, a re-expression of an established idea.

free parameters (5)
  • PageRank damping factor alpha and convergence threshold = not stated
    Eq. 2 exposes them as configurable with defaults 'chosen to perform well on citation networks'; exact values are not reported, so the precise ranking behavior is not reproducible from the text alone.
  • RAM decay parameter gamma = not stated
    Eq. 3, inherited from Ghosh et al. [1]; the paper neither reports the value used in BIP! DB nor benchmarks its effect on scores.
  • ECM chain parameters lambda and H = not stated
    Eq. 4; lambda attenuates chain length and H truncates the matrix-power sum; both are free choices inherited from [1] and unspecified in this paper.
  • AttRank parameters alpha, beta, gamma, rho and normalisation constant c = not stated
    Eq. 5 with alpha+beta+gamma=1; parameters come from the authors' own prior paper [2], and production values are not given here.
  • iCC window N = N=3 (typical)
    Eq. 6; 'typically a small value, e.g., N=3'; the window length is a free decision that directly shapes the momentum score.
assumptions (5)
  • standard math Power-iteration convergence of PageRank and AttRank on a citation graph with the given damping and teleportation parameters
    Section 3.2, Eqs. 2 and 5; the library relies on iterative methods reaching the fixed point of each recurrence within the configured tolerance.
  • domain assumption ECM's H-hop matrix-power truncation of citation chains reproduces the intended chain-recency model
    Eq. 4; the model and its truncation come from Ghosh et al. [1] and are assumed correct and appropriate; no empirical check is offered here.
  • domain assumption Publication years, citation edges, and publication-to-field mappings in the input dataset are complete and reliable
    Eqs. 3, 5, 6, 7 depend on t_i, t_j, f_i, d_i, y_i; the paper treats these as given inputs (Sections 3.1 and 3.2) and does not analyze error or missingness.
  • domain assumption Performance measured on one 32-worker cluster (1,280 vCPUs, 2.44 TB RAM) generalizes to the abstract's 'billions of citations' claim
    Section 3.3; a single run on a single cluster configuration underlies the scaling claim; no sensitivity or robustness analysis is reported.
  • domain assumption Spark checkpointing during iterative PageRank and AttRank runs prevents lineage blowup without changing the fixed point
    Section 3.2, PageRank paragraph; the claim that checkpointing 'enables processing of large citation graphs' assumes cached checkpoints are semantically transparent to the scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BIP! Ranker: A Software Library for Citation-Based Impact Indicators on Large-Scale Graphs." pith.science (2026). https://pith.science/paper/YDVOWISQ

@misc{pith2026260802004,
  author       = {Pith},
  title        = {Pith review of: BIP! Ranker: A Software Library for Citation-Based Impact Indicators on Large-Scale Graphs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YDVOWISQ}},
  note         = {Machine review of arXiv:2608.02004}
}
read the original abstract

Scientific impact is multidimensional: overall influence, current popularity, early citation momentum, and field-relative performance each capture a distinct facet of a publication's impact. Yet, in practice, these dimensions are often reduced to a single metric, such as citation count. Open solutions for computing multiple complementary impact indicators at scale remain scarce, particularly for citation graphs as large as those provided by major scholarly databases. We introduce BIP! Ranker, an open-source, Spark-based library for computing citation-based impact indicators at scale, capable of processing citation networks with billions of citations among hundreds of millions of publications.

Figures

Figures reproduced from arXiv: 2608.02004 by the authors.

Figure 1
Figure 1. BIP! Ranker’s inputs, main components, and out [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 1 linked inside Pith

  1. [1]

    Time-aware ranking in dynamic citation networks

    Rumi Ghosh, Tsung-Ting Kuo, Chun-Nan Hsu, Shou- De Lin, and Kristina Lerman. Time-aware ranking in dynamic citation networks. InProceedings of the IEEE International Conference on Data Mining Workshops (ICDMW), pages 373–380, 2011

    2011

  2. [2]

    Ranking pa- pers by their short-term scientific impact

    Ilias Kanellos, Thanasis Vergoulis, Dimitris Sacharidis, Theodore Dalamagas, and Yannis Vassiliou. Ranking pa- pers by their short-term scientific impact. InProceedings of the 37th IEEE International Conference on Data En- gineering (ICDE), pages 1997–2002. IEEE, 2021

    1997

  3. [3]

    Comparison of two article- level, field-independent citation metrics: Field-weighted citation impact (fwci) and relative citation ratio (rcr)

    Amrita Purkayastha, Eleonora Palmaro, Holly J Falk- Krzesinski, and Jeroen Baas. Comparison of two article- level, field-independent citation metrics: Field-weighted citation impact (fwci) and relative citation ratio (rcr). Journal of informetrics, 13(2):635–642, 2019

    2019

  4. [4]

    Impact- based ranking of scientific publications: A survey and experimental evaluation.IEEE Transactions on Knowl- edge and Data Engineering, 33(4):1567–1584, 2019

    Ilias Kanellos, Thanasis Vergoulis, Dimitris Sacharidis, Theodore Dalamagas, and Yannis Vassiliou. Impact- based ranking of scientific publications: A survey and experimental evaluation.IEEE Transactions on Knowl- edge and Data Engineering, 33(4):1567–1584, 2019

    2019

  5. [5]

    Openaire graph dataset, December 2022

    Paolo Manghi, Claudio Atzori, Alessia Bardi, Miriam Baglioni, Jochen Schirrwagen, Harry Dimitropoulos, Sandro La Bruzzo, Ioannis Foufoulas, Andrea Mannocci, Marek Horst, Andreas Czerniak, Katerina Iatropoulou, Argiro Kokogiannaki, Michele De Bonis, Michele Ar- tini, Antonis Lempesis, Alexandros Ioannidis, Natalia Manola, Pedro Principe, Thanasis Vergoulis...

    2022

  6. [6]

    Ope- nAlex: A fully-open index of scholarly works, au- thors, venues, institutions, and concepts.arXiv preprint arXiv:2205.01833, 2022

    Jason Priem, Heather Piwowar, and Richard Orr. Ope- nAlex: A fully-open index of scholarly works, au- thors, venues, institutions, and concepts.arXiv preprint arXiv:2205.01833, 2022

    arXiv 2022

  7. [7]

    Networkx: Network analysis with python.URL: https://networkx.github.io, 1031, 2020

    Aric Hagberg and Drew Conway. Networkx: Network analysis with python.URL: https://networkx.github.io, 1031, 2020

    2020

  8. [8]

    Package ‘igraph’.Last accessed, 3(09):2013, 2013

    Maintainer Gabor Csardi et al. Package ‘igraph’.Last accessed, 3(09):2013, 2013

    2013

Show all 20 references
  1. [9]

    {GraphX}: Graph processing in a distributed dataflow framework

    Joseph E Gonzalez, Reynold S Xin, Ankur Dave, Daniel Crankshaw, Michael J Franklin, and Ion Stoica. {GraphX}: Graph processing in a distributed dataflow framework. In11th USENIX symposium on operating systems design and implementation (OSDI 14), pages 599–613, 2014

    2014

  2. [10]

    Software survey: V osviewer, a computer program for bibliometric map- ping.Scientometrics, 84(2):523–538, 2010

    Nees Jan Van Eck and Ludo Waltman. Software survey: V osviewer, a computer program for bibliometric map- ping.Scientometrics, 84(2):523–538, 2010

    2010

  3. [11]

    Citnetexplorer: A new software tool for analyzing and visualizing citation networks.Journal of Informetrics, 8(4):802–823, 2014

    Nees Jan Van Eck and Ludo Waltman. Citnetexplorer: A new software tool for analyzing and visualizing citation networks.Journal of Informetrics, 8(4):802–823, 2014

    2014

  4. [12]

    bibliometrix: An r-tool for comprehensive science mapping analysis.Jour- nal of Informetrics, 11(4):959–975, 2017

    Massimo Aria and Corrado Cuccurullo. bibliometrix: An r-tool for comprehensive science mapping analysis.Jour- nal of Informetrics, 11(4):959–975, 2017

    2017

  5. [13]

    Routledge, 2013

    Virgil P Diodato and Peter Gellatly.Dictionary of biblio- metrics. Routledge, 2013

    2013

  6. [14]

    The pagerank citation ranking: Bringing or- der to the web

    Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing or- der to the web. InProc. of the 7th International World Wide Web Conf.–1998, 1999

    1998

  7. [15]

    Bip! ranker, July 2026

    Ilias Kanellos, Thanasis Vergoulis, Claudio Atzori, Andrea Mannocci, Serafeim Chatzopoulos, Sandro La Bruzzo, Natalia Manola, and Paolo Manghi. Bip! ranker, July 2026. URL https://doi.org/10.5281/zenodo. 21721600

    2026 doi

  8. [16]

    Bip! finder: Facilitating scientific literature search by exploiting impact-based ranking

    Thanasis Vergoulis, Serafeim Chatzopoulos, Ilias Kanel- los, Panagiotis Deligiannis, Christos Tryfonopoulos, and Theodore Dalamagas. Bip! finder: Facilitating scientific literature search by exploiting impact-based ranking. In Proceedings of the 28th ACM International Conferen...

    2019

  9. [17]

    Bip4covid19: Releasing impact measures for articles rel- evant to covid-19.Quantitative Science Studies, 2(4): 1447–1465, 2022

    Thanasis Vergoulis, Ilias Kanellos, Serafeim Chat- zopoulos, Danae Pla Karidi, and Theodore Dalamagas. Bip4covid19: Releasing impact measures for articles rel- evant to covid-19.Quantitative Science Studies, 2(4): 1447–1465, 2022

    2022

  10. [18]

    Bip! db: A dataset of impact measures for scientific publications

    Thanasis Vergoulis, Ilias Kanellos, Claudio Atzori, An- drea Mannocci, Serafeim Chatzopoulos, Sandro La 5 Bruzzo, Natalia Manola, and Paolo Manghi. Bip! db: A dataset of impact measures for scientific publications. In Companion Proceedings of the Web Conference (WWW), pages 45...

    2021

  11. [19]

    BIP! Scholar: A service to facilitate fair researcher assessment

    Thanasis Vergoulis, Ilias Kanellos, Serafeim Chatzopou- los, Andrea Mannocci, Paolo Manghi, Konstantinos Vi- chos, and Natalia Manola. BIP! Scholar: A service to facilitate fair researcher assessment. InProceedings of the ACM/IEEE Joint Conference on Digital Libraries (JCDL), ...

    2022

  12. [20]

    The bio.tools registry of software tools and data resources for the life sciences.Genome Biology, 20: 164, 2019

    Jon Ison et al. The bio.tools registry of software tools and data resources for the life sciences.Genome Biology, 20: 164, 2019. 6

    2019

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.