Bilateral and multilateral international scientific collaboration of EU member states: OpenAlex vs Scopus (2000-2024)

Myroslava Hladchenko

arxiv: 2606.06336 · v1 · pith:6KYLWDOJnew · submitted 2026-06-04 · 💻 cs.DL

Bilateral and multilateral international scientific collaboration of EU member states: OpenAlex vs Scopus (2000-2024)

Myroslava Hladchenko This is my paper

Pith reviewed 2026-06-27 22:27 UTC · model grok-4.3

classification 💻 cs.DL

keywords OpenAlexScopusscientific collaborationEU member statesbilateral collaborationmultilateral collaborationRelative Intensity of CollaborationEU framework programs

0 comments

The pith

OpenAlex restricted to cited articles yields EU collaboration metrics comparable to Scopus from 2000 to 2024.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper compares OpenAlex and Scopus data to examine bilateral and multilateral scientific collaborations among EU member states and with external partners over 2000-2024. It establishes that OpenAlex, when limited to cited articles, produces broadly similar results to Scopus for country-level collaboration assessments. The analysis reveals higher collaboration intensity in multilateral partnerships and links increases in intensity to EU framework programs, while documenting persistent structural differences between EU-14 and EU-13 countries in their partnership patterns.

Core claim

When OpenAlex is restricted to cited articles, its measurements of bilateral and multilateral research collaboration among EU member states and with global partners align closely with those from Scopus over 2000-2024. Relative Intensity of Collaboration is higher for multilateral partnerships. EU framework programs appear to have boosted collaboration, though asymmetries remain between EU-14 and EU-13 countries, with specific patterns in partnerships with countries like Russia, China, and the US.

What carries the argument

Relative Intensity of Collaboration (RIC), which measures observed versus expected collaboration based on publication shares, used to compare bilateral and multilateral country pairs across the two databases.

Load-bearing premise

That restricting OpenAlex to cited articles is sufficient to align its coverage, quality, and collaboration signals with Scopus without introducing systematic biases in country-level metrics.

What would settle it

A side-by-side calculation showing that country-level RIC values or collaboration trends diverge substantially between OpenAlex (cited subset) and Scopus for the same EU country pairs.

Figures

Figures reproduced from arXiv: 2606.06336 by Myroslava Hladchenko.

read the original abstract

This study examines the evolution of bilateral and multilateral scientific collaboration among EU Member States and between the EU and global partners from 2000 to 2024 using data from OpenAlex and Scopus. The results show that OpenAlex, when restricted to cited articles, yields findings broadly comparable to those obtained from Scopus for assessing country-level research collaboration. Relative Intensity of Collaboration (RIC) values are consistently higher for multilateral than for bilateral partnerships. Increased collaboration intensity during the final years of FP7, the intermediate and later stages of Horizon 2020, and the final years of the study period suggests that EU FP may have strengthened collaboration among participating countries. With regard to European integration, multilateral collaboration intensity increased between the EU-14 and EU-13, between these groups and EU candidate countries, and within the EU-13. Despite this growth, structural asymmetries persist. Bilateral collaboration among EU-14 countries is concentrated within the group and with EU-13, Brazil, Norway, Switzerland, and the United Kingdom, whereas EU-13 countries collaborate more intensively within the group, with EU candidate countries and Russia. EU-14 countries maintain stronger multilateral collaboration with high-income countries such as Australia, Canada, and the United States than do EU-13 countries. For both groups, collaboration with China remains the weakest. Although multilateral collaboration intensity with Russia has declined, it remained above the expected level for the EU-14 in 2024 and was 2.5 times higher than expected for the EU-13. This persistence may reflect the continued participation of Russian researchers in multilateral projects despite Russia's suspension from Horizon Europe in 2022.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

OpenAlex restricted to cited papers tracks EU collaboration patterns similarly to Scopus over 2000-2024, with the main value in the long-span descriptive comparison.

read the letter

The key point is that OpenAlex, after filtering to cited articles, produces country-level bilateral and multilateral collaboration metrics that line up with Scopus for EU member states across 2000-2024. The paper uses the standard Relative Intensity of Collaboration measure and reports higher values for multilateral than bilateral ties, plus growth around FP7 and Horizon 2020 periods.

What stands out as new is the direct head-to-head on collaboration networks rather than just publication counts, covering 24 years and breaking out EU-14 versus EU-13 patterns, including links to candidates, Russia, China, and high-income partners. It flags persistent asymmetries, such as EU-14's stronger multilateral ties to Australia, Canada, and the US, and notes that Russia collaboration stayed above expected levels into 2024 despite the Horizon Europe suspension.

The work is straightforward on the data side and adds a practical check on whether an open source can substitute for Scopus in this specific use case. The longitudinal angle and the policy tie-in to framework programs give it some applied relevance.

The soft spots are in the methods and evidence for the central claim. The abstract states the results are broadly comparable after the citation filter but supplies no side-by-side tables, no quantitative measure of agreement, and no statistical tests. Without those, it is hard to judge how close the match actually is or whether the filter introduces country-specific biases. The suggestion that EU programs strengthened collaboration rests on timing alone, with no controls for other trends. If the full paper has the raw RIC values or robustness checks, that would help; otherwise the comparability conclusion stays descriptive.

This is mainly for scientometrics researchers or EU policy analysts who need cheap, long-term collaboration data and want to know the practical limits of OpenAlex. It is solid enough on the empirical question to deserve peer review, though it will likely need clearer methods and quantitative support for the match between sources.

Referee Report

3 major / 2 minor

Summary. The paper compares bilateral and multilateral scientific collaboration among EU member states and with global partners from 2000-2024 using OpenAlex and Scopus data. It claims that restricting OpenAlex to cited articles produces broadly comparable country-level results to Scopus on metrics such as Relative Intensity of Collaboration (RIC), with multilateral RIC consistently higher than bilateral; it also reports increased collaboration intensity linked to EU framework programs, growth in multilateral ties between EU-14/EU-13 groups, and persistent structural asymmetries (e.g., EU-14 stronger with high-income countries, EU-13 with candidates and Russia).

Significance. If the comparability claim is substantiated with transparent methods and quantitative validation, the work would support OpenAlex as a viable open-access alternative to Scopus for collaboration studies, which is practically significant given subscription costs and coverage differences. The policy-relevant findings on EU integration effects and post-2022 Russia collaboration patterns add value to science-policy analysis.

major comments (3)

[Abstract] Abstract: the central claim that OpenAlex 'when restricted to cited articles, yields findings broadly comparable' to Scopus is asserted without any reported statistical tests, agreement metrics (e.g., correlation or rank correlation on RIC values), or quantitative thresholds for 'broadly comparable'.
[Methods] Methods: no description is supplied of the citation-filtering procedure (citation window, minimum citation count, handling of self-citations or database-specific citation counts), data exclusion rules, or how country-level aggregates were computed and aligned between the two sources.
[Results] Results: reported RIC patterns and subgroup asymmetries lack error estimates, sensitivity checks, or direct side-by-side tables quantifying divergence between the filtered OpenAlex and Scopus outputs, making it impossible to evaluate whether observed similarities are robust or coincidental.

minor comments (2)

[Abstract] Abstract: 'final years of the study period' is vague; specify the exact years or reference the relevant figure/table.
[Throughout] Throughout: define RIC, FP7, Horizon 2020, EU-14, and EU-13 on first use even if conventional in the field.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed comments. The feedback identifies important gaps in quantitative validation, methodological transparency, and robustness checks that we agree require attention. We respond to each major comment below and will incorporate the suggested improvements in the revised manuscript.

read point-by-point responses

Referee: [Abstract] Abstract: the central claim that OpenAlex 'when restricted to cited articles, yields findings broadly comparable' to Scopus is asserted without any reported statistical tests, agreement metrics (e.g., correlation or rank correlation on RIC values), or quantitative thresholds for 'broadly comparable'.

Authors: We accept the criticism. The claim of broad comparability will be supported in the revision by adding Pearson and Spearman rank correlations between RIC values from the two sources, together with an explicit quantitative threshold (e.g., correlations exceeding 0.75) used to define 'broadly comparable'. These metrics will be reported in both the abstract and the results section. revision: yes
Referee: [Methods] Methods: no description is supplied of the citation-filtering procedure (citation window, minimum citation count, handling of self-citations or database-specific citation counts), data exclusion rules, or how country-level aggregates were computed and aligned between the two sources.

Authors: We agree that these procedural details are necessary for reproducibility. The revised methods section will explicitly describe the citation window applied, any minimum citation thresholds, treatment of self-citations, database-specific citation handling, data exclusion criteria, and the exact steps used to compute and align country-level aggregates across OpenAlex and Scopus. revision: yes
Referee: [Results] Results: reported RIC patterns and subgroup asymmetries lack error estimates, sensitivity checks, or direct side-by-side tables quantifying divergence between the filtered OpenAlex and Scopus outputs, making it impossible to evaluate whether observed similarities are robust or coincidental.

Authors: This observation is correct. In the revision we will add bootstrap-derived confidence intervals to the RIC estimates, perform sensitivity analyses varying the citation filter parameters, and insert side-by-side tables that directly quantify absolute and relative differences between the filtered OpenAlex and Scopus outputs for the main collaboration metrics. revision: yes

Circularity Check

0 steps flagged

No circularity; empirical database comparison

full rationale

The paper conducts a direct empirical comparison of collaboration metrics (RIC, bilateral/multilateral patterns) between two independent external databases, OpenAlex (with cited-article filter) and Scopus. No equations, fitted parameters renamed as predictions, self-citations as load-bearing premises, or definitional equivalences appear in the abstract or described methods. The restriction to cited articles is presented as an alignment step rather than a constructed equivalence, and all reported patterns derive from external data sources without internal reduction to inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on standard bibliometric assumptions about database representativeness and the validity of relative intensity metrics; no free parameters or invented entities are introduced in the abstract.

axioms (2)

domain assumption Cited articles in OpenAlex form a subset whose collaboration signals align with full Scopus records for country-level analysis
Invoked to justify the comparability finding after filtering.
domain assumption Relative Intensity of Collaboration (RIC) accurately captures deviation from expected partnership levels
Used throughout to interpret bilateral versus multilateral intensities.

pith-pipeline@v0.9.1-grok · 5830 in / 1230 out tokens · 27090 ms · 2026-06-27T22:27:27.259272+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

6 extracted references · 2 canonical work pages · 1 internal anchor

[1]

Adams, J. (2013). The fourth age of research. Nature, 497(7451), 557–560. Adams, J., & Szomszor, M. (2022). A converging global research system. Quantitative Science Studies, 3(3), 715-731. Adams, J. (2013). The fourth age of rese arch. Nature, 497(7451), 557 –560. Adams, J., & Szomszor, M. (2022). A converging global research system. Quantitative Science...

work page arXiv 2013
[2]

Belli, S., & Nenoff, J. M. (2022). Cooperation in science and innovation between Latin America and the European Union. Journal of Open Innovation , 8(2),

2022
[3]

Burke, M. (2025). EU’s new research infrastructure strategy. Royal Society of Chemistry. Chinchilla-Rodríguez, Z., Zacca -González, G., Vargas-Quesada, B., & de Moya -Anegón, F. (2016). Benchmarking scientific performance… Scientometrics, 106(3), 1239–1264. Choi, M., Lee, H., & Zoo, H. (2021). Scientific collaboration between Australia and South Korea. Sc...

2025
[4]

Culbert, J. , et al. (2025). Reference coverage analysis of OpenAlex… Scientometrics. EEAS (2023). EU–Chile Cooperation Factsheet. Euraxess (2024). European Commission invites India to explore association to Horizon Europe . https://euraxess.ec.europa.eu/worldwide/india/news/european -commission-invites-india- explore-association-horizon-europe EC (2015)....

2025
[5]

W., Sivertsen, G., & Rousseau, R

Fuchs, J. W., Sivertsen, G., & Rousseau, R. (2021). Measuring the relat ive intensity of collaboration. Scientometrics, 126(10), 8673–8682. Glänzel, W., & Schubert, A. (2001). Double effort = double impact? Scientometrics, 50(2), 199 –

2021
[6]

Glänzel, W., & Schubert, A. (2004). Analysing scientific networks through co -authorship. In Handbook of quantitative science and technology research . Gómez-Espés, A., Färber, M., & Jatowt, A. (2024). Benefits of international collaboration in computer science. Scientometrics, 129(2), 1155–1171. Gueye, A., Choi, E., Guzmán -Valenzuela, C., & Gre gorutti,...

work page internal anchor Pith review Pith/arXiv arXiv 2004

[1] [1]

Adams, J. (2013). The fourth age of research. Nature, 497(7451), 557–560. Adams, J., & Szomszor, M. (2022). A converging global research system. Quantitative Science Studies, 3(3), 715-731. Adams, J. (2013). The fourth age of rese arch. Nature, 497(7451), 557 –560. Adams, J., & Szomszor, M. (2022). A converging global research system. Quantitative Science...

work page arXiv 2013

[2] [2]

Belli, S., & Nenoff, J. M. (2022). Cooperation in science and innovation between Latin America and the European Union. Journal of Open Innovation , 8(2),

2022

[3] [3]

Burke, M. (2025). EU’s new research infrastructure strategy. Royal Society of Chemistry. Chinchilla-Rodríguez, Z., Zacca -González, G., Vargas-Quesada, B., & de Moya -Anegón, F. (2016). Benchmarking scientific performance… Scientometrics, 106(3), 1239–1264. Choi, M., Lee, H., & Zoo, H. (2021). Scientific collaboration between Australia and South Korea. Sc...

2025

[4] [4]

Culbert, J. , et al. (2025). Reference coverage analysis of OpenAlex… Scientometrics. EEAS (2023). EU–Chile Cooperation Factsheet. Euraxess (2024). European Commission invites India to explore association to Horizon Europe . https://euraxess.ec.europa.eu/worldwide/india/news/european -commission-invites-india- explore-association-horizon-europe EC (2015)....

2025

[5] [5]

W., Sivertsen, G., & Rousseau, R

Fuchs, J. W., Sivertsen, G., & Rousseau, R. (2021). Measuring the relat ive intensity of collaboration. Scientometrics, 126(10), 8673–8682. Glänzel, W., & Schubert, A. (2001). Double effort = double impact? Scientometrics, 50(2), 199 –

2021

[6] [6]

Glänzel, W., & Schubert, A. (2004). Analysing scientific networks through co -authorship. In Handbook of quantitative science and technology research . Gómez-Espés, A., Färber, M., & Jatowt, A. (2024). Benefits of international collaboration in computer science. Scientometrics, 129(2), 1155–1171. Gueye, A., Choi, E., Guzmán -Valenzuela, C., & Gre gorutti,...

work page internal anchor Pith review Pith/arXiv arXiv 2004