Non-English language publications in Citation Indexes -- quantity and quality

Ekaterinburg; Mark Akoev (Ural Federal University; Olga Moskaleva (Saint Petersburg State University; Russia); St. Petersburg

arxiv: 1907.06499 · v1 · pith:ZZIGONLPnew · submitted 2019-07-15 · 💻 cs.DL

Non-English language publications in Citation Indexes -- quantity and quality

Olga Moskaleva (Saint Petersburg State University , St. Petersburg , Russia) , Mark Akoev (Ural Federal University , Ekaterinburg This is my paper

Pith reviewed 2026-05-24 21:08 UTC · model grok-4.3

classification 💻 cs.DL

keywords non-English publicationscitation indexesWeb of ScienceScopusresearch evaluationopen accesssocial scienceshumanities

0 comments

The pith

Non-English publications are read and cited less outside their home countries than English ones in Web of Science and Scopus.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper compares native-language and English publications in the two major citation indexes to identify patterns in their distribution, access type, and impact. English-language output grows as a share over time while native-language work shows lower international readership and citation rates except within the country of origin. Open-access status boosts views and citations more for native-language papers than for English ones, and higher-ranked journals publish a larger share of English content. These patterns lead the authors to conclude that evaluations relying only on these indexes undervalue research in the social sciences and humanities produced in non-English-speaking countries.

Core claim

Analysis of publication records in Web of Science and Scopus shows that the share of English publications is rising, that native-language publications receive fewer views and citations outside their country of origin, that open access produces a larger citation and view gain for native-language work, and that multi-language journals with higher rankings contain a larger proportion of English articles; from this the authors conclude that the contribution of non-English publications is underestimated when research evaluation in non-English-speaking countries relies solely on these two indexes, especially in the social sciences and humanities.

What carries the argument

Comparison of shares, citation counts, and view counts for native-language versus English publications across research areas, languages, and access types within Web of Science and Scopus.

If this is right

The proportion of English-language publications in the indexes continues to increase.
Native-language publications receive lower international readership and citation rates than English ones.
Open-access status raises views and citations more for native-language than for English publications.
Higher-ranked multi-language journals publish a larger share of English articles.
Research evaluation in non-English-speaking countries that uses only these indexes underestimates the contribution of social-science and humanities work published in native languages.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Evaluation policies that weight only indexed English output may systematically undervalue locally relevant scholarship in the humanities and social sciences.
Adding language-specific citation sources or adjusted metrics could change funding and promotion decisions in countries where native-language publishing remains common.
The observed open-access advantage for native-language work suggests that language barriers and access barriers interact, so removing paywalls may narrow the visibility gap more than it narrows the English gap.

Load-bearing premise

Citation counts, views, and indexing status in Web of Science and Scopus give an unbiased picture of actual readership and scholarly impact across languages.

What would settle it

A direct comparison of total readership (including local-language sources or altmetrics) showing that non-English publications in social sciences and humanities achieve impact levels comparable to or higher than English ones when language-based coverage gaps are removed.

Figures

Figures reproduced from arXiv: 1907.06499 by Ekaterinburg, Mark Akoev (Ural Federal University, Olga Moskaleva (Saint Petersburg State University, Russia), St. Petersburg.

read the original abstract

We analyzed publications data in WoS and Scopus to compare publications in native languages vs publications in English and find any distinctive patterns. We analyzed their distribution by research areas, languages, type of access and citation patterns. The following trends were found: share of English publications increases over time; native-language publications are read and cited less than English-language outside the origin country; open access impact on views and citation is higher for native languages; journal ranking correlates with the share of English publications for multi-language journals. We conclude also that the role of non-English publications in research evaluation in non-English speaking countries is underestimated when research in social science and humanities is assessed only by publications in Web of Science and Scopus.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper maps language splits in WoS/Scopus citations and views but its underestimation claim for non-English work lacks an external benchmark.

read the letter

The main takeaway is that English share is rising, native-language papers get fewer external citations and views, open access gives them a larger lift, and higher-ranked journals carry more English content. Those patterns come from straightforward database pulls across fields and time. The OA differential by language and the rank-English correlation are the freshest quantitative notes; they build on earlier English-dominance work but add language-specific detail. The trends themselves look consistent with what the abstract describes. The soft spots sit in the methods and the inference. No sample sizes, error bars, or statistical tests appear in the available description, and there is no accounting for how WoS and Scopus decide what to index or link. The conclusion that non-English publications are underestimated in SSH evaluation when only these two indexes are used rests on treating the observed lower counts as genuine impact gaps rather than possible database artifacts. The analysis stays inside the indexed set and brings no national indexes, local citation data, or coverage audit to test the alternative. That gap is real but not fatal for an empirical note; it just limits how far the underestimation claim travels. This is for bibliometricians and research evaluators who work in non-English countries and need to see the scale of the language split inside the main indexes. The patterns are clear enough to warrant referee time even if the authors need to add data-handling steps and an external check. Send it for review.

Referee Report

2 major / 2 minor

Summary. The paper analyzes publication data from Web of Science and Scopus to compare native-language versus English-language outputs, reporting trends including an increasing share of English publications over time, lower external citations and views for native-language papers outside the country of origin, stronger open-access effects on views/citations for native languages, and a correlation between journal ranking and English-publication share in multi-language journals. It concludes that the role of non-English publications in research evaluation (especially SSH) in non-English-speaking countries is underestimated when assessment relies solely on WoS and Scopus.

Significance. If the reported patterns are robust, the work would usefully flag potential limitations of WoS/Scopus-centric evaluation for non-English research contexts. The empirical database-query approach and absence of fitted parameters or circular derivations are strengths. However, the significance is constrained by the lack of statistical tests, sample-size reporting, and external benchmarks needed to support the underestimation inference.

major comments (2)

[Abstract / Conclusion] Abstract (final sentence) and conclusion: the claim that non-English publications' role 'is underestimated' when SSH research is assessed only via WoS/Scopus does not follow from the internal trends alone. The analysis remains entirely within the indexed set and provides no external benchmark (national indexes, local citation data, or coverage audit) that would demonstrate systematic omission of high-impact non-English work; lower observed citations/views could equally reflect database indexing or linking biases favoring English content.
[Methods / Results] Methods / Results sections: no sample sizes, confidence intervals, statistical tests, or handling of potential confounds (e.g., database indexing policies, language-based selection effects, or citation-linking practices) are reported for any of the directional trends. This absence makes it impossible to assess whether the observed patterns (English-share increase, citation differentials, OA effects) are statistically reliable or merely artifacts of query construction.

minor comments (2)

[Abstract] Abstract: the phrase 'we analyzed their distribution by research areas, languages, type of access and citation patterns' is too vague; a brief statement of the exact query parameters or time window used would improve reproducibility.
[Introduction / Methods] Terminology: 'native languages' and 'non-English' are used interchangeably without an explicit definition or list of languages treated as native for each country; this risks ambiguity when multi-language countries are involved.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on the scope of our inferences and the need for clearer methods reporting. We address each major point below and indicate revisions where the manuscript will be updated.

read point-by-point responses

Referee: [Abstract / Conclusion] Abstract (final sentence) and conclusion: the claim that non-English publications' role 'is underestimated' when SSH research is assessed only via WoS/Scopus does not follow from the internal trends alone. The analysis remains entirely within the indexed set and provides no external benchmark (national indexes, local citation data, or coverage audit) that would demonstrate systematic omission of high-impact non-English work; lower observed citations/views could equally reflect database indexing or linking biases favoring English content.

Authors: We agree that the analysis is confined to WoS and Scopus and offers no external benchmark or coverage audit. The underestimation claim rests on the consistent patterns of reduced external visibility and citations for native-language outputs within these databases. To address the concern, we will revise the abstract and conclusion to present the finding as evidence of a potential limitation in WoS/Scopus-only evaluations rather than a proven systematic omission, and we will explicitly note that alternative explanations such as indexing biases cannot be ruled out without additional data sources. revision: yes
Referee: [Methods / Results] Methods / Results sections: no sample sizes, confidence intervals, statistical tests, or handling of potential confounds (e.g., database indexing policies, language-based selection effects, or citation-linking practices) are reported for any of the directional trends. This absence makes it impossible to assess whether the observed patterns (English-share increase, citation differentials, OA effects) are statistically reliable or merely artifacts of query construction.

Authors: The study is descriptive and reports aggregate trends from database queries rather than fitted models or hypothesis tests. We will add explicit sample sizes (total records and breakdowns by language and field) to the results section. We will also expand the methods to describe query construction and discuss possible confounds such as indexing policies and citation-linking practices. Formal statistical tests and confidence intervals were not performed, as the work does not aim at inferential claims; this limitation will be stated clearly. revision: partial

Circularity Check

0 steps flagged

No circularity: purely empirical comparison of external database records

full rationale

The paper conducts an observational analysis of publication counts, language shares, citation rates, and access types drawn directly from WoS and Scopus records. No equations, fitted parameters, predictions, or derivations appear in the reported methods or conclusions. No self-citations are invoked as load-bearing uniqueness theorems or ansatzes. The central inference (underestimation of non-English work when using only these indexes) follows from internal trends within the indexed set and does not reduce to any input by construction. This is a standard empirical study whose validity rests on data coverage rather than any self-referential logic.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claims rest on the domain assumption that WoS and Scopus coverage is sufficiently complete and unbiased for language-level comparisons; no free parameters or invented entities are introduced.

axioms (1)

domain assumption Web of Science and Scopus provide representative samples of global scholarly output suitable for comparing language-based visibility and citation patterns.
Invoked implicitly when drawing conclusions about underestimation from database statistics alone.

pith-pipeline@v0.9.0 · 5658 in / 1090 out tokens · 22149 ms · 2026-05-24T21:08:07.576241+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages

[1]

Akoev, M. (2018). The list of Multi-Language Journals from Web of Science Core Collection masterlist 12 with Impact Factor and Share of English Language Documents In Previous Two YearsNo Title. https://doi.org/10.17632/r6dcf26mvp.2

work page doi:10.17632/r6dcf26mvp.2 2018
[2]

Ammon, U. (2012). Linguistic inequality and its effects on participation in scientific discourse and on global knowledge accumulation – With a closer look at the problems of the second-rank language communities. Applied Linguistics Review, 3(2), 333–355. https://doi.org/10.1515/applirev-2012- 0016 Clarivate Analytics Partners with the Egyptian Knowledge B...

work page doi:10.1515/applirev-2012- 2012
[3]

Diekhoff, T., Schlattmann, P., & Dewey, M. (2013). Impact of Article Language in Multi-Language Medical Journals - a Bibliometric Analysis of Self-Citations and Impact Factor. PLoS ONE, 8(10), e76816. https://doi.org/10.1371/journal.pone.0076816 Ethnologue. (2015). Summary by language size: Language Size. SIL International. Retrieved from https://www.ethn...

work page doi:10.1371/journal.pone.0076816 2013
[4]

Fung, I. C. H. (2008). Citation of non-English peer review publications - Some Chinese examples. Emerging Themes in Epidemiology. https://doi.org/10.1186/1742-7622-5-12 González-Alcaide, G., Valderrama-Zurián, J. C., & Aleixandre-Benavent, R. (2012). The Impact Factor in non-English-speaking countries. Scientometrics, 92(2), 297–311. https://doi.org/10.10...

work page doi:10.1186/1742-7622-5-12 2008
[5]

Gordin, M. D. (2015). Scientific Babel. How Science Was Done Before and After Global English. The University of Chicago Press

work page 2015
[6]

Kulczycki, E., Engels, T. C. E., Pölönen, J., Bruun, K., Dušková, M., Guns, R., … Zuccala, A. (2018). Publication patterns in the social sciences and humanities: evidence from eight European countries. Scientometrics, 1–24. https://doi.org/10.1007/s11192-018-2711-0

work page doi:10.1007/s11192-018-2711-0 2018
[7]

Kumar, N., Panwar, Y., & Mahesh, G. (2016). A snapshot of research papers in non - English languages. CURRENT SCIENCE, 111(1), 9–10. Retrieved from https://en.wikipedia.org/w/index.php?title=

work page 2016
[8]

Liang, L., Rousseau, R., & Zhong, Z. (2013). Non-English journals and papers in physics and chemistry: bias in citations? Scientometrics, 95(1), 333–350. https://doi.org/10.1007/s11192-012-0828-0

work page doi:10.1007/s11192-012-0828-0 2013
[9]

Liu, F., Hu, G., Tang, L., & Liu, W. (2018). The penalty of containing more non-English articles. Scientometrics, 114(1), 359–366. https://doi.org/10.1007/s11192-017-2577-6

work page doi:10.1007/s11192-017-2577-6 2018
[10]

Liu, W. (2017). The changing role of non-English papers in scholarly communication: Evidence from Web of Science’s three journal citation indexes. Learned Publishing, 30(2), 115–123. https://doi.org/10.1002/leap.1089

work page doi:10.1002/leap.1089 2017
[11]

Meneghini, R., & Packer, A. L. (2007). Is there science beyond English? EMBO Reports, 8(2), 112–116

work page 2007
[12]

Montgomery, S. L. (2016). Impacts of a Global Language on Science: Are There Disadvantages? In Language as a Scientific Tool. Shaping Scientific Language Across Time and National Traditions. (pp. 199–218). Routledge. https://doi.org/10.4324/9781315657257

work page doi:10.4324/9781315657257 2016
[13]

Moskaleva, O., & Akoev, M. (2018). Publications on different languages in citation indexes, or If there is a chance for Russian language in science? Universitetskaya Kniga, (3), 42–45

work page 2018
[14]

Moskaleva, O., Pislyakov, V., Sterligov, I., Akoev, M., & Shabanova, S. (2018). Russian index of 13 Science citation: Overview and review. Scientometrics. https://doi.org/10.1007/s11192-018-2758-y

work page doi:10.1007/s11192-018-2758-y 2018
[15]

Pislyakov, V. (2007). What Are National Citation Indices Needed For ? Scientific and Technical Libraries, 2, 66–68

work page 2007
[16]

Shu, F., & Larivière, V. (2015). Chinese-language articles are biased in citations. Journal of Informetrics, 9(3), 526–528. https://doi.org/10.1016/j.joi.2015.05.005 Sīle, L., Guns, R., Sivertsen, G., & Engels, T. C. E. (2017). European databases and repositories for Social Sciences and Humanities research output, (July). https://doi.org/10.6084/m9.figsha...

work page doi:10.1016/j.joi.2015.05.005 2015
[17]

Vinther, S., & Rosenberg, J. (2012). Impact factor trends for general medical journals: non-English- language journals are lacking behind. Swiss Medical Weekly 2012 142:3940, 142(3940). https://doi.org/10.4414/smw.2012.13572

work page doi:10.4414/smw.2012.13572 2012

[1] [1]

Akoev, M. (2018). The list of Multi-Language Journals from Web of Science Core Collection masterlist 12 with Impact Factor and Share of English Language Documents In Previous Two YearsNo Title. https://doi.org/10.17632/r6dcf26mvp.2

work page doi:10.17632/r6dcf26mvp.2 2018

[2] [2]

Ammon, U. (2012). Linguistic inequality and its effects on participation in scientific discourse and on global knowledge accumulation – With a closer look at the problems of the second-rank language communities. Applied Linguistics Review, 3(2), 333–355. https://doi.org/10.1515/applirev-2012- 0016 Clarivate Analytics Partners with the Egyptian Knowledge B...

work page doi:10.1515/applirev-2012- 2012

[3] [3]

Diekhoff, T., Schlattmann, P., & Dewey, M. (2013). Impact of Article Language in Multi-Language Medical Journals - a Bibliometric Analysis of Self-Citations and Impact Factor. PLoS ONE, 8(10), e76816. https://doi.org/10.1371/journal.pone.0076816 Ethnologue. (2015). Summary by language size: Language Size. SIL International. Retrieved from https://www.ethn...

work page doi:10.1371/journal.pone.0076816 2013

[4] [4]

Fung, I. C. H. (2008). Citation of non-English peer review publications - Some Chinese examples. Emerging Themes in Epidemiology. https://doi.org/10.1186/1742-7622-5-12 González-Alcaide, G., Valderrama-Zurián, J. C., & Aleixandre-Benavent, R. (2012). The Impact Factor in non-English-speaking countries. Scientometrics, 92(2), 297–311. https://doi.org/10.10...

work page doi:10.1186/1742-7622-5-12 2008

[5] [5]

Gordin, M. D. (2015). Scientific Babel. How Science Was Done Before and After Global English. The University of Chicago Press

work page 2015

[6] [6]

Kulczycki, E., Engels, T. C. E., Pölönen, J., Bruun, K., Dušková, M., Guns, R., … Zuccala, A. (2018). Publication patterns in the social sciences and humanities: evidence from eight European countries. Scientometrics, 1–24. https://doi.org/10.1007/s11192-018-2711-0

work page doi:10.1007/s11192-018-2711-0 2018

[7] [7]

Kumar, N., Panwar, Y., & Mahesh, G. (2016). A snapshot of research papers in non - English languages. CURRENT SCIENCE, 111(1), 9–10. Retrieved from https://en.wikipedia.org/w/index.php?title=

work page 2016

[8] [8]

Liang, L., Rousseau, R., & Zhong, Z. (2013). Non-English journals and papers in physics and chemistry: bias in citations? Scientometrics, 95(1), 333–350. https://doi.org/10.1007/s11192-012-0828-0

work page doi:10.1007/s11192-012-0828-0 2013

[9] [9]

Liu, F., Hu, G., Tang, L., & Liu, W. (2018). The penalty of containing more non-English articles. Scientometrics, 114(1), 359–366. https://doi.org/10.1007/s11192-017-2577-6

work page doi:10.1007/s11192-017-2577-6 2018

[10] [10]

Liu, W. (2017). The changing role of non-English papers in scholarly communication: Evidence from Web of Science’s three journal citation indexes. Learned Publishing, 30(2), 115–123. https://doi.org/10.1002/leap.1089

work page doi:10.1002/leap.1089 2017

[11] [11]

Meneghini, R., & Packer, A. L. (2007). Is there science beyond English? EMBO Reports, 8(2), 112–116

work page 2007

[12] [12]

Montgomery, S. L. (2016). Impacts of a Global Language on Science: Are There Disadvantages? In Language as a Scientific Tool. Shaping Scientific Language Across Time and National Traditions. (pp. 199–218). Routledge. https://doi.org/10.4324/9781315657257

work page doi:10.4324/9781315657257 2016

[13] [13]

Moskaleva, O., & Akoev, M. (2018). Publications on different languages in citation indexes, or If there is a chance for Russian language in science? Universitetskaya Kniga, (3), 42–45

work page 2018

[14] [14]

Moskaleva, O., Pislyakov, V., Sterligov, I., Akoev, M., & Shabanova, S. (2018). Russian index of 13 Science citation: Overview and review. Scientometrics. https://doi.org/10.1007/s11192-018-2758-y

work page doi:10.1007/s11192-018-2758-y 2018

[15] [15]

Pislyakov, V. (2007). What Are National Citation Indices Needed For ? Scientific and Technical Libraries, 2, 66–68

work page 2007

[16] [16]

Shu, F., & Larivière, V. (2015). Chinese-language articles are biased in citations. Journal of Informetrics, 9(3), 526–528. https://doi.org/10.1016/j.joi.2015.05.005 Sīle, L., Guns, R., Sivertsen, G., & Engels, T. C. E. (2017). European databases and repositories for Social Sciences and Humanities research output, (July). https://doi.org/10.6084/m9.figsha...

work page doi:10.1016/j.joi.2015.05.005 2015

[17] [17]

Vinther, S., & Rosenberg, J. (2012). Impact factor trends for general medical journals: non-English- language journals are lacking behind. Swiss Medical Weekly 2012 142:3940, 142(3940). https://doi.org/10.4414/smw.2012.13572

work page doi:10.4414/smw.2012.13572 2012