The citation advantage of foreign language references for Chinese social science papers

Cassidy R. Sugimoto; Juan Xie; Kaile Gong; Vincent Larivi\`ere; Ying Cheng

REVIEW 3 major objections 5 minor 11 references

The citation advantage of foreign language references for Chinese social science papers

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Chinese social science papers that cite foreign-language literature receive about two to three times as many citations as papers citing only Chinese work, and the advantage resists standard controls.

desk verdict Solid large-scale descriptive finding on foreign-language referencing in Chinese social science, but the regression omits total reference count and the causal language goes beyond the evidence. read the letter →

arxiv 1908.07576 v1 pith:S77ND7ER submitted 2019-08-20 cs.DL

Kaile Gong , Juan Xie , Ying Cheng , Vincent Larivière , Cassidy R. Sugimoto This is my paper · ORCID

classification cs.DL

keywords foreignlanguagereferencescitationadvantagecitationssocialsciencesChinaglobalizationlinguafranca

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the language of the literature a researcher absorbs shapes the impact of the work they produce. Using roughly 950,000 Chinese-language social science papers and the references they cite, it reports that papers with at least one foreign-language reference receive about 2.4 times as many citations within two years and about 3.0 times as many within five years as papers whose references are entirely Chinese. That advantage holds within every discipline and after controlling for journal prestige, institution prestige, and collaboration, and the foreign-language indicator is the strongest predictor in the model. The paper further shows that the advantage narrowed as foreign-language citing became common between 1998 and 2008, then stabilized, suggesting that absorbing foreign work still pays off even as internationalization spreads. If true, this would mean that access to non-Chinese literature is a concrete driver of scientific impact for Chinese social science, with direct implications for language policy and research evaluation.

What carries the argument

The central objects are the binary codes PFLR (paper with at least one foreign-language reference) and PNLR (paper with none), together with the citation advantage $\mathrm{CA}$, defined as the ratio of average citations per paper for PFLR papers to that for PNLR papers. The paper computes $\mathrm{CA}$ for two- and five-year citation windows and tests the raw gap with Mann-Whitney U tests. To rule out confounds, it then fits a multiple linear regression with $\log_{10}(C_5+1)$ as the outcome and PFLR, discipline dummies, leading-journal status, top-institution status, and co-authorship as predictors; the regression is what carries the claim that the language effect is not an artifact of prestige or collaboration.

What would settle it

Add total reference count, article length, and author count to the regression, or match each PFLR paper with a PNLR paper on those variables plus discipline, journal tier, institution, and collaboration; if the foreign-language coefficient drops to zero, the claimed citation advantage is refuted.

Watch

Extended reading notes

Core claim

The central claim is that absorbing foreign-language literature raises a paper's impact inside the Chinese social science community. Using 950,302 Chinese-language papers and 8,151,327 cited references from a comprehensive national citation index, the paper reports that papers with at least one foreign-language reference account for 35.5% of publications but 62.5% of five-year citations, with citation advantage $\mathrm{CA}_2=2.41$ and $\mathrm{CA}_5=3.04$ relative to papers without such references. The advantage appears in every one of thirteen disciplines, in leading and ordinary journals, at top and other institutions, and for single- and co-authored papers; in the full regression, the foreign-language indicator has the largest standardized coefficient ($\beta=0.229$, $p<0.001$), larger than journal prestige (0.144), institution prestige (0.097), or collaboration (0.085). The paper also finds that the advantage declined as the share of papers citing foreign work rose from 11.7% in 1998 to 54.4% in 2013, stabilizing after 2008 at roughly 2 to 2.5, which it interprets as a persistent but homogenizing effect of internationalization.

Load-bearing premise

The load-bearing premise is that a paper's binary status of having at least one foreign-language reference captures the effect of absorbing foreign knowledge, rather than serving as a proxy for longer reference lists, longer papers, or unmeasured quality.

Editorial extensions

If this is right

Chinese-language social science papers that draw on foreign literature will, on average, accumulate domestic citations faster and for longer than papers that cite only Chinese work.
Restricting access to foreign-language journals by subscription gaps, translation delays, or policy would likely depress the production of high-impact Chinese social science.
Because 96.4% of foreign-language references are English, the advantage is effectively an English-reading advantage, so the internationalization of Chinese social science is largely an English-language absorption effect.
The narrowing of the advantage between 1998 and 2008 is consistent with a diffusion process: as more papers cite foreign work, citing it no longer sets a paper apart, yet the stable post-2008 premium indicates the association is not disappearing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An inference beyond the paper: the regression omits total reference count, a factor the paper itself notes predicts citations; if PFLR papers also carry longer reference lists, part of the measured advantage may be a reference-quantity effect rather than a language effect.
An inference beyond the paper: the same design could be applied to other large non-English research communities, such as Japan, France, or Spain, to see whether the foreign-language citation premium generalizes beyond China.
An inference beyond the paper: the post-2008 stability suggests a ceiling or signaling mechanism; once foreign-language citing becomes the norm, the remaining premium may reflect unobserved quality rather than the causal benefit of reading foreign work.
An inference beyond the paper: a direct test would check whether the number of foreign-language references, or the depth with which they are used, scales with the citation advantage, something the binary PFLR coding cannot reveal.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, this is a large, careful descriptive study of 950k Chinese social science papers. The headline finding is real: papers with at least one foreign-language reference get about 2–3 times more citations than those without, and the gap shrinks then plateaus over 1998–2013. What's new here is scale and scope—all 13 CSSCI disciplines over 16 years—and the temporal analysis of the citation advantage. The authors also do a decent job showing the raw association holds up within disciplines, journal prestige strata, institution tiers, and co-authorship splits.

The soft spot is the regression in Eq. 3. It omits total reference count, which the paper's own introduction lists as a well-established predictor of citations. If PFLR papers also cite more references overall—which seems likely, since they have at least one foreign reference and probably longer reference lists—then the standardized coefficient of 0.229 partly captures quantity, not just language. The authors never report reference counts by PFLR status, so we can't tell how much of the advantage is language diversity versus sheer reference volume. That makes the causal language in the abstract and discussion ('absorbed knowledge,' 'scientific impact') premature. The 'decreased as an artifact' explanation for 1998–2008 is also asserted rather than tested with a formal decomposition.

The Mann-Whitney p-values are essentially meaningless at this sample size, but that's minor; the effect sizes (CA 2–3x) are what matter. I'd also like confidence intervals, but the raw means are there.

The self-citations to Gong et al. 2018 and Lariviere et al. 2019 are appropriate given the prior work; no problem there.

All told, the descriptive contribution is solid and worth publishing after revision. The authors should add reference count as a control, or at minimum present reference-count distributions by group and discuss the direction of expected bias. They should soften 'impact' to 'association' unless they can defend a mechanism. If they do that, this becomes a useful paper for science policy and scientometrics readers. I would send it to peer review; with those additions, it would be a reasonable accept.

Referee Report

3 major / 5 minor

Summary. This paper investigates whether Chinese social science papers that cite foreign-language references receive more citations than those that do not, using 950,302 papers from the Chinese Social Sciences Citation Index (CSSCI) published 1998-2013. The authors compute citation counts per paper (CPP2, CPP5) and citation advantages (CA2, CA5), run Mann-Whitney U tests by discipline, journal prestige, institution prestige, and collaboration type, and estimate a multiple linear regression of log10(C5+1) on a binary indicator for foreign-language references plus controls (Eq. 3). They find that papers with foreign-language references (PFLR) receive significantly more citations, that the advantage declines from 1998 to 2008 as the proportion of PFLR papers rises, and that the advantage persists after controlling for discipline, journal prestige, institution prestige, and co-authorship. They interpret the persistent positive coefficient as evidence that incorporating foreign-language literature increases scientific impact.

Significance. The paper provides a large-scale, descriptive picture of language diversity in references for a non-English scientific community, a topic that is often overlooked in bibliometric research. The data collection is substantial and the statistical analyses are transparent: the raw CPP and CA calculations, the Mann-Whitney tests, and the regression are all straightforward and replicable. The paper is valuable as a descriptive empirical contribution to understanding the internationalization of Chinese social science and the relationship between foreign-language referencing and citation outcomes. However, the central inferential claim that foreign-language references have a causal or even net positive effect on impact is not established by the current design, primarily because of an omitted-variable problem in the regression that is directly acknowledged in the paper's own introduction.

major comments (3)

[Section 2.4, Eq. (3), Table 10] The regression in Eq. (3) omits the total number of references per paper, a variable that the authors themselves list in Section 1 as a robust positive predictor of citation counts, citing more than a dozen studies. If PFLR papers tend to have longer reference lists than PNLR papers, the standardized coefficient for PFLR (β=0.229) may partially reflect reference quantity rather than language diversity. The authors do not report descriptive statistics for reference counts by PFLR status, so the magnitude of this potential confound is unknown. A robustness check that adds total reference count as a control, or at least a descriptive comparison of reference counts by PFLR status, is essential to support the claim that the citation advantage holds after controlling for characteristics of the citing papers.
[Abstract and Section 4 (Discussion)] The causal framing of the results goes beyond what the observational design can support. The abstract states that the study explores 'the effect of absorbed knowledge on research impact,' and the discussion says that incorporating foreign-language literature 'continues to increase scientific impact.' Since PFLR status is not randomly assigned and the regression omits numerous potential confounders (e.g., English proficiency of authors, research quality, topic novelty, funding, and international collaboration), the estimates at best show a partial association. The authors should either temper the causal language throughout, or clearly label the design as observational and the interpretation as associational.
[Section 3.3.5, Table 10] The claim that the citation advantage of PFLR is 'the strongest' among all variables rests on comparing standardized coefficients for a binary variable with coefficients for categorical and other binary variables. Since the variables have different units and the table does not include unstandardized coefficients or standard errors, the statement that foreign-language referencing has 'a stronger influence' than journal prestige, institution prestige, or collaboration is not fully supported. Presenting unstandardized coefficients with confidence intervals would allow a more direct comparison.

minor comments (5)

[Section 2.4, Eq. (2)] The definition of CA as a ratio of average citations is clear, but the notation in Eq. (2) is not explicitly shown; writing the formula explicitly would improve readability.
[Section 3.2, Figure 3] Figure 3B shows CA values of 2 and 2.5 after 2008, but the text says 'remained relatively stable around 2 and 2.5' without providing confidence intervals or a formal trend test; adding a simple trend test for the post-2008 period would strengthen the claim of stability.
[Section 3.3.1, Table 5] The descriptive CA values in Table 5 vary substantially across disciplines (e.g., CA2=4.68 for Statistics vs. CA2=1.20 for Sports Science), yet the authors do not discuss why certain disciplines show much larger advantages; a brief comment would help interpret these differences.
[Throughout] There are minor typographical issues, such as 'multiple liner regression' in Section 3.3.5 and 'paper with foreign language references' in Section 3.3.1; a careful proofreading pass is recommended.
[Section 2.2] The sentence 'By 2018, less than 2% (n= 25,504; 1.49%) of CSSCI-indexed papers have authors from other countries' uses inconsistent verb tense; it should be 'had' or 'have' consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper estimates an association between an independently measured reference-language indicator and citation counts; no target quantity is defined in terms of the fitted parameters.

full rationale

The paper's central claim is empirical: papers with foreign language references (PFLR) receive more citations, and this association persists after controlling for discipline, journal prestige, institution prestige, and collaboration. The PFLR indicator is defined from the language tags of cited references, while the outcome is the citation count received by the citing paper in CSSCI; these are measured independently and are not mutually definitional. Eq. 3 is a standard multiple regression of log10(C5+1) on PFLR and control variables, with the PFLR coefficient estimated from data rather than imposed. The citation advantage CA in Eq. 2 is a ratio of observed average citation counts, not a fitted quantity that is later renamed as a prediction. The self-citations (Gong et al. 2018 and Lariviere et al. 2019) are used only as background support for related bibliometric regularities and are not load-bearing in the derivation; the main result does not reduce to those citations. The omission of total reference count from Eq. 3 is a potential omitted-variable concern about confounding, not a circularity, because the paper never constructs the PFLR variable from citation counts or defines the target result in terms of its own inputs.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. It relies on the CSSCI database, which is treated as authoritative, and on standard statistical assumptions. The central unstated assumption is that the binary PFLR variable adequately captures the 'absorbed knowledge' mechanism, which is not validated against reference count or other known citation predictors.

free parameters (1)

No ad hoc free parameters
The empirical estimates (means, CA, regression betas) are outcomes of statistical analysis, not free parameters chosen to make a derivation work. No ad hoc constants are introduced.

assumptions (4)

domain assumption CSSCI reference-language tags accurately reflect the language of each cited reference
The binary PFLR/PNLR classification and all results depend on these tags; misclassification of 'Others' or 'Translation' could bias the comparison. No validation of tag accuracy is given (Section 2.3).
domain assumption Citations received from within CSSCI are a valid proxy for scientific impact
The paper interprets citation counts as impact, following Merton, but domestic citations only measure influence on Chinese social science journals, not broader impact.
ad hoc to paper The binary PFLR indicator isolates language diversity of references from other reference characteristics
The analysis does not control for number of references, age, or type, so the model implicitly assumes these are not confounders (Eq. 3).
standard math The linear regression with log10(C5+1) is an appropriate model for citation data
Citing Thelwall and Wilson (2014), the authors use OLS after log transform; this relies on standard assumptions that are not checked (Section 2.4).

how reviews work

0 comments

Cite this review

Pith. "Pith review of The citation advantage of foreign language references for Chinese social science papers." pith.science (2026). https://pith.science/paper/S77ND7ER

@misc{pith2026190807576,
  author       = {Pith},
  title        = {Pith review of: The citation advantage of foreign language references for Chinese social science papers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S77ND7ER}},
  note         = {Machine review of arXiv:1908.07576}
}

read the original abstract

Contemporary scientific exchanges are international, yet language continues to be a persistent barrier to scientific communication, particularly for non-native English-speaking scholars. Since the ability to absorb knowledge has a strong impact on how researchers create new scientific knowledge, a comprehensive access to and understanding of both domestic and international scientific publications is essential for scientific performance. This study explores the effect of absorbed knowledge on research impact by analyzing the relationship between the language diversity of cited references and the number of citations received by the citing paper. Chinese social sciences are taken as the research object, and the data, 950,302 papers published between 1998 and 2013 with 8,151,327 cited references, were collected from the Chinese Social Sciences Citation Index. Results show that there is a stark increase in the consumption of foreign language material within the Chinese social science community, and English material accounts for the vast majority of this consumption. Papers with foreign language references receive significantly more citations than those without, and the citation advantage of these internationalized work holds when we control for characteristics of the citing papers. However, the citation advantage has decreased from 1998 to 2008, largely as an artifact of the increased number of papers citing foreign language material. After 2008, the decline of the citation advantage subsided and became relatively stable, which suggests that incorporating foreign language literature continues to increase scientific impact, even as the scientific community itself becomes increasingly international. However, internationalization is not without concerns: the work closes with a discussion of the benefits and potential problems of the lack of linguistic diversity in scientific communication.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 11 canonical work pages

[1]

Aksnes, D. W. (2003). Characteristics of highly cited papers. Research Evaluation, 12(3), 159-170. Amara, N., Landry, R., & Halilem, N. (2015). What can university administrators do to increase the publication and citation scores of their faculty members?. Scientometrics, 103(2), 489-530. Antoniou, G. A., Antoniou, S. A., Georgakarakos, E. I., Sfyroeras, ...

work page 2003
[3]

C., & Nowotniak, R

Kulczycki, E., Engels, T. C., & Nowotniak, R. (2017). Publication patterns in the social sciences and humanities in Flanders and Poland. In Proceedings of the16th International Conference of the International Society for Scientometrics and Informetrics, Oct. 16-20, 2017 (pp. 95-104). Wuhan, China. Lariviè re, V. (2018). Le franç ais, langue seconde ? De l...

work page 2017
[5]

Miranda, R., & Garcia-Carpintero, E. (2018). Overcitation and overrepresentation of review papers in the most cited papers. Journal of Informetrics, 12(4), 1015-1030. Montgomery, S. L. (2013). Does science need a global language?: English and the future of research. Chicago: University of Chicago Press. Mou, X., Gong, K., Xie, J., Cheng, Y., & Ke, Q. (201...

work page 2018
[6]

Nieminen, P., Carpenter, J., Rucker, G., & Schumacher, M. (2006). The relationship between quality of research and citation frequency. BMC Medical Research Methodology, 6,

work page 2006
[8]

Accessed 20 Feb

https://www.nsf.gov/statistics/2018/nsb20181/report/sections/academic-research-and-develop ment/outputs-of-s-e-research-publications#publication-output-by-u-s-sector. Accessed 20 Feb

work page 2018
[9]

NobelPrize.org. (2018). Facts on the Nobel Prize in Literature. https://www.nobelprize.org/p rizes/facts/facts-on-the-nobel-prize-in-literature/. Accessed 3 Mar

work page 2018
[10]

Office of Humanities and Social Sciences of Nanjing University. (2017). Catalogue of Leading Journals in Humanities and Social Sciences. http://skch.nju.edu.cn/regulation. Accessed 3 Mar

work page 2017
[11]

Onodera, N., & Yoshikane, F. (2015). Factors affecting citation rates of research articles. Journal of the Association for Information Science and Technology, 66(4), 739–764. Qiu, J., Zhao, R., & Dong, K. (2016). Scientometrics. Beijing: Science Press. (in Chinese). Roth, C., Wu, J., & Lozano, S. (2012). Assessing impact and quality from local dynamics of...

work page 2015

Show all 11 references

[42]

Science and Engineering Indicators

https://doi.org/10.1186/1471-2288-6-42 National Science Foundation (2018). Science and Engineering Indicators

2018 doi
[1900]

project 985

Journal of the Association for Information Science and Technology, 66(7), 1323-1332. Liu, W. (2017). The changing role of non-English papers in scholarly communication: Evidence from web of science's three journal citation indexes. Learned Publishing, 30(2), 115-123. Liu, W., ...

2017
[2019]

E., Zarkali, A., Karageorgopoulos, D

Falagas, M. E., Zarkali, A., Karageorgopoulos, D. E., Bardakas, V., & Mavros, M. N. (2013). The impact of article length on the number of future citations: A bibliometric analysis of general medicine journals. PLoS One, 8(2), e49476. https://doi.org/10.1371/journal.pone.004947...

2013

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Aksnes, D. W. (2003). Characteristics of highly cited papers. Research Evaluation, 12(3), 159-170. Amara, N., Landry, R., & Halilem, N. (2015). What can university administrators do to increase the publication and citation scores of their faculty members?. Scientometrics, 103(2), 489-530. Antoniou, G. A., Antoniou, S. A., Georgakarakos, E. I., Sfyroeras, ...

work page 2003

[2] [3]

C., & Nowotniak, R

Kulczycki, E., Engels, T. C., & Nowotniak, R. (2017). Publication patterns in the social sciences and humanities in Flanders and Poland. In Proceedings of the16th International Conference of the International Society for Scientometrics and Informetrics, Oct. 16-20, 2017 (pp. 95-104). Wuhan, China. Lariviè re, V. (2018). Le franç ais, langue seconde ? De l...

work page 2017

[3] [5]

Miranda, R., & Garcia-Carpintero, E. (2018). Overcitation and overrepresentation of review papers in the most cited papers. Journal of Informetrics, 12(4), 1015-1030. Montgomery, S. L. (2013). Does science need a global language?: English and the future of research. Chicago: University of Chicago Press. Mou, X., Gong, K., Xie, J., Cheng, Y., & Ke, Q. (201...

work page 2018

[4] [6]

Nieminen, P., Carpenter, J., Rucker, G., & Schumacher, M. (2006). The relationship between quality of research and citation frequency. BMC Medical Research Methodology, 6,

work page 2006

[5] [8]

Accessed 20 Feb

https://www.nsf.gov/statistics/2018/nsb20181/report/sections/academic-research-and-develop ment/outputs-of-s-e-research-publications#publication-output-by-u-s-sector. Accessed 20 Feb

work page 2018

[6] [9]

NobelPrize.org. (2018). Facts on the Nobel Prize in Literature. https://www.nobelprize.org/p rizes/facts/facts-on-the-nobel-prize-in-literature/. Accessed 3 Mar

work page 2018

[7] [10]

Office of Humanities and Social Sciences of Nanjing University. (2017). Catalogue of Leading Journals in Humanities and Social Sciences. http://skch.nju.edu.cn/regulation. Accessed 3 Mar

work page 2017

[8] [11]

Onodera, N., & Yoshikane, F. (2015). Factors affecting citation rates of research articles. Journal of the Association for Information Science and Technology, 66(4), 739–764. Qiu, J., Zhao, R., & Dong, K. (2016). Scientometrics. Beijing: Science Press. (in Chinese). Roth, C., Wu, J., & Lozano, S. (2012). Assessing impact and quality from local dynamics of...

work page 2015