{"id":"5dbf5aa7-1abb-4412-9efa-0ad36435f1a6","arxiv_id":"1908.07576","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Chinese social science papers citing foreign language references receive substantially more citations than those that do not, an advantage that weakened until 2008 and has since stabilized.","lead":"An analysis of 950,302 Chinese social science papers finds that papers citing foreign language references, mostly English, receive more than twice as many citations as papers without them. The finding connects the languages scholars read to the impact of their work, and raises questions about the costs of English domination in science.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 3 omits total reference count, a known citation predictor; the PFLR coefficient may be confounded by reference quantity.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the regression omits total reference count, which the introduction itself acknowledges as a robust citation predictor. This is a concrete, testable threat to the central claim because the PFLR binary could simply be a proxy for longer reference lists. The paper's descriptive findings are likely robust, and the large sample and Mann-Whitney tests show a clear association, but the regression-based claim of a persistent advantage 'after controlling for characteristics' is not yet demonstrated. The proposed test directly addresses the omitted-variable bias by adding the missing control; if the coefficient attenuates, the paper's causal and policy implications would need substantial revision. Since the reader already recommends a conditional acceptance pending this additional control, my assessment does not change the verdict. I therefore mark verdict as unchanged and agreement as full.","tokens_in":17006,"tokens_out":2564,"duration_ms":549506,"concrete_test":"Re-estimate the model in Eq. 3 adding a control for total reference count (e.g., log10(Nref+1), where Nref is the number of references for each paper). If the standardized coefficient for PFLR drops substantially (e.g., by more than 30%) or loses statistical significance, the claimed citation advantage is confounded by reference quantity. As a supporting diagnostic, report the mean and median reference counts for PFLR and PNLR papers across the sample and within each discipline, and consider a matching analysis that pairs PFLR and PNLR papers on reference count.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the citation advantage of papers with foreign language references (PFLR) persists after controlling for discipline, journal prestige, institution prestige, and collaboration (Section 3.3.5, Table 10). However, the regression in Eq. 3 does not include the total number of references per paper. The paper's own introduction (Section 1) explicitly lists 'number of references' as a factor positively associated with citation counts, citing numerous studies. If PFLR papers systematically cite more references than PNLR papers, the standardized coefficient for PFLR (β=0.229) will absorb part of the reference-quantity effect, overstating the language-specific advantage. The authors do not report descriptive statistics for reference counts by PFLR status, so the magnitude of this potential confound is unknown. This is a classic omitted-variable bias that directly undermines the claim that the advantage 'holds when we control for characteristics of the citing papers.' A robustness check adding reference count as a control is essential before the causal language in the abstract and discussion ('incorporating foreign language literature continues to increase scientific impact') can be supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether Chinese social science papers that cite foreign-language references receive more citations than those that do not, using 950,302 papers from the Chinese Social Sciences Citation Index (CSSCI) published 1998-2013. The authors compute citation counts per paper (CPP2, CPP5) and citation advantages (CA2, CA5), run Mann-Whitney U tests by discipline, journal prestige, institution prestige, and collaboration type, and estimate a multiple linear regression of log10(C5+1) on a binary indicator for foreign-language references plus controls (Eq. 3). They find that papers with foreign-language references (PFLR) receive significantly more citations, that the advantage declines from 1998 to 2008 as the proportion of PFLR papers rises, and that the advantage persists after controlling for discipline, journal prestige, institution prestige, and co-authorship. They interpret the persistent positive coefficient as evidence that incorporating foreign-language literature increases scientific impact.","tokens_in":17163,"tokens_out":2096,"duration_ms":21720,"significance":"The paper provides a large-scale, descriptive picture of language diversity in references for a non-English scientific community, a topic that is often overlooked in bibliometric research. The data collection is substantial and the statistical analyses are transparent: the raw CPP and CA calculations, the Mann-Whitney tests, and the regression are all straightforward and replicable. The paper is valuable as a descriptive empirical contribution to understanding the internationalization of Chinese social science and the relationship between foreign-language referencing and citation outcomes. However, the central inferential claim that foreign-language references have a causal or even net positive effect on impact is not established by the current design, primarily because of an omitted-variable problem in the regression that is directly acknowledged in the paper's own introduction.","major_comments":[{"comment":"The regression in Eq. (3) omits the total number of references per paper, a variable that the authors themselves list in Section 1 as a robust positive predictor of citation counts, citing more than a dozen studies. If PFLR papers tend to have longer reference lists than PNLR papers, the standardized coefficient for PFLR (β=0.229) may partially reflect reference quantity rather than language diversity. The authors do not report descriptive statistics for reference counts by PFLR status, so the magnitude of this potential confound is unknown. A robustness check that adds total reference count as a control, or at least a descriptive comparison of reference counts by PFLR status, is essential to support the claim that the citation advantage holds after controlling for characteristics of the citing papers.","section":"Section 2.4, Eq. (3), Table 10"},{"comment":"The causal framing of the results goes beyond what the observational design can support. The abstract states that the study explores 'the effect of absorbed knowledge on research impact,' and the discussion says that incorporating foreign-language literature 'continues to increase scientific impact.' Since PFLR status is not randomly assigned and the regression omits numerous potential confounders (e.g., English proficiency of authors, research quality, topic novelty, funding, and international collaboration), the estimates at best show a partial association. The authors should either temper the causal language throughout, or clearly label the design as observational and the interpretation as associational.","section":"Abstract and Section 4 (Discussion)"},{"comment":"The claim that the citation advantage of PFLR is 'the strongest' among all variables rests on comparing standardized coefficients for a binary variable with coefficients for categorical and other binary variables. Since the variables have different units and the table does not include unstandardized coefficients or standard errors, the statement that foreign-language referencing has 'a stronger influence' than journal prestige, institution prestige, or collaboration is not fully supported. Presenting unstandardized coefficients with confidence intervals would allow a more direct comparison.","section":"Section 3.3.5, Table 10"}],"minor_comments":[{"comment":"The definition of CA as a ratio of average citations is clear, but the notation in Eq. (2) is not explicitly shown; writing the formula explicitly would improve readability.","section":"Section 2.4, Eq. (2)"},{"comment":"Figure 3B shows CA values of 2 and 2.5 after 2008, but the text says 'remained relatively stable around 2 and 2.5' without providing confidence intervals or a formal trend test; adding a simple trend test for the post-2008 period would strengthen the claim of stability.","section":"Section 3.2, Figure 3"},{"comment":"The descriptive CA values in Table 5 vary substantially across disciplines (e.g., CA2=4.68 for Statistics vs. CA2=1.20 for Sports Science), yet the authors do not discuss why certain disciplines show much larger advantages; a brief comment would help interpret these differences.","section":"Section 3.3.1, Table 5"},{"comment":"There are minor typographical issues, such as 'multiple liner regression' in Section 3.3.5 and 'paper with foreign language references' in Section 3.3.1; a careful proofreading pass is recommended.","section":"Throughout"},{"comment":"The sentence 'By 2018, less than 2% (n= 25,504; 1.49%) of CSSCI-indexed papers have authors from other countries' uses inconsistent verb tense; it should be 'had' or 'have' consistently.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, this is a large, careful descriptive study of 950k Chinese social science papers. The headline finding is real: papers with at least one foreign-language reference get about 2–3 times more citations than those without, and the gap shrinks then plateaus over 1998–2013. What's new here is scale and scope—all 13 CSSCI disciplines over 16 years—and the temporal analysis of the citation advantage. The authors also do a decent job showing the raw association holds up within disciplines, journal prestige strata, institution tiers, and co-authorship splits.\n\nThe soft spot is the regression in Eq. 3. It omits total reference count, which the paper's own introduction lists as a well-established predictor of citations. If PFLR papers also cite more references overall—which seems likely, since they have at least one foreign reference and probably longer reference lists—then the standardized coefficient of 0.229 partly captures quantity, not just language. The authors never report reference counts by PFLR status, so we can't tell how much of the advantage is language diversity versus sheer reference volume. That makes the causal language in the abstract and discussion ('absorbed knowledge,' 'scientific impact') premature. The 'decreased as an artifact' explanation for 1998–2008 is also asserted rather than tested with a formal decomposition.\n\nThe Mann-Whitney p-values are essentially meaningless at this sample size, but that's minor; the effect sizes (CA 2–3x) are what matter. I'd also like confidence intervals, but the raw means are there.\n\nThe self-citations to Gong et al. 2018 and Lariviere et al. 2019 are appropriate given the prior work; no problem there.\n\nAll told, the descriptive contribution is solid and worth publishing after revision. The authors should add reference count as a control, or at minimum present reference-count distributions by group and discuss the direction of expected bias. They should soften 'impact' to 'association' unless they can defend a mechanism. If they do that, this becomes a useful paper for science policy and scientometrics readers. I would send it to peer review; with those additions, it would be a reasonable accept.","headline":"Solid large-scale descriptive finding on foreign-language referencing in Chinese social science, but the regression omits total reference count and the causal language goes beyond the evidence.","tokens_in":17720,"tokens_out":2062,"would_cite":true,"duration_ms":230000,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Chinese social science papers that cite foreign-language literature receive about two to three times as many citations as papers citing only Chinese work, and the advantage resists standard controls.","keywords":["foreign language references","citation advantage","references","citations","social sciences","China","globalization","lingua franca"],"falsifier":"Add total reference count, article length, and author count to the regression, or match each PFLR paper with a PNLR paper on those variables plus discipline, journal tier, institution, and collaboration; if the foreign-language coefficient drops to zero, the claimed citation advantage is refuted.","tokens_in":16767,"feed_emoji":"🌐","tokens_out":11584,"duration_ms":98688,"temperature":0.7,"pith_summary":"This paper asks whether the language of the literature a researcher absorbs shapes the impact of the work they produce. Using roughly 950,000 Chinese-language social science papers and the references they cite, it reports that papers with at least one foreign-language reference receive about 2.4 times as many citations within two years and about 3.0 times as many within five years as papers whose references are entirely Chinese. That advantage holds within every discipline and after controlling for journal prestige, institution prestige, and collaboration, and the foreign-language indicator is the strongest predictor in the model. The paper further shows that the advantage narrowed as foreign-language citing became common between 1998 and 2008, then stabilized, suggesting that absorbing foreign work still pays off even as internationalization spreads. If true, this would mean that access to non-Chinese literature is a concrete driver of scientific impact for Chinese social science, with direct implications for language policy and research evaluation.","feed_headline":"Foreign-language references double or triple citation counts","feed_subtitle":"For Chinese social science papers, the gain survives controls for discipline, journal, institution, and collaboration.","key_machinery":"The central objects are the binary codes PFLR (paper with at least one foreign-language reference) and PNLR (paper with none), together with the citation advantage $\\mathrm{CA}$, defined as the ratio of average citations per paper for PFLR papers to that for PNLR papers. The paper computes $\\mathrm{CA}$ for two- and five-year citation windows and tests the raw gap with Mann-Whitney U tests. To rule out confounds, it then fits a multiple linear regression with $\\log_{10}(C_5+1)$ as the outcome and PFLR, discipline dummies, leading-journal status, top-institution status, and co-authorship as predictors; the regression is what carries the claim that the language effect is not an artifact of prestige or collaboration.","core_discovery":"The central claim is that absorbing foreign-language literature raises a paper's impact inside the Chinese social science community. Using 950,302 Chinese-language papers and 8,151,327 cited references from a comprehensive national citation index, the paper reports that papers with at least one foreign-language reference account for 35.5% of publications but 62.5% of five-year citations, with citation advantage $\\mathrm{CA}_2=2.41$ and $\\mathrm{CA}_5=3.04$ relative to papers without such references. The advantage appears in every one of thirteen disciplines, in leading and ordinary journals, at top and other institutions, and for single- and co-authored papers; in the full regression, the foreign-language indicator has the largest standardized coefficient ($\\beta=0.229$, $p<0.001$), larger than journal prestige (0.144), institution prestige (0.097), or collaboration (0.085). The paper also finds that the advantage declined as the share of papers citing foreign work rose from 11.7% in 1998 to 54.4% in 2013, stabilizing after 2008 at roughly 2 to 2.5, which it interprets as a persistent but homogenizing effect of internationalization.","pith_inferences":["An inference beyond the paper: the regression omits total reference count, a factor the paper itself notes predicts citations; if PFLR papers also carry longer reference lists, part of the measured advantage may be a reference-quantity effect rather than a language effect.","An inference beyond the paper: the same design could be applied to other large non-English research communities, such as Japan, France, or Spain, to see whether the foreign-language citation premium generalizes beyond China.","An inference beyond the paper: the post-2008 stability suggests a ceiling or signaling mechanism; once foreign-language citing becomes the norm, the remaining premium may reflect unobserved quality rather than the causal benefit of reading foreign work.","An inference beyond the paper: a direct test would check whether the number of foreign-language references, or the depth with which they are used, scales with the citation advantage, something the binary PFLR coding cannot reveal."],"forward_implications":["Chinese-language social science papers that draw on foreign literature will, on average, accumulate domestic citations faster and for longer than papers that cite only Chinese work.","Restricting access to foreign-language journals by subscription gaps, translation delays, or policy would likely depress the production of high-impact Chinese social science.","Because 96.4% of foreign-language references are English, the advantage is effectively an English-reading advantage, so the internationalization of Chinese social science is largely an English-language absorption effect.","The narrowing of the advantage between 1998 and 2008 is consistent with a diffusion process: as more papers cite foreign work, citing it no longer sets a paper apart, yet the stable post-2008 premium indicates the association is not disappearing."],"supporting_citations":[{"why":"Supplies the absorptive-capacity concept that motivates treating references as absorbed knowledge.","marker":"Cohen and Levinthal (1990)"},{"why":"Provides the regression strategy for citation data that the paper adopts.","marker":"Thelwall and Wilson (2014)"},{"why":"Documents the national citation index's design and the reference-language tagging scheme used to classify papers.","marker":"Su et al. (2014)"},{"why":"Reviews the paper-, journal-, and author-level factors that the paper must control for.","marker":"Tahamtan et al. (2016)"},{"why":"Earlier finding that prestigious Chinese journals cite more foreign-language literature, which this study extends to citation impact.","marker":"Gong et al. (2018)"},{"why":"Establishes journal prestige as a predictor of citations, justifying the leading-journal control.","marker":"Callaham et al. (2002)"},{"why":"Frames citations as 'pellets of peer recognition,' the conceptual basis for treating references as knowledge transmission.","marker":"Merton (1988)"},{"why":"Documents national homophily in referencing, motivating the study of Chinese papers' foreign-language citing.","marker":"Larivière et al. (2019)"}],"fun_headline_variants":["Foreign-language refs give Chinese social science papers a 3x citation boost","Foreign-language citations raise Chinese social science impact by up to 3x","Chinese social science papers with foreign refs get 2-3x more citations","The foreign-language citation edge persists in Chinese social science"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a paper's binary status of having at least one foreign-language reference captures the effect of absorbing foreign knowledge, rather than serving as a proxy for longer reference lists, longer papers, or unmeasured quality.","fun_headline_variants_meta":{"raw":{"variants":["Foreign-language refs give Chinese social science papers a 3x citation boost","Foreign-language citations raise Chinese social science impact by up to 3x","Chinese social science papers with foreign refs get 2-3x more citations","The foreign-language citation edge persists in Chinese social science"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001255,"raw_usage":{"total_tokens":5209,"prompt_tokens":1079,"completion_tokens":4130,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":4061}},"tokens_in":695,"tokens_out":4130,"duration_ms":33044,"temperature":1.0,"reasoning_tokens":4061,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:03:35.126433+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Add total reference count, article length, and author count to the regression, or match each PFLR paper with a PNLR paper on those variables plus discipline, journal tier, institution, and collaboration; if the foreign-language coefficient drops to zero, the claimed citation advantage is refuted.","supporting_citations":[],"review_version":1}