{"id":"3a667f6a-b976-41f2-9c69-bce9a8af805d","arxiv_id":"2411.15451","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A bibliometric study of five IITs finds growing publication output, dominant US/Germany/UK collaborations, and uneven Sustainable Development Goal alignment.","lead":"This paper counts publications from five older Indian Institutes of Technology using Scopus data from 1952 to 2024. It reports growth trends, citation impact, collaboration patterns, and alignment with Sustainable Development Goals.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's impact claim relies on citation-per-paper ratios that the paper itself contradicts: §3 credits IIT-KGP with 22.04, while Tables 3/4 yield 20.33, and IIT-K's citations differ by 360 across tables.","rationale":"The reader's weakest assumption points to external validity of Scopus affiliation IDs and SDG tags, which is a genuine limitation. But the more immediate load-bearing problem is internal: the paper's own tables and prose disagree about which IIT has the highest per-paper impact. The abstract's headline claim is exactly the one affected, so the numerical inconsistency directly threatens the central claim regardless of data-source quality. Section 3's '22.04' for IIT-KGP is contradicted by the same paper's Table 3/4 totals (20.33), and Section 4.1 gives the opposite ordering, with IIT-K first. Table 2's IIT-K citation totals also do not match Table 3/4, so the raw data feeding the ratios are not reproducible from the manuscript. This is an internally inconsistent result, not merely a disagreement with prior literature. I nevertheless keep the verdict unchanged at CONDITIONAL: the fix is an erratum plus data release, and the corrected ranking may well confirm the abstract, but the paper as written cannot be trusted on its headline metric. I credit the paper for its explicit limitations section and stated willingness to share data on request, but those do not resolve the arithmetic contradiction.","tokens_in":16021,"tokens_out":4682,"duration_ms":38844,"concrete_test":"Recompute the five citation-to-paper ratios from the row totals in Tables 3 and 4 (IIT-K: 811,476/38,585 = 21.03; IIT-KGP: 1,082,431/53,232 = 20.33; IIT-D: 19.85; IIT-B: 18.13; IIT-M: 17.09) and check whether any combination of Table 2's OA/non-OA split (e.g. IIT-K 811,836) or Table 4 decade sums reproduces the §3 value of 22.04 for IIT-KGP. If no consistent set of totals yields the claimed ranking and ratios simultaneously, the abstract's impact claim should be revised or the full dataset released for independent re-tabulation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—\"IIT-Kanpur excels in research impact, while IIT-Bombay and IIT-Madras show slightly lower per-paper impact\"—depends on accurate citation-per-paper ratios. The paper does not provide a stable set of those ratios. Section 3 says IIT-KGP has the highest overall citation-to-paper ratio (22.04), but Table 3 and Table 4 give IIT-KGP total citations 1,082,431 over 53,232 papers, i.e. 20.33—second to IIT-K, whose Table 3 total is 811,476 over 38,585 papers, i.e. 21.03. The discrepancy is not a rounding artifact: it changes the claimed leader. Moreover, Table 2's IIT-K row sums to 811,836 citations (70,731 + 741,105), 360 more than Tables 3 and 4 report, so even the input totals are internally inconsistent. Section 4.1 later reports the IIT-K-first ordering, so the paper contradicts itself. This is a correctness risk independent of Scopus coverage or SDG mapping: whatever the 'true' bibliometric answer, the paper as written does not support the headline ranking.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a bibliometric analysis of research output from five older Indian Institutes of Technology (Bombay, Delhi, Madras, Kharagpur, and Kanpur) based on Scopus-indexed publications from 1952 to 2024. It reports publication growth trends, citation impact, inter-IIT and international collaboration patterns, authorship team-size and position distributions, funding sources, and mappings of research output to the UN Sustainable Development Goals. The headline claim is that IIT-Kanpur excels in research impact, while IIT-Bombay and IIT-Madras show slightly lower per-paper impact.","tokens_in":30,"tokens_out":2761,"duration_ms":86094,"significance":"If the reported data were internally consistent, the study would offer a useful descriptive reference for policymakers, bibliometricians, and institutional planners. Its long time window, explicit Scopus affiliation-ID-based data collection, and combination of growth, collaboration, funding, and SDG analyses go beyond several earlier studies. The paper also explicitly acknowledges limitations such as reliance on Scopus and exclusion of newer IITs. However, the central impact ranking is not supported by the numbers as reported because of internal contradictions in the citation totals and ratios.","major_comments":[{"comment":"The paper's central impact claim is undermined by an internal contradiction. Section 3 states that IIT-KGP has the highest overall citation-to-paper ratio (22.04), but Table 3 reports IIT-KGP with 1,082,431 total citations over 53,232 papers, which gives 20.33, and Table 4 reports the same totals. Section 4.1 correctly gives the 20.33 figure and ranks IIT-K first. These two orderings are mutually exclusive and cannot both describe the same dataset. The authors must decide which citation totals and ratios are authoritative and correct the abstract, Section 3, and Section 4.1 accordingly.","section":"Section 3 and Table 3/Table 4"},{"comment":"The input totals themselves are inconsistent. For IIT-K, Table 2 reports 70,731 open-access citations and 741,105 non-open-access citations, summing to 811,836, while Tables 3 and 4 report 811,476 total citations. The same discrepancy does not occur for the other four IITs. This 360-citation difference indicates a data entry or aggregation error, and it propagates into any per-paper impact comparison. The authors should audit the raw Scopus query results and ensure that all tables derive from the same underlying dataset.","section":"Table 2 versus Tables 3 and 4"},{"comment":"The SDG mapping table reports total paper counts that do not match the filtered paper counts in Table 1. For IIT-B, IIT-M, and IIT-K, Table 9 lists 47,783, 50,025, and 38,584 mapped papers, whereas Table 1 reports 47,785, 50,027, and 38,585 filtered papers. The differences are small, but they suggest that the SDG analysis either used a slightly different data pull or involved additional filtering that is not documented. Because Section 4.8 uses these counts to compute SDG percentages, the source of this discrepancy must be clarified.","section":"Table 9 (SDG mapping)"},{"comment":"The subsection titled 'Proportion of authorship position in top 3 countries' contains no text or analysis. It appears immediately before Figure 10, and the caption of Figure 10 refers to 'Distribution of authorship position for top three country collaborators,' but no results for this portion are described. Either the subsection should be completed with the relevant findings, or it should be removed along with Figure 10 if it is not intended to be part of the analysis.","section":"Section 4.6.1"}],"minor_comments":[{"comment":"The sentence citing 'Hasan and Singh (2015)' contains the malformed number '2227%' and the text 'with a maximum of 2227% articles indexed in 2013'; this appears to be a typo, likely '22.7%'.","section":"Section 2"},{"comment":"The phrase 'arond 10-12%' should read 'around 10-12%'.","section":"Section 3"},{"comment":"'demonstartes' is misspelled; it should be 'demonstrates'.","section":"Section 4.1"},{"comment":"'ecah' in 'The main themes reported by ecah IIT' should be 'each'.","section":"Section 4.8"},{"comment":"The appendix repeatedly spells 'Institute' as 'Institude' in the list of IITs; this should be corrected.","section":"Appendix"},{"comment":"Several reference formatting issues appear, including inconsistent use of italics and journal titles; a careful copyedit is needed.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The internal inconsistency in the citation-to-paper ratios is not a matter of interpretation: Section 3 and Section 4.1 give different answers to the same question, and the discrepancy changes the identity of the leading institution. The authors must correct this before the paper can be considered further. The other issues, while less severe, should also be addressed in the revision. I would not recommend rejection at this stage, because the descriptive and collaborative findings may still be salvageable with a consistent dataset and a clearly stated authoritative source for the numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a straightforward descriptive bibliometric study of five older IITs, with a 1952–2024 Scopus window. The new material is real: the seven-decade growth series, the inter-IIT collaboration matrix, and the SDG mapping are not in the earlier IIT bibliometric work I know. The paper is clearly organized, the tables are fairly complete, and the limitations section is honest about Scopus-only coverage.\n\nThe problem is the headline. The abstract says IIT-Kanpur excels in per-paper impact. Section 4.1 and Table 4 support that (IIT-K 21.03 vs IIT-KGP 20.33). But Section 3 states IIT-KGP has the highest overall citation-to-paper ratio (22.04), which would make KGP the leader. Tables 3 and 4 give KGP 1,082,431 citations over 53,232 papers—exactly 20.33—not 22.04. Also, Table 2 gives IIT-K total citations of 811,836, while Tables 3/4 give 811,476, a 360-citation difference. These are not rounding effects; they flip the paper's central ranking. As written, the paper does not support its own abstract.\n\nThe SDG mapping is another soft spot. It relies on Scopus's proprietary SDG tags, and there is no validation or sensitivity check. The data availability statement says 'available upon request,' which is not reproducible. No code or processed dataset is released.\n\nThe citation pattern is fine; it cites the relevant earlier bibliometric work (Hasan and Singh, Pradhan and Sahu, etc.). For a science-policy audience, the broad conclusions—exponential growth, engineering dominance, heavy US/UK/Germany collaboration, strong ties among older IITs—are plausible and match the prior literature. The SDG percentages and collaboration counts are new descriptive results that could be useful if the numbers are corrected. But the internal inconsistency is a load-bearing flaw that needs a full correction, not a touch-up.\n\nThe paper deserves a serious referee, but I would send it back for major revision: fix the citation totals and ratios in Section 3, reconcile Table 2 with Tables 3/4, release the underlying data and code, and add a note on Scopus SDG mapping reliability. After those changes, it would be a reasonable descriptive contribution, not a groundbreaking one.","headline":"Useful descriptive bibliometric study of five IITs, but the abstract's impact ranking is contradicted by the paper's own citation totals; fix the numbers and it's a solid policy-facing snapshot.","tokens_in":16789,"tokens_out":5319,"would_cite":false,"duration_ms":42841,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Scopus-based analysis of five older IITs from 1952 to 2024 shows exponential research growth, IIT-Kanpur with the highest per-paper citation impact, and research concentrated on SDG 3, 7, and 11.","keywords":["IITs","research productivity","citation impact","SDGs","collaboration analysis","Scopus","bibliometrics","India"],"falsifier":"A re-analysis using Web of Science or Dimensions data, or using each IIT's own publication lists, that finds a different ranking of citation-to-paper ratios among the five IITs would disprove the paper's central impact claim; for instance, if IIT-Kanpur's Scopus affiliation ID misses a large number of zero-citation papers, its high citation ratio could be an artifact.","tokens_in":15852,"feed_emoji":"📈","tokens_out":5059,"duration_ms":37772,"temperature":0.7,"pith_summary":"The paper tries to establish a quantitative picture of seven decades of research at the five older IITs: Bombay, Delhi, Madras, Kharagpur, and Kanpur. It claims that output grew exponentially, supported by funding and collaborations, and that IIT-Kanpur has the best citation impact per paper while IIT-Bombay and IIT-Madras are more productive but less impactful per paper. It also claims that collaborations concentrate among the older IITs and with the USA, Germany, and the UK, and that research aligns most strongly with SDG 3 (health), SDG 7 (clean energy), and SDG 11 (sustainable cities). The study matters because it provides a long-window benchmark for Indian research policy and identifies gaps such as weak links with newer IITs and low focus on social SDGs.","feed_headline":"IIT-Kanpur leads older IITs in per-paper citation impact","feed_subtitle":"Scopus analysis of 1952–2024 output ties growth to funding, collaborations, and SDG 3, 7, 11 alignment.","key_machinery":"The analysis rests on a Scopus bibliometric dataset built from unique affiliation IDs for each of the five IITs, filtered to articles, reviews, and conference papers. The paper uses citation-to-paper ratios to rank research impact, collaboration counts to map inter-IIT and country networks, and Scopus's built-in SDG mapping to assign papers to Sustainable Development Goals. These three instruments carry the argument: growth trajectories, impact rankings, and SDG alignment percentages all come from them.","core_discovery":"The central discovery is a set of quantitative patterns covering 243,171 papers after filtering across the five IITs. Total output per IIT grew from dozens of papers per year to thousands per year, passing 1,000 papers per year by 2008. IIT-Kanpur, despite publishing the fewest papers (38,585), has the highest citation-to-paper ratio, followed by IIT-Kharagpur and IIT-Delhi; IIT-Bombay and IIT-Madras show higher volume but lower per-paper impact. IIT-Madras leads inter-IIT collaboration with 317 multi-IIT papers, and the USA is the dominant international partner for all five. Across all five, about 35% of papers map to at least one SDG, with SDG 3, 7, and 11 the most common and SDG 1, 5, and 16 below 1%.","pith_inferences":["The citation-to-paper ranking is not normalized for field or publication age; an editorially useful extension would recompute the ratios within discipline-time strata to see whether IIT-Kanpur's lead survives.","Scopus coverage itself expanded over 1952–2024, so part of the measured 'exponential growth' likely reflects growth of the database, not just IIT productivity; a database-growth-adjusted series would separate the two effects.","Scopus's SDG mapping is keyword-based; a manual or altmetric-based validation of a sample of papers could test whether the 35% SDG-aligned figure is an artifact of mapping rules."],"forward_implications":["Research funding and policy can treat IIT-Kanpur as a model for high per-paper impact rather than raw output.","The strong preference for collaborations among older IITs suggests that deliberate joint projects with newer IITs would expand the system's research base.","About 35% SDG mapping means a majority of IIT output is not aligned with SDGs, pointing to room for sustainability-focused research programs.","The dominance of the USA, Germany, and the UK as partners suggests that diversifying toward Asia, Africa, and South America could open new collaboration channels.","Thematic specialization varies by institute, so national strategy could assign niche strengths rather than expecting uniform profiles."],"supporting_citations":[{"why":"It supplies the decade-level productivity baseline for IIT Delhi that this study extends.","marker":"Chaurasia and Chavan (2014)"},{"why":"It provides the highly-cited-publications context for assessing research quality at IITs.","marker":"Awasthi and Sukula (2020)"},{"why":"It offers citation-impact figures for eight newly established IITs that frame the comparison with older IITs.","marker":"Siddaiah et al. (2016)"},{"why":"It establishes the precedent of using Scopus for bibliometric analysis of IIT research output.","marker":"Pradhan and Sahu (2018)"},{"why":"It reports productivity and citation trends for IIT Madras and IIT Bombay that the present study updates and extends.","marker":"Ramesh and Pradhan (2017)"},{"why":"It frames the Indian scientific community's response to SDG research, anchoring the paper's SDG alignment analysis.","marker":"Singh et al. (2022)"}],"fun_headline_variants":["IIT-Kanpur leads all five IITs in per-paper citations","IITs grew from dozens to thousands of papers annually","IIT-Madras tops inter-IIT collaboration with 317 papers","35% of IIT research maps to UN SDGs, led by SDG 3,7,11","USA is the top international research partner for IITs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The analysis assumes that Scopus affiliation IDs and document-type filters capture every IIT-affiliated article, review, and conference paper, and that Scopus's built-in SDG tags correctly assign papers to goals.","fun_headline_variants_meta":{"raw":{"variants":["IIT-Kanpur leads all five IITs in per-paper citations","IITs grew from dozens to thousands of papers annually","IIT-Madras tops inter-IIT collaboration with 317 papers","35% of IIT research maps to UN SDGs, led by SDG 3,7,11","USA is the top international research partner for IITs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000386,"raw_usage":{"total_tokens":2041,"prompt_tokens":950,"completion_tokens":1091,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":991}},"tokens_in":566,"tokens_out":1091,"duration_ms":9443,"temperature":1.0,"reasoning_tokens":991,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:17:02.806089+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A re-analysis using Web of Science or Dimensions data, or using each IIT's own publication lists, that finds a different ranking of citation-to-paper ratios among the five IITs would disprove the paper's central impact claim; for instance, if IIT-Kanpur's Scopus affiliation ID misses a large number of zero-citation papers, its high citation ratio could be an artifact.","supporting_citations":[{"cited_title":"Scientometricsofengineeringresearchatindianinstitutesoftechnologymadrasandbombayduring2006-2015","cited_arxiv_id":null,"evidence_quote":"It reports productivity and citation trends for IIT Madras and IIT Bombay that the present study updates and extends."}],"review_version":1}