Pith. sign in

REVIEW 3 major objections 5 minor 58 references

Quantitative analysis of the value of investment in research facilities, with examples from cyberinfrastructure

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Quantitative accounting methods can show that investment in university research-computing facilities returns more than it costs, with XSEDE as the flagship example.

desk verdict A useful, honest methods review whose central XSEDE claim hinges on an undisclosed break-even credit share; worth publishing after revision. read the letter →

arxiv 2502.07833 v3 pith:Y2DPTWF3 submitted 2025-02-11 econ.GN cs.CYq-fin.EC

Winona G. Snapp-Childs , David Y. Hancock , Preston M. Smith , John Towns , Craig A. Stewart This is my paper · ORCID
classification econ.GNcs.CYq-fin.EC
keywords researchfacilitiescyberinfrastructurereturnoninvestmentcost-benefitanalysisproductionfunctionhighereducationfinanceXSEDEpolicy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the value of investment in research-enabling facilities can be quantified well enough to guide hard budget decisions, and that for cyberinfrastructure the numbers come out positive. It presents a menu of methods\u2014quality-management metrics, bibliometrics, time-savings estimates, econometric multipliers, accounting-based ROI measures, and production-function modeling\u2014and shows that they can be applied retroactively to existing records. Its headline findings are that university-operated high-performance computing is repeatedly cheaper than buying the same capability from commercial cloud providers, that a production-function analysis links HPC capacity and support-staff salaries to grant awards, publications, and PhDs at one university and across five research-intensive universities, and that a comprehensive accounting of XSEDE puts the value it delivered to the US economy and society at or above its cost to US taxpayers. These findings matter because US universities are closing at more than one per month and several have announced deficits above $10 million, so institutions need fact-based, stakeholder-facing arguments about what research facilities return.

What carries the argument

Three mechanisms carry the argument. The first is a proxy for return on investment, $\text{ROI}_{\text{proxy}} = \frac{\text{market value of services delivered and products created}}{\text{cost to deliver services and products}}$, where a value above 1.0 means the operation is cheaper than buying the same services at market prices; it is the basis for the lease-versus-cloud comparisons. The second is an integrated reporting framework organized around six forms of capital\u2014financial, manufactured, intellectual, human, social and relationship, and natural\u2014which ensures that non-financial outputs are counted, together with an 'appeal to reasonableness' that assigns XSEDE a minimum 1% share of credit for end-product value. The third is the Cobb\u2013Douglas production function $Y = f(K_{\mathrm{flops}}, L_{\mathrm{staff}})$, treating on-premises TeraFLOPS and research-computing staff salaries as inputs and academic outputs as outputs, which supplies the statistically significant relationships reported for a single university and across five institutions. A Balanced Scorecard layout ties these results together for stakeholder communication.

What would settle it

An independent accounting that assigns fractional credit across all contributing facilities and investigators, instead of a flat 1% floor to XSEDE, would settle the benefit claim: if the value creditable to XSEDE falls below XSEDE's total cost, the 'at least 1.0' conclusion is wrong. For the production-function claim, a natural experiment at a university that sharply cuts HPC capacity or research-computing staff, showing no subsequent decline in grants, publications, or PhDs once other inputs are controlled, would falsify the claimed relationship.

Watch

Extended reading notes

Core claim

The central claim, stated the way the authors would state it, is that the benefits of research-enabling facilities are measurable, and the measurements consistently justify the investment. The strongest result is about XSEDE, a federally funded US cyberinfrastructure service that operated from 2011 to 2022: applying a six-capital integrated reporting framework to the end products enabled by XSEDE, the authors' prior work valued publications, grants, doctorates, and other outcomes, and showed that even if XSEDE is credited with only 1% of the value of those end products, the benefit-to-cost ratio is at least 1.0. The paper also presents a Cobb\u2013Douglas production function\u2014capital entered as on-premises TeraFLOPS and labor as salaries of research-computing staff\u2014that yields strong, statistically significant relationships with university R&D expenditures, new grant awards, publications, high-impact publications, and PhDs, at one university over 21 years and across five research-intensive universities over 82 total years. The paper does not present new data; its contribution is to consolidate prior results into a reusable evaluative toolkit and to argue the toolkit transfers to other research-enabling facilities and to research enterprises generally.

Load-bearing premise

The load-bearing premise is that XSEDE deserves at least 1% of the credit for the full value of every end product that used its services; that floor is adopted as an appeal to reasonableness rather than derived from data, and if the true share were lower the conclusion that the return was at least 1.0 would no longer follow.

Editorial extensions

If this is right

  • A university comparing local HPC with commercial cloud can expect on-premises operation to be roughly two to three times cheaper for ordinary cluster workloads, so cutting local capacity on cost grounds may increase total expenditures unless workloads are cloud-native.
  • Institutional planners can use the production-function coefficients as starting estimates: about $100,000 of HPC equipment or one additional research-computing staff member corresponds to millions of dollars in research expenditures at a research-intensive university.
  • If XSEDE's accounting holds at the 1% credit floor, any higher attribution of credit only strengthens the conclusion, so the sign of the result is robust to attribution uncertainty above the floor.
  • Because most of the methods work retroactively from existing records, a facility with good accounting can produce a defensible value statement without waiting for a prospective study.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: replacing the flat 1% floor with an explicit fractional-credit rule shared among all contributing facilities would probably shrink the estimated surplus but make the method more persuasive; the report should disclose the break-even credit share as an audit figure.
  • Editorial inference: the production-function results imply a testable natural experiment\u2014universities that cut HPC staff or capacity during recent budget crises should show lagged declines in grant awards and publications, which would strengthen the claimed causal link.
  • Editorial inference: applying the same toolkit to non-computing research facilities, such as animal facilities, observatories, or clinical research cores, would reveal whether the strong positive relationships are specific to cyberinfrastructure or general to research-enabling investment.
  • Editorial inference: the surveys showing cloud use is driven by capability rather than cost suggest a portfolio strategy\u2014on-premises clusters for bulk throughput plus commercial cloud for cloud-native or always-on services\u2014that the paper's cost comparisons do not directly test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper is a survey of quantitative methods for valuing investment in research-enabling facilities, developed largely for US cyberinfrastructure. It reviews: quality-management metrics (user satisfaction, total cost of ownership, operational usage); bibliometrics; researcher-time savings; econometric multiplier methods (RIMS II, IMPLAN); accounting-based methods (leverage, an ROI proxy, and the International <IR> Framework as applied to XSEDE); and a 'production-function' analysis using 21 years of Purdue data extended to five additional R1 institutions, relating TeraFLOPS capacity and RCD staff salary to HERD expenditures, new grant awards, publications, high-impact publications, and PhDs awarded. The headline claims are that (i) a comprehensive accounting assessment shows XSEDE delivered value to the US economy and society that exceeded its cost to US taxpayers, (ii) the production-function results show strong relationships between HPC investment and academic outputs that testify to an underlying causal relationship, and (iii) on-premises HPC is more cost-effective than commercial cloud for the workloads studied. The paper presents no new data; it says that reconsidering five recent publications by the authors yields new insight, and it proposes a Balanced Scorecard template as a communication device for such metrics.

Significance. The manuscript is a genuinely useful and clearly organized survey of an emerging evaluation literature, and it has several strengths that should be acknowledged: the cost equations (Eqs. 3-6) are concrete and usable; the authors are candid that valuations are approximate, that the XSEDE argument rests on an 'appeal to reasonableness,' and that in every area save two the results discussed are their own; the leverage discussion explicitly warns against double counting; and the treatment of von Laszewski et al. candidly identifies a conceptual weakness in the bibliometric approach. The paper also reports significance levels and variance decompositions in Tables 1-4 rather than hiding them. If these methods are to serve the load-bearing policy purpose stated in the Introduction, two things must be repaired: the XSEDE net-benefit threshold must be reported so that the 1% credit assumption can be evaluated rather than taken on faith, and the production-function discussion must present associations as associations, since the design is correlational and secular trends are a plausible confounder.

major comments (3)
  1. [Comprehensive framework for financial value creation; Discussion] The central conclusion that XSEDE's value to US taxpayers and US society exceeded its cost rests on the premise that XSEDE merits at least 1% of the credit for the aggregate value of end products created with its services, a step the authors themselves label an 'appeal to reasonableness.' The manuscript states that Snapp-Childs et al. [8] 'calculated the percentage of credit that would have to be attributable to XSEDE in order for the ROI to be at least 1.0,' but it never reports this break-even credit percentage. Because every component of the end-product value is acknowledged to be approximate, and because the leverage figures cited earlier (USD 4.5B in grants against a USD 200M XSEDE investment) indicate that the break-even share could differ substantially from 1%, the reader cannot tell whether 1% is a comfortable margin or a knife-edge assumption. The paper should report the break-even credit share, the total end-product value and cost base used in [8], and the sensitivity of the ROI conclusion to plausible variation in the credit share and the component valuations.
  2. [A production-function analysis of academic and financial outputs; Discussion] The Discussion claims that the relationships 'remain statistically significant when analyzed across six different R1 universities' and that this is 'a strong testament to the reality of an underlying causal relationship.' Both statements are at odds with the paper's own tables: Table 3 shows the PhD-awarded/TeraFLOPS slope (1.56) is not statistically significant, and Table 4 shows TeraFLOPS explains only 1% of the variance in PhDs across institutions while 'other factors' explain 74%; the text in the same section refers to 'five different R1 institutions' and '82 total years of data from five institutions.' Moreover, the analysis as presented is a set of contemporaneous linear correlations with no controls for common time trends, and the assertion that all other major research-enabling facilities 'stayed relatively consistent over time' is made without supporting data; given that HERD, grant income, and compute capacity grew secularly over 1999-2020, trending confounders are a plausible alternative explanation. The causal language should be replaced with associational language, the five/six inconsistency resolved, and the non-significant PhD result acknowledged.
  3. [Eq. (7), production-function section] The section title and Eq. (7) present the method as a production function Y = f(K_flops, L_staff) in the spirit of Cobb and Douglas, but Tables 1-4 report only bivariate 'fitted slopes of outputs as linear correlates of inputs, measured annually.' No joint estimation of capital and labor inputs, no functional form, and no lag structure is described, so the label 'production-function analysis' claims more economic structure than the reported evidence exhibits. The authors should either describe the actual econometric specification used in [12] and [13] (whether inputs enter jointly, whether lags or controls are included) or describe the method as a correlational analysis.
minor comments (5)
  1. [Discussion, investment guidance] The claim that an investment of USD 100,000 in computational capability (about 200 TeraFLOPS) 'will correspond with an increase in total university research expenditures by the institution of USD 6.45M' is not traceable to the paper's tables: Table 1 implies about USD 2.6M (slope 1.29 per 100 TeraFLOPS at Purdue) and Table 3 implies about USD 6.2M (slope 3.10 per 100 TeraFLOPS across five institutions). Please state which model and which units produce the 6.45 figure.
  2. [Eqs. (2) and (6)] Both equations are labelled ROIproxy but define different quantities: Eq. (2) is the market value of outputs divided by cost, whereas Eq. (6) is the ratio TCO_onprem/TCO_cloud, a cost-avoidance ratio. Using one symbol for two different ratios invites confusion; distinct names would be clearer.
  3. [Abstract] Both the abstract and the Introduction state that 'a publicly funded cyberinfrastructure project delivered to the USA economy and society exceeded the cost to USA taxpayers'; the sentence is missing the noun 'value' and as written attributes the exceeding to the project rather than to the value delivered.
  4. [Title page and reference [1]] The title contains a stray space ('researc h facilities'), and reference [1] says 'republished with new forward,' which should be 'foreword.'
  5. [Prior Related Research, CERN paragraph] The sentence 'The European facilities of CERN ... - is perhaps the best-known example' has a subject-verb agreement problem ('facilities ... is') and awkward dashes; the sentence should be rewritten.

Circularity Check

2 steps flagged · score 4.0 of 10

XSEDE ROI claim is carried by the authors' own prior 1%-credit assumption, and the production-function 'prediction' overlaps definitionally with its HERD output.

  1. self citation load bearing [Section 'A comprehensive framework for financial value creation'; Discussion]
    "After calculating the total end-product value, Snapp-Childs et al. calculated the percentage of credit that would have to be attributable to XSEDE in order for the ROI to be at least 1.0. Starting from the premise that XSEDE deserved at least 1% of the credit for outcomes stemming from the use of XSEDE, they were able to conclude that the return on USA federal investment in XSEDE was at least 1.0."

    The paper's flagship quantitative result is not derived here; it is imported from reference [8], whose authors overlap with the present paper. The cited paper's conclusion is obtained by assuming a 1% credit share, not by estimating one, and the present paper never reports the break-even credit percentage that [8] computed. The central benefit claim therefore reduces to the authors' own prior 'appeal to reasonableness,' which is an unverified premise rather than an externally checkable empirical result.

  2. self definitional [Section 'A production-function analysis of academic and financial outputs', Eq 7 and Table 1]
    "The outputs and metrics (Y) are (in any given time period, typically annually): ... total Higher Education Research and Development (HERD) expenditures as reported to the NSF ... and the inputs are: ... Lstaff: salary costs(labor) for RCD professionals."

    The dependent variable HERD expenditures is the university's total R&D spending, and the labor input Lstaff is the salary cost of RCD professionals. Unless those salaries are explicitly excluded from HERD—and the paper gives no such exclusion—Lstaff is a component of HERD by definition. The regression of HERD on Lstaff therefore contains a guaranteed positive relationship, so the later statement that $100,000 in staff costs 'corresponds to' a $14M increase in HERD is partly forced by variable definitions rather than by an independent causal effect.

full rationale

No complete definitional circularity is present: the 1% credit share is an explicit assumption, and the lease-vs-buy ROI-proxy comparisons are benchmarked against commercial cloud prices, which provides independent content. However, the flagship XSEDE benefit claim is load-bearing on the authors' own prior work, whose key premise is an unreported-margin 'appeal to reasonableness,' and the production-function prediction contains a definitional overlap between HERD output and RCD staff salary input. These are partial circularities, not total equivalence, so the score is 4.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on fitted regression parameters, a hand-chosen credit threshold, and several domain assumptions about stability and valuation. No new entities are introduced. The most consequential free parameter is the 1% credit share, because the XSEDE net benefit conclusion changes if the true share is lower.

free parameters (4)
  • Minimum XSEDE credit percentage = 1%
    Hand-chosen threshold adopted from Snapp-Childs et al. to conclude that XSEDE ROI is at least 1.0. No empirical basis is provided for this percentage.
  • TeraFLOPS regression slopes = See Tables 1 and 3
    Slopes relating HERD expenditures, grants, publications, high-impact publications, and PhDs to HPC capacity are fitted to 21 years of Purdue data and 82 institution-years across five R1 universities, then used to make forward-looking 'will correspond' statements.
  • RCD salary regression slopes = See Tables 1 and 3
    Fitted slopes per USD 100,000 of research computing staff salary, used to claim each additional staff member corresponds to a USD 14 million increase in grant expenditures.
  • Market valuations for non-market outputs = Not specified in this paper
    The accounting framework assigns financial values to publications, PhDs, and quality-of-life improvements drawn from various sources. The paper does not test the sensitivity of conclusions to these valuations.
assumptions (4)
  • domain assumption Academic outputs are a linear production function of HPC capacity and staff costs: Y = f(K_flops, L_staff).
    This is the core specification of the production-function analysis, adopted from Cobb-Douglas tradition without model selection or specification tests in this paper.
  • domain assumption Other major research-enabling facilities at the studied universities stayed relatively consistent over the measurement period.
    Invoked to interpret Purdue correlations as evidence of a causal relationship, but no data are shown to support this stability assumption.
  • domain assumption Valuations of non-market goods are credible and additive within the Integrated Reporting Framework.
    The framework depends on summing the financial value of publications, PhDs, and other outputs, which requires accepting valuations from heterogeneous sources as commensurable.
  • ad hoc to paper XSEDE merits at least 1 percent credit for the aggregate value of all outcomes produced with XSEDE resources.
    This 'appeal to reasonableness' is the linchpin of the XSEDE ROI greater than or equal to 1.0 claim. It is a chosen threshold, not a measured quantity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Quantitative analysis of the value of investment in research facilities, with examples from cyberinfrastructure." pith.science (2026). https://pith.science/paper/Y2DPTWF3

@misc{pith2026250207833,
  author       = {Pith},
  title        = {Pith review of: Quantitative analysis of the value of investment in research facilities, with examples from cyberinfrastructure},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y2DPTWF3}},
  note         = {Machine review of arXiv:2502.07833}
}
read the original abstract

Purpose: How much to invest in research facilities has long been a question in higher education and research policy. We present established and recently developed techniques for assessing the quantitative value created or received as a result of investments in research facilities. This discussion is timely. Financial challenges in higher education may soon force difficult decisions regarding investment in research facilities at some institutions. Clear quantitative analysis will be necessary for such strategic decision-making. Further, institutions of higher education in the USA are currently being called on to justify their value to society. The analyses presented here are extendable to research enterprises as a whole. Results: We present methods developed primarily for analyses of cyberinfrastructure. Most analyses comparing investment in university-based cyberinfrastructure facilities with purchasing services from commercial sources demonstrate positive results for economic and scientific research. A recent assessment, based on a comprehensive accounting approach, has shown that for one large publicly funded cyberinfrastructure project the value delivered to the USA economy and society exceeded the cost to USA taxpayers. Conclusions: Quantitative analyses of the benefits of investment in research and research facilities create a fact-based foundation for discussing the value of research and higher education. These methods enable a quantitative assessment of the relationship between investment in specific research facilities or research projects and economic, societal, and educational outcomes. These methods are of value in quantifying the economic benefit of higher education and in managing investments within such institutions.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 56 canonical work pages

  1. [8]

    Evaluating return on investment for cyberinfrastructure u sing the International Integrated Reporting Framework

    Snapp-Childs WG, Hart DL, Costa CM, Wernert JA, Jankowski HE , Towns J, et al. Evaluating return on investment for cyberinfrastructure u sing the International Integrated Reporting Framework. SN Computer S cience. 2024;5(5):558. doi:10.1007/s42979-024-02889-z

    work page doi:10.1007/s42979-024-02889-z 2024

  2. [12]

    The value proposition of campus high performance com puting facilities to institutional productivity: a production function model

    Smith PM. The value proposition of campus high performance com puting facilities to institutional productivity: a production function model. SN computer science. 2024;5(5). doi:10.1007/s42979-024-02888-0

    work page doi:10.1007/s42979-024-02888-0 2024

  3. [13]

    Application of the Cyberinfrastructure Production Function Model to R1 Institutions

    Smith PM, Gemmill J, Hancock DY, O’Shea BW, Snapp-Childs WG, Wilgenbusch J. Application of the cyberinfrastructure production function model to R1 institutions; 2025. Available from: https://arxiv.org/abs/2501.10264

    work page Pith review arXiv 2025

  4. [1]

    Science, the endless frontier

    Bush V. Science, the endless frontier. Office of Scientific Resea rch and Development; 1945, republished with new forward in 1975. Available f rom: https://nsf-gov-resources.nsf.gov/2023-04/EndlessFr ontier75th_w.pdf

    work page 1945

  5. [2]

    The year in closures and mergers; 2024

    Moody J. The year in closures and mergers; 2024. Inside Higher E d. Available from: https://www.insidehighered.com/news/business/financial-health/2024/12/13/2

    work page 2024

  6. [3]

    The Carnegie classification of inst itutions of higher education; 2023

    American Council on Education. The Carnegie classification of inst itutions of higher education; 2023. American Council on Education. Available fr om: https://carnegieclassifications.acenet.edu/

    work page 2023

  7. [4]

    Investing in education 2024

    European Commission. Investing in education 2024. Luxembourg : Publications Office European Communities/Union; 2024. Available from: http://dx.doi.org/doi/10.2766/969920

    work page doi:10.2766/969920 2024

  8. [5]

    Trump administration freezes $1 billion in funding for Cornell University, $790 million for Northwestern University; 2025

    Waldenberg S, Dam T, Romine T. Trump administration freezes $1 billion in funding for Cornell University, $790 million for Northwestern University; 2025. CNN. Available from: https://www.cnn.com/2025/04/09/us/cornell-northwest ern-federal-funding-free May 27, 2025 22/26

    work page 2025

Show all 58 references
  1. [6]

    A year after cuts, WV still bleeding faculty, administrators; 2024

    Quinn R. A year after cuts, WV still bleeding faculty, administrators; 2024. Inside Higher Ed. Av ailable from: https://www.insidehighered.com/news/faculty-issues/ tenure/2024/09/09/year-

    2024

  2. [7]

    What is cyberinfrastructure

    Stewart CA, Simms S, Plale B, Link M, Hancock DY, Fox GC. What is cyberinfrastructure. In: Proceedings of the 38th Annual ACM S IGUCCS Fall Conference: Navigation and Discovery. SIGUCCS ’10. New York, NY : ACM

  3. [9]

    Our origins; 2025

    A WS. Our origins; 2025. A WS. Available from: https://aws.amazon.com/about-aws/our-origins/

    2025

  4. [10]

    Stewart CA, Hancock DY, Wernert J, Link MR, Wilkins-Diehr N, Mille r T, et al. Return on investment for three cyberinfrastructure facilit ies: a local campus supercomputer; the NSF-funded Jetstream cloud syste m; and XSEDE (the eXtreme Science and Engineering Discovery Environ...

    2018

  5. [11]

    Use of accounting concepts to study research: return on invest ment in XSEDE, a US cyberinfrastructure service

    Stewart CA, Costa CM, Wernert JA, Snapp-Childs W, Bland M, Blo od P, et al. Use of accounting concepts to study research: return on invest ment in XSEDE, a US cyberinfrastructure service. Scientometrics. 2023;128(6) :3225–3255. doi:10.1007/s11192-022-04539-8

    2023 doi

  6. [14]

    Federal R&D funding: the b edrock of national innovation

    Mandt R, Seetharam K, Cheng CHM. Federal R&D funding: the b edrock of national innovation. MIT Science Policy Review. 2020;I. doi:10.38105/spr.n463z4t1u8

    2020 doi

  7. [15]

    Risin g above the gathering storm, revisited

    National Academies of Sciences, Engineering, and Medicine. Risin g above the gathering storm, revisited. The National Academies Press; 2010. Available from: https://nap.nationalacademies.org/catalog/12999/rising-above-the-gathering

    2010

  8. [16]

    The Balanced Scorecard — measures tha t drive performance; 1992

    Kaplan RS, Norton DP. The Balanced Scorecard — measures tha t drive performance; 1992. Harvard Business Review https://hbr.org/1992/01/the-balanced-scorecard-meas ures-that-drive-performa

    1992

  9. [17]

    Theorie der wirtschaftlichen Entwicklung (repr int of 1912 book)

    Schumpeter J. Theorie der wirtschaftlichen Entwicklung (repr int of 1912 book). Boston, MA: Springer; 2003

    1912

  10. [18]

    Inf ormation technology innovation: resurgence, confluence, and continuing im pact; 2020

    National Academies of Sciences, Engineering, and Medicine. Inf ormation technology innovation: resurgence, confluence, and continuing im pact; 2020. https://doi.org/10.17226/25961. May 27, 2025 23/26

    2020 doi

  11. [19]

    The economic benefits of publicly funded ba sic research: a critical review

    Salter AJ, Martin BR. The economic benefits of publicly funded ba sic research: a critical review. Research Policy. 2001;30(3):509–532. doi:10.1016/s0048-7333(00)00091-3

    2001 doi

  12. [20]

    The rate of return to investment in R&D: the case of research infrastructures

    Del Bo CF. The rate of return to investment in R&D: the case of research infrastructures. Technological Forecasting and Social Change. 2016;112:26–37. doi:10.1016/j.techfore.2016.02.018

    2016 doi

  13. [21]

    The evaluation of research infrastructure s: a cost-benefit analysis framework

    Florio M, Sirtori E. The evaluation of research infrastructure s: a cost-benefit analysis framework. Milan European Economic Workshop; 2014. Ava ilable from: http://dx.doi.org/10.2139/ssrn.2722500

    2014 doi

  14. [22]

    Investing in science: social cost-benefit analysis of re search infrastructures

    Florio M. Investing in science: social cost-benefit analysis of re search infrastructures. MIT Press; 2019

    2019

  15. [23]

    High performance computing instrumentation and research productiv ity in US universities; 2010

    Apon A, Ahalt S, Dantuluri V, Gurdgiev C, Limayem M, Ngo L, et al.. High performance computing instrumentation and research productiv ity in US universities; 2010. Journal of Information Technology Impact 10 : 87-98. Available from https://ssrn.com/abstract=1679248

    2010

  16. [24]

    Strohmaier E, Dongarra J, Simon H, Meuer M. Top 500. top500.o rg; 2024. Available from: https://top500.org/

    2024

  17. [25]

    XSEDE value added, cost avoidance, and return on investment

    Stewart CA, Roskies R, Knepper R, Moore RL, Whitt J, Cockerill TM. XSEDE value added, cost avoidance, and return on investment. In: Proc eedings of the 2015 XSEDE Conference on Scientific Advancements Enabled by Enh anced Cyberinfrastructure - XSEDE ’15. New York, New York, US...

    2015

  18. [26]

    XSEDE:Accelerating Scientific discovery

    Towns J, Cockerill T, Dahan M, Foster I, Gaither K, Grimshaw A, et al. XSEDE:Accelerating Scientific discovery. Computing in Science & Engin eering. 2014;16:62–74. doi:10.1109/MCSE.2014.80

    2014 doi

  19. [27]

    Total quality management; 2025

    American Society for Quality. Total quality management; 2025. https://asq.org/quality-resources/total-quality-man agement#:~:text=Overview,

    2025

  20. [28]

    The Baldrige quality system: the do-it-yourself way t o transform your business

    George S. The Baldrige quality system: the do-it-yourself way t o transform your business. New York, NY: Wiley; 1992

    1992

  21. [29]

    Measuring quality, co st, and value of IT services

    Peebles C, Stewart C, Voss B, Workman S. Measuring quality, co st, and value of IT services. In: Proceedings of the 55th Annual Quality Congre ss, American Society for Quality, Charlotte NC; 2001.Available from: https://hdl.handle.net/2022/426

    2001

  22. [30]

    Fair ranking of researchers and research team s

    Vavry¸ cuk V. Fair ranking of researchers and research team s. PLOS ONE. 2018;13(4):1–17. doi:10.1371/journal.pone.0195509

    2018 doi

  23. [31]

    Federal funding and the ra te and direction of inventive activity

    Corredoira RA, Goldfarb BD, Shi Y. Federal funding and the ra te and direction of inventive activity. Research Policy. 2018;47(9):1777–1800. doi:10.1016/j.respol.2018.06.009

    2018 doi

  24. [32]

    Comprehensive evaluation of XSEDE’s scientific impact using semantic scholar data

    von Laszewski G, Wang F, Fox GC. Comprehensive evaluation of XSEDE’s scientific impact using semantic scholar data. In: Practice and Expe rience in Advanced Research Computing. New York, NY, USA: ACM; 2021.Ava ilable from: http://dx.doi.org/10.1145/3437359.3465601. May 27, 2025 24/26

    2021

  25. [33]

    An input-output virtual laboratory in practice - su rvey of uptake, usage and applications of the first operational IELab

    Wiedmann T. An input-output virtual laboratory in practice - su rvey of uptake, usage and applications of the first operational IELab. Economic Sy stems Research. 2017;29:296–312. doi:10.1080/09535314.2017.1283295

    2017

  26. [34]

    RIMS II: An essential tool for regional developers an d planners; 2024

    USBEA. RIMS II: An essential tool for regional developers an d planners; 2024. Available from: https://www.bea.gov/sites/default/files/methodologies/RIMSII_User_Guide.pdf

    2024

  27. [35]

    IMPLAN; 2024

    IMPLAN Group. IMPLAN; 2024. Available from: https://implan.com/

    2024

  28. [36]

    Miller T, Stewart CA. Economic development by the Indiana Univer sity Pervasive Technology Institute, Pervasive Technology Labs, and the Research Technologies Division of University Information Technology Services during FY 2012/2013; 2014. Indiana University http://hdl.handle...

    2012

  29. [37]

    Cost accounting: foundations and evolu tions

    Kinney MR, Raiborn CA. Cost accounting: foundations and evolu tions. Boston, MA: South Western Cengage Learning; 2011

    2011

  30. [38]

    International IR Framework; 2021

    The International Integrated Reporting Council. International IR Framework; 2021. Available from: https://www.integratedreporting.org/resource/international-ir-framework/

    2021

  31. [39]

    A Theory of Production; 1928

    Cobb CW, Douglas PH. A Theory of Production; 1928. https://www.jstor.org/stable/1811556

    1928

  32. [41]

    Higher education research and d evelopment (HERD) survey 2023; 2023

    National Science Foundation. Higher education research and d evelopment (HERD) survey 2023; 2023. https://ncses.nsf.gov/surveys/higher-education-rese arch-development/2023

    2023

  33. [42]

    Nature Index; 2025

    Springer Nature. Nature Index; 2025. https://www.nature.com/nature-index/

    2025

  34. [43]

    Surve y of earned doctorates (SED) 2023

    National Center for Science and Engineering Statistics. Surve y of earned doctorates (SED) 2023. National Science Foundation; 2023. NSF 25-300. Available from: https://ncses.nsf.gov/surveys/doctorate-recipients/ 2023

    2023

  35. [44]

    nVidia Tensor Cores; 2020

    nVidia, Inc . nVidia Tensor Cores; 2020. Available from: https://www.nvidia.com/en-us/data-center/tensor-cor es/

    2020

  36. [45]

    Cloud an d on-premises data center usage, expenditures, and approaches to return on investment: a survey of academic research computing organizatio ns

    Chalker A, Hillegas CW, Sill A, Broude Geva S, Stewart CA. Cloud an d on-premises data center usage, expenditures, and approaches to return on investment: a survey of academic research computing organizatio ns. In: Practice and Experience in Advanced Research Computing 2020. P...

    2020

  37. [46]

    Results from a second longitudinal survey of academic research computing and data center usage: expenditures, utilization patterns, and approach es to return on investment

    Broude Geva S, Chalker A, Hillegas C, Petravick D, Sill A, Stewart C. Results from a second longitudinal survey of academic research computing and data center usage: expenditures, utilization patterns, and approach es to return on investment. In: Practice and Experience in Adv...

    2025

  38. [47]

    Cloud services selection: a syste matic review and future research directions

    Thakur N, Singh A, Sangal AL. Cloud services selection: a syste matic review and future research directions. Computer Science Review. 2022;4 6:100514. doi:https://doi.org/10.1016/j.cosrev.2022.100514

    2022

  39. [48]

    Home Page; 2025

    Coalition for Academic Scientific Computation. Home Page; 2025. Available from: https://casc.org/

    2025

  40. [49]

    Determinants of citation impact: A comparative analysis of the Global South versus th e Global North

    Confraria, H , and M Mira Godinho, and L Wang. Determinants of citation impact: A comparative analysis of the Global South versus th e Global North

  41. [50]

    Insufficient yet improving involvement of the global south in to p sustainability science publications

    Dangles O, Struelens Q, Ba MP, Bonzi-Coulibaly Y, Charvis P, Emma nuel E, et al. Insufficient yet improving involvement of the global south in to p sustainability science publications. PloS one. 2022;17(9):e0273083. doi:10.1371/journal.pone.0273083

    2022 doi

  42. [51]

    Cost effectiveness and return on investment analysis for surgica l care in a conflict-affected region of Sudan

    Nicholson CP, Saxton A, Young K, Smith ER, Shrime MG, Fielder J, e t al. Cost effectiveness and return on investment analysis for surgica l care in a conflict-affected region of Sudan. PLOS global public health. 2024;4(11):e0003712. doi:10.1371/journal.pgph.0003712

    2024 doi

  43. [52]

    Return on in vestments in the Health Extension Program in Ethiopia

    Bowser D, Kleinau E, Berchtold G, Kapaon D, Kasa L. Return on in vestments in the Health Extension Program in Ethiopia. PloS one. 2023;18(11):e0 291958. doi:10.1371/journal.pone.0291958

    2023 doi

  44. [53]

    Cyberinfrastructure, cloud computing, science gateways, visua lization, and cyberinfrastructure ease of use

    Stewart CA, Knepper R, Link MR, Pierce M, Wernert E, Wilkins-Die hr N. Cyberinfrastructure, cloud computing, science gateways, visua lization, and cyberinfrastructure ease of use. In: Advances in Computer and Electrical Engineering. IGI Global; 2018. p. 157–170. Available from...

    2018

  45. [54]

    A longitudinal study of XSEDE H PC workshops

    Miles T, Destefano L, Wernert J. A longitudinal study of XSEDE H PC workshops. In: Practice and Experience in Advanced Research Co mputing. New York, NY, USA: ACM; 2023.Available from: https://dl.acm.org/doi/10.1145/3569951.3593603

    2023

  46. [55]

    Economic value of HPC experience for new STEM professionals: insigh ts from STEM hiring managers

    Snapp-Childs W, Costa CM, Olds D, Snell A, Wernert JA, Stewart CA. Economic value of HPC experience for new STEM professionals: insigh ts from STEM hiring managers. Frontiers in Research Metrics and Analytics. 2024;9:1462329. doi:10.3389/frma.2024.1462329

    2024

  47. [56]

    Real Options applied to infra structure projects: a new approach to value and manage risk and flexibility

    van Rhee, Pieters M, van de Voort. Real Options applied to infra structure projects: a new approach to value and manage risk and flexibility. In : 2008 First International Conference on Infrastructure Systems an d Services. New York, NY: IEEE; 2008. p. 1–6. Available from: htt...

    2008

  48. [57]

    Metrics for Measuring Success of CyberInfrastru cture (CI) Projects

    Aurora R. Metrics for Measuring Success of CyberInfrastru cture (CI) Projects. In: SN Computer Science Topical Collection. SN Computer Science; 2024.Av ailable from: https://link.springer.com/journal/42979/topicalCollection/AC_e8e45e9fd5dab09 May 27, 2025 26/26

    2024

  49. [2010]

    p. 37–44. Available from: https://dl.acm.org/doi/10.1145/1878335.1878347

  50. [2016]

    UNU-MERIT Working Papers No

    UNU-MERIT. UNU-MERIT Working Papers No. 029. Available fr om: https://cris.maastrichtuniversity.nl/ws/portalfiles/portal/127469517/wp2016_

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.