Pith. sign in

REVIEW 4 major objections 5 minor 7 references

Understanding Computational Science and Engineering (CSE) and Domain Science Skills Development in National Laboratory Postgraduate Internships

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A retrospective survey of 24 past participants finds that federally funded postgraduate computational-science internships at a national laboratory improve self-rated computational skills and domain science familiarity, and strengthen…

desk verdict A transparent pilot survey of NREL internships, but the abstract overstates the CSE-skill findings because the analysis uses one-sample tests on self-reported improvement, not pre/post change. read the letter →

arxiv 2501.10601 v2 pith:NZCPQ3W4 submitted 2025-01-17 cs.CY

Morgan M. Fong , Hilary Egan , Marc Day , Kristin Potter , Michael J. Martin This is my paper · ORCID
classification cs.CY
keywords computationalscienceandengineeringhighperformancecomputingnationallaboratoryinternshipgraduateeducationskillsdevelopmentcareerinterestssurveyevaluationdomain
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that federally funded postgraduate internships in computational science and engineering (CSE) at a national laboratory give students training that many universities cannot provide: hands-on experience with high-performance computing (HPC), direct exposure to domain science, and research and professional development. It reports survey evidence from 24 past participants across five internship programs, showing statistically significant self-reported gains in computational skills such as scientific programming, data visualization, and HPC use, as well as increased familiarity with renewable energy and energy efficiency. The authors also find significantly greater interest in careers at NREL and other DOE national laboratories after the internships. They present this as the first evaluation of graduate internships at the intersection of CSE and domain science, and they argue the survey instrument can be adapted for evaluating internships in other disciplines and settings.

What carries the argument

The survey instrument is organized around five themes—computational skills, familiarity with sustainability and renewable energy topics, research skills, professional skills, and career interests—with participants rating their pre-internship level and their internship experience on matched Likert scales in the same questionnaire. One-sided one-sample or paired t-tests at α = 0.05 are used to determine whether reported improvement, familiarity increases, or career-interest shifts exceed chance. The design's central choice is retrospective self-assessment: participants judge their earlier state after the internship, and that same instrument is the sole measure of change.

What would settle it

Conduct a pre/post evaluation using the same survey administered before the internship starts and again immediately after, alongside objective measures such as a standardized programming/HPC task and logs of supercomputer usage. If objective gains are absent while retrospective gains remain, the paper's central conclusion would not survive.

Watch

Extended reading notes

Core claim

Using a retrospective survey of 24 past participants (a 41% response rate across five federally funded programs), the paper reports that interns entered with at least some experience in most computational skills yet still improved significantly in general-purpose programming, scientific programming, data visualization and analysis, using HPC or supercomputers, and using software libraries or open-source code. Familiarity with renewable energy and with energy efficiency also increased significantly, while interest in careers at NREL and at other DOE national laboratories rose significantly. Participants reported that the skills improved during the internship were at least somewhat useful in their subsequent degree programs and positions, and they used those skills at least some of the time. The authors conclude that national-laboratory internships are an effective route for building CSE and domain-science skills that are unevenly available in academic institutions.

Load-bearing premise

The central claim depends on participants accurately remembering their skill levels before the internship when they rate them after it, so the measured gains could reflect memory distortion or a desire to report improvement.

Editorial extensions

If this is right

  • If the claims hold, federally funded internships can supply the CSE and HPC training that universities with limited computational resources cannot offer, narrowing an equity gap in advanced computing education.
  • The results imply that internships structured around educational objectives—rather than project-funded hires matched to existing skills—are more likely to produce broadly useful computational and research skills.
  • The reusable survey instrument could let other national laboratories and research sites measure internship outcomes in a standardized way, enabling comparisons across disciplines and programs.
  • The observed shift in career interest toward national laboratories suggests such internships may serve as a recruitment pipeline for the CSE and HPC workforce, potentially improving workforce diversity if the programs reach underrepresented groups.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The retrospective design means part of the measured gain could be response shift: after learning new skills, participants may judge their earlier skill level as lower than they would have at the time, inflating apparent improvement.
  • The significant gains cluster in skills NREL explicitly trains (e.g., HPC use, software libraries), which suggests structured, hands-on training matters more than mere exposure to a research environment; comparing sites with and without formal HPC training would test this.
  • Given the small, NREL-specific sample, the survey would need validation with pre/post administration and control groups before its findings could be generalized to other national laboratories or to non-CSE domains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a survey-based evaluation of federally funded postgraduate internships in computational science and engineering (CSE) at the National Renewable Energy Laboratory. The authors collected retrospective self-reports from 24 past participants across five internship programs, asking about pre-internship experience, in-internship improvement or experience, and changes in career interest. The paper concludes that participants improve CSE skills and domain science knowledge and become more interested in national-laboratory careers, and it recommends the survey instrument for broader use. The central evidence for skill improvement is one-sample t-tests on retrospective 'improvement' ratings, not paired pre/post comparisons.

Significance. If the causal claims were supported, the study would provide useful evidence for evaluating federal internship programs and for arguments about the value of national-laboratory training in CSE. The study addresses an important, understudied intersection of CSE education and national-laboratory internships, and the detailed survey instrument could be adapted by other programs. The authors also report response rates and acknowledge self-selection and recall limitations. However, the load-bearing inference rests on retrospective self-reports, and the statistical reporting contains an internal inconsistency; the results as presented support only the weaker claim that participants perceive improvement, not that their skills measurably improved.

major comments (4)
  1. [Abstract; Section III-E; Section IV-A/B/C; Table IV] The central claim that participants 'improve CSE skills and domain science knowledge' is not tested as a pre/post change. For computational skills, familiarity with domain science topics, and research skills, the analysis uses one-sample t-tests on post-internship ratings of 'improvement' or 'during internship' experience against a threshold of >1, as stated in Section III-E. The 'Pre-Internship' and 'Post-Internship' columns in Table IV are therefore not compared for these items. The Abstract and Section V overstate the evidence: the data show that participants report perceived improvement, not measured skill gain. Please revise the wording throughout to say 'participants reported improvement' or, if pre- and post-ratings are available, re-analyze as paired comparisons.
  2. [Section IV-D; Table IV (Professional 1)] The reported paired t-test for Professional 1 is internally inconsistent. The post-internship mean (3.67) is larger than the pre-internship mean (3.33), but the reported t-statistic is -1.88 with p=0.036 for a one-sided test of increase. With a negative t-statistic, a one-sided p of 0.036 would indicate a significant decrease, not an increase. This suggests an error in the sign of the t-statistic, the p-value, or the test direction. Please verify all paired t-test statistics in Table IV and the corresponding text, and report corrected values.
  3. [Section IV-A/B/C; Table IV] The columns labeled 'Pre-Internship' and 'Post-Internship' in Table IV are misleading for the one-sample tests. For computational skills and familiarity, the 'Post-Internship' column contains 'improvement' or 'increase' ratings, not post-internship skill levels; for research skills, it contains 'during internship' experience counts. These entries are not commensurable with the 'Pre-Internship' baseline, and no paired comparison is performed. The table should be relabeled to distinguish baseline experience from improvement ratings, or the analysis should be converted to paired comparisons if the data permit.
  4. [Section III-F; Section V] The retrospective self-report design is acknowledged in the limitations, but its implications for the main conclusions are not fully reflected. Because all data are collected after the internship in a single instrument, response shift, social desirability, and the motivation to justify the internship experience can all produce positive 'improvement' ratings even if actual skill levels are unchanged. The Abstract's language 'participants improve CSE skills and domain science knowledge' is too strong; the conclusions should be framed as self-reported perceived gains, with the design limitation stated prominently.
minor comments (5)
  1. [Table IV] There are typographical errors in Table IV: '1.93(1.03' and '1.80(1.32' are missing closing parentheses.
  2. [References] The reference list contains two entries labeled 'Cote et al., 2025'; the in-text citations should be disambiguated (e.g., Cote et al., 2025a/b).
  3. [Section V] The sentence beginning 'This approach may be particularly valuable in CSE, where there is a large disparity in institutional resources and ability to offer cross-training between' is incomplete and appears to be cut off before 'To the best of our knowledge'.
  4. [Section III-E] The description of the statistical tests could be clearer: the text says 'one-sided, paired samples t-tests' for professional skills and career interests, but then the results for these items are presented in the same table as the one-sample tests without a clear visual or textual distinction.
  5. [Figure 2 caption] The caption says 'improvement of familiarity' while the survey wording and Section IV-B use 'increase in familiarity'; please harmonize the terminology.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the survey conclusions rest on self-reported ratings, not on a derivation that reduces to its own inputs.

full rationale

This is an empirical survey study with no mathematical derivation chain in which an output is equivalent to an input by construction. The headline findings are direct summaries of self-reported Likert-scale ratings. The one-sample t-tests on 'improvement' ratings are a measurement-validity concern (the design does not test pre/post change), not a circularity concern: the self-reports are empirical data, not assumptions that make the conclusion true by definition. The authors' involvement in the internship programs is a potential bias, but it is not circular reasoning. No fitted parameter is renamed as a prediction; no uniqueness theorem is invoked; no ansatz is smuggled in via citation; and no known result is merely renamed. The stated limitation that responses from less recent participants 'may be less accurate' is properly acknowledged and does not create a circular step.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the validity of retrospective self-report and the representativeness of the small sample. No free parameters are fitted; the only hand-chosen quantity is the 'some improvement' threshold of 1 used in one-sample t-tests. No new entities are postulated.

free parameters (1)
  • Improvement threshold = 1
    The analysis defines improvement as a post-internship rating > 1 ('Some improvement'), a hand-chosen threshold. The significance of several results (e.g., energy efficiency familiarity, p=0.035) depends on this boundary.
assumptions (3)
  • domain assumption Retrospective self-reports of skill improvement accurately reflect actual skill development
    The entire analysis relies on participants' memory and judgment of their own skill levels before and during the internship, with no external validation. See Section III-F.
  • domain assumption The 24 respondents are representative of all eligible internship alumni
    With a 41% response rate and self-selection, the sample may not represent the full population. The paper claims representative response rates across programs but does not test for non-response bias. See Table II and Section III-F.
  • standard math Normality of rating distributions for t-tests
    One-sample and paired t-tests assume approximately normal data. With n=24 and Likert-scale responses, this assumption is questionable, though the central limit theorem provides some cover.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Computational Science and Engineering (CSE) and Domain Science Skills Development in National Laboratory Postgraduate Internships." pith.science (2026). https://pith.science/paper/NZCPQ3W4

@misc{pith2026250110601,
  author       = {Pith},
  title        = {Pith review of: Understanding Computational Science and Engineering (CSE) and Domain Science Skills Development in National Laboratory Postgraduate Internships},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NZCPQ3W4}},
  note         = {Machine review of arXiv:2501.10601}
}
read the original abstract

Background: Harnessing advanced computing for scientific discovery and technological innovation demands scientists and engineers well-versed in both domain science and computational science and engineering (CSE). However, few universities provide access to both integrated domain science/CSE cross-training and Top-500 High-Performance Computing (HPC) facilities. National laboratories offer internship opportunities capable of developing these skills. Purpose: This student presents an evaluation of federally-funded postgraduate internship outcomes at a national laboratory. This study seeks to answer three questions: 1) What computational skills, research skills, and professional skills do students improve through internships at the selected national laboratory. 2) Do students gain knowledge in domain science topics through their internships. 3) Do students' career interests change after these internships? Design/Method: We developed a survey and collected responses from past participants of five federally-funded internship programs and compare participant ratings of their prior experience to their internship experience. Findings: Our results indicate that participants improve CSE skills and domain science knowledge, and are more interested in working at national labs. Participants go on to degree programs and positions in relevant domain science topics after their internships. Conclusions: We show that national laboratory internships are an opportunity for students to build CSE skills that may not be available at all institutions. We also show a growth in domain science skills during their internships through direct exposure to research topics. The survey instrument and approach used may be adapted to other studies to measure the impact of postgraduate internships in multiple disciplines and internship settings.

Figures

Figures reproduced from arXiv: 2501.10601 by the authors.

Figure 1
Figure 1. Participants’ self-reported prior experience (top) and improvement of computational skills (bottom) [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Participants’ self-reported prior familiarity (top) and improvement of familiarity (bottom) with sustainability and renewable energy topics [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Participants’ self-reported research experience before (left) and during (right) their internship. The open-ended responses echoed similar experiences as our questions. For example, one participant noted “Remote collaboration” as a skill that improved, and another participant mentioned working in teams with varying levels of expertise as an important skill for their post-internship position. D. Professional Skills O… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Participants’ self-reported agreement with being part of a research community and with ability to communicate ideas before (left) and during (right) their internship [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Participants’ self-reported interest in career domain before (left) and after (right) their internship [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Participants’ self-reported interest in career sector before (left) and after (right) their internship. E. Career Interests When asked about their current position at the time of completing the survey, 14 participants reported they were in the same or different degree …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 7 canonical work pages

  1. [1]

    Somewhat related

    Similarly, one-sided one sample t-tests were used to determine if participants current positions are at least “Somewhat related” to NREL domain science topics, if their skills are at least “Somewhat useful” in applications, and if their skills are used at least “Some of the time.” α < 0.05 served as the cutoff for statistical significance. F . Limitations...

    work page 2018

  2. [2]

    Full questions for each theme are provided in Table I

    pathways to future careers, survey questions were organized and designed around the following themes: Computational Skills, Familiarity with NREL-relevant domain science topics such as energy generation and efficiency, Research Skills, Professional Skills, and Career Interests. Full questions for each theme are provided in Table I. For each theme, the res...

    work page 2018

  3. [3]

    Findings: Our results indicate that participants improve CSE skills and domain science knowledge, and are more interested in working at national labs

    Do students’ career interests change after these internships? Design/Method: We developed a survey and collected responses from past participants of five federally-funded internship programs and compare participant ratings of their prior experience to their internship experience. Findings: Our results indicate that participants improve CSE skills and doma...

    work page 2018

  4. [4]

    Research and Professional Skills 7 B. Participant Background Information Participants were also asked to self-report the following academic background information: • Prior experience at other Department of Energy National Labs (i.e., no experience, once, 2-3 times, or 4 or more times) • Prior experience at other federal laboratories (i.e., no experience, ...

    work page 2018

  5. [6]

    Somewhat useful

    Participants’ self-reported interest in career sector before (left) and after (right) their internship. E. Career Interests When asked about their current position at the time of completing the survey, 14 participants reported they were in the same or different degree program, and 10 participants reported they were looking for employment or currently empl...

    work page 2024

  6. [152]

    When I talk about it, my eyes light up!

    [Bautista and Sukhija, 2024] Bautista, E. and Sukhija, N. (2024). Data Analytics Program in Community Colleges in Preparation for STEM and HPC Careers. The Journal of Computational Science Education, 15(1):59–63. 15 [Connor et al., 2016] Connor, C., Bonnie, A., Grider, G., and Jacobson, A. (2016). Next Generation HPC Workforce Development: The Computer Sy...

    work page 2024

  7. [2021]

    Neither EERE HPC4EI nor NSF-MSGI operated in summer of 2024 due to budget uncertainties. III. METHODS A. Survey Design The research team collaboratively designed the survey. Given the purpose of the survey to understand interns’

    work page 2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.