{"id":"5f785f5a-eeec-43d1-afbb-c34f3b8011e2","arxiv_id":"2501.10601","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A retrospective survey of 24 NREL internship alumni reports self-perceived improvement in computational skills and increased interest in national lab careers, but lacks a control group.","lead":"This study surveys 24 alumni of five federally funded internships at NREL, asking them to rate their computational, research, and professional skills before and during their internship. The authors report self-perceived gains in computational skills and renewable energy familiarity, and increased interest in national laboratory careers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline CSE-skill claim is not actually tested as a pre/post change: the analysis uses one-sample tests on self-reported 'improvement' ratings, so the abstract overstates what the data can show.","rationale":"The reader's weakest-assumption identification points to retrospective self-report, which is correct, but the more precise and load-bearing problem is that the reported statistical tests for computational skills and domain-science familiarity do not even compare pre- and post-internship measurements. Section III-E describes one-sample tests on the 'improvement' response item itself, so the abstract's claim of improvement rests entirely on participants' post-hoc perception of gain. This is a stronger reason to be cautious than the reader's general response-shift concern, because it means the central claim is not directly supported by any change score. However, the paper is transparent about its design, uses hedged language in the conclusion ('appear to have improved'), and is framed as an evaluation rather than a causal experiment. The additional sign inconsistency in Table IV for Professional 1 reinforces caution but does not by itself overturn the main CSE finding. A conditional acceptance requiring the authors to reframe the claim as 'self-reported improvement' and to note that the CSE tests are one-sample tests on perceived gain is appropriate. Thus the reader's CONDITIONAL verdict stands, and I recommend no change to the verdict.","tokens_in":13378,"tokens_out":5458,"duration_ms":57426,"concrete_test":"Run a prospective validation substudy: administer the same skill items as a true pre-test at internship start and post-test at internship end (plus a retrospective pre-test at the end) to the next cohort, and wherever possible compare against objective indicators such as code/portfolio evaluations or mentor ratings. If the prospective gain scores correlate poorly with the one-sample 'improvement' ratings, or if the retrospective pre-test differs systematically from the prospective pre-test, the paper's central claims about CSE skill and domain-science gains are not supported by the current instrument.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is more specific than generic retrospective self-report: for the headline CSE and domain-science findings, the paper does not actually test a change from baseline. Section III-E states that computational skills, research skills, and familiarity were analyzed with one-sample t-tests of whether the post-internship 'improvement'/'increase' rating exceeds 1, not paired pre/post tests. Table IV's columns labeled 'Pre-Internship' and 'Post-Internship' for these items are therefore not compared; the 'post' column is a self-report of perceived improvement. Consequently, the abstract's 'participants improve CSE skills and domain science knowledge' is a statement about what participants think they gained, not about measured change. This is especially problematic because both baseline experience and improvement are collected in the same instrument after the internship, so response shift, social desirability, and a general tendency to justify the internship experience can all produce positive 'improvement' ratings even if actual skill levels are unchanged. Additionally, the t-statistic signs for Professional 1 (Section IV-D/Table IV: t=-1.88, p=0.036 with post mean larger than pre) are inconsistent with the stated one-sided paired test for increase, which suggests the reported p-values may not be reliable. The central claim would land only if self-reported improvement tracks actual skill development; nothing in the current design establishes that.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a survey-based evaluation of federally funded postgraduate internships in computational science and engineering (CSE) at the National Renewable Energy Laboratory. The authors collected retrospective self-reports from 24 past participants across five internship programs, asking about pre-internship experience, in-internship improvement or experience, and changes in career interest. The paper concludes that participants improve CSE skills and domain science knowledge and become more interested in national-laboratory careers, and it recommends the survey instrument for broader use. The central evidence for skill improvement is one-sample t-tests on retrospective 'improvement' ratings, not paired pre/post comparisons.","tokens_in":13625,"tokens_out":4347,"duration_ms":40976,"significance":"If the causal claims were supported, the study would provide useful evidence for evaluating federal internship programs and for arguments about the value of national-laboratory training in CSE. The study addresses an important, understudied intersection of CSE education and national-laboratory internships, and the detailed survey instrument could be adapted by other programs. The authors also report response rates and acknowledge self-selection and recall limitations. However, the load-bearing inference rests on retrospective self-reports, and the statistical reporting contains an internal inconsistency; the results as presented support only the weaker claim that participants perceive improvement, not that their skills measurably improved.","major_comments":[{"comment":"The central claim that participants 'improve CSE skills and domain science knowledge' is not tested as a pre/post change. For computational skills, familiarity with domain science topics, and research skills, the analysis uses one-sample t-tests on post-internship ratings of 'improvement' or 'during internship' experience against a threshold of >1, as stated in Section III-E. The 'Pre-Internship' and 'Post-Internship' columns in Table IV are therefore not compared for these items. The Abstract and Section V overstate the evidence: the data show that participants report perceived improvement, not measured skill gain. Please revise the wording throughout to say 'participants reported improvement' or, if pre- and post-ratings are available, re-analyze as paired comparisons.","section":"Abstract; Section III-E; Section IV-A/B/C; Table IV"},{"comment":"The reported paired t-test for Professional 1 is internally inconsistent. The post-internship mean (3.67) is larger than the pre-internship mean (3.33), but the reported t-statistic is -1.88 with p=0.036 for a one-sided test of increase. With a negative t-statistic, a one-sided p of 0.036 would indicate a significant decrease, not an increase. This suggests an error in the sign of the t-statistic, the p-value, or the test direction. Please verify all paired t-test statistics in Table IV and the corresponding text, and report corrected values.","section":"Section IV-D; Table IV (Professional 1)"},{"comment":"The columns labeled 'Pre-Internship' and 'Post-Internship' in Table IV are misleading for the one-sample tests. For computational skills and familiarity, the 'Post-Internship' column contains 'improvement' or 'increase' ratings, not post-internship skill levels; for research skills, it contains 'during internship' experience counts. These entries are not commensurable with the 'Pre-Internship' baseline, and no paired comparison is performed. The table should be relabeled to distinguish baseline experience from improvement ratings, or the analysis should be converted to paired comparisons if the data permit.","section":"Section IV-A/B/C; Table IV"},{"comment":"The retrospective self-report design is acknowledged in the limitations, but its implications for the main conclusions are not fully reflected. Because all data are collected after the internship in a single instrument, response shift, social desirability, and the motivation to justify the internship experience can all produce positive 'improvement' ratings even if actual skill levels are unchanged. The Abstract's language 'participants improve CSE skills and domain science knowledge' is too strong; the conclusions should be framed as self-reported perceived gains, with the design limitation stated prominently.","section":"Section III-F; Section V"}],"minor_comments":[{"comment":"There are typographical errors in Table IV: '1.93(1.03' and '1.80(1.32' are missing closing parentheses.","section":"Table IV"},{"comment":"The reference list contains two entries labeled 'Cote et al., 2025'; the in-text citations should be disambiguated (e.g., Cote et al., 2025a/b).","section":"References"},{"comment":"The sentence beginning 'This approach may be particularly valuable in CSE, where there is a large disparity in institutional resources and ability to offer cross-training between' is incomplete and appears to be cut off before 'To the best of our knowledge'.","section":"Section V"},{"comment":"The description of the statistical tests could be clearer: the text says 'one-sided, paired samples t-tests' for professional skills and career interests, but then the results for these items are presented in the same table as the one-sample tests without a clear visual or textual distinction.","section":"Section III-E"},{"comment":"The caption says 'improvement of familiarity' while the survey wording and Section IV-B use 'increase in familiarity'; please harmonize the terminology.","section":"Figure 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a valuable and understudied topic, and the survey instrument could be a useful contribution. However, the mismatch between the abstract's causal claims and the actual one-sample analysis, together with the internal inconsistency in the Professional 1 t-test, needs careful correction. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection. I would also suggest that the editor ask the authors to have the statistical reporting checked by a quantitative methodologist before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a small retrospective survey of postgraduate interns at NREL. The headline claim—that internships improve CSE skills—is weaker than the abstract suggests. For computational skills and domain familiarity, the analysis does not compare before/after ratings; it runs one-sample t-tests on self-reported improvement ratings to see if they exceed 'some improvement.' That establishes 'participants believe they gained skills,' not 'participants gained skills.' The methods section is explicit about this, so the authors aren't hiding it, but the abstract and conclusion push a causal reading.\n\nWhat is new: this appears to be the first evaluation specifically targeting CSE and domain-science skill development in national laboratory postgraduate internships. The survey instrument is adaptable, the HPC-access context is useful, and the authors clearly list limitations. Given how little published evaluation exists for these programs, that is a real contribution.\n\nThe soft spots are proportionate. The one-sample design is the main issue and is fixable by reframing the claims. Table IV also has an inconsistency: Professional 1 shows t=-1.88, p=0.036 with post mean higher than pre, which doesn't match the stated one-sided paired test for increase. That needs correction. Multiple testing isn't addressed, and the NNSA-MSIIP program had zero respondents, so cross-program generalization is risky. These are common issues in program evaluation, not disqualifying ones. The career-interest results use paired pre/post tests, so those are on firmer ground.\n\nWho should read it: people studying workforce development, STEM education, or national lab training. It deserves peer review with revisions; I'd ask the authors to fix the statistics and moderate the causal language before publication.","headline":"A transparent pilot survey of NREL internships, but the abstract overstates the CSE-skill findings because the analysis uses one-sample tests on self-reported improvement, not pre/post change.","tokens_in":14133,"tokens_out":3595,"would_cite":false,"duration_ms":35739,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A retrospective survey of 24 past participants finds that federally funded postgraduate computational-science internships at a national laboratory improve self-rated computational skills and domain science familiarity, and strengthen…","keywords":["computational science and engineering","high performance computing","national laboratory internship","graduate education","skills development","career interests","survey evaluation","domain science"],"falsifier":"Conduct a pre/post evaluation using the same survey administered before the internship starts and again immediately after, alongside objective measures such as a standardized programming/HPC task and logs of supercomputer usage. If objective gains are absent while retrospective gains remain, the paper's central conclusion would not survive.","tokens_in":13194,"feed_emoji":"💻","tokens_out":3531,"duration_ms":36827,"temperature":0.7,"pith_summary":"The paper argues that federally funded postgraduate internships in computational science and engineering (CSE) at a national laboratory give students training that many universities cannot provide: hands-on experience with high-performance computing (HPC), direct exposure to domain science, and research and professional development. It reports survey evidence from 24 past participants across five internship programs, showing statistically significant self-reported gains in computational skills such as scientific programming, data visualization, and HPC use, as well as increased familiarity with renewable energy and energy efficiency. The authors also find significantly greater interest in careers at NREL and other DOE national laboratories after the internships. They present this as the first evaluation of graduate internships at the intersection of CSE and domain science, and they argue the survey instrument can be adapted for evaluating internships in other disciplines and settings.","feed_headline":"National-lab internships boost computational skills, survey finds","feed_subtitle":"Past participants report stronger CSE skills, domain knowledge, and lab-career interest after federally funded programs.","key_machinery":"The survey instrument is organized around five themes—computational skills, familiarity with sustainability and renewable energy topics, research skills, professional skills, and career interests—with participants rating their pre-internship level and their internship experience on matched Likert scales in the same questionnaire. One-sided one-sample or paired t-tests at α = 0.05 are used to determine whether reported improvement, familiarity increases, or career-interest shifts exceed chance. The design's central choice is retrospective self-assessment: participants judge their earlier state after the internship, and that same instrument is the sole measure of change.","core_discovery":"Using a retrospective survey of 24 past participants (a 41% response rate across five federally funded programs), the paper reports that interns entered with at least some experience in most computational skills yet still improved significantly in general-purpose programming, scientific programming, data visualization and analysis, using HPC or supercomputers, and using software libraries or open-source code. Familiarity with renewable energy and with energy efficiency also increased significantly, while interest in careers at NREL and at other DOE national laboratories rose significantly. Participants reported that the skills improved during the internship were at least somewhat useful in their subsequent degree programs and positions, and they used those skills at least some of the time. The authors conclude that national-laboratory internships are an effective route for building CSE and domain-science skills that are unevenly available in academic institutions.","pith_inferences":["The retrospective design means part of the measured gain could be response shift: after learning new skills, participants may judge their earlier skill level as lower than they would have at the time, inflating apparent improvement.","The significant gains cluster in skills NREL explicitly trains (e.g., HPC use, software libraries), which suggests structured, hands-on training matters more than mere exposure to a research environment; comparing sites with and without formal HPC training would test this.","Given the small, NREL-specific sample, the survey would need validation with pre/post administration and control groups before its findings could be generalized to other national laboratories or to non-CSE domains."],"forward_implications":["If the claims hold, federally funded internships can supply the CSE and HPC training that universities with limited computational resources cannot offer, narrowing an equity gap in advanced computing education.","The results imply that internships structured around educational objectives—rather than project-funded hires matched to existing skills—are more likely to produce broadly useful computational and research skills.","The reusable survey instrument could let other national laboratories and research sites measure internship outcomes in a standardized way, enabling comparisons across disciplines and programs.","The observed shift in career interest toward national laboratories suggests such internships may serve as a recruitment pipeline for the CSE and HPC workforce, potentially improving workforce diversity if the programs reach underrepresented groups."],"supporting_citations":[{"why":"Supplies the definition of computational science and engineering and the CSE education context that motivates the study, and inspired the computational skills questions in the survey.","marker":"[Rude et al., 2018]"},{"why":"Provides prior evaluation evidence that national laboratory internships increase research skills and STEM retention, which this study extends to CSE-specific skills.","marker":"[Foltz et al., 2011]"},{"why":"Documents impacts of national laboratory internships on community college student success and career trajectories, a key comparison and motivation for studying postgraduate internships.","marker":"[Cote et al., 2025]"},{"why":"Supplies evidence that computer-science internships develop technical skills, the closest prior work to the CSE skill-development claim made here.","marker":"[Kang and Girouard, 2022]"},{"why":"Provides the call for graduate students to do internships as supplemental training, which frames the paper's relevance to graduate education.","marker":"[Murguía Burton and Cao, 2022]"},{"why":"Documents the concentration of top-500 supercomputers at government facilities and their scarcity at universities, the structural rationale for learning HPC through national lab internships.","marker":"[TOP500.org, 2024]"},{"why":"Argues for cross-training scientists in both CSE and domain science, which the paper positions as the educational gap its internship outcomes address.","marker":"[McInnes et al., 2023]"}],"fun_headline_variants":["National lab internships boost computing and science skills","Survey: Lab internships improve CSE and domain expertise","Interns report sharper computational skills after lab stints","Postgrad internships at national labs build CSE skills","Lab internships: gains in computing and career interest"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on participants accurately remembering their skill levels before the internship when they rate them after it, so the measured gains could reflect memory distortion or a desire to report improvement.","fun_headline_variants_meta":{"raw":{"variants":["National lab internships boost computing and science skills","Survey: Lab internships improve CSE and domain expertise","Interns report sharper computational skills after lab stints","Postgrad internships at national labs build CSE skills","Lab internships: gains in computing and career interest"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1516,"prompt_tokens":980,"completion_tokens":536,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":463}},"tokens_in":596,"tokens_out":536,"duration_ms":5911,"temperature":1.0,"reasoning_tokens":463,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:01:26.271655+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Conduct a pre/post evaluation using the same survey administered before the internship starts and again immediately after, alongside objective measures such as a standardized programming/HPC task and logs of supercomputer usage. If objective gains are absent while retrospective gains remain, the paper's central conclusion would not survive.","supporting_citations":[],"review_version":1}