{"id":"4e25c7a2-b2c2-4cea-9a24-d21472707683","arxiv_id":"2411.09240","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A global survey of 101 top-ranked universities finds that most named cybersecurity programs under-cover non-technical knowledge areas and hands-on learning compared with CSEC2017 and CC2020 guidelines.","lead":"The authors analyzed descriptions of 133 study programs at 101 top-ranked universities across 24 countries and found that only 45 have \"cyber\" in the name, with most of those missing non-technical topics such as law, policy, and risk management, and only 11% requiring internships. The paper argues this gap may leave graduates under-prepared and provides a checklist for program directors to improve their curricula.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Coverage coding in Section 4.3 is under-specified and Section 6 overstates 'only five cover all eight KAs'; the headline counts may shift under a reasonable re-coding.","rationale":"The Reader's weakest_assumption correctly identifies that public course descriptions may not reflect real curricula, and that the counting rule could bias the results. My concern is more specific: the coding procedure is not auditable because the essential-concept list and rubric are omitted, and the conclusion in Section 6 ('only five cover all eight knowledge areas') exceeds what was measured, since the five technical KAs were never coded. This strengthens the case for a CONDITIONAL verdict rather than ACCEPT, because the central percentages could shift under a transparent re-coding. However, the underlying direction of the finding is plausible and the authors provide a dataset and acknowledge the outdated-website limitation, so I would not reject the paper. The proposed re-coding test would settle whether the 'only five' figure is robust; until then, the conditional verdict remains appropriate. My assessment therefore does not change the Reader's verdict, but it sharpens the reason for caution and identifies a concrete validation step the authors should perform or enable.","tokens_in":882,"tokens_out":754,"duration_ms":79708,"concrete_test":"Download the companion dataset (reference [40]) and have two independent raters, blind to the paper's conclusions, re-code all 45 cyber-named programs with a pre-registered rubric that lists the CSEC2017 essential concepts for human, organizational, and societal security; counts a knowledge area as covered if any essential concept appears in any required course description; and records whether coverage is a full course or a one-off topic. Report Cohen's kappa for the two raters, and recompute the number of programs covering all three non-technical KAs under the paper's exclusion rule and under the inclusive rule. Also independently verify whether the five programs highlighted in Section 4.3 actually cover the five technical KAs before endorsing the 'all eight' statement. If the inclusive count exceeds about 10 or kappa is below 0.6, the headline percentages are not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's main quantitative anchor is the claim that only five of the 45 cyber-named programs cover the three non-technical CSEC2017 knowledge areas (human, organizational, societal security). This rests on an under-specified coding procedure: Section 4.3 says the authors 'searched for essential concepts' and did not count concepts appearing as 'only one or two topics out of ten or more in a single, typically introductory course,' but the list of essential concepts, the coding rubric, and inter-rater reliability are not reported. Section 6 then converts this into the stronger statement that 'only five cover all eight knowledge areas defined by CSEC2017,' without measuring the five technical KAs at all. That inference is not supported: a program could have at least one topic in each non-technical KA and still miss technical essentials, and conversely programs whose non-technical content is embedded as one or two topics in a large course are excluded. Since the central conclusion is numerically anchored to these counts, a small change in counting rule could move the 'only five' figure and weaken the headline percentages. The released dataset provides links but not auditable coding decisions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper surveys 133 study programs at 101 universities selected from five international rankings and focuses on the 45 programs whose names contain \"cyber.\" Based on manual inspection of public program and course descriptions, the authors code whether programs cover the human, organizational, and societal security knowledge areas of CSEC2017 and whether they require internships or final projects/theses. They report that only five cyber-named programs cover the three non-technical knowledge areas, only five mandate internships, and only four require both an internship and a final project/thesis. The paper concludes that top-ranked universities have not widely adopted existing curricular guidelines and offers a ten-item checklist for program directors. A dataset and Python notebook are released as supplementary material.","tokens_in":12204,"tokens_out":3673,"duration_ms":41688,"significance":"If the descriptive claims hold, the paper provides a useful global snapshot of cybersecurity program design that complements prior US-centric work, and the released dataset is a reproducible community artifact. The study is anchored to external standards (CSEC2017 and CC2020) rather than to parameters fitted by the authors, so circularity is not a concern. The authors are also transparent about sampling limitations and about using publicly available descriptions. However, the paper's headline quantitative claim that only five programs cover all eight CSEC2017 knowledge areas is not supported by the measurements actually reported, which cover only three non-technical knowledge areas. The coding procedure is also under-specified, with no inter-rater reliability and no auditable concept-level coding decisions. These issues are fixable within the paper's scope, so the central descriptive message remains plausible after revision.","major_comments":[{"comment":"The statement that 'only five cover all eight knowledge areas defined by CSEC2017' is not supported by the methods described in Sections 3.2 and 4.3. The authors report coding only for the human, organizational, and societal security knowledge areas; they do not report any measurement of the five technical knowledge areas (for example, data security, software security, component security, connection security, and system security). A program could therefore be counted as 'not covering all eight' even if it covers the technical areas well, and no program is positively verified as covering all eight. Please reword the headline to 'only five cover the three non-technical knowledge areas' or collect and report evidence for all eight knowledge areas.","section":"Section 6 (Conclusion)"},{"comment":"The coding procedure is under-specified and not independently auditable. The authors do not report the list of CSEC2017 essential concepts used for each knowledge area, the operational definition of 'one or two topics out of ten or more in a single, typically introductory course,' or inter-rater reliability. The released dataset contains links to programs and a processing notebook, but not per-program coding decisions. Since the central percentages (for example, 5/45 and 12/45) are counts produced by this coding, a reasonable re-coding could shift the headline numbers. Please publish a concept-level coding sheet with evidence per program and report agreement statistics from at least two coders on a sample of programs.","section":"Section 4.3"},{"comment":"The treatment of the 'N/A' category in Figure 3 is ambiguous and affects the reported counts. The figure distinguishes 'N/A' from 'Yes' and 'No,' but Section 4.4.1 states that mandatory internships are present in only five programs (11%), and Section 6 states that six programs require neither an internship nor a final project/thesis. If 'N/A' means the program descriptions do not state the requirement, then absence of stated information is not evidence of absence from the curriculum. Please state explicitly how 'N/A' observations were mapped into the yes/no totals, and report ranges or sensitivity analyses where the classification is uncertain.","section":"Section 4.4 and Figure 3"},{"comment":"The claim that graduates 'may not meet employer expectations and may require additional training' goes beyond the data collected. The study measures presence or absence of certain course topics and experiential-learning components in public descriptions; it does not measure employer satisfaction, graduate outcomes, or the actual relationship between those curriculum features and job readiness. Please reframe this as an inference or hypothesis motivated by prior literature rather than a finding of the present survey, or add direct evidence linking the coded features to employment outcomes.","section":"Section 1.3 and Abstract"}],"minor_comments":[{"comment":"The phrase 'the least represented (6×)' is informal; consider writing 'six programs' or 'six occurrences' for consistency with the other counts.","section":"Section 4.3"},{"comment":"The bar labels in Figure 3 are dense and the exact per-category counts are not repeated in the prose, which makes the figure hard to verify; please restate the counts for each group in the text or in a table.","section":"Figure 3"},{"comment":"The statement that 'only 6 out of 17 cyber programs offered in the USA have been validated by NSA' would benefit from a brief explanation of what 'validated' means in this context, since the cited CAE designation is a program-level designation rather than a simple binary check.","section":"Section 4.2"},{"comment":"Reference [31] is cited as 'Cybersecurity Curricular Guideline' in the reference list but as 'CSEC2017' throughout the text; please standardize the title and ensure the URL is the version used for coding.","section":"References"},{"comment":"There are minor spelling and formatting inconsistencies, such as 'post-graduate' versus 'postgraduate,' 'cyber crime' versus 'cybercrime,' and the phrase 'InProceedings' in reference [25]; these should be cleaned up in the camera-ready version.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computing-education venue and the dataset is a genuinely useful contribution. The main risk is that the conclusion currently overstates what was measured: the 'all eight knowledge areas' phrasing must be corrected or supported, and the coding procedure needs to be auditable. Neither issue appears to be unfixable, and I do not see grounds for rejection. I would also encourage the editor to ask the authors to clarify the treatment of missing information in the internship and project counts before final acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper for its dataset, not for its headline. The authors manually surveyed 133 study programs at 101 universities across 24 countries and released the dataset and a processing notebook. That is a real, reusable contribution, and it extends Crabb et al.'s US-only review to a global sample. The finding that most cyber-named programs under-cover the non-technical CSEC2017 knowledge areas (human, organizational, societal security) and rarely require internships is consistent with earlier national studies and is worth taking seriously.\n\nThe descriptive core is solid. The authors are transparent about their sampling from five rankings, and they acknowledge that unranked institutions are excluded. The checklist in Section 5 is practical and grounded in their own teaching experience rather than in the data alone. I also appreciate that they double-checked the collected information, even though they do not report inter-rater reliability.\n\nNow the soft spots, in proportion. The coding procedure in Section 4.3 is under-specified: no list of which essential concepts were searched for, no coding rubric, no reliability check. The threshold for ignoring concepts that appear as one or two topics in a large introductory course is ad hoc. This matters because the 'only five programs cover the three KAs' figure is the empirical anchor of the paper. A small change in counting rules could move that number. More importantly, Section 6 converts this into a stronger claim: 'only five cover all eight knowledge areas defined by CSEC2017.' But the authors only searched for the three non-technical KAs in this analysis. They have not measured the five technical KAs, so the 'all eight' claim is not supported by their own method. That is a genuine overstatement, though it does not sink the paper's central point about non-technical under-coverage. The inference from curriculum descriptions to graduate employability is also beyond the data, but the authors frame it as a concern, not a proof, so I would call that a minor overreach.\n\nWho gets value from this? Program directors, cybersecurity education researchers, and anyone comparing curricula across countries. The checklist alone makes it worth a skim. It deserves serious peer review, but the authors need to fix the Section 6 wording and either make the coding audit available or temper the precision of the counts. I would engage with it and cite the dataset.","headline":"A useful, genuinely new global dataset on cybersecurity programs, but the Section 6 'all eight KAs' claim overstates what the coding actually measured.","tokens_in":12750,"tokens_out":1143,"would_cite":true,"duration_ms":13790,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Top-ranked universities have not widely adopted cybersecurity curricular guidelines, and most 'cyber'-named degree programs omit non-technical knowledge areas and mandatory internships.","keywords":["cybersecurity education","CSEC2017","CC2020","curricular guidelines","university programs","experiential learning","internships","curriculum analysis"],"falsifier":"Inspect the official curriculum records or accreditation self-studies of the same 45 'cyber' programs and count how many have required courses that substantially cover the human, organizational, and societal security knowledge areas of CSEC2017; if that count is substantially larger than five, the paper's central finding is an artifact of relying on public course descriptions.","tokens_in":11821,"feed_emoji":"🎓","tokens_out":8632,"duration_ms":78687,"temperature":0.7,"pith_summary":"The paper asks whether university programs whose names contain 'cyber' actually deliver what the cybersecurity curricular guidelines prescribe. Examining 101 top-ranked universities across 24 countries, the authors found only 45 programs with 'cyber' in their name, and of those only five cover the three non-technical CSEC2017 knowledge areas—human, organizational, and societal security—and only five require internships. Only four are undergraduate programs. The authors argue that most such programs teach technical knowledge but skip the non-technical and hands-on learning that employers and guidelines say are essential, so graduates may leave unprepared for the workforce. If the claim holds, the 'cyber' label is not a reliable signal of a complete cybersecurity education, and program directors have a concrete checklist to close the gap.","feed_headline":"Only 5 of 45 'cyber' degree programs require internships","feed_subtitle":"Survey of 101 universities finds most cyber-named programs omit law, risk, and hands-on training.","key_machinery":"The argument is carried by the CSEC2017 Knowledge Areas, used as the benchmark for what a complete cybersecurity curriculum should contain, and a strict counting rule that credits a program with covering a knowledge area only when its essential concepts appear as a substantial part of core or core-elective courses—not as one or two topics in a single introductory course. The authors apply this rule to publicly available program websites and course catalogs for 45 'cyber'-named programs, and separately check program requirements for internships and final projects or theses. The survey set itself is built from five university rankings to ensure global coverage.","core_discovery":"The paper's central discovery is that the existing cybersecurity curricular guidelines have not been broadly adopted by top-ranked universities. Among the 45 programs with 'cyber' in their name at 101 institutions in 24 countries, only five programs cover the human, organizational, and societal security knowledge areas defined by CSEC2017; the rest concentrate on the technical knowledge areas such as data, connection, and system security. Mandatory internships are required by only five programs, and only four programs are bachelor's degrees. The authors conclude that most of these 'cyber' programs lack essential non-technical content and experiential learning, and that graduates may therefore need additional training to meet employer expectations.","pith_inferences":["My reading: the strict counting rule (ignoring a knowledge area that appears as only one or two topics in an introductory course) is a design choice that may undercount borderline programs; a more lenient counting of any mention in a course description could push the number of 'covering' programs higher, so the exact percentages should be treated as lower bounds under the authors' own criterion.","The paper assumes employers value the CSEC2017 non-technical areas because job-ad studies emphasize soft skills, but it does not directly link graduates of the five 'full coverage' programs to better employment outcomes; that link is a testable next step.","By limiting the detailed analysis to programs with 'cyber' in the name, the paper leaves open the possibility that security tracks inside general computer science programs cover these areas; a broader scope might change the picture of global coverage.","A natural extension is to map the job-market skill categories from the cited ad analyses onto the CSEC2017 knowledge areas and then check which programs' descriptions align with which job roles; that would turn the curriculum check into a workforce-readiness forecast."],"forward_implications":["Students cannot rely on the word 'cyber' in a degree title to mean the program covers the full CSEC2017 curriculum, so they should check the required courses for law, policy, risk, and human factors.","Employers hiring from these programs should expect to provide additional on-the-job training, because most of the surveyed programs do not require internships and therefore do not guarantee workplace experience.","The scarcity of bachelor's programs (only four at top universities) means students seeking an early, deep specialization in cybersecurity have few options at leading institutions.","Program directors can use the paper's 10-item checklist and the highlighted good-practice examples to add the missing non-technical and experiential components.","The paper's dataset and Python notebook are publicly available, so prospective students and researchers could re-examine or extend the analysis."],"supporting_citations":[{"why":"Defines the eight knowledge areas and the essential concepts used as the benchmark for what a complete cybersecurity program should cover.","marker":"[31]"},{"why":"CC2020 argues for competency-based curricula and experiential learning, the standard used to judge internships and projects.","marker":"[8]"},{"why":"Prior review of US cybersecurity programs that this global survey extends to institutions outside the USA.","marker":"[11]"},{"why":"Job-ad analysis showing employers ask for non-technical skills, used to motivate why the missing knowledge areas matter.","marker":"[17]"},{"why":"Interviews with cybersecurity professionals on the skills students should learn in school, supporting the importance of non-technical and hands-on competencies.","marker":"[19]"},{"why":"Topic modeling of cybersecurity job postings showing human- and organization-oriented roles, reinforcing the relevance of the non-technical areas.","marker":"[43]"}],"fun_headline_variants":["Most cyber degree programs lack law and risk training","Cyber programs miss key security guidelines, study finds","Only 5 of 45 cyber programs require internships","Study: Cyber degrees skimp on non-technical skills","Top universities fall short on cybersecurity curriculum"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The findings assume that what appears in publicly available program descriptions and course catalogs is an accurate reflection of what a program actually requires and teaches; if those descriptions are outdated or misleading, the reported coverage numbers would be wrong.","fun_headline_variants_meta":{"raw":{"variants":["Most cyber degree programs lack law and risk training","Cyber programs miss key security guidelines, study finds","Only 5 of 45 cyber programs require internships","Study: Cyber degrees skimp on non-technical skills","Top universities fall short on cybersecurity curriculum"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1400,"prompt_tokens":855,"completion_tokens":545,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":473}},"tokens_in":471,"tokens_out":545,"duration_ms":6178,"temperature":1.0,"reasoning_tokens":473,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:51:27.769309+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the official curriculum records or accreditation self-studies of the same 45 'cyber' programs and count how many have required courses that substantially cover the human, organizational, and societal security knowledge areas of CSEC2017; if that count is substantially larger than five, the paper's central finding is an artifact of relying on public course descriptions.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the eight knowledge areas and the essential concepts used as the benchmark for what a complete cybersecurity program should cover."},{"cited_title":"Matt Graham and Yonggang Lu","cited_arxiv_id":null,"evidence_quote":"Job-ad analysis showing employers ask for non-technical skills, used to motivate why the missing knowledge areas matter."},{"cited_title":"Jones, Akbar Siami Namin, and Miriam E","cited_arxiv_id":null,"evidence_quote":"Interviews with cybersecurity professionals on the skills students should learn in school, supporting the importance of non-technical and hands-on competencies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Topic modeling of cybersecurity job postings showing human- and organization-oriented roles, reinforcing the relevance of the non-technical areas."}],"review_version":1}