{"id":"cb4b39c8-11ca-40aa-8aff-cb1cc060b4af","arxiv_id":"2501.09709","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper introduces CyberMentor, an open-source LLM-based tutoring platform for cybersecurity education, and reports automated evaluation scores that are not independently validated.","lead":"CyberMentor is an AI tutoring platform that combines retrieval-augmented generation and an agentic workflow to answer cybersecurity students' questions about knowledge, skills, and careers. It is reported to score well on an LLM-run evaluation, but the evaluation lacks human validation and baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Table II scores cannot support the central effectiveness claim because they come from an unvalidated LLM judge, and Section X.C defers actual student pilot studies to future work.","rationale":"After reading the paper in good faith, I agree with the reader's verdict. The platform design, use cases, and open-source release are genuine contributions to the educational technology space. However, the central claim that the system 'confirmed its strong performance' and 'underscores its ability to support students' depends entirely on Section IX's LLM-judge evaluation. The reader identified the unvalidated LLM judge as the weakest assumption, and my reading reinforces that: the paper does not describe the judge's prompt, model, or reference answers; no human graders are involved; and the paper itself, in Section X.C, places pilot studies and user feedback in future work. This creates an internal inconsistency between the abstract's strong claim and the authors' own statement that the framework currently offers 'demonstration capabilities.' My concrete test, human expert grading of a sample, would directly validate the LLM judge. If human experts agree with the LLM judge, the scores gain credibility; if not, the central claim is unsupported. I also note the low cryptography correctness (0.63) shows the system's limitations, but that alone would not reject the paper if other evidence were strong. The overclaim about equity and sustainability is another symptom of the same problem: these outcomes are not measured. Therefore, the reader's REJECT verdict remains appropriate; I would not change it, though a revised version with human evaluation and baselines could change the verdict.","tokens_in":13365,"tokens_out":4655,"duration_ms":47905,"concrete_test":"Have three cybersecurity educators with teaching experience independently grade a stratified random sample of 30 responses from the 115-question eval set (10 per category) using the paper's stated rubrics for helpfulness, correctness, and completeness, blinding them to whether a response was produced by CyberMentor, a generic LLM, or a reference answer. Compute the correlation and Cohen's kappa between the average human scores and the reported LangChain LLM-judge scores. If the agreement is low (e.g., kappa < 0.6) or human graders identify factual errors in answers the LLM judge scored correct, then the Table II scores are not valid evidence for the strong performance claim. For an even more decisive test, the same human graders should also score the reference answers; if the reference answers are themselves LLM-generated or unreachable, correctness scores are uninterpretable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CyberMentor 'confirmed its strong performance' (abstract, Section IX). The only evidence is Table II, produced by a 'LangChain prompt-based evaluation methodology, which involved an independent LLM to cross-validate the performance of the responses' (Section IX). This evaluation is not anchored to any human judgment, validated benchmark, or baseline system, and the reference answers used for correctness are not described in the paper. The correctness metric is especially problematic: an LLM judge checking consistency with an unseen reference answer can be biased by stylistic similarity or by the same parametric knowledge used to generate the response, inflating the reported 0.83 average. The paper's own Section X.C states 'Currently, our framework offers fully functional demonstration capabilities' and that 'Pilot studies and feedback from users will be crucial for further development,' implicitly acknowledging that the evaluation did not involve students using the platform. The abstract nonetheless claims the results 'underscore the system's ability to support students in developing practical cybersecurity skills while improving equity and sustainability,' outcomes that are not operationalized or measured anywhere in the paper. Table II also shows a correctness of 0.63 for cryptography, so the system is not uniformly 'strong' even under its own judge. Without external validation, the evaluation does not establish that the platform provides educational value; it only shows that an LLM can produce answers that another LLM rates well.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CyberMentor, an AI-powered tutoring platform for cybersecurity education that combines a retrieval-augmented generation (RAG) knowledge base with agentic workflow tools for cryptography, scripting, and machine-learning-based threat detection. The authors identify six pain points for non-traditional cybersecurity students, describe the system architecture and three illustrative use cases, and evaluate the platform on 115 questions using a LangChain prompt-based LLM judge with metrics of helpfulness, correctness, and completeness. They report average scores of 0.85, 0.83, and 0.90 respectively and claim the results confirm the system's strong performance in supporting students.","tokens_in":13578,"tokens_out":3017,"duration_ms":32638,"significance":"If the claimed effectiveness were properly established, the platform would address a genuine need for scalable, personalized support for non-traditional cybersecurity students, and its open-source, modular design is a practical strength. The system description, use cases, and initial evaluation demonstrate a functional prototype. However, the current evidence does not support the paper's central effectiveness claim: the evaluation relies entirely on an LLM-as-judge with no human validation, baseline comparison, or statistical analysis, and the paper explicitly defers user pilot studies to future work. The contribution at this stage is a system design and a preliminary internal evaluation, not a validated educational intervention.","major_comments":[{"comment":"The reference answers used for the correctness metric are not described; their provenance (human-authored, LLM-generated, or hybrid) is unknown. Without this information, the correctness scores in Table II are not interpretable, since consistency with an unseen reference is a meaningful measure only if the reference itself is trustworthy and independent of the system's generation process.","section":"Section IX, Table II"},{"comment":"The paper's own conclusion states that 'Currently, our framework offers fully functional demonstration capabilities' and that 'Pilot studies and feedback from users will be crucial for further development.' This is an explicit acknowledgment that no student users evaluated the system. Yet the abstract claims the results 'underscore the system's ability to support students in developing practical cybersecurity skills while improving equity and sustainability within higher education.' These outcomes (skill development, equity, sustainability) are not operationalized or measured anywhere in the paper. The claims overreach the evidence, and the evaluation does not establish that the platform provides educational value to students.","section":"Section X.C"},{"comment":"The cryptography category received a correctness score of 0.63, which is notably lower than the other categories and below the level one would expect for a 'strong' or 'robust' educational tool. Use Case 1 prominently features the CryptoSolver tool for a modular inverse problem, making cryptography a core part of the platform's claimed value. The paper acknowledges that LLMs struggle with mathematical reasoning, but then still presents the aggregate scores as confirming strong performance. Given that the system is intended to support students in this domain, the low cryptography score needs a direct discussion of whether the tool meets educational needs for cryptography, not merely a generic note about LLM limitations.","section":"Section IX, Table II, cryptography row"}],"minor_comments":[{"comment":"The phrase 'Imbalance Balancing Foundational and Advanced Knowledge' appears to be a typo; it should likely read 'Imbalance in Balancing Foundational and Advanced Knowledge' or similar.","section":"Section II.A, P2"},{"comment":"The sentence 'Another significant challenges encountered by students' contains a subject-verb agreement error ('challenges' should be 'challenge').","section":"Section II.B"},{"comment":"The evaluation section does not specify the exact prompt template used for the LangChain evaluator, the specific LLM model used as judge, or the temperature/sampling settings. These details are necessary for reproducibility and for assessing the risk of judge bias.","section":"Section IX"},{"comment":"The reference answers for the 115-question dataset are not described in the paper; the authors point to a repository, but the paper itself should summarize how these answers were created and validated.","section":"Section IX"},{"comment":"The term 'sustainability' is used in the abstract without definition or measurement; if the authors intend a specific meaning (e.g., cost-effectiveness, long-term maintainability), it should be stated and operationalized.","section":"Abstract and Section X.C"},{"comment":"The phrase 'a low-cost, 24/7, multilingual, personalized and effective learning support' lists several qualities, but the evaluation does not address multilingual support or cost; these claims should be either evidenced or qualified.","section":"Section IV.B"}],"recommendation":"reject","confidential_remarks":"The paper describes a potentially useful open-source platform and identifies a real need, but the validation is fundamentally insufficient for the claims made. The LLM-as-judge evaluation with no human grounding, no baseline, and no statistical analysis, combined with the paper's own admission that pilot studies are future work, means the central effectiveness claim is not supported. The manuscript could be reconsidered after a rigorous human evaluation or after substantially reframing the contribution as a system design and demonstration. I would also note that the stress-test concern about the unvalidated LLM evaluation is well founded and directly lands on the manuscript's load-bearing claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a legitimate systems paper with an evaluation that does not support its central claim. The artifact is real, open-source, and thoughtfully designed for a genuine population (non-traditional cybersecurity students). The three skill tools and the 115-question benchmark are the most useful parts. But Table II, produced by an LLM judge checking against unseen reference answers, cannot carry the \"robust performance\" claim.\n\nWhat's actually new: the specific integration of RAG with a ReAct-style agent and purpose-built tools for cryptography, scripting, and ML classification, plus a domain-specific question set. That is a real contribution, though it is an application of known methods. The paper also does a good job of mapping well-documented pain points (limited mentorship, outdated materials, career guidance gaps) onto concrete design decisions. I have no quarrel with the architecture or the open-source release.\n\nThe soft spot is exactly where the reader put it. The evaluation in Section IX uses a LangChain prompt-based LLM judge. There is no human grading, no baseline system, no variance reporting, and the reference answers are not described. The correctness metric is particularly fragile: an LLM checking consistency with an unseen reference can be fooled by stylistic similarity. The paper's own Section X.C says pilot studies are future work, so the system has not been tested with students. The 0.63 correctness on the 10-question cryptography subset shows the system is not uniformly strong even by its own judge. And the abstract's talk of improving equity and sustainability is not operationalized anywhere.\n\nI would not reject this outright. The artifact is useful, the benchmark may be reusable, and the flaws are fixable. But as submitted, the effectiveness claim is not established. A revision with human evaluation (even a small one), a baseline comparison, and measured learning outcomes would change my mind. If the authors drop the overclaiming and present CyberMentor as a functional prototype with promising initial results, it could be acceptable.\n\nWho is this for? People building LLM tutoring systems, especially in STEM education. It is not a methods paper. It deserves a serious referee, but the referee should treat the evaluation as the main issue.\n\nRecommendation: send it to peer review, but with a clear expectation that the evaluation has to be redone.\n\nRegards.","headline":"A useful open-source tutoring platform whose evaluation is too weak to support the effectiveness claim; the artifact is real but the central claim is not established.","tokens_in":14152,"tokens_out":2209,"would_cite":false,"duration_ms":22979,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CyberMentor claims an LLM-powered platform with retrieval-augmented knowledge and skill tools can supply the mentorship that non-traditional cybersecurity students lack.","keywords":["cybersecurity education","non-traditional students","large language models","retrieval-augmented generation","agentic workflow","virtual mentoring","career development","coding skills"],"falsifier":"Run the same 115 questions through CyberMentor and have three human cybersecurity instructors independently grade helpfulness, correctness, and completeness without seeing the LLM judge's scores; if their averages come in materially below 0.85, 0.83, and 0.90, or if human-LLM agreement on cryptography is near chance, the claimed performance fails to replicate.","tokens_in":13158,"feed_emoji":"🛡️","tokens_out":8735,"duration_ms":82091,"temperature":0.7,"pith_summary":"CyberMentor is a proposed answer to a specific educational equity problem: non-traditional cybersecurity students—commuters, part-timers, veterans—often lack the mentors, peers, and updated materials that traditional students can reach. The paper argues that an LLM-powered platform combining retrieval-augmented generation with specialized coding and cryptography tools can provide the missing 24/7 personalized support for knowledge acquisition, skill practice, and career guidance. To back this, it reports an evaluation of 115 questions across three domains, with average scores of 0.85 for helpfulness, 0.83 for correctness, and 0.90 for completeness. The authors' central claim is that such a system can be effective and equitable enough to matter in real cybersecurity programs, while being open and modular enough to extend to other disciplines.","feed_headline":"AI mentor posts 0.85 helpfulness for cybersecurity students","feed_subtitle":"Retrieval-augmented knowledge plus coding tools gives 24/7 support to commuters, part-timers, and veterans.","key_machinery":"The architecture combines three layers. A knowledge base holds curated cybersecurity documents—course-level knowledge units, certification study material, federal career pathway documents, and a local program catalog—chunked and embedded into a vector database for retrieval-augmented generation. A skill base provides agentic tools that walk students through well-defined tasks: CryptoSolver guides cryptographic problem solving, ScriptCoder builds scripts for network anomaly detection, and MLClassifier leads machine-learning threat-detection exercises. An LLM agent parses natural-language queries, selects the appropriate knowledge source or tool, verifies the generated response, and returns it with sources; the workflow follows a reasoning-and-acting pattern in which the model's reasoning traces are interleaved with tool actions.","core_discovery":"The paper claims that a single platform—CyberMentor—can cover the main pain points of non-traditional cybersecurity students: limited access to updated material, difficulty balancing foundational and advanced knowledge, unclear career alignment, weak math and coding foundations, and lack of timely mentoring. On its own 115-question evaluation spanning knowledge, cryptography, and coding, it reports average helpfulness 0.85, correctness 0.83, and completeness 0.90, with the knowledge category strongest (0.94 helpfulness, 0.91 correctness) and cryptography correctness weakest (0.63); the authors attribute the difference to known LLM limits in precise mathematical reasoning. The paper frames these results as evidence that RAG-grounded, tool-augmented LLM mentoring can support skill development and career preparation while improving equity and sustainability in higher education.","pith_inferences":["The reported evaluation does not compare CyberMentor to a generic LLM chatbot, so it remains possible that the retrieval and tool-selection layers add less than the authors attribute to them; a head-to-head comparison on the same 115 questions would settle this.","Because the judge is itself an LLM, the scores may reflect stylistic similarity to the system's outputs rather than educational value; human grading of the same responses would be the natural check.","The cryptography result suggests a hybrid design: let the LLM explain the method but delegate exact arithmetic and modular inverse computation to a symbolic calculator.","If the transferability claim is right, the same knowledge-base-plus-skill-tools pattern could be applied to other fast-changing fields where curricula lag industry practice, such as AI ethics or cloud engineering, with an equally modular open-source release."],"forward_implications":["Non-traditional students would gain round-the-clock, low-cost, multilingual mentoring without needing to align with faculty office hours.","Faculty can generate course plans aligned with current industry frameworks by querying the knowledge base, reducing manual curation time.","Students can practice cryptography and coding through structured, step-by-step tool workflows rather than receiving only text answers.","Institutions and instructors can extend the open-source platform with new documents and custom tools, so the pattern could move to other STEM disciplines.","If the scores hold, the weakest area is math-heavy cryptography, where correctness falls to 0.63; the authors expect chain-of-thought reasoning and stronger LLMs to close that gap."],"supporting_citations":[{"why":"Establishes the nontraditional-student population and its enrollment characteristics, which define who CyberMentor is for.","marker":"[1]"},{"why":"Documents the risks and access gaps high-risk students face, the problem CyberMentor claims to mitigate.","marker":"[2]"},{"why":"Introduces retrieval-augmented generation, the method used to ground answers in the knowledge base.","marker":"[27]"},{"why":"Defines the agentic workflow pattern of interleaved reasoning and action that the platform's LLM agent follows.","marker":"[35]"},{"why":"Provides the certification-guide content loaded into the knowledge base for retrieval.","marker":"[36]"},{"why":"Shows a prior RAG-based system for cybersecurity expertise that CyberMentor extends with skill tools and career guidance.","marker":"[30]"},{"why":"Cited as the chain-of-thought technique expected to strengthen mathematical reasoning, where the evaluation found the lowest correctness.","marker":"[28]"},{"why":"Cited as a way to scale inference-time reasoning for harder problems, informing future improvements.","marker":"[33]"}],"fun_headline_variants":["CyberMentor hits 0.85 helpfulness for non-traditional students","RAG-powered AI mentor boosts cybersecurity learning equity","One platform, three use cases: CyberMentor aids career prep","AI mentor tackles knowledge, coding, and career guidance gaps","Open-source CyberMentor: 24/7 support for diverse learners"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported scores rest on the assumption that an automated LLM judge grading answers against reference answers of unstated origin is a valid measure of educational quality; if that judge is biased toward the system's own style, the numbers could be inflated.","fun_headline_variants_meta":{"raw":{"variants":["CyberMentor hits 0.85 helpfulness for non-traditional students","RAG-powered AI mentor boosts cybersecurity learning equity","One platform, three use cases: CyberMentor aids career prep","AI mentor tackles knowledge, coding, and career guidance gaps","Open-source CyberMentor: 24/7 support for diverse learners"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000138,"raw_usage":{"total_tokens":1183,"prompt_tokens":1001,"completion_tokens":182,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":91}},"tokens_in":617,"tokens_out":182,"duration_ms":2673,"temperature":1.0,"reasoning_tokens":91,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:43:34.762316+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 115 questions through CyberMentor and have three human cybersecurity instructors independently grade helpfulness, correctness, and completeness without seeing the LLM judge's scores; if their averages come in materially below 0.85, 0.83, and 0.90, or if human-LLM agreement on cryptography is near chance, the claimed performance fails to replicate.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the nontraditional-student population and its enrollment characteristics, which define who CyberMentor is for."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the risks and access gaps high-risk students face, the problem CyberMentor claims to mitigate."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the certification-guide content loaded into the knowledge base for retrieval."}],"review_version":1}