{"id":"34231f1f-7055-4994-bb40-d9e7dfba06fe","arxiv_id":"2608.03118","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An LLM-based system maps SQL submission errors to concepts in an automatically built course knowledge graph; expert ratings support basic validity but reveal edge-direction and granularity flaws.","lead":"The paper describes a platform that automatically builds a knowledge graph of course concepts from lecture materials and links students' SQL errors to specific concepts and prerequisites. A five-expert review rated most extracted concepts and relations as valid, but flagged reversed edges and coarse error labels; no learning-impact results are reported yet.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reversed DEPENDS_ON edges are the load-bearing risk: if direction accuracy is low, prerequisite-gap tracing sends students to the wrong concepts, and the evaluation does not quantify this failure.","rationale":"I read the paper as a systems/AI-in-education contribution whose central value proposition is connecting SQL submission traces to a course concept graph and using that graph to identify both the error and its prerequisite context. The architecture is coherent, the execution-based grading is deterministic, and the expert quotes provide some support for the perceived alignment of the graph. The paper also honestly reports the biggest threat: both courses' evaluators found reversed prerequisite edges. The reader's weakest assumption correctly identifies this as the point where diagnostic value can fail even if node validity is high. My concern is a narrower version: the evaluation aggregates triplet correctness, but directional errors are not separately quantified, so the headline metric of 63.8% fully correct does not reveal how often the system would recommend the wrong prerequisite. The concrete check above would settle this. I do not see an internal inconsistency in the derivation; the issue is insufficient evidence for the prerequisite-tracing sub-claim. Because the authors already condition their conclusions on these limitations and propose instructor review as a mitigation, I keep the CONDITIONAL verdict rather than escalating to REJECT. I agree with the reader's weakest assumption.","tokens_in":9537,"tokens_out":5143,"duration_ms":59883,"concrete_test":"Obtain the full set of proposed DEPENDS_ON edges for both courses (not just the sampled triplets) and have two independent database instructors label the correct direction for each edge. Compute directed-edge accuracy before and after the platform's reconciliation workflow. Then, for the ten error cases (five Course A real, five Course B simulated), recompute the top-3 prerequisite concepts reachable via corrected directed edges and compare them with the recommendations produced by the original graph. If directed accuracy is below, say, 90%, or if the corrected and original top-3 lists differ in more than one case, the prerequisite-gap diagnosis is not robust and the central claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is not just that errors can be tagged with concepts, but that the platform can trace a failed submission to the specific concepts involved and to prerequisite concepts that may require reinforcement (Section III-D). That traceability depends entirely on the orientation of the DEPENDS_ON (and PART_OF) edges that an LLM proposes from lecture slides. The paper itself reports that evaluators from both courses flagged reversed prerequisite arrows (Section IV-C and Section V), and only 63.8% of triplets were rated fully correct (Table II). If edge direction is wrong, the graph may tell a student to review the wrong prerequisite concept, which breaks the core diagnostic value even when node validity is high. The instructor-review workflow described in Section III-B can catch such errors, but then the 'automated extraction' claim is weakened, and more importantly the evaluation does not separate raw extraction quality from post-review quality. We therefore cannot tell whether the prerequisite-tracing function is reliable as deployed. This is a correctness risk in the causal chain from graph to diagnosis, not a disagreement with the educational-knowledge-graph research direction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an LLM-assisted platform that extracts course concepts and relations (PART_OF, DEPENDS_ON) from lecture materials, links them to SQL student submissions via a Neo4j graph, and classifies errors at the concept level to provide diagnostic feedback. The system combines execution-based grading with LLM-based semantic feedback and a fine-tuned no-reveal filter, then synchronizes attempts into a knowledge graph for instructor and student analytics. The evaluation consists of an expert study with five participants across two courses (Course A with real submissions and four evaluators; Course B with simulated submissions and one evaluator) plus an automated LLM-as-judge assessment. Reported results are 95.7% of extracted nodes rated at least somewhat valid and 63.8% of triplets rated fully correct by experts, with the LLM judge showing a consistent positive bias. The authors acknowledge that reversed prerequisite edges were flagged by both courses and state that evaluating impact on student learning remains future work.","tokens_in":9666,"tokens_out":3306,"duration_ms":37694,"significance":"If the prerequisite-tracing functionality were reliable, the platform would be a useful bridge between SQL tutoring systems and educational knowledge graphs, and the architecture's separation of deterministic execution grading from model-based semantic interpretation is a sound design choice. The explicit instructor-review reconciliation workflow is a practical strength, as is the authors' transparent reporting of LLM-judge bias and the small evaluation scale. However, the core diagnostic claim—that errors can be traced to specific concepts and prerequisite gaps—depends entirely on the correctness of automatically extracted edge directions, and the paper reports that experts found reversed prerequisite arrows in both courses. This makes the significance conditional: the platform is a plausible and well-motivated prototype, but the current evidence does not yet establish the reliability of the concept-level diagnosis it promises.","major_comments":[{"comment":"The central diagnostic claim—that a failed submission can be traced to the specific concepts involved and to prerequisite concepts that may require reinforcement—depends on the orientation of DEPENDS_ON and PART_OF edges extracted from lecture materials. The paper itself reports that evaluators from both courses flagged reversed prerequisite arrows (Section IV-C) and that determining which concept is the prerequisite 'requires pedagogical reasoning that current LLMs do not always handle reliably' (Section V). Table II shows that only 63.8% of triplets were rated fully correct, yet no separate accuracy figure is reported for edge direction specifically. As a result, the evaluation does not quantify how often the system would direct a student to the wrong prerequisite concept, which is a load-bearing failure mode for the main contribution. Please report per-edge-type and per-direction accuracy, separate raw extraction quality from post-review quality, and either fix the edge-orientation problem or substantially soften the prerequisite-tracing claim.","section":"Section III-D, Section IV-C, Section V"},{"comment":"The evidence for the overall claims is very thin relative to their strength. Course A has four evaluators and real submissions, but Course B has a single evaluator and simulated submissions; the combined percentages in Table II pool these very different conditions. The 95.7% node figure uses the low bar of 'at least somewhat valid' (a nonzero rating), which hides the fact that only 60.9% of nodes were rated fully valid. The higher Course B scores are acknowledged by the authors as possibly reflecting the absence of inter-rater variability rather than a genuine difference. Please report per-evaluator and per-course results, exact item counts, inter-rater agreement (e.g., Cohen's kappa or percent agreement with chance correction), and avoid pooling Course A and Course B in headline numbers. Without this, the claim that the graphs 'align with instructor mental models' is not supported at the level asserted.","section":"Section IV-A, Section IV-B, Table II"},{"comment":"The automated LLM-as-judge evaluation cannot serve as independent confirmation of the knowledge-graph or error-classification results. The judge is an LLM (GPT-4o-mini) evaluating outputs of an LLM-based extraction pipeline, and the authors report that it never assigned a score of 0 and exhibited a consistent positive bias. This is a circularity risk: the judge may share systematic biases with the extraction model, and the paper's own data show the LLM ratings diverge from expert ratings (90.7% vs. 60.9% fully valid nodes; 4.00 uniform overall quality vs. expert range 3.25–4.75). I recommend either removing the LLM-judge results from the main evidence for correctness, or explicitly repositioning them as a screening tool with the reported bias acknowledged as a limitation, not as corroboration.","section":"Section IV-B, Section V"}],"minor_comments":[{"comment":"The example 'GROUP BY is part of SELECT clause' is semantically confusing, especially because PART_OF and DEPENDS_ON are the two relation types being evaluated; please clarify whether this is a compositional or prerequisite relation and make the figure consistent with the definitions in the text.","section":"Section III-D, Figure 2"},{"comment":"The definition of P_c is unclear: 'N possible_c' is described as the number of eligible student-question opportunities, but the text does not define 'eligible' or specify whether multiple attempts by the same student count once or multiple times. Please state the counting rule explicitly.","section":"Section III-E, Eq. (1)"},{"comment":"The paper says 'up to 27 sampled concept nodes and up to 23 sampled triplets' but Table II reports percentages without exact counts; please include the actual number of items rated per course and per metric so the percentages can be interpreted.","section":"Section IV-A"},{"comment":"The phrase 'approximately 123 human-curated non-revealing feedback examples' should give the exact count or a clear justification for the approximation, since the fine-tuned model's behavior is part of the system description.","section":"Section III-C"},{"comment":"The term 'HAVING' is rendered as 'HA VING' in the abstract and near the introduction examples; please fix the spacing.","section":"Abstract and Section I"}],"recommendation":"major_revision","confidential_remarks":"This is a well-motivated systems paper for an innovative-practice track, and the authors are transparent about several limitations. However, the load-bearing issue of reversed prerequisite edges is acknowledged but not quantified, and the evaluation rests on five experts (with one course having a single evaluator) plus an LLM judge that the authors themselves show to be positively biased. I would advise the editor that the paper is not ready as is; a revision should either add a rigorous edge-direction accuracy analysis (or an ablation with instructor-corrected edges) or restate the central contribution as a pilot platform with instructor-in-the-loop verification. The topic fits the venue, and the underlying architecture has merit, so I do not see grounds for rejection if the authors address these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my honest take on arXiv:2608.03118.\n\nThe paper deserves a serious referee. Its contribution is modest but real: it integrates three things that previously lived in separate literatures—semantic SQL error classification, educational knowledge graphs, and submission-trace analytics—into one working platform. No single component is new (LLM extraction, execution grading, graph storage), but the combination is. The architecture is described clearly, with sensible separation between the low-latency grading path and the asynchronous knowledge-graph sync.\n\nThe evaluation is small but honestly reported. Five experts is thin, especially with one evaluator on simulated submissions for Course B, yet the paper openly says what those limits are. It also does the right thing in reporting that the LLM judge never assigns zero and is systematically lenient; that makes the automated evaluation weak evidence, not a hidden failure. The qualitative feedback is actually useful: instructors found the graph aligned with their mental models, and the GROUP BY example shows the diagnostic concept, not just a correctness label.\n\nThe soft spot the stress test flagged is real and is confirmed by the paper itself: evaluators in both courses reported reversed DEPENDS_ON edges. The Discussion mentions this but doesn't quantify how often direction is wrong, or how much it degrades the 'trace to prerequisites' feature. Since that feature is part of the claimed value, this is a genuine limitation, not a nitpick. The instructor review workflow mitigates it, but then the platform is less 'automatic' than the abstract implies, and the evaluation doesn't separate raw extraction quality from post-review quality. I'd also flag the absence of released code/data; for a systems paper that's not fatal, but it makes reimplementation harder.\n\nThe citation pattern looks fine; self-citations are to the authors' own prior work (CodeLens, InstructKG), which is directly relevant. The paper doesn't overclaim learning impact; it explicitly defers that to future work. In summary: a solid, well-scoped paper with one load-bearing but acknowledged weakness. I'd send it to peer review, expecting heavy revision on evaluation depth, and I'd cite it if I worked on SQL education or educational graphs.","headline":"A genuinely new integration of SQL error classification and educational knowledge graphs, with an acknowledged prerequisite-edge risk that needs better quantification; worth a serious review.","tokens_in":10232,"tokens_out":4065,"would_cite":true,"duration_ms":40420,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A knowledge-graph pipeline traces failed SQL submissions back to the course concepts a student has not mastered.","keywords":["SQL education","knowledge graph","automated feedback","learning analytics","error classification","large language models","concept dependencies","student submission analysis"],"falsifier":"Run the pipeline on a course where instructors have published a definitive prerequisite ordering, then measure how often the extracted DEPENDS_ON direction disagrees with that ordering; if disagreement is common, concept-level diagnosis will systematically recommend the wrong review material.","tokens_in":9292,"feed_emoji":"🔗","tokens_out":4007,"duration_ms":38847,"temperature":0.7,"pith_summary":"The paper argues that the reason a SQL query fails is often a gap in a course concept, not a mere syntax slip, and that autograders cannot see that gap. It presents a platform that builds a knowledge graph of course concepts and their prerequisite relations from lecture slides, logs each student's submission as a typed path through that graph, and labels each semantic error with the concepts involved. The goal is to turn isolated grading events into a diagnostic map: an instructor can see which concepts the class is missing, and a student can be told which prerequisite concept to revisit. The authors report that expert evaluators rated 95.7% of extracted concept nodes as at least somewhat valid and 63.8% of concept triplets as fully correct, with qualitative feedback that the graphs match instructor mental models.","feed_headline":"Failed SQL queries mapped to the concepts students haven't mastered","feed_subtitle":"A knowledge-graph platform links each wrong query to the course concept and prerequisite a student needs to review.","key_machinery":"The load-bearing object is the course knowledge graph: concept nodes extracted from lecture slides by an LLM, canonicalized through a concept table, and connected by two relation types, PART_OF for compositional structure and DEPENDS_ON for prerequisite dependencies. Student submissions enter the graph asynchronously as typed paths linking student, assessment, question, and error entities, while a question-tagging layer associates each assessment question with the concept nodes it exercises. The error-to-concept associations come from a two-stage feedback module: a general-purpose LLM detects the semantic error, and a fine-tuned filter removes solution-revealing content. Progress analytics then compute per-concept completion as $P_c = \\frac{N^{\\text{correct}}_c}{N^{\\text{possible}}_c} \\times 100\\%$ and surface low-mastery concepts as personalized focus targets. Instructors can accept, reject, or modify proposed relations through a reconciliation step before they enter the graph.","core_discovery":"The central claim is that mapping student SQL errors to nodes in an extracted course knowledge graph yields actionable diagnostic insight that binary pass/fail grading cannot provide. The platform combines deterministic execution-based grading with LLM-generated semantic feedback, then synchronizes attempts into a graph database as typed student-assessment-question-error paths connected to concept nodes through PART_OF and DEPENDS_ON edges. Evaluated across two database courses, the system produced graphs that aligned with instructor mental models and error labels that pointed directly to what a question is testing. The authors position the work as bridging two previously separate strands: SQL error research and educational knowledge graphs.","pith_inferences":["The diagnostic value hinges on edge direction; if the LLM reverses prerequisite arrows at the rate experts observed, mastery estimates for prerequisite concepts will be systematically wrong even when node and triplet validity scores look high.","A stronger test than expert ratings would be a longitudinal comparison of whether students who receive concept-linked feedback show fewer repeated errors on the flagged prerequisite concepts.","The reported quality percentages likely overstate graph quality when the same model family used for extraction also serves as the automated judge, since the paper's own LLM judge never assigned a zero rating and showed consistent positive bias.","The progress metric counts any correct concept-linked attempt toward mastery, so a student who guesses correctly once is marked as having the concept; a threshold based on repeated attempts might change the focus-concept lists."],"forward_implications":["A recurring error such as misusing NATURAL JOIN can be traced not just to JOIN but along DEPENDS_ON and PART_OF edges to prerequisite concepts like SELECT, GROUP BY, and HAVING that a student may need to revisit.","Instructors get a cohort-level view of persistent concept gaps across assessments rather than isolated per-attempt grades.","The graph representation lets the system distinguish whether an error is localized to one question or propagates across related concepts.","The architecture separates deterministic grading from model-assisted interpretation, so immediate feedback remains low-latency while analytics run asynchronously.","The authors state the design is domain-agnostic and could be extended beyond SQL with adapted extraction prompts and domain-specific ontologies."],"supporting_citations":[{"why":"Documents that students make recurring semantic SQL mistakes across query types, motivating the paper's focus on semantic rather than syntactic errors.","marker":"[1]"},{"why":"Categorizes systematic errors and complications in SQL query formulation, providing the taxonomy the platform's error classification builds on.","marker":"[2]"},{"why":"Shows student difficulties are visible in how solutions evolve across attempts, justifying the analysis of submission traces rather than final correctness alone.","marker":"[4]"},{"why":"Demonstrates structural hints and partial credit for autograded SQL, serving as the prior-work baseline for richer feedback generation.","marker":"[9]"},{"why":"Provides an expert-in-the-loop method for building educational knowledge graphs with prerequisite relations, which the platform extends by connecting the graph to student traces.","marker":"[11]"},{"why":"Introduces the CodeLens approach to semantic SQL error detection and non-revealing feedback, which the platform's two-stage feedback module reuses.","marker":"[17]"},{"why":"Proposes the InstructKG framework for extracting concepts and inferring prerequisite and compositional dependencies from lecture materials, the extraction approach the platform adopts.","marker":"[19]"}],"fun_headline_variants":["AI knowledge graph links SQL errors to concept gaps","SQL mistakes traced to missing concepts by AI graph","From SQL errors to learning gaps: a knowledge graph","Knowledge graph maps SQL queries to untested concepts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system's diagnosis is only as reliable as the prerequisite arrows between concepts, and the language model sometimes draws those arrows backwards, as experts in both courses reported.","fun_headline_variants_meta":{"raw":{"variants":["AI knowledge graph links SQL errors to concept gaps","SQL mistakes traced to missing concepts by AI graph","From SQL errors to learning gaps: a knowledge graph","Knowledge graph maps SQL queries to untested concepts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1511,"prompt_tokens":927,"completion_tokens":584,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":524}},"tokens_in":543,"tokens_out":584,"duration_ms":6243,"temperature":1.0,"reasoning_tokens":524,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T00:50:58.459358+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the pipeline on a course where instructors have published a definitive prerequisite ordering, then measure how often the extracted DEPENDS_ON direction disagrees with that ordering; if disagreement is common, concept-level diagnosis will systematically recommend the wrong review material.","supporting_citations":[{"cited_title":"Students’ semantic mistakes in writing seven different types of sql queries,","cited_arxiv_id":null,"evidence_quote":"Documents that students make recurring semantic SQL mistakes across query types, motivating the paper's focus on semantic rather than syntactic errors."},{"cited_title":"Errors and complications in sql query formulation,","cited_arxiv_id":null,"evidence_quote":"Categorizes systematic errors and complications in SQL query formulation, providing the taxonomy the platform's error classification builds on."},{"cited_title":"Insights from student solutions to sql homework prob- lems,","cited_arxiv_id":null,"evidence_quote":"Shows student difficulties are visible in how solutions evolve across attempts, justifying the analysis of submission traces rather than final correctness alone."},{"cited_title":"Enhancing feedback gener- ation for autograded sql statements to improve student learning,","cited_arxiv_id":null,"evidence_quote":"Demonstrates structural hints and partial credit for autograded SQL, serving as the prior-work baseline for richer feedback generation."},{"cited_title":"Ace: Ai-assisted con- struction of educational knowledge graphs with prereq- uisite relations,","cited_arxiv_id":null,"evidence_quote":"Provides an expert-in-the-loop method for building educational knowledge graphs with prerequisite relations, which the platform extends by connecting the graph to student traces."},{"cited_title":"Codelens: A generative ai framework for automated feedback on sql assignments,","cited_arxiv_id":null,"evidence_quote":"Introduces the CodeLens approach to semantic SQL error detection and non-revealing feedback, which the platform's two-stage feedback module reuses."},{"cited_title":"Instructor-aligned knowledge graphs for personalized learning,","cited_arxiv_id":null,"evidence_quote":"Proposes the InstructKG framework for extracting concepts and inferring prerequisite and compositional dependencies from lecture materials, the extraction approach the platform adopts."}],"review_version":1}