{"id":"40790750-0f70-4490-95ff-466e3e06f2e3","arxiv_id":"2505.18467","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Pedagogy-R1 distills pedagogical reasoning into small models from QwQ-32B and evaluates them with a new five-domain educational benchmark, but the gains are modest and the benchmark is only partially public.","lead":"A team trains small AI models to reason like teachers by filtering correct answers from a larger model, and introduces a five-part benchmark for educational tasks. The resulting models beat their instruction-tuned baselines on some teaching-related tasks, but remain far behind frontier models and the evaluation has unresolved gaps.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The AES dimension of WBEB is an unexplained metric with a suspicious 7.02% floor, so the claimed balanced pedagogical improvement over five domains is not established.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the AES metric is undefined and produces suspiciously identical values for many models, making the claimed AES improvements and the 'balanced' portrayal of WBEB unreliable. I agree that this is the most load-bearing issue because AES is one of five benchmark pillars and one of the few dimensions where Pedagogy-R1-7B shows a large, headline-worthy gain over its instruction-tuned control. The manuscript itself contains no description of the AES prompt, parsing rule, or scoring metric, and the exact coincidence of many models at 7.02% strongly suggests a default-fallback artifact rather than genuine model performance. I also note that Section 5.3 explicitly defers machine-human agreement analysis for the GPT-4.1 qualitative coding, which is a real limitation but secondary to the quantitative AES problem. The paper does have positive aspects: the distillation and filtering pipeline is a standard and reasonable approach, the five-domain benchmark structure is a useful framing, and the public components (cj-eval-T, dbe-kt22, xes3g5m-simple, csedm19-spring, learning-agency-lab-aes) allow partial external checks. However, those strengths do not repair the unexplained AES column, and the central claim that pedagogical alignment significantly improves feedback and balanced performance cannot be accepted while one of its supporting metrics may be meaningless. The appropriate verdict remains REJECT, so no adjustment to the reader's verdict is needed.","tokens_in":17227,"tokens_out":6036,"duration_ms":55300,"concrete_test":"Obtain the authors' AES evaluation script and run the exact pipeline on the 3,462 learning-agency-lab-aes test essays with Llama-3.1-8B-Instruct. Print the raw model responses, the prompt, and the parser; compute what fraction of responses fail to parse and whether 7.02% equals the accuracy of assigning a fixed default label to all unparseable outputs. Then recompute both the baseline and Pedagogy-R1-7B AES scores using the official competition metric (quadratic weighted kappa) and report per-prompt agreement. If the floor reproduces as a parse-failure fallback and QWK is near zero, the AES dimension is an artifact and the 'balanced' claim should be restricted to the remaining four domains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Pedagogy-R1 achieves balanced and effective performance across educational tasks relies on all five WBEB dimensions, but the AES dimension is not actually defined. Section 3.4 describes only the Kaggle dataset and never specifies the scoring prompt, the output parser, or the metric behind the 'AES (ACC)' column. Table 2 makes the problem concrete: nearly every non-pedagogical model and several Pedagogy-R1 variants sit at exactly 7.02%, a value consistent with a fallback label assigned when model output cannot be parsed, rather than with any meaningful essay-scoring ability. The only model that escapes this floor is Pedagogy-R1-7B (15.83%), which is also the model whose AES gain over its control (+8.75 points) is used to support the feedback-improvement claim. Without the protocol, this gain cannot be distinguished from format compliance—e.g., the model emitting a parseable numeric score—rather than superior essay-scoring quality. The same ambiguity contaminates the training pipeline: Section 2.1.3 says responses were kept only if 'correct', as verified by automatic evaluation or rule-based heuristics, and for AES that correctness criterion is never described. If the AES metric is broken, one of the five WBEB dimensions is invalid, the 'balanced' characterization is unsupported, and the claimed AES improvements are an artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Pedagogy-R1, a family of instruction-tuned reasoning models for educational tasks, trained by distilling a teacher model (QwQ-32B) on responses filtered for correctness, and evaluated on a new Well-balanced Educational Benchmark (WBEB) covering subject knowledge, pedagogical knowledge, knowledge tracing, automated essay scoring, and classroom decision-making. The authors also propose a Chain-of-Pedagogy (CoP) prompting strategy intended to elicit teacher-like reasoning, and they report quantitative and qualitative analyses of the resulting reasoning traces. The central claim is that pedagogically aligned instruction tuning significantly improves educational reasoning, feedback, and decision-making compared with standard instruction tuning or general-purpose models, and that CoP prompting further enhances pedagogical alignment.","tokens_in":17476,"tokens_out":3898,"duration_ms":32676,"significance":"If the WBEB were a sound and reproducible benchmark, the paper would make a useful contribution to the AI-in-education community by offering a multi-dimensional evaluation of LRMs and a training recipe for pedagogical alignment. The idea of evaluating five distinct educational capabilities is timely, and the qualitative analysis through a reflective-practice lens is an interesting departure from purely accuracy-based evaluation. However, the current manuscript does not support these contributions because several load-bearing components are either unspecified, unreproducible, or conflicting with the reported data: the AES protocol is undefined with a suspicious 7.02% floor, error bars and significance tests are absent, the GPT-4.1-based qualitative coding is unvalidated, and the PK and DM datasets are small and private. These issues prevent an assessment of whether the claimed balanced and effective performance is real or an artifact of the evaluation setup.","major_comments":[{"comment":"The automated essay scoring (AES) evaluation protocol is not defined. Section 3.4 describes only the Kaggle dataset and never specifies the scoring prompt, the output parser, or the exact formula for the reported 'AES (ACC)' column. The near-universal 7.02% value across unrelated models suggests a parse-failure fallback rather than a meaningful essay-scoring accuracy. This issue is load-bearing because the claimed AES improvement of Pedagogy-R1-7B (+8.75 points over its control) and the 'balanced' characterization of WBEB depend on this dimension. Without a detailed protocol and a sanity check that the metric behaves like essay-scoring accuracy, the AES results cannot be interpreted, and the central claim of balanced improvement across five domains is not established.","section":"3.4 / Table 2"},{"comment":"Table 2 presents results without error bars, confidence intervals, or significance tests, which is especially problematic because several datasets are very small (PK subdomains have 64-124 items, DM splits have 85-471 items). The report in the text and in the table caption overstates the findings: the caption claims the proposed models 'consistently achieve the best results across all domains except for SK,' but the table shows, for example, that Pedagogy-R1-7B has a KT-AUC of 54.99% versus 64.73% for Qwen2.5-32B, and its SK score (27.20%) is below many baselines. The authors should report variance estimates and correct the summary statements to match the data.","section":"4.2 / Table 2"},{"comment":"The quantified qualitative analysis relies entirely on GPT-4.1 for coding thought-token traces, yet the paper explicitly states that a machine-human agreement analysis is 'left for future work.' Without any validation that GPT-4.1 follows the codebook reliably, the code distribution in Figure 5 and the subsequent claims about CoP effects on metacognitive codes cannot be taken as evidence. The authors should provide at least a validation subsample with human-coded agreements (e.g., Cohen's kappa or similar) before interpreting the GPT-4.1-based distributions.","section":"5.3"},{"comment":"The response-filtering step that defines D_filtered uses a correctness criterion 'Correct(yi)' that is never operationalized per task. For AES, it is unclear what constitutes a correct generated essay score; for KT, it is unclear whether correctness refers to next-token prediction or to a separate answer extraction. The filtering step selects the training data, so if it is based on a flawed or format-dependent check, the resulting models may be optimized for parseability rather than pedagogical quality. The authors should specify the exact automatic evaluation or rule-based heuristic used for each WBEB category and report the category-wise retention rates.","section":"2.1.3 / 4.1"},{"comment":"Two of the five benchmark dimensions rest on very small and private datasets. The PK dataset has only 124, 104, and 64 items for the three subdomains, and the DM dataset is based on NCTE transcripts that are not publicly released; the paper states that 'ncte-eval' is private and available only under data-sharing agreements. With such small samples, the reported accuracies (e.g., PK 31.67% vs. 30.60%) are not reliably distinguishable, and the inability to access these data prevents independent verification. The authors should make these datasets available under appropriate licenses or replace them with larger public alternatives, and they should report exact sample sizes and variance.","section":"3.2 / 3.5"}],"minor_comments":[{"comment":"The text of Theme 3 is duplicated verbatim in Sections 5.2.3 and 5.2.4; one copy should be removed.","section":"5.2.3 / 5.2.4"},{"comment":"The word 'underthikning' is a typo; it should be 'underthinking.'","section":"5.1"},{"comment":"The radar chart is difficult to read without axis labels and value annotations; consider replacing it with a table that includes error bars.","section":"Figure 1"},{"comment":"The quoted prompt transformation should be typeset consistently, and the distinction between CoP used for data generation ('pedagogy distillation') and CoP used at inference time should be made clearer in the main text.","section":"2.2"},{"comment":"The dataset name 'dbe-kt22' is written inconsistently as 'DBE-KT-22' in Table 1; unify the spelling throughout.","section":"3.3"},{"comment":"The paper would benefit from a dedicated limitations subsection discussing the small private datasets, the lack of error bars, and the unvalidated automated coding.","section":"General"}],"recommendation":"reject","confidential_remarks":"The paper has several signs of being rushed: duplicated sections, typos, and an overreaching table caption. More importantly, the private nature of the PK and DM datasets, combined with the undefined AES metric, means that the benchmark cannot be independently reproduced or trusted in its current form. The central claim of balanced pedagogical improvement is not supported by the evidence as presented, and I do not see a minor revision that could fix this without essentially redoing the evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a real artifact—a five-domain educational benchmark and a small fine-tuned model family—but the evaluation is not reliable enough to support the central claim. The most important problem is the AES column. Nearly every model sits at exactly 7.02% accuracy, which looks like a parse-failure fallback rather than a measured score. The paper never describes the scoring prompt or metric. Pedagogy-R1-7B alone escapes the floor at 15.83%, and the +8.75 improvement over its instruction-tuned control is the load-bearing evidence for the feedback claim. Without the protocol, that gain could just be format compliance. The stress-test note is right: one of the five dimensions is effectively unexplained, so 'balanced' is unsupported.\n\nA second issue you should know about: the text claims Pedagogy-R1-7B achieves the highest score in DM, PK, and AES, but Table 2 shows o4-mini beats it on all three (65.52 vs 54.76, 50.09 vs 31.67, 20.05 vs 15.83). That is a factual contradiction inside the paper. It also weakens the rhetorical contrast with 'general-purpose models'—the strongest general-purpose model outperforms the distilled model everywhere.\n\nWhat is genuinely good: the benchmark composition is a sensible attempt to cover subject knowledge, pedagogy, tracing, essay scoring, and decision-making, and the KICE/NCTE curation with human verification is real work. The qualitative framework using Schön's reflection types is a thoughtful way to interpret reasoning traces, though the GPT-4.1 coding without agreement analysis means those results are illustrative at best.\n\nMinor quibbles: no error bars, tiny private datasets (PK and DM), and a duplicated Theme 3 are signs of a hasty write-up. The instruction-tuned baseline is weak because it was trained to output problem-answer pairs, so the comparison flatters the reasoning models.\n\nNet: the paper deserves a serious referee—the benchmark is a useful contribution to AIED—but it needs a major revision to fix the AES protocol, correct the o4-mini overclaim, and report variance. Right now I wouldn't rely on any single number from it.","headline":"A useful educational benchmark and model family undermined by an unexplained AES metric and a factual overclaim about o4-mini.","tokens_in":18045,"tokens_out":3015,"would_cite":false,"duration_ms":23223,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes that filtering teacher-model reasoning traces for correctness and instruction-tuning small student models on them yields balanced pedagogical improvements across subject knowledge, pedagogical knowledge, knowledge…","keywords":["pedagogical reasoning","large reasoning models","instruction tuning","distillation","Chain-of-Pedagogy","educational benchmark","knowledge tracing","automated essay scoring"],"falsifier":"Rerun the AES evaluation with a documented scoring prompt and output format; if the 7.02% cluster persists across all models regardless of response content, the AES dimension is a formatting artifact. Independently, a human–machine agreement study on the 42 coded traces would test whether the CoP-induced increase in metacognitive codes is real.","tokens_in":17039,"feed_emoji":"🎓","tokens_out":8949,"duration_ms":61996,"temperature":0.7,"pith_summary":"This paper claims that large reasoning models can be adapted for teaching by distilling only the reasoning traces that end in a correct answer, and that this works better than standard instruction tuning when the goal is educational reasoning rather than raw subject accuracy. It introduces the Well-balanced Educational Benchmark (WBEB), spanning subject knowledge, pedagogical knowledge, knowledge tracing, automated essay scoring, and real-world classroom decision-making, to test that claim. On that benchmark, a 7-billion-parameter model trained this way (Pedagogy-R1-7B) reaches the top scores in pedagogical knowledge, essay scoring, and decision-making, and outperforms standard instruction tuning by double-digit percentage points on several dimensions. The paper also introduces the Chain-of-Pedagogy (CoP) prompt, “consider the pedagogical step by step,” which shifts model reasoning toward teacher-style reflection when applied during both data generation and inference.","feed_headline":"Teacher-style reasoning training lifts small AI models' judgment","feed_subtitle":"Five-domain benchmark shows pedagogy-filtered tuning improves classroom decision-making and feedback.","key_machinery":"The load-bearing mechanism is a correctness-filtered distillation loop wrapped in pedagogical prompting. A large teacher model generates responses to seed questions under the Chain-of-Pedagogy prompt; only outputs whose answers pass automatic or rule-based correctness checks are kept, shrinking 5,000 seeds to 1,948 training examples. Small student models are then instruction-tuned on those traces plus answers, and the same CoP prompt (“Consider the pedagogical step by step, and then pick the best option”) is used at inference time to elicit teacher-style reasoning. This isolates the contribution of pedagogical reasoning from raw content knowledge, because the control models are trained on the same examples with only question–answer pairs.","core_discovery":"On its own terms, the paper's discovery is that pedagogically aligned instruction tuning—filtering a teacher model's outputs for correctness, then fine-tuning smaller models on those filtered traces with an explicit pedagogical reasoning prompt—produces models that are consistently better than standard instruction tuning at pedagogical knowledge, automated essay scoring, and classroom decision-making, without sacrificing subject knowledge. The authors report that Pedagogy-R1-7B scores 54.76% on decision-making versus 23.42% for the same-size instruction-tuned baseline, and 15.83% versus 7.08% on essay scoring, with gains in pedagogical knowledge around six percentage points. Qualitatively, they report that CoP prompting elicits metacognitive codes such as self-questioning and planning that never appear under generic reasoning prompts, although roughly two-thirds of reasoning tokens are filler noise.","pith_inferences":["Editorial inference: a similar filter-and-distill loop could be applied to other professions where reasoning traces are audited, such as medicine, law, or counseling, replacing “correct answer” with “expert-validated decision” as the filter.","Editorial inference: because the paper never describes the AES scoring prompt or metric, and many models sit at exactly 7.02%, the AES dimension should be treated as unvalidated until the protocol is released.","Editorial inference: the CoP effect on metacognitive codes is measured by GPT-4.1 coding with no agreement analysis, so the qualitative claim is best read as a hypothesis about prompt-induced reflection rather than an established measurement.","Editorial inference: one could test whether filtering on pedagogical quality rather than answer correctness produces even larger PK and DM gains, since the current filter only guarantees right answers, not good teaching."],"forward_implications":["Small models (1.5B and 7B) can match or beat much larger general-purpose models on pedagogical knowledge and decision-making, so education-specific deployment costs could drop substantially.","Standard instruction tuning on problem–answer pairs is not enough; the combination of correct-answer filtering and pedagogical reasoning traces is what drives the gains, since the instruction-tuned baselines score far lower.","CoP prompting is a zero- and few-shot intervention that can be applied at inference time without retraining and still shifts reasoning patterns, so existing models can be steered toward pedagogy.","Reasoning-augmented models still underperform on knowledge tracing AUC, so pedagogical reasoning is not a silver bullet for modeling student learning dynamics.","The finding that about 65% of reasoning tokens are filler noise leaves substantial headroom for cheaper inference in pedagogical tasks."],"supporting_citations":[{"why":"Supplies the distillation and response-filtering methodology that the pipeline adapts for pedagogy.","marker":"(Guo et al., 2025)"},{"why":"Provides the STILL-2 filtering step, keeping only correct model responses, used to build the training set.","marker":"(Min et al., 2024)"},{"why":"Identifies the QwQ-32B teacher model that generated the filtered training data.","marker":"(Team, 2025b)"},{"why":"Source of the translated cj-eval-T subject-knowledge questions from Chinese junior high exams.","marker":"(Zhang et al., 2024)"},{"why":"Provides the dbe-kt22 university database-education knowledge tracing dataset used in WBEB.","marker":"(Abdelrahman et al., 2022)"},{"why":"Provides the XES3G5M knowledge tracing dataset, sampled to xes3g5m-simple for LLM evaluation.","marker":"(Liu et al., 2024)"},{"why":"Source of the Learning Agency Lab automated essay scoring dataset used for AES evaluation.","marker":"(Crossley et al., 2024)"},{"why":"Provides the NCTE classroom transcripts that ground the real-world decision-making dimension.","marker":"(Demszky and Hill, 2023)"},{"why":"Supplies the reflection-in-action and reflection-on-action framework used to code reasoning traces.","marker":"(Schön, 2017)"},{"why":"Defines the underthinking score and transition-word analysis used in the quantitative reasoning analysis.","marker":"(Wang et al., 2025)"}],"fun_headline_variants":["Pedagogy-tuned AI models score higher on classroom tests","Filtered teacher reasoning boosts small models in school tasks","Pedagogical tuning lifts AI judgment in five-domain test","Small models gain teacher smarts via pedagogy-filtered tuning","CoP prompting elicits teacher-style reasoning in AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automated essay-scoring numbers measure essay quality, since the scoring protocol is never described and most models sit at 7.02%, and that GPT-4.1's automatic coding of reasoning traces reliably identifies reflection without an agreement test.","fun_headline_variants_meta":{"raw":{"variants":["Pedagogy-tuned AI models score higher on classroom tests","Filtered teacher reasoning boosts small models in school tasks","Pedagogical tuning lifts AI judgment in five-domain test","Small models gain teacher smarts via pedagogy-filtered tuning","CoP prompting elicits teacher-style reasoning in AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000133,"raw_usage":{"total_tokens":1082,"prompt_tokens":841,"completion_tokens":241,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":161}},"tokens_in":457,"tokens_out":241,"duration_ms":2654,"temperature":1.0,"reasoning_tokens":161,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:30:32.894354+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the AES evaluation with a documented scoring prompt and output format; if the 7.02% cluster persists across all models regardless of response content, the AES dimension is a formatting artifact. Independently, a human–machine agreement study on the 42 coded traces would test whether the CoP-induced increase in metacognitive codes is real.","supporting_citations":[],"review_version":1}