{"id":"5749a8f0-0dfb-44ac-ac19-5200eac4ffb5","arxiv_id":"2602.01415","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"EDF organizes LLM tutoring agents into evidence, decision, and feedback modules, and its Copa instantiation shows within-system correlations between task mastery, scaffold fading, and interpretable feedback in 33 dyads.","lead":"A team proposes EDF, a three-part framework—evidence, decision, feedback—for building LLM tutoring agents that adapt to what a student shows they know, and tests it in Copa with 33 high-school student pairs. A generalist might read it to see whether LLM tutors can be structured around transparent pedagogical reasoning rather than one-size-fits-all answers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical case for EDF rests on automated dialogue-state labels and LLM CoT 'evidence' that are never validated against human judgment; if those labels are mastery-contaminated or post-hoc, the RQ2/RQ4 claims lose their foundation.","rationale":"The reader's weakest assumption and my identified concern coincide: the paper's central empirical claims depend on treating the LLM's automated dialogue-state labels and CoT evidence as faithful, yet no independent gold standard or inter-rater reliability check is provided. I read the paper in good faith and find the EDF framework coherent, with a concrete multi-agent instantiation and a real classroom deployment; those are genuine strengths. However, RQ2 and RQ4 are the load-bearing empirical supports for the abstract's claims about understanding-mastery alignment and interpretability, and both rely on unvalidated internal constructs. My concrete test would settle whether the concern lands: if human-annotated labels agree with Copa's labels and preserve the RQ2 correlation, the alignment claim is substantially strengthened; if not, the central empirical claims are not supported. This does not move the verdict because the reader already set CONDITIONAL with essentially this reservation identified as the weakest assumption. The correct status remains CONDITIONAL, pending the proposed validation.","tokens_in":10571,"tokens_out":6088,"duration_ms":66929,"concrete_test":"Conduct a blind human-annotation study on a stratified random sample of the 238 student-agent turns, oversampling near the mastery>0.7 boundary. Two or more coders, blind to Copa's labels and to task mastery, should (a) classify each student reply as demonstrating understanding or not, and (b) rate whether the CoT evidence excerpt is supported by the immediately preceding log/chat context. Compute Cohen's kappa among coders and against Copa's DEMONSTRATES_UNDERSTANDING labels, then recompute the RQ2 Spearman correlation using human labels. If human-coder agreement with Copa's labels is below roughly 0.6, or if the human-label-based RQ2 correlation is not significant, the alignment and interpretability claims do not survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RQ2's key dependent variable is the DialogueAgent's own DEMONSTRATES_UNDERSTANDING classification of student responses, made by the same agent that also consults task mastery in the learner model. The reported positive correlation (rho=0.40) between this 'success rate' and mastery deciles may therefore reflect the LLM labeling high-mastery students' utterances as demonstrating understanding, rather than any genuine alignment between understanding and mastery. No human gold-standard coding, inter-rater reliability, or manipulation check is reported for this construct, so the 'not merely gaming the system' conclusion is unsupported. RQ4 is similarly self-referential: Grounding is token overlap between logs and the LLM's own CoT 'evidence'; Alignment and Faithfulness are SBERT similarities between the LLM's own policy/evidence/feedback texts. These metrics capture internal consistency, not interpretability to stakeholders, and CoT can be a post-hoc rationalization rather than the actual decision basis. The paper itself concedes in Section 6 that 'rigorous stakeholder-centric evaluation across these axes remains future work.' Additionally, footnote 4 truncates RQ2 by suppressing probes when mastery >0.7, so even the restricted correlation is only available in the lower mastery range. If the automated labels or CoT evidence are not faithful, the central claims about understanding-mastery alignment and evidence-grounded interpretability lack empirical support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Evidence-Decision-Feedback (EDF), a three-module framework (Evidence, Decision, Feedback) for theory-driven adaptive scaffolding in LLM-based pedagogical agents, and instantiates it as Copa, a multi-agent peer deployed in the C2STEM open-ended learning environment. In a six-week high school study with 33 dyads, the authors analyze 7,017 logged actions and 238 student-agent conversation turns to address four research questions: scaffold adaptivity, alignment of verbalized understanding with task mastery, student reliance on the agent, and interpretability of agent feedback. They report significant Spearman correlations between dialogue policies and task mastery (e.g., PROBE_UNDERSTANDING decreases, SUGGEST_ACTION and PUSH_LIMIT increase), a positive correlation between `DEMONSTRATES_UNDERSTANDING` success rate and mastery deciles (rho=0.40), decreasing support requests with mastery (rho=-0.26), and internal traceability scores for Grounding, Alignment, and Faithfulness that exceed shuffled baselines. The paper concludes that EDF supports personalized, effective, and interpretable scaffolding without fostering overreliance.","tokens_in":10867,"tokens_out":4767,"duration_ms":52549,"significance":"The EDF framework is a constructive synthesis of ITS/learning-theory principles and agentic LLM design; the explicit separation of evidence gathering, pedagogical decision-making, and feedback generation, coupled with a shared learner model, is a concrete architectural contribution that could guide future LLM tutoring systems. The participatory design process, authentic classroom deployment, and public supplementary materials are commendable strengths. However, the empirical support for two central claims is currently incomplete: RQ2's understanding-mastery alignment rests on an unvalidated automated dialogue-state label produced by the same agent whose decisions it is used to validate, and RQ4's interpretability evidence is entirely internal to the LLM's own outputs. If these issues are addressed by external validation or by appropriately tempering the claims, the paper would be a valuable contribution to AIED and multi-agent LLM systems. As it stands, the evidence supports EDF as an architecture for system-internal consistency and adaptive behavior, but not yet the broader notions of validated understanding-mastery alignment and stakeholder-facing interpretability asserted in the","major_comments":[{"comment":"The RQ2 'success rate' is not an externally validated measure: it is the DialogueAgent's own DEMONSTRATES_UNDERSTANDING classification made immediately after a PROBE_UNDERSTANDING policy. The same agent also has access to task mastery in the learner model (Section 4). No human gold-standard coding, inter-rater reliability, or manipulation check is reported. The positive correlation (rho=0.40) can therefore reflect the classifier labeling high-mastery students' utterances as demonstrating understanding rather than a genuine understanding-mastery alignment. The claim that students were 'not merely gaming the system' requires independent coding or a clear separation between the evidence used for labeling and the mastery variable. At minimum, the claim should be rephrased as internal/exploratory, and the limitation should be stated explicitly in addition to the general caveats in Section 6.","section":"§5.2, Fig. 4a; §4"},{"comment":"Interpretability is measured with internal traceability metrics: Grounding uses token overlap between student logs and the LLM's own CoT 'evidence'; Alignment and Faithfulness use SBERT similarity between the LLM's own policy, evidence, and feedback texts. These statistics show the pipeline is internally consistent, not that stakeholders can understand or trust the reasoning; CoT outputs are not guaranteed to reflect the actual decision basis. The abstract's claim that EDF 'support[s] interpretable, evidence-grounded explanations' therefore goes beyond what RQ4 establishes. I recommend either (i) adding a small human evaluation with teachers/students scoring the trace chain, or (ii) explicitly reframing RQ4 as 'internal traceability' and reserving 'interpretability' for future work, as the paper itself does in Section 6.","section":"§5.4, Table 3"},{"comment":"The statistical analyses treat policy frequencies and success rates as independent across mastery levels, but the data are repeated measures from 33 dyads over three sessions (238 dialogue turns). Spearman correlations are reported without confidence intervals or clustering adjustments; the effective sample size is much smaller than the number of data points. Additionally, footnote 4 reports that Copa suppressed PROBE_UNDERSTANDING when mastery was greater than 0.7, so Figure 4a's correlation is only identified on the lower-mastery range; the paper should report this restriction and discuss how it affects the 'understanding-mastery alignment' conclusion. These issues do not invalidate the framework, but transparency about them is needed and the strength of the empirical conclusions should be tempered accordingly.","section":"§5.1–§5.3, Tables 2 and Fig. 4"}],"minor_comments":[{"comment":"Report standard deviations and sample sizes for the Likert-scale items (e.g., means of 3.81 and 2.69 are given without variability or N).","section":"§5.5"},{"comment":"Define all column abbreviations in the caption (OE, MAS, SR, INT, etc.) and ensure the legend is self-contained; some symbols are not intuitive without cross-referencing the text.","section":"Table 1"},{"comment":"Clarify whether the mastery > 0.7 cutoff for not probing is a fixed design rule, and state explicitly its consequence: RQ2 analyses are restricted to lower mastery levels.","section":"Footnote 4 / §5.2"},{"comment":"Specify the exact GPT-5 model version/date used for reproducibility, and indicate where prompts and full model configurations are available in the supplement.","section":"§4"},{"comment":"The statement 'Over four years of participatory design' would benefit from a citation to the supplementary materials or a brief description of the PD process, since it is a key grounding claim.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript relies heavily on the authors' own prior work (e.g., CoTAL, earlier adaptive scaffolding theory), and the novelty of EDF relative to that prior theory should be clarified. The external-validity concerns are substantive but addressable: a human-coded validation sample for the dialogue-state labels and stakeholder ratings of the trace chains would materially strengthen the paper. The paper's scope and framing fit an AIED/learning-sciences venue; for a general CS or multi-agent journal, the evaluation's internal focus would need to be justified more explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper: EDF is a genuinely useful organizing framework for LLM-based adaptive scaffolding, and Copa is a concrete, inspectable instantiation. But the abstract's empirical claims go beyond what the evidence supports. The core problem is not the framework; it's that the key alignment and interpretability results depend on the agent's own automated labels and self-similarity metrics, with no human gold standard.\n\nThe new contribution is real: separating evidence, decision, and feedback into explicit modules connected by a shared learner model, with traceability links grounded in ECD, Stealth Assessment, SCT, and ZPD. That synthesis is not present in the cited prior work, and the Copa architecture maps each module to distinct sub-agents in a way that is concrete and inspectable. The four-year participatory design is a strength, and the authors are transparent about missing RCT, small sample, and correlational findings. The adaptivity result (probe decreases, suggest/push increases with mastery) is internally plausible and less contaminated by self-reference than the other claims.\n\nNow the soft spots, in proportion. The stress-test concern lands. RQ2's success rate is defined by the DialogueAgent's own DEMONSTRATES_UNDERSTANDING classification of student responses, made by the same agent that consults mastery. The observed rho=0.40 could partly reflect the LLM labeling consistent with its model of mastery rather than genuine verbal understanding. No human coding, no inter-rater reliability, no manipulation check. Footnote 4 compounds this: probes are suppressed when mastery >0.7, so the correlation only covers the lower range. RQ4 is also self-referential: keyword recall and SBERT similarity compare the agent's outputs to its own CoT evidence. That measures internal consistency, not stakeholder interpretability. The paper concedes this in Section 6. These are significant limitations, but they are disclosed, and they don't invalidate the framework — they do mean the abstract overreaches.\n\nThe minor points: 238 turns across 33 dyads is thin for repeated measures, and the analysis doesn't model clustering. But for a framework paper, the empirical work is a demonstration, not a decisive test.\n\nWho gets value? AIED researchers and ed-tech builders designing LLM tutors, especially those wanting to connect learning theory to agent architecture. It deserves a serious referee: the framework is coherent and the evaluation, while flawed, is honest enough to be a starting point. I'd send it to review with a clear request to validate the automated labels externally or soften the claims. Reading group? Maybe — useful for discussion of evaluation pitfalls in AIED. I wouldn't cite it in my own work yet, but I'd point to it as a framework worth watching.","headline":"A useful framework for LLM tutoring with an evaluation that overreaches; the EDF architecture is worth serious attention, but the empirical claims need human-grounded validation.","tokens_in":11430,"tokens_out":1765,"would_cite":false,"duration_ms":20484,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EDF, a three-stage evidence-decision-feedback framework, makes LLM tutoring agents adapt their scaffolding to students' demonstrated mastery in a real classroom.","keywords":["adaptive scaffolding","LLM pedagogical agents","Evidence-Decision-Feedback","agentic AI","multi-agent architectures","scaffold fading","learner modeling","interpretability"],"falsifier":"A version of Copa with its chain-of-thought evidence blocks stripped out that produces identical student-facing feedback would show the interpretability chain is decorative rather than functional.","tokens_in":10457,"feed_emoji":"🎓","tokens_out":3796,"duration_ms":41813,"temperature":0.7,"pith_summary":"EDF is a three-stage framework—evidence, decision, feedback—for building LLM tutoring agents that adapt to what students actually demonstrate. The paper instantiates it in Copa, a peer agent for STEM+C kinematics, and shows in a real high-school classroom that the agent's scaffolding shifts from probing understanding to suggesting actions and pushing limits as task mastery rises. It also reports that students' verbalized explanations align with task performance, that students request less support as mastery grows, and that the agent's reasoning chain is traceable from student input to feedback. If true, this is a concrete recipe for making LLM tutors both adaptive and interpretable rather than one-size-fits-all answer machines.","feed_headline":"Fades help as mastery grows in LLM tutor classroom test","feed_subtitle":"Copa, an EDF-based peer agent, probed less and pushed more as students' task scores rose—without breeding overreliance.","key_machinery":"EDF's evidence-decision-feedback loop, instantiated in Copa as four sub-agents (Strategy, Assessment, Knowledge, Dialogue) operating over a shared learner model. The DialogueAgent maps evidence to a dialogue policy (e.g., PROBE_UNDERSTANDING, SUGGEST_ACTION, PUSH_LIMIT) that generates a talk move. The workhorse is the chain-of-thought 'evidence' output that traces each decision back to student data.","core_discovery":"The paper claims that LLM pedagogical agents can be structured around three linked stages—evidence, decision, feedback—so that scaffolding is personalized and traceable. In a six-week classroom study with 33 dyads using the Copa agent in an open-ended kinematics environment, the framework produced dialogue-policy shifts that correlated with task mastery, understanding-mastery alignment, reduced help-seeking, and interpretability chain scores significantly above baselines.","pith_inferences":["Because the design is correlational and single-system, the observed adaptivity could partly reflect students self-selecting which moments to ask for help; a controlled comparison against a non-EDF tutor would test the framework's unique contribution.","The 'helpfulness paradox'—students wanting direct answers while the agent probes—likely generalizes beyond this setting; future agents may need to explain their pedagogical intent or negotiate roles to sustain student buy-in.","The interpretability metrics are automated proxies; a stakeholder study asking teachers whether the evidence-to-policy traces actually aid their trust would be a natural next step.","The EDF separation of evidence, decision, and feedback could be ported to other open-ended learning environments, but the specific policies and evidence types would need re-authoring per domain."],"forward_implications":["Copa's scaffolding policy shifts from probing to suggesting actions and pushing limits as students' task mastery rises, consistent with ZPD-based scaffold fading.","Students' success at verbally explaining their code increases with task mastery, suggesting progress reflects understanding rather than gaming.","Students request agent support less often as mastery grows, indicating reduced reliance and developing autonomy.","All three interpretability links—grounding, alignment, faithfulness—show statistically significant non-random structure, supporting that feedback is traceable to student input.","EDF offers a reusable blueprint for connecting learning theory to concrete, inspectable components in LLM-based pedagogical agents."],"fun_headline_variants":["LLM tutor adapts help to mastery, fading as skills grow","Copa agent tunes scaffolding to understanding in class test","Adaptive LLM peer agent cuts overreliance, study finds","Evidence-decision-feedback framework personalizes LLM tutoring","Scaffold fades as learners master tasks in Copa classroom trial"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's claims rest on the LLM's self-reported chain-of-thought evidence and automated dialogue-state labels being faithful—that they accurately capture both the agent's reasoning and the student's genuine understanding.","fun_headline_variants_meta":{"raw":{"variants":["LLM tutor adapts help to mastery, fading as skills grow","Copa agent tunes scaffolding to understanding in class test","Adaptive LLM peer agent cuts overreliance, study finds","Evidence-decision-feedback framework personalizes LLM tutoring","Scaffold fades as learners master tasks in Copa classroom trial"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00013,"raw_usage":{"total_tokens":897,"prompt_tokens":615,"completion_tokens":282,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":359,"completion_tokens_details":{"reasoning_tokens":210}},"tokens_in":359,"tokens_out":282,"duration_ms":3866,"temperature":1.0,"reasoning_tokens":210,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:39:25.350978+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A version of Copa with its chain-of-thought evidence blocks stripped out that produces identical student-facing feedback would show the interpretability chain is decorative rather than functional.","supporting_citations":[],"review_version":1}