Pith. sign in

REVIEW 3 major objections 4 minor 32 references

Auditable Release Control for Pedagogical Leakage in LLM Tutors

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Answer leakage in LLM tutors can be mechanically gated, the paper argues, but the gate's measured effect and its utility cost must be read in panel-flag units, not human ground truth.

desk verdict Genuinely new release-control architecture with a model limitations section, but every headline number runs through an LLM judge panel with .104 human sensitivity — magnitudes provisional, direction corroborated. read the letter →

arxiv 2608.00515 v1 pith:WJ7GWSZ3 submitted 2026-08-01 cs.CR cs.AIcs.CL

Nizam Kadir This is my paper
classification cs.CRcs.AIcs.CL
keywords pedagogicalleakageLLMtutorsanswerdisclosurereleasecontrolcompletemediationcontractsauditabletracesafety-utilitytradeoff
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language model tutors can leak the answer while sounding helpful: they disclose the target answer or the decisive calculation before the pedagogy authorizes it. The paper argues that this is a state- and action-dependent failure, and that it can be controlled mechanically at release time by a complete-mediation boundary that separates what to disclose from how to say it. A single release function checks every candidate response against one of five disclosure contracts, substitutes a deterministic fallback when a check fails, and logs the reason so failures can be attributed to action selection, generation, verification, or enforcement. In matched tests the strict gate cut a three-model judge panel's leakage flags from 181 to 0 but replaced 581 of 599 responses and lowered helpfulness; an externally timestamped prospective replication cut majority leaks from 42 to 8, with seven persisting and one introduced. The paper's claim is narrower than 'AI tutors are safe': it establishes an auditable release boundary and failure attribution under declared contracts, not universal semantic safety or learning gains.

What carries the argument

The central mechanism is the fail-closed release function G_a,m(y,H,z) = C_a(y,z) ∧ (m=P ∨ V_a(y,H,z)=1), which releases raw renderer output only when a deterministic contract checker passes and, in high-assurance mode, a semantic verifier also passes; any failure substitutes an action-specific deterministic fallback F_a. A selector first emits one of five disclosure contracts A0–A4, an authorization gate controls the privileged A4 mode, and a renderer proposes language. The release invariant is that every recorded raw release implies the guard passed, and every trace logs the pre-action state, selected action, authorization scope, raw output, checker/verifier reasons, intervention source, f

What would settle it

Run a matched blind evaluation of the same raw versus strictly gated responses with expert human raters on all 599 cases. If human-majority leakage flags do not fall nearly to zero, the release gate's headline effect is an artifact of the panel's differential response to fallback text; the paper itself reports the panel misses roughly 90% of human-flagged leaks, so this experiment is directly decisive.

Watch

Extended reading notes

Core claim

The core discovery is empirical: in a matched factorial holding prompts, proposals, disclosure contracts, and the judge rubric fixed, fallback enforcement is what removes leakage flags; the semantic verifier adds no reliable marginal benefit. Strict mediation reduces majority leakage flags from 181 to 0 (paired problem-cluster difference −30.22 points) while replacing 581 of 599 responses and lowering helpfulness; in a prospectively timestamped 480-sequence replication, high-assurance release reduces majority flags from 42 to 8 (−7.08 points), leaves eight, introduces one, and lowers helpfulness by .192. The author interprets this as establishing a mechanically enforced, replayable release b

Load-bearing premise

The numeric results assume that the three-model judge panel's majority vote measures real pedagogical leakage; against human raters the panel was 98.9% specific but only 10.4% sensitive, so most human-flagged leaks go undetected and every effect size in the study is expressed in panel-flag units.

Editorial extensions

If this is right

  • Deploying tutors with this release boundary makes unauthorized answer disclosure measurable and attributable instead of a hidden property of fluent generation.
  • Because fallback enforcement, not semantic verification, drove the measured reduction, adding a semantic verifier should not be assumed to improve safety without independent calibration.
  • The safety gain is not free: strict mediation replaced 581 of 599 responses and reduced helpfulness in matched tests, and the prospective study lowered helpfulness by .192.
  • A simple global scaffold (A1) outperformed the learned fitted-Q selector on the reported automatic safety and utility metrics, so learned action selection is not the source of the runtime safety gain.
  • The gate provides a precise attribution trace but does not establish learning gains or universal semantic safety; the guarantees are mechanical, not semantic.
  • Every component failure, including a check that wrongly passes or wrongly blocks, is logged and replayable, giving operators a concrete audit trail for investigating incidents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The architecture's audit ledger could support compliance and accountability requirements for educational platforms, since every rejected or released response records a human-auditable reason; the paper does not test such compliance uses.
  • If leakage measurement were switched from the automatic panel to expert human raters on the matched set, the magnitude of the effect could change substantially because the panel agreed with a human majority on only about one in ten human-flagged leaks; institutions should treat the headline numbers as panel-relative.
  • A natural testable extension is to couple this release boundary with a purpose-built, independently calibrated verifier; the paper's own evidence suggests any marginal benefit would need to be demonstrated per deployment, and the architecture is structured to make that demonstration direct.
  • The same complete-mediation pattern—contracts, a gate, a fallback, and replayable traces—could transfer to other settings where a model may disclose information only under authorization, such as graded exam support or clinical decision support, though the paper does not test those settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes an auditable release-control architecture for LLM tutors that mediates the release of tutor responses according to five 'disclosure contracts' (A0–A4). A selector picks a contract, a trusted gate controls privileged answer-release modes, and a release guard applies a deterministic checker, optionally a semantic verifier, and an action-specific fallback. The core empirical claims are: (i) on 599 fixed Gemini 3.5 proposals, strict mediation reduces a blinded three-model panel-majority 'leakage flag' rate from 181 to 0, but at large utility cost; (ii) a matched factorial shows that checker-triggered fallback explains most of this safety reduction, while the semantic verifier adds no reliable marginal benefit; (iii) a global A1 scaffold outperforms a fitted-Q selector on the reported automatic metrics; and (iv) an externally timestamped prospective replication on 40 unseen clusters reduces majority flags from 42 to 8, with paired cluster intervals excluding zero. The paper explicitly disclaims universal semantic safety and learning-gain claims, and it discloses that all model judges come from one provider family and that the semantic verifier is one of the three evaluation models.

Significance. The architectural decomposition—separating selector, generator, guard, and fallback, with replayable traces—addresses a real and underexplored failure mode in LLM tutors: unauthorized answer disclosure that is state- and action-dependent. If the empirical claims hold, the paper provides a useful 'release boundary' that makes leakage measurable and attributable, and its matched-ablation methodology is a good template for component-level safety evaluation. The externally timestamped prospective protocol with predeclared endpoints and cluster-bootstrapped confidence intervals is a notable strength, as is the paper's consistent reluctance to overclaim. However, the significance is conditional on the validity of the automatic leakage measure; the paper's own human calibration indicates that this measure is highly insensitive (sensitivity .104), which threatens the quantitative conclusions and, in particular, the attribution of the safety effect to fallback enforcement and the claim that the semantic verifier has no marginal benefit.

major comments (3)
  1. The primary endpoint for the headline results (181→0, 42→8, and the factorial attribution) is a same-provider three-model panel whose majority has sensitivity .104 and specificity .989 against a three-rater human majority (Section 4.4). Because the fallback is a short deterministic scaffold, the panel may be systematically less likely to flag fallback text even when leakage persists, and the semantic verifier's 'no reliable marginal benefit' (Section 4.2) could be an artifact of an outcome measure that detects only a small, possibly biased subset of leaks. This is load-bearing for the claim that fallback enforcement 'explains the large safety effect.' I request that, on the 120 matched raw/released sequences in Section 4.4, the authors report the automatic panel's paired raw-vs-released difference alongside the human majority difference, and, where feasible, recompute the verifier margin
  2. The paper discloses that the semantic verifier is also one of the three evaluation model versions, creating dependence despite blinded inputs. This matters specifically for the claim that adding the verifier yields no reliable marginal benefit: the panel majority includes a judge that shares the verifier's model, so the comparison is not independent. The robustness analysis in Section 4.2 reports per-judge intervals for strict-vs-no-guard, but not for the verifier marginal comparison. Please report per-judge outcomes (or at least the majority excluding the verifier-model judge) for the checker-only vs checker-plus-verifier conditions in the matched factorial, so the marginal-benefit claim can be assessed independently of the overlap.
  3. A central contribution is 'failure attribution under declared contracts': the release trace should reveal whether a failure came from selection, authorization, generation, checking, verification, or fallback. Yet the paper does not demonstrate this attribution on the eight final failures in the prospective replication or on the matched-ablation failures. The trace architecture may indeed provide this information, but no trace-level attribution summary is reported. I request an example or summary table that uses the replayable traces to attribute the eight final majority failures (and, if possible, the one introduced failure) to specific components, since this is the paper's main architectural claim.
minor comments (4)
  1. Typo: 'aspedagogical leakage' should be 'as pedagogical leakage'.
  2. The table formatting is difficult to read in the preprint (e.g., values run together). Please ensure columns are clearly separated and that the 'Guard int.' column values (0, 581, 505, etc.) are distinct from the failure counts.
  3. The human calibration section reports sensitivity/specificity of the panel against the human majority, but not the panel's paired raw-vs-released difference on the same 120 sequences. Reporting this would directly address the main measurement concern and would strengthen the exploratory validation.
  4. The reward weights in Eq. (4) are described as diagnostic, but the choice of 10, −0.1, −5 is not justified beyond being ad hoc. Since the paper explicitly says these are not causal learning-gain estimates, a brief sentence on how the weights were chosen (e.g., to balance event frequencies) would be helpful.

Circularity Check

1 steps flagged · score 3.0 of 10

Measurement dependence, not derivation-level circularity: the semantic verifier is also one of the three judge models, so the verifier's marginal-benefit attribution is partly self-assessed; the core auditable-release claims remain independently supported.

  1. other [Section 6 (Limitations) and Section 4.2 (Matched Runtime Ablation)]
    "The verifier is also one of the three evaluation model versions, creating dependence despite blinded inputs; the other judges show substantial disagreement."

    The matched factorial attributes a null result to the semantic verifier: 'adding the semantic verifier yields 14 majority and 248 any-judge failures' with 'no reliable marginal benefit.' But the outcome is measured by a three-model panel that includes the same model version used as the verifier. The intervention under test and one of the judges share weights, so the marginal-benefit estimate contains a self-assessment component: the judge's rating behavior is not independent of the component whose effect is being measured. This is not an equation-level tautology, but it is a load-bearing dependence for the specific attribution that the verifier adds no reliable benefit. The paper discloses the dependence and reports per-model results, so the overall architectural claim is not reduced to a

full rationale

The paper is largely self-contained and unusually explicit about its limitations. The release function in Eqs. (1)-(2) is a mechanical control-flow invariant, and the authors repeatedly state it is not a semantic-safety guarantee. The fitted-Q selector is presented as a diagnostic with known construct-confounded rewards, not as a predictive or causal claim. The prospective replication is externally timestamped, uses held-out problem clusters, and reports per-model outcomes rather than treating the panel majority as ground truth; the observed reductions are empirical and not forced by construction. No self-citation is load-bearing: the only self-citation (Kadir 2026) is a related-work pointer. The significant caveat is the verifier-judge overlap: because one of the three evaluation models is also the semantic verifier, the component-attribution conclusion about the verifier's marginal benefit is partly self-referential. The paper discloses this, but disclosure does not remove the dependence. Additionally, the automatic panel's low human sensitivity (.104) is a construct-validity threat, though it is a measurement weakness rather than circularity. Overall, the central safety/utility frontier and auditable-decomposition claims rest on independent comparisons and honest uncertainty bounds, so the circularity score is moderate, not severe.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The framework rests on the threat-model assumptions that reference answers and authorization records are correct, that user text cannot alter TCB state, and that all candidate output reaches the release function. The empirical claims additionally assume that ASSISTments event rows are usable proxies for disclosure contracts in the diagnostic lane and that a same-provider LLM panel majority is a valid leakage measurement; the latter is weakened by the paper's own human-calibration sensitivity of .104. The five disclosure contracts and the pedagogical-leakage definition are implemented constructs with an external handle through human ratings.

free parameters (5)
  • Diagnostic reward weights (10, -0.1, -5) in Eq. (4) = 10, -0.1, -5
    Hand-chosen constants in the fitted-Q diagnostic reward; they shape selector comparisons but not the runtime guard.
  • Fitted tabular Q(s,a) values = not reported
    Fit on 942,816 ASSISTments rows across five learner splits; used as a selector diagnostic and shown to be dominated by global A1.
  • Action-support count penalty 1/sqrt(n(s,a)) and minimum of 5 rows = 1/sqrt(n); min n=5
    Ad hoc regularizers in Section 3.3 that mask unsupported actions; they affect which actions the fitted-Q selector emits.
  • Lexical checker rule revision = Rules fit to 200-case author-built development set
    The rule revision reached 100% sensitivity on the development set but 0% on a fresh encoding challenge, so it is a fitted parameter, not a semantic guarantee.
  • A4 denial threshold (h_t < 2) = 2 prior non-bottom support events
    Hand-chosen threshold in Eq. (4) that denies unauthorized answer release in the diagnostic; not used in the strongest runtime results.
assumptions (4)
  • domain assumption Reference answers are correct and authorization records are authentic and scoped to the problem
    Stated TCB assumption in Section 3.1; if false, the gate can enforce the wrong target or authorize the wrong release.
  • domain assumption User text cannot modify TCB state and all candidate output passes through the release function
    The complete-mediation invariant in Section 3.1 depends on this; compromised platform code or unlogged output channels are explicitly out of scope.
  • domain assumption ASSISTments event rows are usable proxies for disclosure contracts in the diagnostic lane
    Table 1 states the runtime contracts and logged event proxies are not equivalent constructs, so this assumption is weak and only used for descriptive diagnostics.
  • domain assumption Majority votes of a same-provider three-model Gemini panel are a valid measure of pedagogical leakage
    Used for all headline endpoints; Section 4.4 human calibration shows only .104 sensitivity, so this assumption is partially contradicted.
invented entities (2)
  • Disclosure contracts A0-A4 (elicit, scaffold, concept, procedure, answer release) independent evidence
    purpose: Define the discrete authorization levels that the release gate enforces and that trace records attribute failures to.
    The contracts are directly implemented in the checker, fallback templates, and runtime tests, giving them a falsifiable operational handle rather than being purely decorative.
  • Pedagogical leakage independent evidence
    purpose: Names the protected failure mode: unauthorized answer disclosure, decisive calculation disclosure, or over-constraining the search.
    Operationalized through checker rules and human ratings; the human calibration provides an external handle, though the automatic panel is insensitive.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Auditable Release Control for Pedagogical Leakage in LLM Tutors." pith.science (2026). https://pith.science/paper/WJ7GWSZ3

@misc{pith2026260800515,
  author       = {Pith},
  title        = {Pith review of: Auditable Release Control for Pedagogical Leakage in LLM Tutors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WJ7GWSZ3}},
  note         = {Machine review of arXiv:2608.00515}
}
read the original abstract

Large language model tutors can be correct and helpful yet disclose an answer or decisive reasoning before that disclosure is authorized. We formalize this state- and action-dependent failure as pedagogical leakage and introduce an authorization-aware complete-mediation boundary. A selector emits one of five disclosure contracts, trusted policy gates privileged modes, and a renderer proposes language. A single release function applies inspectable checks, optional cumulative verification, and action-specific fallback; replayable traces separate selection, generation, verification, and enforcement failures. Matched component attribution exposes a safety-utility frontier. On 599 fixed Gemini 3.5 proposals, strict mediation reduces blinded three-model panel-majority leakage flags from 181 to 0 (paired problem-cluster difference -30.22 points, 95% CI [-35.00,-25.72]), while replacing 581 responses and lowering helpfulness. Checker-triggered fallback alone yields 11 majority flags; adding the semantic verifier yields 14 and no reliable marginal gain. A global A1 scaffold yields 0 majority and 54 any-judge flags, outperforming fitted Q on automatic safety and utility. In an externally timestamped replication over 40 unseen problem clusters and 480 attack sequences, high-assurance release reduces majority flags from 42 to 8 (-7.08 points, 95% CI [-13.13,-2.29]); seven failures persist, one is introduced, and mean helpfulness falls by .192. These results establish an auditable release boundary and failure attribution under declared contracts, not universal semantic safety or learning gains.

Figures

Figures reproduced from arXiv: 2608.00515 by the authors.

Figure 1
Figure 1. Auditable release path. Permissive mode applies the deterministic contract checker; high-assurance mode also requires [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 12 canonical work pages

  1. [10]

    Yilin Jiang, Mingzi Zhang, Xuanyu Yin, Sheng Jin, Suyu Lu, Zuocan Ying, Zengyi Yu, and Xiangjie Kong

    doi: 10.18653/v1/P17-1141. Yilin Jiang, Mingzi Zhang, Xuanyu Yin, Sheng Jin, Suyu Lu, Zuocan Ying, Zengyi Yu, and Xiangjie Kong. EduGuardBench: A holistic benchmark for evaluating the pedagogical fidelity and adversarial safety of LLMs as simulated teachers. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 31356– 31364,

    doi:10.18653/v1/p17-1141

  2. [13]

    URL https://aclanthology

    doi: 10.18653/v1/2026.acl-long.875. URL https://aclanthology. org/2026.acl-long.875/. Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 1179–1191,

    work page doi:10.18653/v1/2026.acl-long.875 2026

  3. [15]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643,

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643,

    arXiv 2005

  4. [16]

    VerifyBench: A systematic benchmark for evaluating reasoning verifiers across domains

    Xuzhao Li, Xuchen Li, Shiyu Hu, Yongzhen Guo, and Wentao Zhang. VerifyBench: A systematic benchmark for evaluating reasoning verifiers across domains. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 31796–31804, 2026a. URL https://ojs.aaai.org/index.php/AAAI/article/view/40448. Zechen Li, Qiannan Zhu, Mei Wang, Jia Li, an...

    work page 2026

  5. [17]

    Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bha- gavatula, and Yejin Choi

    URL https://aclanthology.org/2024.lrec-main.237/. Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bha- gavatula, and Yejin Choi. NeuroLogic decoding: (un)supervised neural text generation with predicate logic constraints. InProceed- ings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hum...

    work page 2024

  6. [18]

    Alexander Scarlatos, Jaewook Lee, Simon Woodhead, and Andrew Lan

    URL https://arxiv.org/abs/2503.06424. Alexander Scarlatos, Jaewook Lee, Simon Woodhead, and Andrew Lan. Simulated students in tutoring dialogues: Substance or il- lusion? InProceedings of the 64th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Pa- pers), pages 42349–42385. Association for Computational Lin- guistics,

    arXiv

  7. [19]

    URL https://aclanthology.org/2023.findings-emnlp.372/

    doi: 10.18653/v1/2023.findings-emnlp.372. URL https://aclanthology.org/2023.findings-emnlp.372/. Jakub Macina, Nico Daheim, Ido Hakimi, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. MathTutorBench: A bench- mark for measuring open-ended pedagogical capabilities of LLM tutors. InProceedings of the 2025 Conference on Empirical Meth- ods in Natural Langua...

    doi:10.18653/v1/2023.findings-emnlp.372 2023

  8. [20]

    URL https://aclanthology.org/ 2025.emnlp-main.11/

    doi: 10.18653/v1/2025.emnlp-main.11. URL https://aclanthology.org/ 2025.emnlp-main.11/. Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications to educational games. InProceedings of the 13th International Conference on Autonomous Agents and Multiagent Systems, pages 1...

    doi:10.18653/v1/2025.emnlp-main.11 2025

Show all 32 references
  1. [22]

    URL https://aclanthology.org/2025.naacl-long.57/

    doi: 10.18653/v1/2025.naacl-long.57. URL https://aclanthology.org/2025.naacl-long.57/. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wain- wright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Kata- rina Slama, Alex Ray, et al. Training language models to follow i...

    2025 doi

  2. [23]

    Towards the pedagogical steering of large language models for tutoring: A case study with modeling productive failure

    8 Romain Puech, Jakub Macina, Julia Chatain, Mrinmaya Sachan, and Manu Kapur. Towards the pedagogical steering of large language models for tutoring: A case study with modeling productive failure. InFindings of the Association for Computational Linguistics: ACL 2025, pages 262...

    2025

  3. [24]

    URL https://aclanthology.org/2025.findings-acl.1348/

    doi: 10.18653/v1/2025.findings-acl.1348. URL https://aclanthology.org/2025.findings-acl.1348/. Alexander Scarlatos, Naiming Liu, Jaewook Lee, Richard Baraniuk, and Andrew Lan. Training LLM-based tutors to improve student learning outcomes in dialogues. InArtificial Intelligenc...

    2025 doi

  4. [26]

    URL https://aclanthology.org/2026.acl-long.1960/

    doi: 10.18653/v1/2026.acl-long.1960. URL https://aclanthology.org/2026.acl-long.1960/. Weiyan Shi, Shivam Srivastava, Kaixiang Zhang, Tong Yu, Dinesh Goyal, and Aldo Lipani. Beyond pedagogical principles: Multi- horizon preference optimization for efficient socratic tutoring. ...

    2026 doi

  5. [27]

    URL https://aclanthology.org/ 2026.acl-long.518/

    doi: 10.18653/v1/2026.acl-long.518. URL https://aclanthology.org/ 2026.acl-long.518/. Shashank Sonkar, Kangqi Ni, Sapana Chaudhary, and Richard Bara- niuk. Pedagogical alignment of large language models. InFind- ings of the Association for Computational Linguistics: EMNLP 2024...

    2026 doi

  6. [28]

    URL https://aclanthology.org/2024.findings-emnlp.797/

    doi: 10.18653/v1/2024.findings-emnlp.797. URL https://aclanthology.org/2024.findings-emnlp.797/. Rakshith S. Srinivasa, Zora Che, Chen Bo Calvin Zhang, Diego Mares, Ernesto Hernandez, et al. Tutorbench: A benchmark to as- sess tutoring capabilities of large language models.arX...

    2024

  7. [29]

    The AI teacher test: Measuring the pedagogical ability of blender and GPT-3 in educational dialogues

    Ana¨ıs Tack and Chris Piech. The AI teacher test: Measuring the pedagogical ability of blender and GPT-3 in educational dialogues. arXiv preprint arXiv:2205.07540,

    arXiv

  8. [30]

    Replacing judges with juries: Evaluating LLM generations with a panel of diverse models.arXiv preprint arXiv:2404.18796,

    Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Alek- sandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating LLM generations with a panel of diverse models.arXiv preprint arXiv:2404.18796,

    arXiv

  9. [32]

    Jin Zhao, Marta Kne ˇzevi´c, and Tanja K ¨aser

    URL https://ojs.aaai.org/index.php/AAAI/ article/view/40785. Jin Zhao, Marta Kne ˇzevi´c, and Tanja K ¨aser. Evaluating answer leakage robustness of LLM tutors against adversarial student at- tacks. InProceedings of the 64th Annual Meeting of the As- sociation for Computationa...

    2026 doi

  10. [1957]

    Open problems and fundamen- tal limitations of reinforcement learning from human feedback

    Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jeremy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamen- tal limitations of reinforcement learning from human feedback. arXiv preprint arXiv...

    arXiv

  11. [1976]

    Xiaoyun Zhang, Zhengyue Zhao, Wenxuan Shi, Kaidi Xu, Di Huang, and Xing Hu

    doi: 10.1111/j.1469-7610.1976.tb00381.x. Xiaoyun Zhang, Zhengyue Zhao, Wenxuan Shi, Kaidi Xu, Di Huang, and Xing Hu. Safety alignment of large language models via contrasting safe and harmful distributions. InProceedings of the AAAI Conference on Artificial Intelligence, volum...

    1976

  12. [1995]

    From problem-solving to teach- ing problem-solving: Aligning LLMs with pedagogy using re- inforcement learning

    David Dinucu-Jianu, Jakub Macina, Nico Daheim, Ido Hakimi, Iryna Gurevych, and Mrinmaya Sachan. From problem-solving to teach- ing problem-solving: Aligning LLMs with pedagogy using re- inforcement learning. InProceedings of the 2025 Conference on Empirical Methods in Natural ...

    2025

  13. [2008]

    Enkelejda Kasneci, Kathrin Sessler, Stefan K¨uchemann, Maria Ban- nert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan G¨unnemann, Eyke H¨ullermeier, et al

    doi: 10.1080/07370000802212669. Enkelejda Kasneci, Kathrin Sessler, Stefan K¨uchemann, Maria Ban- nert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan G¨unnemann, Eyke H¨ullermeier, et al. ChatGPT for good? on opportunities and challenges of large language m...

    doi

  14. [2009]

    Ge Gao, Xi Yang, and Min Chi

    doi: 10.1007/s11257-009-9063-7. Ge Gao, Xi Yang, and Min Chi. Get a head start: On-demand peda- gogical policy selection in intelligent tutoring.Proceedings of the AAAI Conference on Artificial Intelligence, 38(11):12136–12144,

    doi

  15. [2014]

    Unifying AI tutor evaluation: An evaluation taxonomy for pedagogical ability assessment of LLM-powered AI tutors

    Kaushal Kumar Maurya, Kv Aditya Srivatsa, Kseniia Petukhova, and Ekaterina Kochmar. Unifying AI tutor evaluation: An evaluation taxonomy for pedagogical ability assessment of LLM-powered AI tutors. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of ...

    2025

  16. [2017]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    arXiv

  17. [2018]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al

    doi: 10.1609/aaai.v32i1.11797. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprin...

    arXiv

  18. [2020]

    LearnLM: Improving Gemini for learning

    LearnLM Team et al. LearnLM: Improving Gemini for learning. arXiv preprint arXiv:2412.16429,

    arXiv

  19. [2021]

    naacl-main.339

    doi: 10.18653/v1/2021. naacl-main.339. Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. Math- Dial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems. InFindings of ...

    2021 doi

  20. [2022]

    Ryan S. J. d. Baker and Kalina Yacef. The state of educational data mining in 2009: A review and future visions.Journal of Educational Data Mining, 1(1):3–17,

    2009

  21. [2023]

    Sebastian Kobler, Matthew Clemson, Angela Sun, and Jonathan K

    doi: 10.1016/j.lindif.2023.102274. Sebastian Kobler, Matthew Clemson, Angela Sun, and Jonathan K. Kummerfeld. Your students don’t use LLMs like you wish they did. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),...

    2023

  22. [2024]

    Google AI for Developers

    doi: 10.1609/aaai.v38i11.29102. Google AI for Developers. LearnLM. https://ai.google.dev/gemini- api/docs/learnlm,

    doi

  23. [2025]

    URL https://aclanthology.org/ 2025.emnlp-main.15/

    doi: 10.18653/v1/2025.emnlp-main.15. URL https://aclanthology.org/ 2025.emnlp-main.15/. Zhiang Dong, Zhenlong Dai, Xiangwei Lv, and Jingyuan Chen. Learning from long-term engagement: Adaptive tutoring dia- logue planning for personalized education. InProceedings of the AAAI Co...

    2025 doi

  24. [2026]

    SafeTutors: Benchmarking pedagogical safety in AI tutoring systems.arXiv preprint arXiv:2603.17373,

    Rima Hazra, Bikram Ghuku, Ilona Marchenko, Yaroslava Tokarieva, Sayan Layek, Somnath Banerjee, Julia Stoyanovich, and Mykola Pechenizkiy. SafeTutors: Benchmarking pedagogical safety in AI tutoring systems.arXiv preprint arXiv:2603.17373,

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.