pith. sign in

arxiv: 2605.16207 · v1 · pith:MUPZHHOXnew · submitted 2026-05-15 · 💻 cs.AI · cs.CL

Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most

Tahreem Yasir , Wenbo Li , Sam Gilson , Sutapa Dey Tithi , Xiaoyi Tian , Tiffany Barnes This is my paper

Pith reviewed 2026-05-20 18:54 UTC · model grok-4.3

classification 💻 cs.AI cs.CL
keywords LLM tutoring agentspropositional logicfeedback evaluationknowledge graph ground truthdiagnostic precisionoptimal vs suboptimal solutionshybrid AI tutoringsolution classification
0
0 comments X

The pith

LLM tutoring agents correctly identify optimal solutions but over-reject valid suboptimal reasoning and over-validate incorrect ones in propositional logic.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper evaluates seven LLM feedback agents using a benchmark of more than ten thousand solution-feedback pairs drawn from propositional logic problems. Ground truth labels come from a knowledge graph that sorts student steps into optimal, valid but suboptimal, or incorrect categories. Models reach near-perfect accuracy when a step is already optimal, yet they routinely reject sound but imperfect reasoning and accept flawed solutions. This pattern is especially costly for tutoring because adaptive help requires spotting opportunities to improve without dismissing progress or allowing errors to stand. The study also finds that correct diagnosis does not automatically produce useful instructional responses, pointing toward hybrid designs that pair structured knowledge models with LLMs.

Core claim

Models achieved near-ceiling performance on optimal steps but systematically over-rejected valid but suboptimal reasoning and over-validated incorrect solutions, precisely where adaptive tutoring matters most. These failures persisted across models regardless of solution context, suggesting architectural rather than informational limits. Moreover, accurate diagnosis did not reliably produce pedagogically actionable feedback, revealing a gap between diagnostic judgment and instructional effectiveness.

What carries the argument

Benchmark of seven LLM feedback agents scored against knowledge-graph-derived ground truth that classifies each student solution step as optimal, valid-suboptimal, or incorrect across three feedback conditions.

If this is right

  • LLMs are better suited for hybrid architectures where knowledge-graph models handle precise diagnosis while LLMs manage open-ended scaffolding and dialogue.
  • Diagnostic accuracy in isolation does not ensure pedagogically effective feedback.
  • The observed misclassifications hold across different models and contexts, indicating limits that are not fixed by adding more information.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Students using these agents might receive discouraging rejections for reasonable work or continue with undetected mistakes, slowing learning.
  • The same diagnostic imbalance could appear in tutoring for mathematics or coding where distinguishing good attempts from errors is equally central.
  • Replacing the knowledge graph with other structured representations might test whether the diagnosis gap is specific to the current grounding method.

Load-bearing premise

The knowledge-graph-derived ground truth accurately and comprehensively distinguishes optimal, valid but suboptimal, and incorrect student solutions across the tested propositional logic problems.

What would settle it

Re-evaluating the same models on the same problems but with human tutor labels replacing the knowledge-graph ground truth, then checking whether the over-rejection of valid-suboptimal steps and over-validation of incorrect steps remain.

Figures

Figures reproduced from arXiv: 2605.16207 by Sam Gilson, Sutapa Dey Tithi, Tahreem Yasir, Tiffany Barnes, Wenbo Li, Xiaoyi Tian.

Figure 1
Figure 1. Figure 1: Optimal and valid-alternative solutions (blue nodes represent abbreviated inference rule names, ex￾plained in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: KG-grounded evaluation pipeline. (1) Proof state and expert solution are extracted from tutoring system. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Mean feedback quality scores across rubric dimen￾sions (N=100). 3) [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: –6 illustrates a representative student in￾teraction in the propositional logic tutor. These screenshots demonstrate forward chaining, rule ap￾plication, and goal completion within the tutor in￾terface [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Rule application with guided simplifica￾tion. After deriving an intermediate conjunction via Modus Ponens, the student applies the Simplification rule. The interface prompts the learner to select the appropriate resulting literal (G or ¬H), illustrating fine￾grained, step-level decision making supported by rule constraints [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Successful proof completion. The student derives J via Disjunctive Syllogism and applies the Addition rule to reach the target conclusion J ∨ K. The system confirms completion, reinforcing correct rule sequencing and alignment with the goal state. D Representative Proof State This appendix presents an illustrative propositional logic proof instance [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Illustrative proof instance showing intermedi [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 10
Figure 10. Figure 10: Teacher (complete hint) feedback reveals solution on incorrect student response Proof State (Excerpt) (S → D), ((¬S ∨ Q) → Y ), ¬D, (¬D → ¬I), ... Student Reasoning: ¬S is optimal because we already have (S → D) from INTERMEDI￾ATE STEPS, and simplifying it to ¬S allows us to construct (¬S ∨ Q)... Peer Solution context: ¬S Peer Feedback: Excellent reasoning! You correctly identified how simplifying (S → D)… view at source ↗
Figure 8
Figure 8. Figure 8: Example for Vague Peer Feedback - repeats student’s incorrect solution Proof state: 1.(B ↔ ¬J), 2.(¬N ∨ J), 3.(B ∨¬N), 4.((B → ¬J)∧(¬J → B)), 5.(N → J), 6.(¬B → ¬N) Student Reasoning: (B → ¬N) is optimal because it is the exact form of the conclusion, it can be derived from the contrapositive of (¬B → ¬N)... Reasoning error: Incorrect derivation Solution context:Derive(J∨¬N) from (¬N∨ J) using Commutative … view at source ↗
Figure 11
Figure 11. Figure 11: Judge Verification Response, extending In￾correct Peer feedback (anchoring on incorrect student reasoning) [PITH_FULL_IMAGE:figures/full_fig_p015_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Example for Judge Feedback repeating Peers’ mistakes - also directs towards the hint without recognizing the reasoning error H.1 Student Agent Response We show a representative Student agent response to demonstrate candidate generation, explicit rea￾soning, and next-step generation. The example corresponds to a single proof instance and high￾lights how intermediate reasoning is externalized for downstream… view at source ↗
Figure 13
Figure 13. Figure 13: Sample problem instance and corresponding [PITH_FULL_IMAGE:figures/full_fig_p016_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Base System Prompt for Student [PITH_FULL_IMAGE:figures/full_fig_p018_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Base System prompt for Peer [PITH_FULL_IMAGE:figures/full_fig_p019_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Base System prompt for Teacher [PITH_FULL_IMAGE:figures/full_fig_p020_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Base System prompt for Judge [PITH_FULL_IMAGE:figures/full_fig_p021_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Over-validation (OV) and over-rejection (OR) rates by model across complexity tiers. [PITH_FULL_IMAGE:figures/full_fig_p021_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Over-validation (OV) and over-rejection (OR) rates by model across distance-to-goal tiers. [PITH_FULL_IMAGE:figures/full_fig_p022_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Classification F1 by inference rule and model. DS: Disjunctive Syllogism, HS: Hypothetical Syllogism, [PITH_FULL_IMAGE:figures/full_fig_p022_20.png] view at source ↗
read the original abstract

Effective tutoring requires distinguishing optimal, valid but suboptimal, and incorrect student solutions, a distinction central to intelligent tutoring systems (ITS) but untested for LLM-based tutors. As LLMs are increasingly explored as conversational complements to ITS, evaluating their diagnostic precision is essential. We present a benchmark of seven LLM feedback agents in propositional logic using knowledge-graph-derived ground truth across 10,836 solution--feedback pairs and three feedback conditions. Models achieved near-ceiling performance on optimal steps but systematically over-rejected valid but suboptimal reasoning and over-validated incorrect solutions, precisely where adaptive tutoring matters most. These failures persisted across models regardless of solution context, suggesting architectural rather than informational limits. Moreover, accurate diagnosis did not reliably produce pedagogically actionable feedback, revealing a gap between diagnostic judgment and instructional effectiveness. Our findings suggest that LLMs are better suited for hybrid architectures where KG-grounded models handle diagnosis while LLMs support open-ended scaffolding and dialogue.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper evaluates seven LLM feedback agents for propositional logic tutoring using a benchmark of 10,836 solution-feedback pairs with knowledge-graph-derived ground truth across three feedback conditions. It claims models achieve near-ceiling performance on optimal steps but systematically over-reject valid but suboptimal reasoning and over-validate incorrect solutions, with accurate diagnosis not reliably yielding pedagogically actionable feedback, indicating architectural limits and favoring hybrid KG-LLM architectures.

Significance. If the results hold, the work is significant for ITS and LLM tutoring research by providing large-scale evidence of specific LLM weaknesses in adaptive feedback, a core requirement for effective tutoring. The consistency across models and use of external KG ground truth (avoiding self-referential evaluation) are strengths. The findings support practical recommendations for hybrid systems combining structured diagnosis with LLM dialogue.

major comments (1)
  1. The central claim depends on the KG ground truth accurately partitioning the 10,836 solutions into optimal / valid-but-suboptimal / incorrect categories. Propositional logic admits multiple logically equivalent derivations that differ in step count or intermediate choices. The manuscript must detail (in the evaluation or methods section) how alternative valid paths are enumerated, how logical soundness of non-canonical paths is validated, and how edge cases such as redundant but correct steps are classified; absent this, reported over-rejection of valid-suboptimal reasoning may reflect label strictness rather than tutoring limitations.
minor comments (2)
  1. The abstract states 'three feedback conditions' without naming or briefly describing them; adding this detail would improve immediate clarity.
  2. Statistical details, data exclusion criteria, and any inter-annotator agreement for the KG labels are not mentioned in the provided abstract or summary; these should be reported to support assessment of result robustness.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their positive evaluation of the work's significance and for the constructive comment regarding the construction of the knowledge-graph ground truth. We address the point below and will revise the manuscript to incorporate additional methodological detail.

read point-by-point responses

  1. Referee: The central claim depends on the KG ground truth accurately partitioning the 10,836 solutions into optimal / valid-but-suboptimal / incorrect categories. Propositional logic admits multiple logically equivalent derivations that differ in step count or intermediate choices. The manuscript must detail (in the evaluation or methods section) how alternative valid paths are enumerated, how logical soundness of non-canonical paths is validated, and how edge cases such as redundant but correct steps are classified; absent this, reported over-rejection of valid-suboptimal reasoning may reflect label strictness rather than tutoring limitations.

    Authors: We agree that greater transparency is required on this point. The current manuscript describes the KG-derived labels but does not provide a sufficiently explicit account of path enumeration and edge-case handling. In the revised version we will expand the Methods section to specify that the knowledge graph is constructed by exhaustively applying all valid inference rules to generate every reachable state, with paths enumerated via breadth-first search from the initial premises to the target conclusion. Soundness of non-canonical paths is verified by an embedded SAT solver that confirms each intermediate step preserves logical entailment. Redundant yet correct steps are labeled valid-but-suboptimal precisely when they increase path length beyond the shortest derivation found in the graph. These additions will make clear that the reported over-rejection reflects model behavior rather than label artifacts. revision: yes

Circularity Check

0 steps flagged

No significant circularity: evaluation uses independent external ground truth

full rationale

The paper's central evaluation compares LLM feedback agents against a knowledge-graph-derived ground truth on 10,836 solution-feedback pairs, partitioning solutions into optimal, valid-but-suboptimal, and incorrect categories. This ground truth is constructed independently of the tested LLMs and does not rely on any fitted parameters, self-definitions, or self-citation chains from the present work. The reported performance patterns (near-ceiling on optimal steps, over-rejection of valid-suboptimal reasoning) are direct empirical measurements against this external benchmark rather than reductions of outputs to inputs by construction. No load-bearing step in the abstract or described methodology collapses into a tautology or renames a fitted result as a prediction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that knowledge graphs provide reliable ground truth for categorizing student reasoning steps in propositional logic.

axioms (1)
  • domain assumption Knowledge graphs can accurately represent and distinguish optimal, valid but suboptimal, and incorrect reasoning paths in propositional logic.
    Invoked to generate the ground truth labels for the 10,836 solution-feedback pairs.

pith-pipeline@v0.9.0 · 5705 in / 1154 out tokens · 48945 ms · 2026-05-20T18:54:22.825927+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages

  1. [1]

    Qinjin Jia, Jialin Cui, Ruijie Xi, Chengyuan Liu, Parvez Rashid, Ruochi Li, and Edward Gehringer

    Logiclearner: A tool for the guided prac- tice of propositional logic proofs.arXiv preprint arXiv:2503.19280. Qinjin Jia, Jialin Cui, Ruijie Xi, Chengyuan Liu, Parvez Rashid, Ruochi Li, and Edward Gehringer. 2024. On assessing the faithfulness of llm-generated feedback on student assignments. InProceedings of the 17th International Conference on Education...

    work page arXiv 2024

  2. [2]

    Great job recognizing that (B→ ¬N) is the contrapositive of (¬B→ ¬N) and using that to reach the con- clusion

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information pro- cessing systems, 36:46595–46623. A Inference Rule List We employ a fixed set of propositional inference rules used in Logic Tutor for the dataset. The short names were used for consistent response genera- tion and evaluation. The complete list of inference rules, al...

    work page 1970

  3. [3]

    Review the givens and intermediate steps

    work page

  4. [4]

    Propose 2–3 candidate next steps

    work page

  5. [5]

    Select the candidate that most directly advances toward the conclusion

    work page

  6. [6]

    Constraints: • Output exactly one next step insymbolic notation only

    Justify your choice and output the selected next step. Constraints: • Output exactly one next step insymbolic notation only. • Use only predefined inference rules (e.g., MP, MT, Conj, DS). • Parent statements must be actual expressions, not line numbers. Response Format: •CANDIDATES: 2–3 candidate steps with brief justification •REASONING: Why the selecte...

    work page

  7. [7]

    Analyze how the optimal step is derived (rule and parent statements)

    work page

  8. [8]

    Evaluate the student’s candidates, reasoning, and chosen next step

    work page

  9. [9]

    Classify the student’s step asCorrect,Valid Alternative, orIncorrect

    work page

  10. [10]

    Constraints: • Do not reveal the optimal step, its rule, or parent statements

    Provide brief, scaffolded feedback guiding the student toward the optimal step. Constraints: • Do not reveal the optimal step, its rule, or parent statements. • Acknowledge what the student did correctly before addressing errors. • Use Socratic questions to guide reasoning; keep feedback concise (2–3 sentences). • Use predefined inference rule short names...

    work page

  11. [12]

    Identify errors in the student’s logic, rule usage, or reasoning

    work page

  12. [14]

    Constraints: • Do not reveal the exact next step, rule, or parent statements from the solution

    Provide brief, scaffolded feedback guiding the student toward the correct solution. Constraints: • Do not reveal the exact next step, rule, or parent statements from the solution. • Acknowledge correct aspects of the student’s attempt before addressing errors. • Use Socratic questions to guide reasoning; keep feedback concise (2–3 sentences). • Refer to t...

    work page

  13. [15]

    Compare the student’s response against the knowledge-base solution

    work page

  14. [16]

    Identify errors in the student’s reasoning, if any

    work page

  15. [17]

    Classify the student’s next step asCorrect,Valid Alternative, orIncorrect

    work page

  16. [18]

    Evaluate whether the Teacher’s feedback correctly guides the student

    work page

  17. [19]

    Constraints: • Do not reveal the exact next step, rule, or parent statements from the solution

    Either enhance the Teacher’s feedback or override it with corrected guidance. Constraints: • Do not reveal the exact next step, rule, or parent statements from the solution. • Acknowledge correct aspects of the student’s attempt before addressing errors. • Use Socratic questions to guide reasoning; scaffold rather than instruct. • Override Teacher feedbac...

    work page