Pith. sign in

REVIEW 3 major objections 6 minor 25 references

Constructing a Question-Answering Simulator through the Distillation of LLMs

T0 review · 3 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LDSim distills an LLM's domain knowledge and reasoning into a lightweight network that predicts students' responses faster and more accurately than either LLM-free or LLM-based simulators.

desk verdict A plausible and useful lightweight LLM-distillation simulator with a real temporal-mismatch problem in its training target and an under-specified evaluation; deserves peer review for the subfield. read the letter →

arxiv 2509.09226 v1 pith:5WGMQDFF submitted 2025-09-11 cs.LG cs.IR

Haipeng Liu , Ting Long , Jing Fu This is my paper · ORCID
classification cs.LGcs.IR
keywords question-answeringsimulatorknowledgetracingLLMdistillationeducationalrecommendersystemsconceptprerequisitegraphmasterypredictionlightweightstudentmodelmulti-stepsimulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes LDSim, a method that transfers the knowledge and reasoning skills of a large language model into a small, fast student simulator. The goal is to give educational recommender systems a cheap, safe way to train against simulated students instead of real ones, while keeping the accuracy that only LLM-based simulators had achieved. Across four datasets and two simulation settings, LDSim reports accuracy and AUC gains of 2% to 20% over ten existing methods, and it completes a 30-step student simulation in 0.73 seconds compared to 10 seconds to nearly 50 minutes for LLM-based baselines. If correct, this means the benefit of LLM-driven student modeling can be captured in a model small enough for real-time deployment.

What carries the argument

The central mechanism is the two-stage distillation pipeline plus a light-weight simulation module. Knowledge distillation converts LLM judgments into a prerequisite graph over concepts; reasoning distillation converts LLM judgments into per-step mastery scores (on real and pseudo QA records); and the simulation module—built on a graph attention network over the concept-question graph and a sampled mastery-level embedding—learns to predict both mastery and response correctness. The mastery regression loss (Eq. 15) is what injects the LLM's reasoning into the small model, and it is the component whose removal causes the largest drop in performance.

What would settle it

Train LDSim with the reasoning-distillation module fed mastery labels computed from an LLM that is given only the student's past records (no future responses). If accuracy and AUC drop to the level of the 'w/o RD' ablation, that would show the lookahead information in the original labels is what carries the reported gains. Alternatively, shuffle the mastery labels randomly while keeping the model pipeline intact; if performance remains near the reported numbers, the reasoning distillation is not actually contributing.

Watch

Extended reading notes

Core claim

LDSim treats the LLM not as the simulator itself but as a teacher that generates two kinds of distilled training signals. A knowledge distillation module prompts the LLM to judge whether pairs of concepts are related and which are prerequisites, producing a concept-relation graph. A reasoning distillation module prompts the LLM to estimate a student's mastery of each concept at each step, using both real records and synthesized pseudo-questions, then trains a lightweight graph-attention network to predict those mastery scores from the student's past-only state. In the final stage, the compact model is fine-tuned to predict response correctness while also predicting mastery, so that the disti

Load-bearing premise

The reasoning-distillation module assumes that the LLM's numerical mastery scores, produced with access to the student's future responses, are accurate enough and can be learned by the small model from the student's past-only state; if those labels are noisy or depend on information not available at inference time, the central performance gain collapses.

Editorial extensions

If this is right

  • Educational recommender systems can be trained offline against simulated students, removing the risk of students receiving harmful or random recommendations from an undertrained recommender.
  • Real-time interactive tutoring becomes feasible with LLM-level simulation accuracy at sub-second inference and modest GPU memory, replacing slow LLM-based simulators in live settings.
  • The distilled concept-prerequisite graph is a reusable knowledge artifact that could improve other educational models that need curriculum structure.
  • Because the method is evaluated on both the simulation task and knowledge tracing, the same distilled model can serve both as an environment for recommender training and as a student-state estimator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the LLM is prompted with the student's full record including the target response and later questions (Eq. 6), the distilled mastery labels carry lookahead information; the lightweight model must learn to infer them from a past-only state, so the method implicitly assumes that 'hindsight' mastery is learnable from pre-response state—a transfer assumption not explicitly tested in the paper.
  • The pseudo-question augmentation, which can be extended to arbitrary questions, may function as a general data-augmentation trick for knowledge tracing, not limited to the four datasets tested.
  • Using a stronger or larger LLM than GLM-4-Flash as the teacher might yield better concept graphs and mastery labels; whether the gains scale with teacher quality is a direct, testable extension.
  • If the concept graph were obtained from a cheaper source (e.g., textbooks or curated knowledge graphs) and mastery labels from a smaller model, the method might still retain a usable fraction of the benefit, which the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes LDSim, a QA simulator that distills LLM domain knowledge and reasoning into a lightweight model. Knowledge distillation (KD) builds a concept prerequisite graph via LLM prompts; reasoning distillation (RD) generates per-step student mastery scores via LLM prompts; a simulation module (Sim), based on GAT and attention, predicts response correctness. Experiments on four datasets compare LDSim with ten LLM-free and LLM-based baselines in multi-step and single-step simulation. LDSim reports higher ACC/AUC than all baselines, with inference times far below LLM-based methods. Ablations attribute the largest gain to RD, and a case study illustrates the learned concept graph and embeddings.

Significance. If the results hold, the paper provides a practical way to retain much of the benefit of LLM-based knowledge tracing in a lightweight simulator, which is relevant for safely training educational recommender systems. Strengths include the multi-dataset evaluation, the multi-step and single-step settings, the deployment-cost measurements, and the public code link. The evaluation is against real student responses, so the headline result is not circular. However, the central distillation mechanism is weakened by a temporal mismatch in the RD target, and the statistical support is incomplete. The work is therefore promising but needs a major revision before the claims can be accepted.

major comments (3)
  1. [§4.2, Eq. (6), Fig. 5] The RD training target uses non-causal information. Eq. (6) defines the LLM mastery label m_i from the entire QA record (q_t,C_t,r_t)|_{t=1}^T, and Figure 5 explicitly asks the LLM to re-evaluate mastery 'after answering this question' using 'subsequent questions'. At inference, Eq. (10) computes the state s_i only from records strictly before step i, and Eq. (15) trains MLP(s_i) to predict m_i. Thus the model is trained to predict a label that depends on the current response and future records it can never observe. Since the ablation (Figure 6) shows that removing RD causes the largest performance drop, the claimed gain is largely attributed to this distillation signal. The paper does not demonstrate that the future-dependent component is negligible or recoverable. Please re-run RD with a causal prompt using only records before step i (and possibly the current question, but not its resp
  2. [§5.4, Tables 1 and 4] The statistical support is underspecified. The tables mark results with '* p-value < 0.01', but no significance test is described: no test name, no number of seeds/runs, no error bars, no measure of variance. Some differences are modest (e.g., single-step ACC on Junyi is 0.8217 vs 0.8022 for SinKT), and without variance or a described paired test the claimed universal superiority is not established. Please report standard deviations over multiple runs and specify the significance test used.
  3. [§5.2 and Tables 1/4] The LLM-based baseline setup is underspecified and may handicap those methods. LLM-KT is described as a fine-tuning-based method, yet its reported AUC values are near or below 0.5 (0.4767–0.5421), which is suspicious for a published KT method. The text says 'for baselines that only require prompting an LLM via API calls, we uniformly adopt GLM-4-Flash', but it is unclear whether LLM-KT was actually fine-tuned, and whether SinKT and Agent4Edu were given concept descriptions. Please provide the exact input format, base model, and training procedure for each LLM-based baseline. If LLM-KT's poor performance is due to an implementation mismatch, the comparison understates a strong baseline.
minor comments (6)
  1. [Abstract and Section 1] Typos: 'enhence' should be 'enhance'; 'Sim devices' should be 'Sim devises'; 'trainning' should be 'training'.
  2. [Figure 6] Panel (d) is labeled 'algebra0809' instead of a consistent dataset name, and the figure lacks y-axis labels and error bars; please add them.
  3. [Eq. (6) and Section 4.2] The notation is inconsistent: Eq. (6) uses m_t and s_t, while the text and Eq. (15) use m_i and s_i. Also, the 'credit score' s_i output by the LLM is never used in any objective; either remove it or explain its role.
  4. [Eq. (11) and Eq. (18)] The training objective involving discrete sampling is not fully specified. Eq. (11) samples a mastery level k, but the paper does not explain how gradients flow through this discrete operation, and L_c in Eq. (18) uses an indicator I(r_i = \hat{r}_i) that makes the loss active only when the prediction is correct. Please clarify the optimization (e.g., straight-through estimator, REINFORCE, or another method).
  5. [Section 5.3] The hyperparameter β in Eq. (16) is set to 40 without justification; a sensitivity analysis would help. Also, the text says 'we set n=30 in Eq. (3)' but the role of n in the multi-step evaluation should be stated more explicitly.
  6. [Related Work] The claim 'first attempt to apply LLM distillation in the educational domain' is too broad. There is existing work on distilling LLMs for educational tasks; please narrow the claim to the specific task of QA simulation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: final objective is ground-truth BCE; LLM signals are auxiliary.

full rationale

The paper's central claim is that LDSim outperforms baselines in simulating student responses. The final training and evaluation objective (Eq. 17, Tables 1 and 4) is binary cross-entropy against real student responses r_i; it is not measured against any LLM output. The LLM-generated concept graph (KD) and mastery scores (RD) enter only as auxiliary training signals. In particular, Eq. 15 trains a mastery estimator to match LLM mastery, but this mastery is not the evaluation target, and the final prediction (Eq. 12) uses a sampled mastery level whose classifier is also trained with a self-consistency loss (Eq. 18) gated on the model's own correctness. No equation reduces to its own input by construction: the KD graph is generated from an LLM prompt, not from the model's embeddings; the RD mastery label is generated by an LLM, not from the model's state. The self-citations (DSim [15], HierLLM [13], and the loss-inspiration citation [14]) are non-load-bearing: they are baseline comparisons or general inspiration, and none supplies a theorem or assumption on which the derivation depends. The RD training target does depend on future responses (Eq. 6 includes (q_t,C_t,r_t)|_{t=1}^T), which raises a causal-validity question about the auxiliary loss, but this is a methodological concern, not a circularity, because the model is not defined in terms of the target and the final metric is external.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper's central claim rests on assumptions about LLM reliability and the validity of distillation targets, plus several hand-set hyperparameters. No new physical or external entities are introduced.

free parameters (6)
  • beta (β in Eq. 16) = 40
    Weight balancing L_c and L_p, set by hand without sensitivity analysis.
  • simulation horizon n = 30
    Number of future steps to simulate; also used for PQA augmentation count, apparently.
  • embedding dimension d = 128
    Hidden size for concept and question embeddings.
  • max history length = 200
    QA history truncated to 200 records per student.
  • mastery level count l = not specified
    Number of discrete mastery levels used in Eq. 11; not reported.
  • LLM temperature / randomness control = not specified
    LLM generation randomness is acknowledged but temperature or version is not given.
assumptions (4)
  • domain assumption A student's latent mastery of concepts is a key factor influencing correctness (IRT assumption)
    Invoked in Section 4.2 and Eq. 8-12 to justify modeling mastery.
  • domain assumption The LLM can reliably assess concept relatedness, prerequisites, and student mastery from the prompted history
    KD and RD modules depend on this; the paper only swaps positions to reduce randomness, with no validation of the LLM judgments.
  • domain assumption Teacher-forcing to free-running transfer: a simulator trained on ground-truth histories remains accurate when conditioned on its own synthetic predictions
    Multi-step evaluation replaces ground-truth responses with predicted ones, while training uses only real history; no scheduled sampling or free-running training is applied.
  • standard math GAT and attention architectures are valid building blocks for concept-question encoding
    Eq. 7-10 rely on established architectures without further justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Constructing a Question-Answering Simulator through the Distillation of LLMs." pith.science (2026). https://pith.science/paper/5WGMQDFF

@misc{pith2026250909226,
  author       = {Pith},
  title        = {Pith review of: Constructing a Question-Answering Simulator through the Distillation of LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5WGMQDFF}},
  note         = {Machine review of arXiv:2509.09226}
}
read the original abstract

The question-answering (QA) simulator is a model that mimics real student learning behaviors and predicts their correctness of their responses to questions. QA simulators enable educational recommender systems (ERS) to collect large amounts of training data without interacting with real students, thereby preventing harmful recommendations made by an undertrained ERS from undermining actual student learning. Given the QA history, there are two categories of solutions to predict the correctness, conducting the simulation: (1) LLM-free methods, which apply a traditional sequential model to transfer the QA history into a vector representation first, and make predictions based on the representation; (2) LLM-based methods, which leverage the domain knowledge and reasoning capability of LLM to enhence the prediction. LLM-free methods offer fast inference but generally yield suboptimal performance. In contrast, most LLM-based methods achieve better results, but at the cost of slower inference speed and higher GPU memory consumption. In this paper, we propose a method named LLM Distillation based Simulator (LDSim), which distills domain knowledge and reasoning capability from an LLM to better assist prediction, thereby improving simulation performance. Extensive experiments demonstrate that our LDSim achieves strong results on both the simulation task and the knowledge tracing (KT) task. Our code is publicly available at https://anonymous.4open.science/r/LDSim-05A9.

Figures

Figures reproduced from arXiv: 2509.09226 by the authors.

Figure 1
Figure 1. Illustration of the role and tasks of the QA simula [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The pipeline of our method. In an online education platform, there are learning materials covering various concepts. You are an education expert. We would like you to determine whether the knowledge concept [concept i] and the knowledge concept [concept j] belong to the same domain, and whether they are related. Please first output your judgment, then provide an explanation. Your judgment should be chosen from the f… view at source ↗
Figure 3
Figure 3. prompt template of P𝑟 , where [concept i] and [con￾cept j] correspond to TEXT(𝑐𝑖) and TEXT(𝑐𝑗) in Equation 4. and synthesize pseudo QA (PQA) records (𝑞 𝑖 𝑡 , C 𝑖 𝑡 , 𝑟𝑖 𝑡 ,𝑚𝑖 𝑡 , 𝑠𝑖 𝑡 )|𝑛 𝑖=1 by conducting the operation in Eq. 6, assuming the student answered the another 𝑛 questions in parallelized step. For each PQA record (𝑞 𝑖 𝑡 , C 𝑖 𝑡 , 𝑟𝑖 𝑡 ,𝑚𝑖 𝑡 , 𝑠𝑖 𝑡 ), 𝑞 𝑖 𝑡 is the randomly selected question. C 𝑖 𝑡 is the c… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: An example of P𝑚, where the mastery information in the prompt at step 𝑖 comes from 𝑚𝑡 | 𝑖−1 𝑡=1 . we will discuss how the simulation module makes predictions to simulate students’ QA behavior first, then we will discuss how to optimize the simulation module. Correctnes…
Figure 6
Figure 6. Figure 6: Ablation Study : Ordering Positive Decimals : Addition and Subtraction Positive Decimals : Percent Increase or Decrease : Area Parallelogram : Surface Area Of Prism t+1 t+2 t+3 t+4 t+5 0.85 0.82 0.70 0.65 0.80 0.92 0.94 0.86 0.87 0.89 step concept response LLM LDSim (a…
Figure 7
Figure 7. Figure 7: Case study 5.6 Ablation Study To further investigate the contribution of each module in LDSim, we conducted an ablation study by removing the world knowledge dis￾tillation module (w/o KD) and the reasoning capability distillation module (w/o RD). For w/o KD, we replace…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 9 linked inside Pith

  1. [1]

    Berk Atil, Alexa Chittams, Liseng Fu, Ferhan Ture, Lixinyu Xu, and Breck Baldwin

  2. [2]

    Xianyu Chen, Jian Shen, Wei Xia, Jiarui Jin, Yakun Song, Weinan Zhang, Wei- wen Liu, Menghui Zhu, Ruiming Tang, Kai Dong, et al. 2023. Set-to-Sequence Ranking-based Concept-aware Learning Path Recommendation.arXiv preprint arXiv:2306.04234(2023)

    arXiv 2023

  3. [3]

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2021. Glm: General language model pretraining with autoregressive blank infilling.arXiv preprint arXiv:2103.10360(2021)

    arXiv 2021

  4. [4]

    Mingyu Feng, Neil Heffernan, and Kenneth Koedinger. 2009. Addressing the assessment challenge with an online system that tutors as it assesses.User modeling and user-adapted interaction19 (2009), 243–266. Constructing a Question-Answering Simulator through the Distillation of LLMs Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Table 4: ACC and AU...

    2009

  5. [5]

    Lingyue Fu, Hao Guan, Kounianhua Du, Jianghao Lin, Wei Xia, Weinan Zhang, Ruiming Tang, Yasheng Wang, and Yong Yu. 2024. Sinkt: A structure-aware inductive knowledge tracing model with large language model. InProceedings of the 33rd ACM International Conference on Information and Knowledge Management. 632–642

    2024

  6. [6]

    Weibo Gao, Qi Liu, Linan Yue, Fangzhou Yao, Rui Lv, Zheng Zhang, Hao Wang, and Zhenya Huang. 2025. Agent4edu: Generating learner response data by generative agents for intelligent education systems. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 23923–23932

    2025

  7. [7]

    Aritra Ghosh, Neil Heffernan, and Andrew S Lan. 2020. Context-aware atten- tive knowledge tracing. InProceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 2330–2339

    2020

  8. [9]

    Xiaopeng Guo, Zhijie Huang, Jie Gao, Mingyu Shang, Maojing Shu, and Jun Sun

Show all 25 references
  1. [10]

    Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory.Neural computation9, 8 (1997), 1735–1780

    1997

  2. [11]

    Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic opti- mization.arXiv preprint arXiv:1412.6980(2014)

    2014 arXiv

  3. [12]

    InProceedings of the 29th ACM international conference on multimedia

    Enhancing knowledge tracing via adversarial training. InProceedings of the 29th ACM international conference on multimedia. 367–375

  4. [13]

    Yuxuan Liu, Haipeng Liu, and Ting Long. 2024. HierLLM: Hierarchical Large Language Model for Question Recommendation.arXiv preprint arXiv:2409.06177 (2024)

    2024 arXiv

  5. [14]

    Ting Long, Yunfei Liu, Jian Shen, Weinan Zhang, and Yong Yu. 2021. Tracing knowledge state with individual cognition and acquisition estimation. InProceed- ings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 173–182

    2021

  6. [15]

    Yoshiki Kubotani, Yoshihiro Fukuhara, and Shigeo Morishima. 2021. RLTutor: Reinforcement Learning Based Adaptive Tutoring System by Modeling Virtual Student with Fewer Interactions.arXiv preprint arXiv:2108.00268(2021)

    2021 arXiv

  7. [16]

    2012.Applications of item response theory to practical testing problems

    Frederic M Lord. 2012.Applications of item response theory to practical testing problems. Routledge

    2012

  8. [17]

    Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2025. An empirical study of the non-determinism of chatgpt in code generation.ACM Transactions on Software Engineering and Methodology34, 2 (2025), 1–28

    2025

  9. [18]

    Ting Long, Li’ang Yin, Yi Chang, Wei Xia, and Yong Yu. 2025. Simulating Question- answering Correctness with a Conditional Diffusion. InProceedings of the ACM on Web Conference 2025. 5173–5182

    2025

  10. [19]

    Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. 2015. Deep knowledge tracing. Advances in neural information processing systems28 (2015)

    2015

  11. [20]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information processing systems30 (2017)

    2017

  12. [21]

    S Pandey and G Karypis. 2019. A self-attentive model for knowledge tracing. arXiv 2019.arXiv preprint arXiv:1907.06837(2019)

    2019 arXiv

  13. [22]

    Ziwei Wang, Jie Zhou, Qin Chen, Min Zhang, Bo Jiang, Aimin Zhou, Qinchun Bai, and Liang He. 2025. LLM-KT: Aligning Large Language Models with Knowledge Tracing using a Plug-and-Play Instruction.arXiv preprint arXiv:2502.02945(2025)

    2025 arXiv

  14. [23]

    Chun-Kit Yeung. 2019. Deep-IRT: Make deep learning based knowledge tracing explainable using item response theory.arXiv preprint arXiv:1904.11738(2019)

    2019 arXiv

  15. [24]

    Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks.arXiv preprint arXiv:1710.10903(2017)

    2017 arXiv

  16. [27]

    Yiyun Zhou, Zheqi Lv, Shengyu Zhang, and Jingyuan Chen. 2025. Disentangled knowledge tracing for alleviating cognitive bias. InProceedings of the ACM on Web Conference 2025. 2633–2645. Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

    2025

  17. [2024]

    LLM Stability: A detailed analysis with some surprises.arXiv e-prints(2024), arXiv–2408

    2024

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.