ConvoLearn: A Learning Sciences Grounded Dataset for Fine-Tuning Dialogic AI Tutors

Hari Subramonyam; Mayank Sharma; Roy Pea

arxiv: 2601.08950 · v4 · submitted 2026-01-13 · cs.AI · cs.HC· cs.LG

ConvoLearn: A Learning Sciences Grounded Dataset for Fine-Tuning Dialogic AI Tutors

Mayank Sharma , Roy Pea , Hari Subramonyam This is my paper

Reviewed by Pith2026-05-16 14:23 UTCgrok-4.3open to challenge →

classification cs.AI cs.HCcs.LG

keywords dialogic tutoringAI tutorsdatasetfine-tuninglearning sciencesknowledge buildingpedagogical signalLLM alignment

0 comments

The pith

Dimension-labeled dialogues capture pedagogical signals that generalize to authentic classrooms and improve AI tutor behavior.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents ConvoLearn, a dataset of over two thousand semi-synthetic tutor-student dialogues in middle school Earth Science, each labeled according to six dimensions of dialogic tutoring from knowledge-building theory. It establishes that these labels carry real pedagogical signal by showing that a classifier trained on the dataset produces scores that correlate with expert ratings of teaching quality in actual classroom recordings across several measures. As proof of concept, the authors fine-tune an open 7 billion parameter model on the labeled data and find that credentialed teachers rate its dialogic tutoring performance as competitive with a leading proprietary system. This work aims to better align large language models with the interactive, knowledge-construction approach that defines good tutoring rather than simple information transmission.

Core claim

ConvoLearn is a dataset of 2,134 semi-synthetic tutor-student dialogues that operationalize six dimensions of dialogic tutoring grounded in knowledge-building theory within a middle school Earth Science curriculum. Scores from a classifier trained on ConvoLearn correlate significantly with expert-coded instructional quality in authentic classrooms across multiple subscales. Fine-tuning Mistral-7B on ConvoLearn using dimension-level labels steers the model to dialogic tutoring behavior that credentialed teachers rate as competitive with a strong proprietary baseline.

What carries the argument

Six dimensions of dialogic tutoring drawn from knowledge-building theory, applied as labels to semi-synthetic dialogues to create training data for classifiers and fine-tuned models.

If this is right

Classifiers trained on the dataset can evaluate instructional quality in real classrooms.
Dimension-specific fine-tuning aligns open-weight models with dialogic principles.
The resulting AI tutors receive competitive ratings from credentialed teachers.
This method supports creation of AI tutors that engage in knowledge construction rather than one-directional teaching.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Extending the dataset to additional subject areas could broaden its applicability.
Combining the labeled synthetic data with real interaction logs might strengthen the transfer to live settings.
The dimension labels provide a structured way to interpret and debug AI tutor responses.
Testing the fine-tuned models in actual student sessions would reveal practical effectiveness.

Load-bearing premise

The semi-synthetic dialogues capture the key dynamics of real student-tutor interactions well enough that patterns learned from them transfer to authentic classroom data and yield better teacher ratings.

What would settle it

Finding no significant correlation between classifier scores from ConvoLearn training and expert ratings on real classroom data, or teachers rating the fine-tuned model as substantially worse than the proprietary baseline on dialogic quality, would disprove the central claim.

read the original abstract

Despite their growing adoption in education, LLMs remain misaligned with the core principle of effective tutoring: the dialogic construction of knowledge. We introduce ConvoLearn, a dataset of 2,134 semi-synthetic tutor-student dialogues operationalizing six dimensions of dialogic tutoring grounded in knowledge-building theory, situated in a middle school Earth Science curriculum. We show that dimension-labeled dialogic training data captures meaningful pedagogical signal that generalizes beyond its semi-synthetic domain: scores from a classifier trained on ConvoLearn correlate significantly with expert-coded instructional quality in authentic classrooms across multiple subscales. As a proof of concept, we fine-tune Mistral-7B on ConvoLearn and show that dimension-level fine-tuning can steer a 7B open-weight model toward dialogic tutoring behavior that credentialed teachers rate as competitive with a strong proprietary baseline. With this work, we support the development of AI tutors capable of more dialogic interactions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

ConvoLearn gives a theory-grounded labeled dataset and a fine-tuning demo, but the generalization to real classrooms needs more methodological detail to evaluate.

read the letter

The main takeaway is a dataset of 2,134 semi-synthetic dialogues labeled on six dialogic dimensions from knowledge-building theory, along with evidence that a classifier trained on it correlates with expert ratings on authentic classroom data and that fine-tuning Mistral-7B on it produces outputs teachers rate competitively. The paper does a good job connecting the work to learning sciences literature and providing a concrete scale of labeled data that can be used for fine-tuning. The cross-validation against real expert-coded transcripts is a positive step toward showing the labels capture something transferable rather than just synthetic artifacts. The fine-tuning experiment demonstrates a practical way to steer an open model toward more dialogic behavior. Where it falls short is in the specifics of the semi-synthetic generation process and the exact statistics of the correlations. The abstract and available details do not include the prompting methods used to create the dialogues, any validation of their quality, the number of authentic transcripts involved, or the correlation coefficients and p-values. This makes it difficult to fully assess the strength of the generalization claim, as noted in the stress-test. The teacher rating results are encouraging but would be stronger with more information on the evaluation setup. This paper would be of interest to researchers in AI for education who are looking for ways to incorporate pedagogical theory into LLM training. It provides a starting point for building datasets that align AI tutors with dialogic principles. I recommend it for peer review. The contribution is concrete enough that referees could help strengthen the methods and clarify the results.

Referee Report

3 major / 1 minor

Summary. The paper introduces ConvoLearn, a dataset of 2,134 semi-synthetic tutor-student dialogues labeled with six dimensions of dialogic tutoring grounded in knowledge-building theory and situated in middle-school Earth Science. It claims that a classifier trained on this dataset yields scores that correlate significantly with expert-coded instructional quality on authentic classroom transcripts across multiple subscales, and that dimension-level fine-tuning of Mistral-7B produces dialogic tutoring behavior that credentialed teachers rate as competitive with a strong proprietary baseline.

Significance. If the reported correlations and teacher ratings hold under full methodological disclosure, the work supplies a publicly usable, learning-sciences-grounded resource that could accelerate development of dialogic AI tutors. The external validation on authentic transcripts and the open-weight fine-tuning demonstration are concrete strengths that distinguish the contribution from purely synthetic or proprietary-only approaches.

major comments (3)

[Dataset Construction] The description of semi-synthetic dialogue generation (implicitly in the dataset-construction section) omits the exact prompting templates, templating procedure, and any human validation steps used to assign the six dimension labels. Without these details it is impossible to determine whether the observed classifier correlations reflect the intended pedagogical constructs or surface-level artifacts of the synthesis process.
[Results / Correlation Analysis] The correlation study (reported in the results section and abstract) states that scores correlate significantly with expert-coded instructional quality but supplies neither the number of authentic transcripts, the per-subscale Pearson or Spearman coefficients, exact p-values, nor the classifier architecture, training procedure, or evaluation protocol (cross-validation vs. held-out real data). These omissions are load-bearing for the central generalization claim.
[Fine-Tuning Experiments] The fine-tuning experiment (proof-of-concept section) asserts that teachers rate the dimension-tuned Mistral-7B competitively with a proprietary baseline, yet reports neither the rating instrument, number of credentialed raters, inter-rater reliability, nor any statistical test comparing the two conditions. This information is required to substantiate the claim of competitive performance.

minor comments (1)

[Abstract] The abstract would be clearer if it named the six dialogic dimensions explicitly rather than referring to them only as 'six dimensions of dialogic tutoring.'

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback. We address each major comment below and will revise the manuscript to provide the requested details, thereby improving transparency, reproducibility, and the strength of our claims.

read point-by-point responses

Referee: [Dataset Construction] The description of semi-synthetic dialogue generation (implicitly in the dataset-construction section) omits the exact prompting templates, templating procedure, and any human validation steps used to assign the six dimension labels. Without these details it is impossible to determine whether the observed classifier correlations reflect the intended pedagogical constructs or surface-level artifacts of the synthesis process.

Authors: We agree that these details are essential for evaluating construct validity. The semi-synthetic dialogues were generated via structured prompts explicitly derived from knowledge-building theory to target the six dialogic dimensions. In the revised manuscript we will include the complete prompting templates, a precise description of the templating procedure, and full information on the human validation steps (including annotator instructions and any reliability checks). These additions will allow readers to assess whether the labels reflect the intended pedagogical constructs. revision: yes
Referee: [Results / Correlation Analysis] The correlation study (reported in the results section and abstract) states that scores correlate significantly with expert-coded instructional quality but supplies neither the number of authentic transcripts, the per-subscale Pearson or Spearman coefficients, exact p-values, nor the classifier architecture, training procedure, or evaluation protocol (cross-validation vs. held-out real data). These omissions are load-bearing for the central generalization claim.

Authors: We accept that these specifics are required to substantiate the generalization claim. The correlation analysis was performed on authentic middle-school Earth Science classroom transcripts. In the revision we will report the exact number of transcripts, the per-subscale Pearson and Spearman coefficients with exact p-values, and a complete description of the classifier architecture, training procedure, and evaluation protocol (specifying cross-validation versus held-out real data). revision: yes
Referee: [Fine-Tuning Experiments] The fine-tuning experiment (proof-of-concept section) asserts that teachers rate the dimension-tuned Mistral-7B competitively with a proprietary baseline, yet reports neither the rating instrument, number of credentialed raters, inter-rater reliability, nor any statistical test comparing the two conditions. This information is required to substantiate the claim of competitive performance.

Authors: We acknowledge that these details are necessary to support the teacher-rating results. The evaluation used a structured rubric aligned with the six dialogic dimensions and involved credentialed teachers. In the revised manuscript we will describe the rating instrument, state the number of raters, report inter-rater reliability (e.g., Cohen’s kappa or ICC), and include the results of statistical tests comparing the fine-tuned Mistral-7B against the proprietary baseline. revision: yes

Circularity Check

0 steps flagged

No significant circularity; central claims rest on external validation data

full rationale

The paper's core claims rest on training a classifier on the 2,134 semi-synthetic ConvoLearn dialogues and showing statistically significant correlations with expert-coded instructional quality on separate authentic classroom transcripts, plus teacher ratings of fine-tuned model outputs. These validation steps use independent human annotations and ratings collected outside the dataset construction process. No equations, fitted parameters, or self-citations are presented that reduce the reported correlations or generalization results to quantities defined by the authors' own inputs. The derivation chain from dimension labeling to external correlation metrics remains self-contained against the provided benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The work rests on the domain assumption that knowledge-building theory supplies valid, operationalizable dimensions for tutoring quality; no numerical free parameters or new invented entities are introduced in the abstract.

axioms (1)

domain assumption Knowledge-building theory supplies six valid dimensions that capture effective dialogic tutoring
The six dimensions are taken directly from the cited learning-sciences framework and used to label all dialogues.

pith-pipeline@v0.9.0 · 5469 in / 1382 out tokens · 26363 ms · 2026-05-16T14:23:15.094802+00:00 · methodology

ConvoLearn: A Learning Sciences Grounded Dataset for Fine-Tuning Dialogic AI Tutors

Core claim

What carries the argument

If this is right

Where Pith is reading between the lines

Load-bearing premise

What would settle it

discussion (0)