Pith. sign in

REVIEW 4 major objections 6 minor 17 references

Conversation Forests: The Key to Fine Tuning Large Language Models for Multi-Turn Medical Conversations is Branching

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training LLMs on branched conversation trees improves multi-turn diagnostic interviewing.

desk verdict Plausible branching idea, but the reported accuracy gains rest on a self-referential GPT-4o evaluation loop and no variance estimates, so the central claim is not yet established. read the letter →

arxiv 2507.04099 v2 pith:BOCCKWDO submitted 2025-07-05 cs.CL cs.AI

Thomas Savage This is my paper
classification cs.CLcs.AI
keywords multi-turnmedicalconversationsreinforcementlearningconversationforestsbranchedtrainingarchitecturediagnosticinterviewingGRPOLLMfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that how a language model is asked to imagine alternative conversation paths during fine-tuning matters more than the underlying reward method. It introduces a training framework in which each doctor turn branches into several possible continuations, so early questions are rewarded by what they lead to later rather than by the immediate exchange alone. In two-turn simulated doctor-patient interviews, this branched training scores higher on diagnostic accuracy than the same method restricted to linear conversations, for two open 8-billion-parameter models. The result is a concrete argument that multi-turn medical interviewing needs training signals that connect turns, not isolated single-turn completions.

What carries the argument

The central object is the branched conversation tree, a training structure in which each doctor response fans out into multiple patient continuations instead of a single linear trajectory. The two supporting mechanisms are sibling-relative reward calculation, which compares a leaf completion only with leaves sharing the same parent branch, and depth-wise normalization, which standardizes rewards separately at each conversational depth. Together they let a reward at a leaf propagate back to the earlier turns that led to it, so the policy can learn that a broad opening question pays off in later diagnostic accuracy.

What would settle it

Have board-certified physicians blindly re-score the same test conversations; if the branched model's advantage over the linear model disappears or reverses under human grading, the reported gain is an artifact of the automatic grader rather than a genuine accuracy improvement.

Watch

Extended reading notes

Core claim

The paper claims that a branched conversation architecture produces a better diagnostic interviewer than a linear architecture under the same reinforcement-learning procedure. SCF adapts GRPO by sampling groups of multi-turn conversations arranged as a tree: at each doctor turn, four continuations are generated, all sharing the early history but diverging later. A frozen GPT-4o instance plays the patient, a frozen diagnostician proposes a final diagnosis at each leaf, and a frozen grader scores the diagnosis against a gold label taken from MedQA; a parent branch's reward is the average of its leaves. Because branches create unequal numbers of completions at different depths, SCF computes each leaf's reward relative only to its sibling leaves and normalizes rewards separately by conversational depth. On the test set, branched SCF reaches 49.2% diagnostic accuracy with Llama-3.1-8B-Instruct and 48.8% with Ministral-8B-Instruct, versus 45.4% and 36.2% for the linear variant and roughly 45% for both base models, which the paper takes as evidence that branching captures inter-turn dependencies.

Load-bearing premise

Every training reward and final test score is produced by GPT-4o simulating the patient, generating the diagnosis, and grading that diagnosis, so the claim stands on the assumption that this loop measures real clinical accuracy rather than the internal preferences of one model family.

Editorial extensions

If this is right

  • Branched SCF improves GPT-4o-graded diagnostic accuracy over both linear SCF and the base model in two-turn interviews.
  • Branched training shifts question style toward broader, more open-ended questions (mean Likert 2.7 vs 3.1, p=0.04), which the paper links to better information gathering.
  • Because leaf rewards propagate to parent branches, early conversational turns receive training signal about their downstream consequences, a dependency single-turn methods cannot represent.
  • The paper predicts the accuracy gap will widen with more conversation turns, since longer dialogues amplify the value of inter-turn reasoning.
  • The same tree-based training structure should transfer to other multi-turn LLM applications where outcomes depend on a sequence of exchanges.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If branching genuinely helps, the reported 3-4% gain is likely a conservative lower bound: the experiments cap depth at two turns and 20 tokens, and longer interviews should increase the advantage while also making the exponential branching cost a real barrier.
  • A direct test of the mechanism would hold total completions constant per conversational depth and vary only the tree structure, isolating branching from the sibling-relative and depth-wise normalization choices.
  • The same reward scheme could be applied to any tree-structured dialogue with a long-horizon outcome, such as tutoring, legal intake, or technical support, where a single-turn preference signal is too weak.
  • Because the patient, diagnostician, and grader models all come from the same model family, part of the apparent gain may reflect how that family responds to open-ended wording; a test using a different patient model would clarify whether the learned questioning strategy generalizes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Savage Conversation Forests (SCF), a reinforcement-learning fine-tuning framework for multi-turn LLM dialogue that replaces linear conversation trajectories with branched conversation trees. The method builds on GRPO but computes rewards with sibling-relative comparisons and depth-wise normalization across a tree of continuations. The author adapts MedQA into simulated doctor-patient conversations, using GPT-4o as label extractor, simulated patient, diagnostician, and grader, and compares branched SCF (branching factor 4) with linear SCF (branching factor 1) on Llama-3.1-8B-Instruct and Ministral-8B-Instruct. Reported diagnostic accuracies are 49.2% vs. 45.4% for Llama and 48.8% vs. 36.2% for Mistral, with the base models at 45.1% and 45.5%. A human rating of question broadness finds branched SCF significantly more open-ended (2.7 vs. 3.1, p=0.04). The paper concludes that branched training architectures improve multi-turn diagnostic interviewing.

Significance. If the central claim were properly validated, SCF would be a useful contribution to multi-turn LLM training: the idea of branching conversation trees with non-uniform normalization is a reasonable extension of GRPO, the code is publicly available, and the study tests two base model families. The human broadness scoring is a genuine, if narrow, external signal. However, the main endpoint is not currently established. Every reported accuracy number is generated by the same GPT-4o-based loop that produced the training reward, there are no repeated runs or confidence intervals, and the completion-parity argument contains an arithmetic discrepancy. These issues are load-bearing because the paper's contribution is precisely the claim that branching improves diagnostic accuracy, and that claim rests on an unvalidated and possibly self-referential evaluation metric.

major comments (4)
  1. [Model Evaluation Metrics] The primary endpoint, diagnostic accuracy, is entirely internal to a GPT-4o loop: GPT-4o extracts the gold-standard diagnosis from MedQA, acts as the simulated patient, generates the predicted diagnosis as the diagnostician, and grades that prediction as the grader. Because the same grader also supplies the RL reward used for training, the reported accuracies are a measure of agreement with one proprietary model, not an independent clinical endpoint. If GPT-4o's grading is lenient or biased toward longer or more open-ended transcripts, the branched model's advantage in Table 1 could reflect stylistic preferences rather than improved diagnostic reasoning. The human evaluation reported in Response Evaluation Metrics scores broadness only, not diagnostic correctness. I do not see any validation of the grader against clinician judgment or an external medical QA benchmark. This is a load-bearing concern: without an independent endpoint, the central claim that branching improves diagnostic accuracy is not supported.
  2. [Results / Table 1] Each condition is reported as a single number with no repeated runs, confidence intervals, or significance tests. The Llama comparison is 49.2% vs. 45.4%, a 3.8-point gap that could easily arise from seed variance in a single fine-tuning run; the Mistral comparison is larger, but the linear variant actually degrades below the base model (36.2% vs. 45.5%), which is unexplained and consistent with training instability. In addition, there is no standard GRPO, DPO, or PPO baseline for the same multi-turn task, so the comparison is only SCF-linear vs. SCF-branched. The paper's claim that SCF with branching 'outperforms' linear architectures would be much more convincing with at least three seeds per condition, a standard deviation or confidence interval, and a comparison against a conventional multi-turn RL baseline.
  3. [Hyperparameters] The stated completion parity appears arithmetically inconsistent. The branched variant uses a branching factor of 4 over a conversational depth of two turns, which yields 4^2 = 16 leaf trajectories per tree. Four trees per case therefore produce 64 leaf trajectories, not the claimed 20 total completions. The linear variant, with branching factor 1 and two turns, produces one trajectory per tree, so ten trees yield 10 trajectories, not 20. Either the definition of 'completion' is different from the leaf-trajectory count implied by the tree description, or the parity claim in this section is wrong. Since the fairness of the comparison is explicitly argued from this 20-completion matching, the discrepancy needs to be resolved and the consequence for training-signal balance analyzed.
  4. [Medical Conversation Dataset] The gold-standard diagnoses used for both reward and evaluation are 'inferred' from MedQA answer keys by GPT-4o rather than read directly from MedQA's structured answer choices. No validation or agreement measure is reported for this extraction step. If the extracted labels, clinical information, or case introductions contain errors, those errors propagate into the reward signal and the final accuracy numbers. Given that the entire evaluation pipeline depends on this extraction, the paper should report a human-validated sample (for instance, expert review of 100 extracted cases) and ad-hoc extraction accuracy; otherwise the reliability of the ground-truth labels is unknown and the comparison may be measuring consistency with GPT-4o's own case reconstruction rather than diagnostic correctness.
minor comments (6)
  1. [Table 1] The model is called 'Ministral-8B-Instruct' in Table 1 but 'Mistral-8B-Instruct' elsewhere; the correct model name and version should be verified and used consistently.
  2. [Figure 5] The training-reward plot has no axis labels, no step numbers, and no smoothing or error bands; adding these would make the claimed training dynamic visible and reproducible.
  3. [Introduction] The statement that PPO and GRPO are 'designed for single-turn completions' is imprecise: these algorithms are commonly applied to multi-turn chat trajectories by treating the whole trajectory as one episode. The paper should clarify which specific limitation is being claimed.
  4. [Hyperparameters] Several training details are missing: number of training steps or epochs, per-device batch size, number of gradient accumulation steps, and the number of cases skipped because of insufficient reward variability. This information is needed for reproducibility and to assess selection bias from the skipping rule.
  5. [Response Evaluation Metrics] The broadness scores are averaged over three medical providers, but no inter-rater reliability statistic is reported; a Cohen's kappa or similar measure would strengthen the claim that the observed difference is robust.
  6. [References] Reference 11 ('Du, Y. & Chen, W. Branching Reinforcement Learning') lacks a publication venue, arXiv identifier, or year, and reference 17 is cited as [17] but is only listed in the reference list without an in-text citation; complete both entries.

Circularity Check

1 steps flagged · score 4.0 of 10

Diagnostic accuracy is scored by the same GPT-4o grader used to generate training rewards, making the central metric self-referential; the branching-vs-linear comparison itself remains an empirical claim under that metric.

  1. self definitional [Methods, Model Evaluation Metrics (with Methods, SCF Architecture reward loop)]
    "Performance was assessed using the same diagnostician and grader models employed during training. For each test conversation, the model generated a linear conversation transcript and the diagnostician model generated a corresponding predicted diagnosis. This prediction was then scored by the grader model against the ground-truth diagnosis label. The grader model assigned one of three possible values: 1.0 for an exact or clinically equivalent diagnosis, 0.5 for a partially correct or related diagnosis, 0.0 for an incorrect diagnosis."

    The reported 'diagnostic accuracy' is the aggregate of the GPT-4o grader's 1.0/0.5/0.0 scores, and the same grader's scalar is the reward signal used during SCF training. The paper explicitly states that test performance was assessed with 'the same diagnostician and grader models employed during training,' so the evaluation metric is, by construction, the same function the RL loop optimizes. Thus the headline claim that branching improves diagnostic accuracy reduces to the claim that the branched policy obtains higher scores from the same GPT-4o grader that supplied its rewards. No clinician validation of the diagnostic metric is provided; the only human evaluation measures question broadness, not diagnostic correctness.

full rationale

The paper's derivation chain is not circular in the usual sense of a fitted parameter being renamed as a prediction or a conclusion being assumed in its premises. The branching architecture, sibling-relative rewards, and depth-wise normalization are concretely specified, the code and dataset are public, and the comparison between branched and linear SCF is an empirical result on held-out test conversations. No load-bearing argument rests on the author's own prior work: reference [2] is only cited as an example of medical fine-tuning and is not used to justify the central claim. The main circularity concern is the evaluation loop: the same GPT-4o grader provides both the training reward and the final 'diagnostic accuracy' score, and GPT-4o also extracts the labels and simulates the patient. As a result, the reported accuracy is an internal, self-referential proxy rather than an externally validated clinical endpoint. This is a validity limitation as much as a circularity, and it is not a mathematical equivalence that forces the outcome; the branched model could in principle have scored lower under the same grader. The score reflects that partial self-referentiality while acknowledging that the core algorithmic comparison is independently meaningful.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The central empirical claim depends on several domain assumptions about the GPT-4o simulation and grading loop. No equations or derivations are used; the method is an algorithmic contribution. Free parameters are experimental hyperparameters, not fitted constants.

free parameters (5)
  • Branching factor (number of branches per turn) = 4
    Chosen by hand; the branched variant generates 4 sibling completions per doctor turn, and the comparison depends on this architecture parameter.
  • Conversational depth = 2 turns
    All dialogues contain two doctor questions and two patient responses; short depth is acknowledged to limit observed gains.
  • Number of conversation trees per case = 4 (branched), 10 (linear)
    Set so both variants produce 20 total completions per training case; this matching is central to the fairness claim but does not match unique trajectories.
  • Learning rate = 2e-7
    Standard hyperparameter chosen for all runs; no sweep or sensitivity analysis is reported.
  • Grading rubric weights = 1.0, 0.5, 0.0
    The grader assigns these scores for exact, partial, and incorrect diagnoses; reported point percentages depend on this chosen rubric.
assumptions (5)
  • domain assumption A two-turn dialogue captures the multi-turn learning signal that distinguishes branched from linear training.
    The experiments use depth 2; the paper argues longer dialogues would amplify gains, which would undermine the current test's sensitivity.
  • domain assumption GPT-4o-simulated patient responses are sufficiently realistic and informative for training diagnostic interviewing.
    The patient model is frozen GPT-4o prompted with GPT-4o-extracted case data; no validation against real patient transcripts is provided.
  • domain assumption GPT-4o grading against extracted MedQA labels is a valid measure of diagnostic accuracy.
    The entire reward and evaluation pipeline uses GPT-4o as diagnostician and grader; grader accuracy is not checked against clinician judgment.
  • domain assumption Matched total completions (20 per case) makes the branched versus linear comparison fair.
    The linear variant gets 10 independent dialogues; the branched variant gets 4 trees with shared prefixes and 16 unique two-turn paths, so trajectory diversity is not matched.
  • domain assumption Sibling-relative rewards and depth-wise normalization correctly extend GRPO advantage estimation to trees.
    No theoretical or ablative evidence shows this normalization preserves GRPO's unbiased advantage estimates.
invented entities (3)
  • Savage Conversation Forests (SCF) independent evidence
    purpose: A tree-structured reinforcement learning training framework for multi-turn LLM fine-tuning, with branching rollouts, sibling-relative reward, and depth-wise normalization.
    The public repository and dataset provide a falsifiable handle, but effectiveness currently rests only on this paper's experiments.
  • Sibling-relative reward calculation
    purpose: Computes each leaf completion's relative reward only against sibling leaves sharing the same parent, to handle unequal branch counts.
    It is a methodological component described in the paper; no external benchmark establishes its behavior.
  • Depth-wise normalization
    purpose: Normalizes rewards at each conversational depth separately so parent branches and leaves are compared only within their level.
    Introduced by the paper without independent validation or formal analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Conversation Forests: The Key to Fine Tuning Large Language Models for Multi-Turn Medical Conversations is Branching." pith.science (2026). https://pith.science/paper/BOCCKWDO

@misc{pith2026250704099,
  author       = {Pith},
  title        = {Pith review of: Conversation Forests: The Key to Fine Tuning Large Language Models for Multi-Turn Medical Conversations is Branching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BOCCKWDO}},
  note         = {Machine review of arXiv:2507.04099}
}
read the original abstract

Fine-tuning methods such as Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO) have demonstrated success in training large language models (LLMs) for single-turn tasks. However, these methods fall short in multi-turn applications, such as diagnostic patient interviewing, where understanding how early conversational turns influence downstream completions and outcomes is essential. In medicine, a multi-turn perspective is critical for learning diagnostic schemas and better understanding conversation dynamics. To address this gap, I introduce Savage Conversation Forests (SCF), a reinforcement learning framework that leverages a branched conversation architecture to fine-tune LLMs for multi-turn dialogue. SCF generates multiple possible conversation continuations at each turn, enabling the model to learn how different early responses affect downstream interactions and diagnostic outcomes. In experiments simulating doctor-patient conversations, SCF with branching outperforms linear conversation architectures on diagnostic accuracy. I hypothesize that SCF's improvements stem from its ability to provide richer, interdependent training signals across conversation turns. These results suggest that a branched training architecture is an important strategy for fine tuning LLMs in complex multi-turn conversational tasks.

Figures

Figures reproduced from arXiv: 2507.04099 by the authors.

Figure 2
Figure 2. Visual representation of the SCF method. The multi-turn conversation shown here has been simplified to enhance readability. A comprehensive depiction of SCF’s branched multi-turn conversation structure is provided in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A comparison of linear conversation structure [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. In SCF the branched conversation structure requires a non-uniform normalization mechanism. The figure above demonstrates the sibling relative reward and depth-wise normalization groupings used in SCF. Related Work - Branched Architecture in Reinforcement Learning In the broader reinforcement learning (RL) literature, outside the context of large language models, several studies have explored branching architectures.… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 8 canonical work pages

  1. [1]

    O’Sullivan, J. W. et al. Towards Democratization of Subspeciality Medical Expertise. Preprint at https://doi.org/10.48550/arXiv.2410.03741 (2024)

    work page Pith review arXiv doi:10.48550/arxiv.2410.03741 2024

  2. [2]

    Savage, T. et al. Fine Tuning Large Language Models for Medicine: The Role and Importance of Direct Preference Optimization. Preprint at https://doi.org/10.48550/arXiv.2409.12741 (2024)

    arXiv doi:10.48550/arxiv.2409.12741 2024

  3. [3]

    Fleming, S. L. et al. MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records. Proceedings of the AAAI Conference on Artificial Intelligence 38, 22021–22030 (2024)

    work page 2024

  4. [4]

    Wang, J., Yang, Z., Yao, Z. & Yu, H. JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capability. Preprint at https://doi.org/10.48550/arXiv.2402.17887 (2024)

    arXiv doi:10.48550/arxiv.2402.17887 2024

  5. [5]

    Rafailov, R. et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Preprint at http://arxiv.org/abs/2305.18290 (2023)

    arXiv 2023

  6. [6]

    & Klimov, O

    Schulman, J., Wolski, F ., Dhariwal, P ., Radford, A. & Klimov, O. Proximal Policy Optimization Algorithms. Preprint at https://doi.org/10.48550/arXiv.1707.06347 (2017)

    arXiv doi:10.48550/arxiv.1707.06347 2017

  7. [7]

    Shao, Z. et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Preprint at https://doi.org/10.48550/arXiv.2402.03300 (2024)

    arXiv doi:10.48550/arxiv.2402.03300 2024

  8. [8]

    https://arxiv.org/html/2504.04717v1?utm_source=chatgpt.com

    Beyond Single-Turn: A Survey on Multi-Turn Interactions with Large Language Models. https://arxiv.org/html/2504.04717v1?utm_source=chatgpt.com

    arXiv

Show all 17 references
  1. [9]

    US Elsevier Health https://www.us.elsevierhealth.com/the-medical-interview-9780323052214.html

    The Medical Interview - 9780323052214. US Elsevier Health https://www.us.elsevierhealth.com/the-medical-interview-9780323052214.html

  2. [10]

    Scavuzzo, L. et al. Learning to branch with Tree MDPs. Preprint at https://doi.org/10.48550/arXiv.2205.11107 (2022)

    2022 doi arXiv

  3. [11]

    & Chen, W

    Du, Y . & Chen, W. Branching Reinforcement Learning

  4. [12]

    Zhou, Y . et al. SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks. Preprint at https://doi.org/10.48550/arXiv.2503.15478 (2025)

    2025 doi arXiv

  5. [13]

    Jin, D. et al. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams. Preprint at https://doi.org/10.48550/arXiv.2009.13081 (2020)

    2020 doi arXiv

  6. [14]

    https://huggingface.co/meta-llama/Llama- 3.1-8B-Instruct (2024)

    meta-llama/Llama-3.1-8B-Instruct · Hugging Face. https://huggingface.co/meta-llama/Llama- 3.1-8B-Instruct (2024)

    2024

  7. [15]

    https://huggingface.co/mistralai/Ministral- 8B-Instruct-2410

    mistralai/Ministral-8B-Instruct-2410 · Hugging Face. https://huggingface.co/mistralai/Ministral- 8B-Instruct-2410

  8. [16]

    https://platform.openai.com

    Model - OpenAI API. https://platform.openai.com

  9. [17]

    & Dolan, B

    Li, J., Galley, M., Brockett, C., Gao, J. & Dolan, B. A Diversity-Promoting Objective Function for Neural Conversation Models. Preprint at https://doi.org/10.48550/arXiv.1510.03055 (2016). Appendix I The file used to adapt the MedQA question-answer dataset to a dataset for med...

    2016 doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.