Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancement

Heyan Huang; Lizi Liao; Xiaofeng Zhou

REVIEW 6 major objections 5 minor 20 references

Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancement

T0 review · 6 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A small language model that debates stronger teachers, then trains on preference trees extracted from those debates, outperforms single-teacher distillation on knowledge and reasoning benchmarks.

desk verdict Useful integration of debate, feedback, and tree-structured DPO, but the headline margins are not yet secured: T-DPO's edge over RPO is 0.02 on average, and the runs are unseeded. read the letter →

arxiv 2506.03541 v1 pith:NSOQQE42 submitted 2025-06-04 cs.CL cs.AI

Xiaofeng Zhou , Heyan Huang , Lizi Liao This is my paper · ORCID

classification cs.CLcs.AI

keywords knowledgedistillationmulti-agentdebatedirectpreferenceoptimizationtree-structuredself-reflectionteacherfeedbacksmalllanguagemodelsreasoningbenchmarks

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a small language model can be made substantially better at knowledge and reasoning tasks by letting it debate stronger teacher models, collecting the debate interactions as structured logs, and distilling those logs back into the student's weights. The key move is to include the student itself in the debate, along with explicit self-reflection from the student and corrective feedback from teachers, so the training signal targets the student's actual mistakes. The paper then converts each debate log into a 'preference tree' and trains with a tree-structured variant of Direct Preference Optimization (T-DPO), so the student learns not just the right answer but why one reasoning path is preferred over another. On the MMLU-Pro and MATH benchmarks, the authors report an average gain of 14.18 points over the original Mistral-7B-Instruct, an edge of at least 2.95 points over the best single-teacher distillation, and an extra 2.16 points from T-DPO over supervised fine-tuning alone. If these results hold, distillation becomes a cheaper and more targeted route to capable small models than static teacher-generated training data.

What carries the argument

The central object is the preference tree extracted from a Multi-Agent Interaction Graph (MAG), a record of the debate. Each tree root is the input prompt plus the structured information from the previous debate round (prior responses, self-reflections, and teacher feedback); its children are the correct and incorrect responses generated in that round. T-DPO applies the standard DPO objective to these trees, treating the correct response as the chosen completion and the incorrect one as the rejected completion. The machinery does two jobs at once: it preserves the debate's error-correction history in the prompt, and it trains the student to prefer the reasoning path that was actually corrected, rather than only imitating final answers.

What would settle it

Run the same T-DPO pipeline on a teacher-only debate dataset of equal size (no student in the loop, no self-reflection or teacher feedback) and compare; if that dataset matches or beats D&R's results, the student-in-the-loop and feedback data are not what drives the reported gains.

Watch

Extended reading notes

Core claim

The paper's central discovery claim is that the quality of distillation data matters more than the distillation objective alone, and that debate is a way to manufacture high-quality data targeted at a specific student's weaknesses. Concretely, the authors claim that when a student model engages in multi-round debates with stronger teachers, and the debate includes the student's self-reflection and teachers' explicit feedback on the student's errors, the resulting logs support better distillation than teacher-only debate logs. The tree-structured objective T-DPO then leverages the hierarchical structure of these logs: each preference tree has the previous round's structured information as the root, a correct response as the chosen node, and an incorrect response as the rejected node, so the student learns from both the contrast and the reasoning history. The reported experiments show D&R with T-DPO beating single-teacher SFT distillation, the multi-teacher graph-distillation baseline MAGDI, and its own SFT-only variant, while also lowering per-problem token cost at inference.

Load-bearing premise

The paper assumes that debate logs that include the student model plus self-reflection and teacher feedback are higher-quality training data than teacher-only debate logs; this is supported by a qualitative example in the appendix, not by a quantitative measure of data quality.

Editorial extensions

If this is right

Small models can reach accuracy close to the debate-level scores of the API teachers on the tested categories: D&R brings Mistral-7B from a 23.98 average to 38.16, while the debate itself scores 84.29 to 90.38 on MMLU Pro categories.
Including the student in the debate and requiring explicit self-reflection and teacher feedback is what enables the distilled model to correct its own mistakes at inference; ablations that remove this data drop accuracy by up to 6.46 points and the model no longer self-corrects.
T-DPO adds consistent gains over SFT across data scales (100 to 974 debates) and across two student models, so the benefit is not tied to one dataset size or one student architecture.
Distilled D&R models are more token-efficient than running a debate at inference and also than the compared distillation baselines, because the student learns more concise reasoning paths.
A stronger student model continues to benefit from T-DPO on MATH where Mistral plateaus, suggesting the framework transfers to students with larger reasoning capacity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same preference-tree construction could be applied to logs from other interactive settings, such as code repair, tool-use dialogues, or human-in-the-loop corrections, not only to knowledge and reasoning benchmarks.
The RPO comparison hints that rising chosen-completion log probabilities do not reliably predict downstream accuracy; a cheaper proxy for preference-tree quality could be tested directly.
A direct test of the data-quality premise would be to measure whether later debate rounds produce more correct responses than earlier ones, or whether self-reflections actually identify the specific error that teacher feedback then corrects; the paper reports neither.
The Llama-3.1 result suggests the MATH bottleneck with Mistral is a capacity effect rather than a debate-data effect, but this is only tested on one additional student model and one dataset.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short take: this is a competently executed paper with a genuinely new integration, but the headline numbers are not yet established. The specific contribution of T-DPO over the existing RPO objective is 0.02 points on average, and every result in the paper comes from a single unseeded run.

What's new: D&R puts the student model into the debate loop with self-reflection and teacher feedback, then turns the debate logs into preference trees and trains with DPO. MAGDI used multi-teacher debate graphs but no student-in-the-loop feedback; Eurus and CPO used tree-structured DPO but not on debate logs. That combination is new, and the ablations in Table 2 give real—if thin—evidence that self-reflection and teacher feedback matter: removing them costs 5+ points on physics. I also give credit for the honest limitations section, the attempt with Llama-3.1-8B to test the capacity-bottleneck explanation for the MATH result, and the token-cost comparison.

Soft spots, in order. First, no error bars, no multiple seeds, no significance tests anywhere. MMLU Pro category-level test sizes are 205-325 questions; a 2-4 point gap is within one binomial standard error. The paper says 'significantly outperforms' but nothing in it is statistically tested. Second, splitting the MMLU Pro test set for training is a real benchmark-hygiene issue; even with a held-out part, training on a public test set needs more discussion. Third, the quantitative case for T-DPO collapses under inspection. Table 4 shows T-DPO beats RPO by 0.02 on average (38.16 vs 38.14), and on MATH it is 0.32 below the SFT-only stage. The paper's claim that T-DPO is 'most suitable' is not supported by its own table. Fourth, the core training data—debate logs and preference trees—is not released, so the quality claim in Appendix B cannot be checked beyond the one example. The efficiency result is largely a tautology: a distilled model uses fewer tokens than live debate.

Overall: the method is plausible and the experiments are consistent with 'this works', but the magnitude claims and component attribution are not secured. This deserves a serious referee, but the paper needs seeds, paired significance tests, and data release before the strong claims can stand. Who is this for? Researchers working on distillation of reasoning into small models; they'll read it as a proof-of-concept, not as a benchmark result.

Referee Report

6 major / 5 minor

Summary. The paper presents D&R, a multi-agent distillation framework in which a Mistral-7B student model debates three commercial teacher models over up to four rounds, generating self-reflections and teacher feedback, and records the interactions in a Multi-Agent Interaction Graph. From these graphs the authors extract SFT instances and preference trees, then train the student with SFT followed by T-DPO, a DPO variant that conditions on the previous debate round's structured information. Experiments on three MMLU-Pro categories and MATH compare D&R with no distillation, single-teacher SFT distillation, and a reimplemented MAGDI baseline. The reported results give an average gain of 14.18 points over Mistral-7B-Instruct, an advantage of at least 2.95 points over the best single-teacher baseline, and a 2.16-point additional gain from T-DPO over SFT-only D&R.

Significance. If the reported gains hold under replication, the D&R framework is a practically useful contribution: it converts an expensive multi-agent debate process into a one-time training signal, releases the student from live teacher involvement at inference, and provides concrete details (model versions, prompt templates, hyperparameters, token costs) that support reproduction. The conversion of interaction graphs into preference trees and the transfer experiment to a stronger student (Llama-3.1-8B) are useful additions. The data construction uses external gold labels and held-out evaluation, so I do not see a derivation-from-fit circularity. The main unresolved question is quantitative: the central comparisons rest on single runs per condition with no reported seed variation on small MMLU-Pro test splits, and the specific advantage of T-DPO over an existing preference objective (RPO) is at most 0.02 points on average, so the strength of the empirical claims currently exceeds the evidence.

major comments (6)

[§4.2, Table 1, Appendix A Table 5] The headline comparisons are reported without error bars, multiple seeds, or significance tests. The MMLU-Pro test splits are small (205, 325, and 239 problems), so the per-category differences of 2.44 to 4.60 points against Single-Teacher Claude3.5 are within roughly one or two binomial standard errors; the 'at least 2.95 average improvement' claim is carried by pooling over categories and MATH. I ask the authors to report multiple training/evaluation seeds or paired bootstrap confidence intervals for the main comparisons, and to state how many test examples each accuracy number is based on.
[§4.3.5, Table 4] The conclusion that T-DPO is 'the most suitable objective' for debate distillation is not supported by Table 4. T-DPO exceeds RPO by only 0.02 points on average (38.16 vs. 38.14), and on MATH the T-DPO model is 0.32 points below the SFT-only model (17.32 vs. 17.64). Without variance estimates or a significance test, this cannot separate T-DPO from RPO, and the 2.16-point contribution of T-DPO over SFT may not be robust. The experiment should be replicated with multiple seeds and, ideally, with a small hyperparameter search for RPO rather than a single alpha=1 setting.
[§4.1, Appendix A] The authors split the MMLU-Pro test set into training and test portions because no dedicated training set exists, but the paper does not justify why this split yields a valid held-out evaluation. Using part of the official test set as training material raises the risk of distributional leakage or contamination; a defense of the split, or an additional evaluation on an independent benchmark, is needed to support the generalization claims drawn from MMLU-Pro.
[Limitations section] The paper explicitly limits its evaluation to final-answer correctness rather than the validity of reasoning paths. Because the stated contribution is about transferring and improving reasoning, this limitation is load-bearing: the accuracy gains could in principle come from better answer selection or format changes rather than from the reasoning improvements the framework is claimed to produce. The authors should add process-level evidence (e.g., reasoning-step accuracy, correction quality, or a qualitative analysis of reasoning paths) or explicitly soften the reasoning claims.
[§4.3.2, Table 2] The ablation of self-reflection and teacher feedback is reported under an 'Additional Correction Accuracy' protocol that mixes inference-time correction methods with the training-data ablation. The table does not state which dataset or split it refers to, and the 'without SR/TF' rows are compared against different reference points in different correction columns. To make the contribution of SR and TF transparent, the same ablation should be reported on the main evaluation protocol (the setting used in Table 1) for each of the four benchmarks.
[§4.2, Appendix B] The claim that student-in-the-loop debate and SR/TF produce higher-quality training data is used to explain the 11.01-point gain over MAGDI, but the only direct evidence is one qualitative example in Appendix B; the ablations in Table 2 measure the effect of removing SR/TF training data under inference-time correction, not data quality directly. A quantitative comparison of learning signals (e.g., training D&R on teacher-only debate logs that lack student participation and SR/TF) would make this explanation load-bearing.

minor comments (5)

[Abstract and §1] Typographical errors such as 'a novelDebate andReflect' should be corrected throughout the paper.
[Table 1] The header 'Multi-Teaher Distillation' contains a typo; also, the MAGDI row appears to use inconsistent formatting for the model name.
[§4.3.1, Table 6] The efficiency comparison reports average token costs without variance or sample sizes; a brief note on stability would help, since the Pareto-frontier claim in Figure 3 depends on these means.
[Appendix F, Figure 7] The prompt template contains a typo ('Fianl answer') in the incorrect-answer placeholder; this should be fixed for reproducibility.
[Appendix E, Figure 5] The preference-tree construction involves several choices (how many previous-round nodes to include, how to segment long structured information, when to create a new tree) that are not ablated; a short discussion of their sensitivity would strengthen the method section.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported gains are held-out measurements against external gold labels, and no load-bearing claim reduces to a fitted input or self-citation.

full rationale

The paper's derivation chain is empirical rather than definitional. Gold answers are external labels: debates compare agent responses to these labels (Sec. 3.1: 'We use training sets as our data sources to obtain gold answers'), and T-DPO pairs are selected by that external correctness signal (Sec. 3.3: 'The correct and incorrect child nodes act as the chosen and rejected responses'). The T-DPO loss is the standard DPO objective with the reference policy fixed to the SFT model; no free parameter is fitted to the test labels, and all headline numbers are reported on held-out MMLU-Pro splits or the MATH test set. The reported MATH degradation of T-DPO relative to D&R_SFT (17.32 vs 17.64) is inconsistent with any mechanism that would force T-DPO to help by construction. Ablations that remove self-reflection/teacher-feedback data are true input removals, not re-labelings. Citations to prior work, including the authors' own related-work citations (e.g., Zhou et al. 2024; Li et al. 2024; He et al. 2024), are background and do not carry the central argument; there is no imported uniqueness theorem. The closest definitional flavor is the efficiency comparison, since a single distilled model is inherently cheaper at inference than the multi-agent debate that produced the training data (efficiency is defined as 1/avg(token cost), Table 6), but the paper presents this as a measured token-cost trade-off, not as a prediction derived from its own definitions. The Limitations section's caveat that only final-answer correctness is measured is a validity limitation, not circularity; the unquantified debate-data-quality claim in Appendix B is an evidence gap. Concerns about single unseeded runs and the 0.02 average T-DPO/RPO gap are statistical robustness issues, not evidence that the claimed improvements are inputs in disguise.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper is an empirical pipeline rather than a derivation. The main unpaid inputs are gold-labeled debate data, teacher model outputs, hand-chosen hyperparameters, and the assumption that preference trees built from a single correct/incorrect split are a useful training signal.

free parameters (6)

max_debate_rounds = 4
Chosen to balance data richness and API cost; affects the amount of structured information and number of preference trees.
MMLU Pro train/test split ratios = 1:1 CS, 3:1 physics, 2:1 biology
Chosen per category to ensure enough test data; training on parts of the MMLU Pro test set is an ad hoc decision.
SFT epochs and learning rate = 2 epochs, 2e-4
Training hyperparameters for the SFT stage; no validation split is reported.
T-DPO epochs and learning rate = MMLU Pro: 3 epochs, 5e-6; MATH: 1 epoch, 1e-6
Training hyperparameters for preference optimization; different per dataset.
LoRA rank and alpha = rank 16, alpha 32
LoRA configuration used for all distillation; central to the parameter-efficient fine-tuning.
sequence length limits = total length 2000, prompt length 1400
Chosen to keep training feasible; truncation may discard structured information.

assumptions (5)

standard math The DPO loss formula from Rafailov et al. is a valid preference optimization objective for reasoning.
The T-DPO loss in Section 3.3 is built directly on the DPO objective; the paper relies on its established behavior without re-deriving it.
domain assumption Gold answers in MMLU Pro and MATH are correct and can be used to label debate responses.
Correct/incorrect labels drive both SFT data selection and preference tree construction in Sections 3.2 and 3.3.
domain assumption Teacher models (GPT-4o, Claude 3.5, Gemini 1.5 Pro) provide useful corrective feedback to the student.
The D&R value proposition depends on teacher feedback being actionable; Appendix B gives only a qualitative example.
ad hoc to paper Splitting the MMLU Pro test set into training and test parts yields a valid evaluation of generalization.
MMLU Pro has no training set, so the authors train on part of the test set in Section 4.1; this assumption is questionable for benchmark validity.
ad hoc to paper Preference tree construction rules (choose a correct and an incorrect node plus previous-round nodes) capture useful learning signal.
The tree construction in Figure 5 is a design choice not derived from theory; its benefit is shown only through end-task accuracy.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancement." pith.science (2026). https://pith.science/paper/NSOQQE42

@misc{pith2026250603541,
  author       = {Pith},
  title        = {Pith review of: Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NSOQQE42}},
  note         = {Machine review of arXiv:2506.03541}
}

read the original abstract

Large Language Models (LLMs) continue to set new standards in knowledge-intensive and complex reasoning tasks, yet their high computational demands limit widespread adoption. While distilling large models into smaller ones offers a sustainable solution, current techniques--such as static knowledge distillation, resource-intensive reinforcement learning from human feedback, or limited self-reflection--struggle to yield substantial and lasting performance gains. In this paper, we present a novel Debate and Reflect (D&R) framework that orchestrates multi-turn debates between smaller models and stronger teacher models, eliciting actionable feedback (e.g., error analysis, corrective strategies) to guide student models. Further, we introduce Tree-structured Direct Preference Optimization (T-DPO) to efficiently leverage these debate logs, organizing interactions into a hierarchical format for effective training. Empirical evaluations across diverse NLP benchmarks demonstrate that our approach significantly improves smaller-model accuracy, robustness, and generalization, outperforming conventional baselines by a large margin.

Figures

Figures reproduced from arXiv: 2506.03541 by the authors.

**Figure 1.** Overview of D&R. The student model debates with teacher models, receiving self-reflection and teacher feedback. A Multi-Agent Interaction Graph (MAG) records debate content, from which preference trees are extracted. All information is then distilled into the student, enhancing its knowledge and reasoning. Existing strategies for enhancing smaller models generally fall into three categories. First, knowledge disti… view at source ↗

**Figure 2.** A debate example of our proposed D&R. The left part demonstrates the MAG of a debate involving [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Trade-off between performance and efficiency. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Performance with different training debates [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: The schematic diagram showing how to con [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

**Figure 6.** Figure 6: The prompt template for generating debates. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

**Figure 7.** Figure 7: The prompt template for self-reflecting in debates. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

**Figure 8.** Figure 8: The prompt template for generating feedback in debates. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

**Figure 9.** Figure 9: The prompt template for D&R distillation. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

**Figure 10.** Figure 10: An example from the MATH dataset. Given the question, we show a response from [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 19 canonical work pages

[1]

Select a correct node, an incorrect node, and several nodes from the previous round

work page
[2]

From them, we trace back their debate history nodes as struc- tured information

The correct and incorrect nodes are chosen and rejected, respectively. From them, we trace back their debate history nodes as struc- tured information

work page
[3]

InThe Twelfth International Con- ference on Learning Representations

MiniLLM: Knowledge Distillation of Large Language Models. InThe Twelfth International Con- ference on Learning Representations. Tao He, Lizi Liao, Yixin Cao, Yuanxing Liu, Ming Liu, Zerui Chen, and Bing Qin. 2024. Planning Like Hu- man: A Dual-process Framework for Dialogue Plan- ning. InProceedings of the 62nd Annual Meeting of the Association for Comput...

work page 2024
[4]

The greatest common divisor (gcd) is the last non-zero value of b

work page
[5]

InAd- vances in Neural Information Processing Systems, volume 36, pages 53728–53741

Direct Preference Optimization: Your Lan- guage Model is Secretly a Reward Model. InAd- vances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc. Mucheng Ren, Heyan Huang, Yuxiang Zhou, Qianwen Cao, Yuan Bu, and Yang Gao. 2022. TCM-SD: A Benchmark for Probing Syndrome Differentiation via Natural Language Proces...

work page 2024
[8]

F Designed Prompt Templates in D&R In this section, we provide the prompt templates that are used in our D&R for generating debates and distillation in Figure 6, 7, 8, and 9

Repeat steps 1-2 to construct more preference trees. F Designed Prompt Templates in D&R In this section, we provide the prompt templates that are used in our D&R for generating debates and distillation in Figure 6, 7, 8, and 9. ### Instruction You are a helpful assistant aiming to answer the following question with reasoning. You are debating the input qu...

work page
[9]

Find the GCD of the first two numbers

work page
[10]

The greatest power of 2 that divides both numbers is 2^4

Then find the GCD of that number and the third number GCD(128, 144) 128 = 2^7 144 = 2^4 * 3^2 The prime factors are 2 and 3. The greatest power of 2 that divides both numbers is 2^4. GCD(128, 144) = 16 GCD(16, 480) 480 = 2^5 * 3 * 5 16 and 480 are both divisible by 16. Therefore, GCD(128, 144, 480) = 16 ----------------------------------------------------...

work page

Show all 20 references

[11]

First, let's find GCD(128, 144) * 144 = 1 × 128 + 16 * 128 = 8 × 16 + 0 * So, GCD(128, 144) = 16
[12]

Now, let's find GCD(16, 480) * 480 = 30 × 16 + 0 * So, GCD(16, 480) = 16
[13]

Therefore, GCD(128, 144, 480) = 16 * We can verify this: * 128 = 16 × 8 * 144 = 16 × 9 * 480 = 16 × 30 Final answer: 16 Student response in D&R training debates:
[14]

Find the greatest common divisors of 128 and 144 using the Euclidean algorithm
[15]

Set up the equation: a = bq + r, where a is the larger number, b is the smaller number, q is the quotient, and r is the remainder
[16]

Repeat step 1 with the new pair of numbers (b, r) until r is zero
[18]

Apply the same process to find the gcd of the gcd of 128 and 144, and 480. Final answer: $128 = 4 \cdot 32^2 \quad 144 = 4 \cdot 36^2 \quad 480 = 2^4 \cdot 5^2 \cdot 36^2$ Using the Euclidean algorithm: $128 = 4 \cdot 144 + 112 \quad 144 = 1 \cdot 112 + 32 \quad 112 = 3 \cdot ...
[19]

Instead, they attempted to use prime factorization, which was unnecessary and led to incorrect calculations

The agent listed steps of the Euclidean algorithm but didn't actually execute them. Instead, they attempted to use prime factorization, which was unnecessary and led to incorrect calculations
[20]

The agent should have followed through with the Euclidean algorithm steps they outlined and provided a single number as the GCD

The prime factorizations provided are incorrect: * 128 = 2^7 (not 4 · 32^2) * 144 = 2^4 · 3^2 (not 4 · 36^2) * 480 = 2^5 · 3 · 5 (not 2^4 · 5^2 · 36^2) ...... The agent should have followed through with the Euclidean algorithm steps they outlined and provided a single number a...

2024
[2015]

CoRR, abs/1503.02531

Distilling the Knowledge in a Neural Network. CoRR, abs/1503.02531. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. InThe Tenth Inter- national Conference o...

2022 arXiv
[2023]

arXiv preprint

Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback. arXiv preprint. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang
[2024]

InThe Twelfth International Conference on Learning Representa- tions

ChatEval: Towards Better LLM-based Eval- uators through Multi-Agent Debate. InThe Twelfth International Conference on Learning Representa- tions. Justin Chen, Swarnadeep Saha, Elias Stengel-Eskin, and Mohit Bansal. 2024. MAGDi: Structured Distillation of Multi-Agent Interactio...

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Select a correct node, an incorrect node, and several nodes from the previous round

work page

[2] [2]

From them, we trace back their debate history nodes as struc- tured information

The correct and incorrect nodes are chosen and rejected, respectively. From them, we trace back their debate history nodes as struc- tured information

work page

[3] [3]

InThe Twelfth International Con- ference on Learning Representations

MiniLLM: Knowledge Distillation of Large Language Models. InThe Twelfth International Con- ference on Learning Representations. Tao He, Lizi Liao, Yixin Cao, Yuanxing Liu, Ming Liu, Zerui Chen, and Bing Qin. 2024. Planning Like Hu- man: A Dual-process Framework for Dialogue Plan- ning. InProceedings of the 62nd Annual Meeting of the Association for Comput...

work page 2024

[4] [4]

The greatest common divisor (gcd) is the last non-zero value of b

work page

[5] [5]

InAd- vances in Neural Information Processing Systems, volume 36, pages 53728–53741

Direct Preference Optimization: Your Lan- guage Model is Secretly a Reward Model. InAd- vances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc. Mucheng Ren, Heyan Huang, Yuxiang Zhou, Qianwen Cao, Yuan Bu, and Yang Gao. 2022. TCM-SD: A Benchmark for Probing Syndrome Differentiation via Natural Language Proces...

work page 2024

[6] [8]

F Designed Prompt Templates in D&R In this section, we provide the prompt templates that are used in our D&R for generating debates and distillation in Figure 6, 7, 8, and 9

Repeat steps 1-2 to construct more preference trees. F Designed Prompt Templates in D&R In this section, we provide the prompt templates that are used in our D&R for generating debates and distillation in Figure 6, 7, 8, and 9. ### Instruction You are a helpful assistant aiming to answer the following question with reasoning. You are debating the input qu...

work page

[7] [9]

Find the GCD of the first two numbers

work page

[8] [10]

The greatest power of 2 that divides both numbers is 2^4

Then find the GCD of that number and the third number GCD(128, 144) 128 = 2^7 144 = 2^4 * 3^2 The prime factors are 2 and 3. The greatest power of 2 that divides both numbers is 2^4. GCD(128, 144) = 16 GCD(16, 480) 480 = 2^5 * 3 * 5 16 and 480 are both divisible by 16. Therefore, GCD(128, 144, 480) = 16 ----------------------------------------------------...

work page