REVIEW 6 cited by
Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Chain-of-thought prompting (e.g., "Let's think step-by-step") primes large language models to verbalize rationalization for their predictions. While chain-of-thought can lead to dramatic performance gains, benefits appear to emerge only for sufficiently large models (beyond 50B parameters). We show that orders-of-magnitude smaller models (125M -- 1.3B parameters) can still benefit from chain-of-thought prompting. To achieve this, we introduce Symbolic Chain-of-Thought Distillation (SCoTD), a method to train a smaller student model on rationalizations sampled from a significantly larger teacher model. Experiments across several commonsense benchmarks show that: 1) SCoTD enhances the performance of the student model in both supervised and few-shot settings, and especially for challenge sets; 2) sampling many reasoning chains per instance from the teacher is paramount; and 3) after distillation, student chain-of-thoughts are judged by humans as comparable to the teacher, despite orders of magnitude fewer parameters. We test several hypotheses regarding what properties of chain-of-thought samples are important, e.g., diversity vs. teacher likelihood vs. open-endedness. We release our corpus of chain-of-thought samples and code.
Forward citations
Cited by 6 Pith papers
-
Rationales Are Not Silver Bullets: Measuring the Impact of Rationales on Model Performance and Reliability
Rationale-augmented finetuning can hurt accuracy while improving calibration, with the sizes of both effects tied linearly to task difficulty.
-
iReDev: A Knowledge-Driven Multi-Agent Framework for Intelligent Requirements Development
A knowledge-driven, event-triggered multi-agent framework called iReDev generates software requirements artifacts that outperform zero-shot prompting, MetaGPT, and Elicitron on ten small projects.
-
AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes
AgentDistill distills agent capabilities without any training by having a teacher generate reusable MCP tool boxes that small-model students invoke at inference time.
-
Multi-MLLM Knowledge Distillation for Out-of-Context News Detection
A two-stage LoRA plus DPO distillation from two large MLLMs lets a 7B student detect out-of-context news with 90.04% accuracy on NewsCLIPpings while using only 8.61% labeled data.
-
AI-Augmented LLMs Achieve Therapist-Level Responses in Motivational Interviewing
A custom prompt built from machine-learning-identified therapy behavior features improved GPT-4's motivational interviewing quality scores, though the model remained slightly below human therapists on the paper's own metric.
-
Robust pid sliding mode control for dc servo motor speed control
An abstract-only claim that SMC-PID outperforms PID for DC servo motor speed on the CE110 trainer; the submitted body text is an unrelated paper, so the result is unverifiable.
Discussion (0). Sign in to comment.