Pith. sign in

REVIEW 12 cited by

In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2212.10670 v1 pith:BF353SZE submitted 2022-12-20 cs.CL cs.LG

Yukun Huang , Yanda Chen , Zhou Yu , Kathleen McKeown This is my paper · ORCID
classification cs.CLcs.LG
keywords in-contextlearningobjectiveslanguagemodelsfew-shotmultitask-ictability
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Given the success with in-context learning of large pre-trained language models, we introduce in-context learning distillation to transfer in-context few-shot learning ability from large models to smaller models. We propose to combine in-context learning objectives with language modeling objectives to distill both the ability to read in-context examples and task knowledge to the smaller models. We perform in-context learning distillation under two different few-shot learning paradigms: Meta In-context Tuning (Meta-ICT) and Multitask In-context Tuning (Multitask-ICT). Multitask-ICT performs better on multitask few-shot learning but also requires more computation than Meta-ICT. Our method shows consistent improvements for both Meta-ICT and Multitask-ICT on two benchmarks: LAMA and CrossFit. Our extensive experiments and analysis reveal that in-context learning objectives and language modeling objectives are complementary under the Multitask-ICT paradigm. In-context learning objectives achieve the best performance when combined with language modeling objectives.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sample-Efficient Learning from Agent Experience

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Agents can consolidate in-context trial-and-error learning into their weights by distilling one-step teacher decisions at recorded histories, needing no additional environment interaction.

  2. FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing

    cs.CL 2024-12 reject novelty 6.0 of 10

    A token-wise pruner with a learned router and a genetic-algorithm sparsity scheduler claims near-lossless LLM inference at 22-40% token sparsity.

  3. Can Smaller LLMs do better? Unlocking Cross-Domain Potential through Parameter-Efficient Fine-Tuning for Text Summarization

    cs.CL 2025-09 reject novelty 5.0 of 10

    PEFT adapters trained on high-resource summarization domains can improve Llama-3-8B's summaries on unseen domains, but the reported gains are weakened by test-set selection and missing significance tests.

  4. AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes

    cs.AI 2025-06 reject novelty 5.0 of 10

    AgentDistill distills agent capabilities without any training by having a teacher generate reusable MCP tool boxes that small-model students invoke at inference time.

  5. Extracting Interpretable Task-Specific Circuits from Large Language Models for Faster Inference

    cs.LG 2024-12 conditional novelty 5.0 of 10

    A greedy, ablation-based pruning method extracts a standalone task-specific subnetwork from GPT-2 Small, reducing parameters by up to 82.77% while keeping accuracy on three synthetic single-token tasks.

  6. Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks

    cs.CL 2025-07 conditional novelty 4.0 of 10

    Distilled students at 43% to 50% of teacher size keep over 90% of teacher Exact Match on SQuAD and MLQA, though one-shot gains reverse on SQuAD test for Pythia.

  7. RLRC: Reinforcement Learning-based Recovery for Compressed Vision-Language-Action Models

    cs.RO 2025-06 conditional novelty 4.0 of 10

    RLRC combines 90% structured pruning, supervised fine-tuning, PPO reinforcement learning, and optional 4-bit quantization to recover OpenVLA's success rate on ManiSkill while cutting memory up to 8x and boosting throu...

  8. SWSC: Shared Weight for Similar Channel in LLM

    cs.LG 2025-01 conditional novelty 4.0 of 10

    SWSC combines channel K-means clustering with an SVD low-rank error correction to compress LLM weights, and reports lower perplexity than RTN quantization on Llama-2-7B Q and K projections at 2 to 3 average bits.

  9. LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment

    cs.CL 2024-12 conditional novelty 4.0 of 10

    LSAQ assigns higher quantization precision to layers deemed important by the overlap of top-k input and output token sets, and reports small accuracy and perplexity gains over a cosine-similarity baseline.

  10. Scaling New Frontiers: Insights into Large Recommendation Models

    cs.IR 2024-12 conditional novelty 4.0 of 10

    Deep HSTU models tend to improve recall, ranking, multi-behavior, and multi-domain performance on public data, while GPT and SASRec fail to scale, though the evidence lacks error bars.

  11. FASTNav: Fine-tuned Adaptive Small-language-models Trained for Multi-point Robot Navigation

    cs.RO 2024-11 conditional novelty 4.0 of 10

    Fine-tuned small language models, coached by a GPT-4 teacher through iterative prompting, can perform multi-point robot navigation on edge devices with success rates approaching larger models.

  12. Large Language models for Time Series Analysis: Techniques, Applications, and Challenges

    cs.LG 2025-05 reject novelty 3.0 of 10

    A review of LLM-based time series analysis that proposes several taxonomies, but is undermined by citation errors and a lack of systematic methodology.

Pith tools