REVIEW 13 cited by
Tuning Language Models by Proxy
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Despite the general capabilities of large pretrained language models, they consistently benefit from further adaptation to better achieve desired behaviors. However, tuning these models has become increasingly resource-intensive, or impossible when model weights are private. We introduce proxy-tuning, a lightweight decoding-time algorithm that operates on top of black-box LMs to achieve the same end as direct tuning, but by accessing only its predictions over the output vocabulary, not its parameters. Our method tunes a smaller LM, then applies the difference between the predictions of the small tuned and untuned LMs to shift the original predictions of the larger untuned model in the direction of tuning, while retaining the benefits of larger-scale pretraining. In experiments, when we apply proxy-tuning to Llama2-70B using proxies of only 7B size, we can close 88% of the gap between Llama2-70B and its truly-tuned chat version, when evaluated across knowledge, reasoning, and safety benchmarks. We then demonstrate the generality of proxy-tuning by applying it to domain adaptation on code, and task-specific finetuning on question-answering and math problems. Finally, we show how to proxy-tune a truly black-box LM, GPT-3.5, for temporal adaptation, increasing its knowledge about recent events. Our work demonstrates the promise of using small tuned LMs to efficiently customize large, potentially proprietary LMs through decoding-time guidance.
Forward citations
Cited by 13 Pith papers
-
Weak-to-Strong Generalization via Direct On-Policy Distillation
Transferring the log-ratio of a small model's pre-RL and post-RL checkpoints provides a dense implicit reward that improves stronger student models at a fraction of the cost of direct RL.
-
Decoupled Contrastive Decoding via Expert-Aligned Drafting
DCD, an expert-aligned proposal path with unchanged contrastive verification, speeds up contrastive decoding by 1.65 to 1.95x on 8B models while preserving the vanilla-CD output distribution.
-
Weak-to-Strong On-Policy Distillation
A strong LLM is improved by distilling from the logit difference of two weaker models instead of from a stronger teacher.
-
Stream Aligner: Efficient Sentence-Level Alignment via Distribution Induction
A sentence-level iterative correction method is claimed to improve large language model alignment and math reasoning, but the evidence is compromised by GPT-4 serving as both trainer and judge.
-
Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning
LE-MCTS searches over reasoning steps generated by different LLMs, using a process reward model to select the best trajectory, and reports accuracy gains on math benchmarks such as MATH and MathQA.
-
Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals
Relative policy-improvement signals from a weak proxy model, after simple calibration, can be transferred to improve stronger primary LLMs without re-exploring on the primary.
-
Logits are All We Need to Adapt Closed Models
Plugin trains a small autoregressive model to multiply its softmax into a frozen LLM's softmax, adapting the output distribution to a target domain using only logits and limited data.
-
A Survey on Training-free Alignment of Large Language Models
A survey that catalogs and categorizes training-free LLM alignment methods into pre-decoding, in-decoding, and post-decoding, with a limited experimental comparison on one model.
-
Logit Arithmetic Elicits Long Reasoning Capabilities Without Training
ThinkLogit blends logits from a small reasoning guider into a frozen 32B model, improving math pass@1 by up to 29% without training the large model.
-
Scalable, Symbiotic, AI and Non-AI Agent Based Parallel Discrete Event Simulations
PDES orchestration of small language models with non-AI verifier agents raises accuracy on four toy tasks from about 23 percent to 68 percent, with the verifiers supplying most of the correctness.
-
Neural Incompatibility: The Unbridgeable Gap of Cross-Scale Parametric Knowledge Transfer in Large Language Models
Directly transferring parameters between differently-sized language models is unreliable; the paper proposes a pre-alignment method (LaTen) and explains the failure via 'Neural Incompatibility'.
-
Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks
The paper organizes large-small model collaboration into downward, upward, and inference-time transfer, and advocates multi-objective benchmarks for private-domain tasks.
-
Best Practices for Large Language Models in Radiology
The paper recommends starting LLM use in radiology with prompt optimization and retrieval augmentation, fine-tuning only when needed, and preferring locally hosted open models with human expert evaluation.
Discussion (0). Continue with ORCID to comment.