REVIEW 5 cited by
PPT: Pre-trained Prompt Tuning for Few-shot Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Prompts for pre-trained language models (PLMs) have shown remarkable performance by bridging the gap between pre-training tasks and various downstream tasks. Among these methods, prompt tuning, which freezes PLMs and only tunes soft prompts, provides an efficient and effective solution for adapting large-scale PLMs to downstream tasks. However, prompt tuning is yet to be fully explored. In our pilot experiments, we find that prompt tuning performs comparably with conventional full-model fine-tuning when downstream data are sufficient, whereas it performs much worse under few-shot learning settings, which may hinder the application of prompt tuning in practice. We attribute this low performance to the manner of initializing soft prompts. Therefore, in this work, we propose to pre-train prompts by adding soft prompts into the pre-training stage to obtain a better initialization. We name this Pre-trained Prompt Tuning framework "PPT". To ensure the generalization of PPT, we formulate similar classification tasks into a unified task form and pre-train soft prompts for this unified task. Extensive experiments show that tuning pre-trained prompts for downstream tasks can reach or even outperform full-model fine-tuning under both full-data and few-shot settings. Our approach is effective and efficient for using large-scale PLMs in practice.
Forward citations
Cited by 5 Pith papers
-
GRID: Scaling Task-Agnostic Inference in Continual Prompt Tuning
GRID combines output-space constrained decoding with gradient-guided prompt compression for task-agnostic, bounded-memory continual prompt tuning.
-
Fast or Slow? Integrating Fast Intuition and Deliberate Thinking for Enhancing Visual Question Answering
FOCUS improves VQA accuracy by routing easy questions through fast zero-shot answering and hard questions through question-conditioned image segmentation before the final answer.
-
Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptation
Mettle distills frozen transformer layer features into compact meta-tokens via parallel cross-attention and linear projection, cutting training memory dramatically while retaining competitive accuracy on three audio-v...
-
ReqBrain: Task-Specific Instruction Tuning of LLMs for AI-Assisted Requirements Generation
ReqBrain, a LoRA-fine-tuned Zephyr-7b-beta model, produces software requirements that human evaluators could not reliably tell apart from human-authored ones, with automatic metrics favoring it over untuned ChatGPT-4o.
-
Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectives
A survey proposes a macro-meso-micro value framework for agentic AI alignment and maps applications, methods, and benchmarks onto it.
Discussion (0). Sign in to comment.