Pith. sign in

REVIEW 63 cited by

AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts

Not yet reviewed by Pith; the record is open.

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2010.15980 v2 pith:5J6CR6ZL submitted 2020-10-29 cs.CL cs.LG

classification cs.CLcs.LG
keywords modelspromptsknowledgelanguageautopromptmlmsautomaticallyfinetuning
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining. Reformulating tasks as fill-in-the-blanks problems (e.g., cloze tests) is a natural approach for gauging such knowledge, however, its usage is limited by the manual effort and guesswork required to write suitable prompts. To address this, we develop AutoPrompt, an automated method to create prompts for a diverse set of tasks, based on a gradient-guided search. Using AutoPrompt, we show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning, sometimes achieving performance on par with recent state-of-the-art supervised models. We also show that our prompts elicit more accurate factual knowledge from MLMs than the manually created prompts on the LAMA benchmark, and that MLMs can be used as relation extractors more effectively than supervised relation extraction models. These results demonstrate that automatically generated prompts are a viable parameter-free alternative to existing probing methods, and as pretrained LMs become more sophisticated and capable, potentially a replacement for finetuning.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 63 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 63 Pith citations

  1. Jailbreaking to Jailbreak

    cs.CL 2025-02 conditional novelty 7.0 of 10

    A transferable multi-turn jailbreak turns refusal-trained black-box LLMs into willing automated jailbreakers, with high attack success against other models and against themselves.

  2. Leveraging Large Vision-Language Model as User Intent-aware Encoder for Composed Image Retrieval

    cs.IR 2024-12 conditional novelty 7.0 of 10

    CIR-LVLM fine-tunes Qwen-VL-Chat with LoRA and hybrid task and instance-specific prompts to produce query and target embeddings, achieving new state-of-the-art recall on Fashion-IQ, Shoes, and CIRR.

  3. Degradation-Aware Prompt Learning with Cross-Modal Compensation for Adverse Weather Removal

    cs.CV 2026-08 conditional novelty 6.0 of 10

    DCMPC-Net couples LLaMA-generated weather captions with visual features through cross-modal prompts to restore images degraded by rain, snow, fog, and raindrops in a single model.

  4. TAPR: Enhancing LLM Performance with a Task-Aware Prompt Rewriter

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A small LLM trained with GRPO and LLM-judge rewards rewrites simple prompts into more effective ones, improving question-answering and arithmetic accuracy over base prompts while giving mixed, often negligible gains o...

  5. Efficient Safety Alignment of Language Models via Latent Personality Traits

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Latent adversarial training on 66 harm-agnostic Big-Five personality statements yields near-zero HarmBench ASR across direct requests and five jailbreaks while preserving utility.

  6. Between Knowledge and Care: A Mixed-Methods Evaluation of Generative AI for T2DM Self-Management from Patient and Physician Perspectives

    cs.HC 2026-07 conditional novelty 6.0 of 10

    Generative AI aids T2DM self-management on facts and lifestyle but fails on meds and emotion; patients and physicians converge on role limits, emotional gaps, and personalization needs, informing four design directions.

  7. DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection

    cs.CV 2026-03 conditional novelty 6.0 of 10

    Detection Prompt Optimization (DetPO) improves few-shot object detection with black-box MLLMs by iteratively refining text prompts from TP/FP/FN errors on few-shot examples, gaining up to 9.7 mAP over prior black-box methods.

  8. Adaptive Content Restriction for Large Language Models via Suffix Optimization

    cs.CL 2025-08 conditional novelty 6.0 of 10

    SOP appends an optimized suffix to prompts, reducing generation of user-specified restricted terms across several LLMs while keeping output quality close to baseline prompting methods.

  9. DEMONSTRATE: Zero-shot Language to Robotic Control via Multi-task Demonstration Learning

    cs.RO 2025-07 conditional novelty 6.0 of 10

    DEMONSTRATE learns a zero-shot mapping from natural-language embeddings to MPC cost parameters from demonstrations, achieving tabletop manipulation success rates comparable to prior LLM-based pipelines.

  10. What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests

    cs.CL 2025-07 conditional novelty 6.0 of 10

    WikiMem, a Wikidata-derived canary dataset and a calibrated NLL-ranking metric, identifies which human-fact associations an LLM has memorized, with higher rates for famous people and larger models.

  11. Stabilizing Black-Box Prompt Optimization with Textual Regularization and Signal Aggregation

    cs.LG 2025-07 conditional novelty 6.0 of 10

    TRAS adds success-based textual regularization and Monte Carlo signal aggregation to black-box prompt optimization, improving accuracy and reducing instruction loss when moving prompts across models.

  12. ConceptMix++: Leveling the Playing Field in Text-to-Image Benchmarking via Iterative Prompt Optimization

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Iteratively optimized prompts improve compositional text-to-image scores by up to 20% across three models, and the improved prompts transfer across models.

  13. MT2-CSD: A New Dataset and Multi-Semantic Knowledge Fusion Method for Conversational Stance Detection

    cs.CL 2025-06 conditional novelty 6.0 of 10

    The paper presents a large new English conversational stance detection dataset and a model that fuses LLM-generated relation and act knowledge, reporting state-of-the-art F1.

  14. Test3R: Learning to Reconstruct 3D at Test Time

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Test3R improves 3D reconstruction by optimizing visual prompts at test time so that pointmaps from different image pairs are geometrically consistent.

  15. TwiUSD: A Benchmark Dataset and Structure-Aware LLM Framework for User Stance Detection

    cs.SI 2025-06 conditional novelty 6.0 of 10

    TwiUSD is a new manually annotated user-level stance benchmark with follower links, and MRFG, which filters followee tweets with an LLM and routes features by graph usefulness, reports top in-target performance.

  16. Because we have LLMs, we Can and Should Pursue Agentic Interpretability

    cs.AI 2025-06 conditional novelty 6.0 of 10

    Agentic interpretability, using LLMs as proactive conversational teachers that model the user, is offered as a needed complement to black-box interpretability.

  17. Lifelong Safety Alignment for Language Models

    cs.CR 2025-05 conditional novelty 6.0 of 10

    A co-evolutionary attacker-defender loop, warmed up by strategies extracted from jailbreak papers, reduces jailbreak success rate on a robust model from 73% to 7% in two iterations.

  18. Transferable Adversarial Attacks on Black-Box Vision-Language Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Targeted, barely visible image perturbations transfer from open-source surrogate models to proprietary black-box VLLMs like GPT-4o, Claude, and Gemini, achieving high attack success on captioning, VQA, and receipt tex...

  19. Diff-Prompt: Diffusion-Driven Prompt Generator with Mask Supervision

    cs.CV 2025-04 conditional novelty 6.0 of 10

    Diff-Prompt uses a mask-supervised diffusion model to generate input-specific visual and textual prompts, improving GLIP on referring expression comprehension beyond existing prompt tuning methods.

  20. Diverse Prompts: Illuminating the Prompt Space of Large Language Models with MAP-Elites

    cs.CL 2025-04 conditional novelty 6.0 of 10

    CFG plus MAP-Elites produces structurally diverse high-performing prompts and shows task-specific effects, with zero-shot prompts winning on logic tasks.

  21. Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities

    cs.CR 2025-02 conditional novelty 6.0 of 10

    Model tampering attacks, especially few-shot fine-tuning, reliably re-elicit unlearned capabilities in Llama-3-8B and can bound the success of held-out input-space attacks.

  22. Adversarial Reasoning at Jailbreaking Time

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A loss-guided 'reason, verify, search' loop with three LLM modules surpasses prior semantic jailbreak methods on several defended models.

  23. OptiSeq: Ordering Examples On-The-Fly for In-Context Learning

    cs.LG 2025-01 conditional novelty 6.0 of 10

    OptiSeq selects the in-context example ordering whose output gets the highest zero-shot log-likelihood, improving few-shot accuracy by up to 10.5 points in tests on API sequencing and classification.

  24. Online Prompt Selection for Program Synthesis

    cs.AI 2025-01 conditional novelty 6.0 of 10

    An online multi-armed bandit that selects among symbolic solvers and LLM-prompt combinations for program synthesis solves 37.2% more queries than the best single solver and reaches 96% of the virtual best solver's per...

  25. Auto-RT: Automatic Jailbreak Strategy Exploration for Red-Teaming Large Language Models

    cs.CR 2025-01 conditional novelty 6.0 of 10

    Auto-RT uses early-terminated exploration plus reward shaping from progressively weakened copies of the target model to automatically discover jailbreak strategies, reporting up to 16.63% higher attack success than baselines.

  26. Differentiable Prompt Learning for Vision Language Models

    cs.LG 2024-12 conditional novelty 6.0 of 10

    DPL searches the per-layer prompt length for CLIP with differentiable architecture search, and reports higher few-shot accuracy than fixed-length prompt baselines.

  27. DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A diffusion-based prompt rewriter that pushes rewritten prompts toward harmless regions of a target model's hidden states achieves higher jailbreak success than existing suffix and template attacks.

  28. Mr. DETR++: Instructive Multi-Route Training for Detection Transformers with Mixture-of-Experts

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Multi-route training with instructive self-attention tokens and a route-aware mixture-of-experts raises detection mAP by 2 to 4 points across several DETR baselines at no inference cost.

  29. GReaTer: Gradients over Reasoning Makes Smaller Language Models Strong Prompt Optimizers

    cs.CL 2024-12 conditional novelty 6.0 of 10

    A gradient-based discrete prompt optimizer that uses reasoning chains to let small LMs self-optimize prompts, outperforming text-feedback baselines on reasoning benchmarks.

  30. A Noise is Worth Diffusion Guidance

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A one-step learned noise refinement replaces classifier-free guidance at inference on Stable Diffusion 2.1, giving comparable image quality at about 1.7x lower cost.

  31. Universal and Context-Independent Triggers for Precise Control of LLM Outputs

    cs.CL 2024-11 conditional novelty 6.0 of 10

    A single trained token pair inserted around any target text forces Qwen-2 7B and Llama-3.1 8B to output that text on 54 to 75 percent of unseen prompts.

  32. IntentGPT: Few-shot Intent Discovery with Large Language Models

    cs.CL 2024-11 conditional novelty 6.0 of 10

    A training-free LLM prompting pipeline with semantic few-shot retrieval and feedback of discovered intents outperforms trained baselines on few-shot intent discovery benchmarks.

  33. On the Privacy Risk of In-context Learning

    cs.LG 2024-11 conditional novelty 6.0 of 10

    A confidence-based membership inference attack identifies prompt demonstration data with AUC 0.69-0.86, more than fine-tuned models leak at matched utility, and ensembling reduces this to near random.

  34. PROBE: Benchmarking Code Generation in Large Language Models

    cs.SE 2026-07 conditional novelty 5.0 of 10

    A multi-language evaluation framework measuring correctness, solution proximity, and code quality finds current LLMs pass at most ~0.70 per language and worsen sharply with problem difficulty.

  35. Data-Efficient Adaptation of LLMs via Attention Head Reweighting

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Learning a single scalar per attention head lets LLMs adapt to few-shot text classification better than LoRA, with 200–1000x fewer trainable parameters.

  36. Metadata Management for AI-Augmented Data Workflows

    cs.DB 2025-08 conditional novelty 5.0 of 10

    TableVault is a metadata governance framework that records ingestion events, operation status, execution parameters, and lineage for human-AI data workflows, demonstrated on a document classification case study.

  37. Causality-aligned Prompt Learning via Diffusion-based Counterfactual Generation

    cs.AI 2025-07 conditional novelty 5.0 of 10

    DiCap generates diffusion-based counterfactual images and uses them as hard negatives in contrastive prompt learning, reporting modest gains over existing prompt learning baselines on unseen classes.

  38. Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Tree

    cs.CR 2025-07 conditional novelty 5.0 of 10

    GCG-optimized trigger strings embedded in HTML can command LLM web agents to perform attacker-chosen actions, including credential exfiltration.

  39. VERA: Variational Inference Framework for Jailbreaking Large Language Models

    cs.CR 2025-06 conditional novelty 5.0 of 10

    VERA frames black-box jailbreaking as variational inference, training a LoRA-tuned attacker that samples diverse fluent prompts; reported ASRs are high but several evaluation choices weaken the SOTA claims.

  40. Adversarial Attack on Large Language Models using Exponentiated Gradient Descent

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Exponentiated gradient descent over relaxed one-hot token encodings finds adversarial suffixes that jailbreak several open-source LLMs with higher success rate and lower runtime than GCG, PGD, and SoftPromptThreats.

  41. FrogDogNet: Fourier frequency Retained visual prompt Output Guidance for Domain Generalization of CLIP in Remote Sensing

    cs.CV 2025-04 conditional novelty 5.0 of 10

    FrogDogNet applies Fourier filtering and self-attention to CLIP visual features before prompt learning, reporting new state-of-the-art remote sensing domain generalization results on four benchmarks.

  42. CodeSCM: Causal Analysis for Multi-Modal Code Generation

    cs.CL 2025-02 conditional novelty 5.0 of 10

    A causal framework with latent mediators quantifies how prompt modalities affect code LLMs, finding that input-output examples and function-header names are influential beyond natural language instructions.

  43. LLMs to Support a Domain Specific Knowledge Assistant

    cs.CL 2025-02 conditional novelty 5.0 of 10

    A synthetic QA dataset for IFRS sustainability reporting is created with LLMs and used to build and evaluate two QA pipelines, with the fully LLM-based pipeline scoring highest.

  44. A Sequential Optimal Learning Approach to Automated Prompt Engineering in Large Language Models

    cs.CL 2025-01 conditional novelty 5.0 of 10

    A Bayesian 'knowledge gradient' policy for sequentially choosing which prompts to evaluate finds better language-model prompts within 30 evaluations than evolutionary, bandit, and greedy baselines on instruction-induc...

  45. LLM-Virus: Evolutionary Jailbreak Attack on Large Language Models

    cs.CR 2024-12 conditional novelty 5.0 of 10

    LLM-Virus uses an evolutionary algorithm with an LLM as crossover, mutation, and fitness operator to evolve jailbreak templates, reporting state-of-the-art attack success on HarmBench and AdvBench.

  46. Prompt Categories Cluster for Weakly Supervised Semantic Segmentation

    cs.CV 2024-12 conditional novelty 5.0 of 10

    LLM-prompted category clusters, injected as a single learnable token, raise PASCAL VOC segmentation mIoU from 68.6 to 72.2 in a weakly supervised ViT-BiLSTM framework.

  47. ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts

    cs.CV 2024-12 conditional novelty 5.0 of 10

    ALoRE aggregates multiple low-rank experts in a Kronecker-product space and merges them into the frozen backbone, reporting top accuracy on FGVC and VTAB-1k with only 0.15M trainable parameters.

  48. Panther: Illuminate the Sight of Multimodal LLMs with Instruction-Guided Visual Prompts

    cs.CV 2024-11 conditional novelty 5.0 of 10

    Panther improves multimodal LLMs by converting the user's text question into visual prompts that steer a frozen image encoder toward instruction-relevant regions, gaining about 2 to 3 points on several VQA benchmarks ...

  49. Scaling Personality Control in LLMs with Big Five Scaler Prompts

    cs.CL 2025-08 conditional novelty 4.0 of 10

    Numeric Big Five trait values placed in prompts shift LLMs' self-reported and dialogue-expressed personality, with simple prompts and low intensity scales working best.

  50. Context, Credibility, and Control: User Reflections on AI Assisted Misinformation Tools

    cs.HC 2025-06 conditional novelty 4.0 of 10

    In a 14-participant study of a prototype misinformation tool, 79% preferred debate-style AI interaction over a standard chatbot.

  51. Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings

    cs.AI 2025-05 conditional novelty 4.0 of 10

    DEEVO evolves better LLM prompts by debating outputs and selecting survivors with Elo ratings, without requiring labeled data or a hand-written fitness function.

  52. Adversarial Preference Learning for Robust LLM Alignment

    cs.LG 2025-05 conditional novelty 4.0 of 10

    APL iteratively trains an attacker to generate adversarial prompt rewrites and a defender to resist them, using the defender's own preference probabilities as the attack signal.

  53. System Prompt Extraction Attacks and Defenses in Large Language Models

    cs.CR 2025-05 conditional novelty 4.0 of 10

    A benchmarking study shows that chain-of-thought, few-shot, and modified sandwich queries can recover LLM system prompts with high similarity-based success, and output filtering is the most reliable tested defense.

  54. MODP: Multi Objective Directional Prompting

    cs.CC 2025-04 reject novelty 4.0 of 10

    MODP is a metrics-driven, multi-objective prompt engineering framework that, in this paper, improves ReCoRD fill-in-the-blank accuracy from 48% to 73% on Mixtral by adding instructions, toxicity handling, and model-sp...

  55. CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey

    cs.CV 2025-04 conditional novelty 4.0 of 10

    CLIP-powered domain generalization and domain adaptation methods are surveyed and categorized into prompt-learning versus backbone use, and source-available versus source-free settings.

  56. LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Seconds

    cs.CL 2024-12 reject novelty 4.0 of 10

    LIAR shows that best-of-N sampling of suffixes from a GPT-2 model jailbreaks several aligned LLMs with low-perplexity prompts and far faster time-to-attack than training-based attacks.

  57. The Evolution of Natural Language Processing: How Prompt Optimization and Language Models are Shaping the Future

    cs.CL 2025-06 reject novelty 3.0 of 10

    A review that categorizes 45 prompt optimization strategies into 11 classes and surveys their use across NLP tasks, models, and datasets, but with inconsistent counts and overlapping categories.

  58. SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation

    cs.CR 2025-06 conditional novelty 3.0 of 10

    A systematization-of-knowledge survey that categorizes LLM privacy risks into training data, prompts, outputs, and agents, and reviews limitations of current mitigations.

  59. Transforming Expert Knowledge into Scalable Ontology via Large Language Models

    cs.AI 2025-06 conditional novelty 3.0 of 10

    An LLM-based taxonomy alignment framework reaches 0.97 F1 using many-shot prompting and expert calibration, but the claimed superiority over the 0.68 human benchmark is based on a non-comparable baseline.

  60. Large Language models for Time Series Analysis: Techniques, Applications, and Challenges

    cs.LG 2025-05 reject novelty 3.0 of 10

    A review of LLM-based time series analysis that proposes several taxonomies, but is undermined by citation errors and a lack of systematic methodology.

See all 63 Pith citations

Pith tools