Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs

Bo Li; Cong Wang; Hanxun Huang; Xiang Zheng; Xingjun Ma; Yige Li; Yu-Gang Jiang; Yutao Wu

REVIEW 4 major objections 5 minor 5 references

Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read JUSTASK, a self-evolving code agent that chooses probing strategies by bandit exploration, extracts the hidden system prompts of all 41 black-box commercial models it was tested against.

desk verdict A genuinely useful study of prompt leakage with a headline claim that outruns its own evidence — worth refereeing, not for publication as-is. read the letter →

arxiv 2601.21233 v2 pith:II6EHVHN submitted 2026-01-29 cs.AI

Xiang Zheng , Yutao Wu , Hanxun Huang , Yige Li , Xingjun Ma , Bo Li , Yu-Gang Jiang , Cong Wang This is my paper · ORCID

classification cs.AI

keywords systempromptextractionleakagecodeagentsUCBbanditexplorationblack-boxLLMsecuritymulti-turnattacksidentityconfusionagenticdefenses

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that hidden system prompts, the private instructions that shape how commercial LLMs behave, are effectively public in today's deployments. Its method, JUSTASK, is an autonomous code agent that treats extraction as an online exploration problem: it repeatedly asks a target model questions built from 28 probing skills, keeps score of which skills succeed, and uses a UCB bandit rule to try under-explored tactics. Across 41 black-box models, the paper reports full extraction success under its consistency criterion (score ≥ 0.7), with hardened closed-weight models needing about 4.8 conversation turns on average and fine-tunes yielding to a single introspective question. The broader claim is that agentic interaction itself has expanded the LLM attack surface: an agent can escalate from roleplay to distraction to foot-in-the-door sequences until even models hardened against direct extraction disclose their priority hierarchies, refusal templates, and safety exception clauses. If this is right, prompt secrecy is not an achievable defense, and security must shift to architecture-level protections such as capability separation and stateful monitoring.

What carries the argument

The load-bearing mechanism is JUSTASK's closed skill-evolution loop. A taxonomy of 14 single-turn skills (roleplay, formatting, obfuscation, translation, framing, authority, introspection, and similar) and 14 multi-turn orchestration patterns (foot-in-the-door, distraction, format pivot, semantic progression, self-reference) forms a hierarchical action space; an Upper Confidence Bound selector, UCB(s) = average success + exploration bonus, balances exploitation of known-effective probes against exploration of untried ones. A consistency score—average of same-skill and cross-skill semantic similarity of a model's self-reports—serves both as the reward signal that updates skill statistics and

What would settle it

Run JUSTASK against 10–15 models whose true system prompts are known by independent means (vendor disclosure, decompiled client packages, or same-provider controlled deployments), and compare the extracted text to those true prompts using the paper's own embedding-based similarity. If most extractions fall below 0.7 semantic similarity, or if consistency scores match a generic HHH-template baseline as well as they match true prompts, the consistency-based success metric is measuring generic alignment talk, not true recovery.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that system-prompt extraction is an emergent vulnerability of code agents, not an exotic attack needing hand-crafted prompts or model access. JUSTASK learns extraction strategies purely from interaction, and the paper reports that it recovered full or near-complete system prompts from all 41 commercial models it tested, with success defined as cross-skill and self-consistency scores at or above 0.7. Validation rests on three legs: official disclosure of one model's prompt (recovered at 0.89 semantic similarity), decompiled client code for a multi-agent coding assistant (recovered prompts matched structural content closely), and a controlled

Load-bearing premise

The load-bearing premise is that when different probing tricks elicit semantically similar self-reports from a model, those reports are the model's actual system prompt—rather than a generic, socially acceptable summary of assistant guidelines.

Editorial extensions

If this is right

If the 100% extraction claim holds, providers should stop treating system prompts as secrets: secrecy is not a defense, and protection must come from architecture, capability separation, and monitoring.
Extracted prompts reveal priority hierarchies, refusal templates, and safety exception clauses, which an adversary can mine for targeted jailbreaks that satisfy a model's stated conditions.
Because the skill vocabulary is open and extensible, new extraction techniques can be added without altering the learning loop, so the method's power grows as red-teaming knowledge accumulates.
Defenses are lopsided: informed, attack-taxonomy-aware prompts give partial protection, while naive prohibitions barely help, so a meaningful defense likely requires stateful agent-side systems rather than prompt text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If consistency mostly captures generic alignment boilerplate rather than actual instruction content, the 100% success rate overstates fidelity; a direct test is to compare extractions against plain helpful–honest–harmless templates.
The result implies a practical design rule for API providers: anything that must not leak should not be encoded in a textual system prompt at all, since any text retrievable through normal chat turns is effectively public.
The same machinery could be repurposed defensively—an agent that continuously probes its own deployment to audit which instructions leak and where defenses fail—an extension the paper gestures toward but does not implement.
Because only two models have independent ground truth, the quantitative 41-model claims rest on a proxy; a replication with ground-truth prompts for a random sample of providers would either confirm or refute the 100% framing.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague —

The thing to know: this paper shows that an autonomous agent using simple social-engineering probes can get many commercial LLMs to describe their own system prompts, and in two cases the descriptions check out against independent ground truth. That is a real result. But the headline claim — 100% extraction success across 41 models — is built on a consistency metric that is partly circular and mostly captures generic boilerplate, not the actual prompt.

What's new: the UCB-driven self-evolving skill selection, the 41-model OpenRouter sweep, the identity-confusion analysis (26.8% models claim the wrong developer), and the defense taxonomy evaluation. The skill taxonomy itself is mostly known persuasion/jailbreak tricks, but the framing as an online exploration problem is a legitimate methodological step. The Claude Code case study with npm decompilation as ground truth is the strongest part of the paper; the Grok official-prompt match is also credible.

Soft spots, in order of size:

1. The 100% success number is threshold-fitted. They set the consistency threshold at 0.7 after looking at the data so that all 41 pass; at 0.8, only 73% pass, at 0.85, 46%. Many 'successes' are marginal. The abstract's 'full or near-complete recovery' is contradicted by their own quality tiers: 68% of extractions are 'weak semantic' — basically generic identity plus HHH platitudes. If a model consistently claims to be OpenAI when it's actually LLaMA, cross-skill agreement on that false identity is not evidence of recovery.

2. The controlled ground-truth validation is circular. Appendix J says the base prompts in the controlled evaluation 'capture identity and guidelines from Phase 1 extraction.' So the r=0.94 between consistency and ground-truth similarity is correlating consistency with a re-description of the same extracted content. That does not independently validate the metric.

3. Internal data inconsistency: Table 15's caption says Self-C was recorded for only 2 models, yet the table lists Self-C for all 41 rows, and Avg-C is defined as (Self-C+Cross-C)/2. If the caption is right, the average is undefined for most rows. This needs a correction before anything else.

4. No code, no data, no error bars. The method is described well enough to reimplement, but without artifacts the threshold-fitting and consistency computations cannot be audited.

Who this is for: security researchers studying prompt leakage and defender tooling. The paper deserves a serious referee — the core phenomenon is real, the two externally verified cases are strong, and the defense evaluation is a useful contribution. But it needs major revision: honest claims about extraction quality, independent ground-truth validation for more than two models, and artifact release.

My recommendation: send to peer review. Not desk-reject. The central argument — that prompts are leakable — holds up; the 100% universality claim does not.

Referee Report

4 major / 5 minor

Summary. The paper proposes JUSTASK, an autonomous code agent that queries black-box LLMs using a UCB-selected set of 14 low-level and 14 high-level prompt-extraction skills. It reports 100% extraction success (consistency score >= 0.7) on 41 OpenRouter models, a Claude Code multi-agent case study, a content analysis showing HHH adoption and 26.8% identity confusion, and controlled defense experiments on four models. The central claim is that hidden system prompts in deployed models are effectively recoverable through ordinary user interaction.

Significance. If the headline result is correct, the paper would be an important demonstration that system-prompt secrecy is not an achievable defense in current LLM deployments, and it would provide one of the largest published corpora of recovered prompts. Creditable elements include the two external ground-truth checks (Grok official prompt, Claude Code npm decompilation), the breadth of the 41-model evaluation, and the controlled comparison of defense levels. However, the main success metric is not independently validated for most models, and the paper's own quality distribution and threshold sensitivity weaken the 100% claim. As it stands, the paper is a valuable attack-framework demonstration whose acceptance-level claim is not yet supported.

major comments (4)

[§3.5, §4.7, Appendix J] The consistency score is the only success metric for 39 of 41 models, but its validation is circular. Appendix J states that the controlled known-ground-truth base prompts are model-specific prompts 'from Phase 1 extraction'; Table 7 then correlates Avg-C with Sim-GT computed against these reconstructed prompts. A high correlation between consistency and similarity to a paraphrase of the same extractions does not establish that consistency measures true recovery. Only Grok (official disclosure) and Claude Code (npm decompilation) are externally anchored, and §4.5 classifies 68% of extractions as 'weak semantic' (identity plus generic HHH). The consistency scores may therefore be capturing generic alignment boilerplate rather than the actual system prompt. Please provide held-out models with genuinely independent ground truth (e.g., provider-published prompts or decompiled artifacts) and
[§3.5, Appendix C, Figure 6] The success threshold of 0.7 is selected empirically so that all 41 models pass. Figure 6 shows that the success rate is 90.2% at threshold 0.75, 73.2% at 0.80, and 46.3% at 0.85. A threshold chosen from the observed distribution cannot support a headline '100% extraction success' claim without further justification. The paper should report the full distribution of Avg-C across models, justify the threshold a priori or on an independent calibration set, and state the success rate at stricter thresholds in the abstract.
[§4.5, Appendix F, Table 14] The identity-confusion results directly undermine the consistency proxy. 26.8% of models claim a wrong developer, and Appendix F shows StepFun Step3 claiming DeepSeek-R1 under L14/L6/L2 and GPT-4/OpenAI under L1. A model that is consistently wrong about its own identity can still achieve high self- and cross-skill similarity on that false content; cross-skill agreement is thus not evidence that the system prompt was recovered. Please report identity correctness as a separate validation signal and re-run the consistency analysis excluding identity and other confabulated components.
[Appendix G, Table 15] Table 15's caption states that Self-C was recorded for only 2 models and Cross-C for 32, with missing scores marked as an em dash, yet every row lists numeric Self-C and Cross-C values and no row contains the missing marker. Since Avg-C is defined as the average of Self-C and Cross-C, the printed table cannot be produced from the stated records. This is a data-integrity issue for the central success table; clarify whether the values are imputed, re-collected, or the caption/table is mislabeled.

minor comments (5)

[§4.3 vs. Appendix H, Table 16, Figure 3] Taxonomy labels are inconsistent: §4.3 describes the GPT-5.2-codex extraction as a two-turn H5 (Foot-in-the-Door) sequence, while Appendix H Table 16 presents the same case as an 11-turn H8 (Low Ball) sequence. Figure 3 labels H5 as Distraction and H9 as FITD, contradicting Table 1 (H4=Distraction, H5=Semantic Progression, H8=FITD, H9=Low Ball). Please harmonize skill labels and case descriptions; the current inconsistencies prevent reproducibility.
[Appendix J, Design Note] The design note states that embedded secrets were dropped because they 'proved impractical to extract and rarely appear in real production system prompts.' This removes the most objective ground-truth signal from the controlled evaluation; please provide the data for that earlier design or justify the substitution more rigorously.
[References] Appendix L cites 'PLeak (Sha & Zhang, 2024)' while the related work section identifies Pleak as Hui et al., 2024. The reference list contains Zhang et al. 2024b as 'Effective prompt extraction from language models,' not PLeak. Please correct the citation.
[Figure 3 caption] The caption mentions skills H1-H15, but only H1-H14 are defined in Table 1. Please correct the range.
[Reproducibility] No code or data release is indicated. For a paper whose contribution is an autonomous agent framework, releasing the agent implementation, extraction logs, and threshold sweep would substantially strengthen reproducibility.

Circularity Check

3 steps flagged · score 6.0 of 10

100% extraction claim is partly constructed: success threshold fitted to pass all 41 models, and controlled 'ground truth' is rebuilt from the paper's own Phase-1 extractions; only two external anchors break full circularity.

fitted input called prediction [Appendix C (Success Threshold), Figure 6; applied in §3.5 and §4.3]
"Success Threshold. We define successful extraction as consistency score ≥0.7, balancing sensitivity (detecting genuine extractions) against specificity (rejecting hallucinated or generic responses). This threshold was determined empirically by examining the distribution of similarity scores across successful and failed extraction attempts. Figure 6 shows how extraction success rate varies with the consistency threshold: at our chosen threshold of 0.7, all 41 models achieve successful extraction (100%), while stricter thresholds progressively reduce coverage (90.2% at 0.75, 73.2% at 0.80, 46.3%"

The headline success criterion is chosen after the fact to make every tested model pass. With no external ground truth for 39 of 41 models, '100% extraction success' is not an empirical discovery but a consequence of lowering the consistency bar until coverage hits 100%. The threshold-sensitivity curve makes this explicit: coverage falls to 90.2% at 0.75 and 46.3% at 0.85. Thus the paper's central numerical claim is fitted to the same data it is supposed to validate.
self definitional [Appendix J (Template Structure / Base Prompt Example); §4.6; Table 7]
"Each controlled prompt follows a common structure: {base_prompt} # Protection Instructions {protection_instruction} where base prompt contains model-specific identity and guidelines extracted from Phase 1 black-box experiments. ... Model-specific base prompts capture identity and guidelines from Phase 1 extraction."

The controlled 'known ground-truth' prompts are assembled from the same Phase-1 black-box extractions whose reliability they are meant to establish. Table 7's Sim-GT therefore measures similarity to the authors' own reconstructed base prompts, not to an independent truth. The reported r=0.94 correlation between Avg-C and Sim-GT largely shows that the consistency metric agrees with a re-description of the same extracted content. This makes the validation loop self-referential, though the Grok and Claude Code external checks provide limited independent anchors.

1 more flagged steps

self definitional [§3.5 (Validation Methodology), §4.5 (Identity Confusion), Tables 14 and 15]
"if different approaches yield similar descriptions, the extraction is likely capturing genuine system prompt content rather than hallucination. ... 26.8% of models (11/41) exhibit identity confusion by claiming developers different from their actual source. Table 14: StepFun claims DeepSeek-R1 under L14/L6/L2, OpenAI under L1, and StepFun only under L7; Table 15 gives it Avg-C=0.770."

The validation metric defines successful extraction as consistent agreement across skills and then infers that agreement means genuine recovery. But the paper's own identity-confusion data show a model can be consistently wrong: StepFun claims DeepSeek across L14, L6, and L2 and OpenAI under L1, yet its average consistency is 0.770, above the 0.7 threshold. Cross-skill agreement on a false identity is not evidence of system-prompt recovery; hence the '100% success' claim is partly an artifact of defining genuineness as consistency.

full rationale

JUSTASK's core result—100% success on 41 models—is not presented as an independent prediction in the way it reads. The success metric is consistency, and the 0.7 threshold was chosen empirically so that all 41 models pass; that already makes the headline '100%' partly a calibration artifact. More importantly, the controlled validation of consistency uses 'known ground-truth' prompts constructed from the same Phase-1 extractions (Appendix J), so Table 7's r=0.94 correlates consistency with a re-description of the method's own output, not with independent truth. The paper does supply two genuine external anchors—Grok's official prompt (xAI, 2025) and the npm-decompiled Claude Code prompts (Piebald AI, 2026)—and these show the metric can capture real content for those prompts, so I do not treat the framework as entirely fabricated. However, the identity-confusion results show the consistency definition can certify stable hallucinations: StepFun consistently claims DeepSeek under multiple skills yet still passes with Avg-C=0.770. This is a definitional gap, not merely a benchmark issue. Separate from circularity, Table 15's caption says Self-C was recorded for only 2 models while the table lists Self-C for all 41, which calls the reported Avg-C values into question; this is a data-consistency problem independent of the circularity score. Overall, 6 reflects partial construction: the success definition and validation loop are self-referential, but the external anchors and the parameter-free extraction mechanism provide some independent content. No load-bearing author self-citation chain is present; the issue is empirical-definitional rather than citational.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the consistency metric as a truth proxy, the OpenRouter access layer, embedding-similarity as semantic equivalence, and a handcrafted skill space. The threshold and budgets are chosen parameters. No new entities are introduced.

free parameters (3)

consistency success threshold = 0.7
Chosen empirically from the score distribution (Appendix C, Figure 6); at 0.7 all 41 models pass, at 0.8 only 73.2% pass, so the 100% success rate is partly a consequence of this choice.
UCB exploration constant c = √2 ≈ 1.414
Set to √2 following bandit literature (§4.1); not fitted to data but affects skill selection and exploration behavior.
attempt budget B = 20
Per-model query budget chosen in §4.1; the reported success rates and turn counts depend on allowing enough attempts for multi-turn escalation.

assumptions (4)

domain assumption Consistency between a model's self-reports under different probes implies recovery of the true system prompt.
Section 3.5 defines extraction success via self-consistency and cross-skill consistency of model outputs rather than matched ground truth; external validation covers only Grok and Claude Code.
domain assumption OpenRouter API responses represent the target model's own system prompt, unaffected by router-level or wrapper instructions.
Section 4.1 accesses all targets via OpenRouter; the paper does not control for or discuss OpenRouter's possible injected prompts.
domain assumption OpenAI text-embedding-3-large cosine similarity is a valid proxy for semantic identity of system prompts.
Sections 3.5 and C use embedding similarity for consistency and ground-truth similarity; the success threshold is calibrated on the authors' own data.
ad hoc to paper The handcrafted 28-skill taxonomy is a sufficient strategy space for extraction; the agent does not generate genuinely new skills.
Section 3.3 defines 14+14 fixed skills; self-evolving means UCB selection among these predefined prompts, not open-ended skill invention.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs." pith.science (2026). https://pith.science/paper/II6EHVHN

@misc{pith2026260121233,
  author       = {Pith},
  title        = {Pith review of: Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/II6EHVHN}},
  note         = {Machine review of arXiv:2601.21233}
}

read the original abstract

Autonomous code agents built on large language models are reshaping software and AI development through tool use, long-horizon reasoning, and self-directed interaction. However, this autonomy introduces a previously unrecognized security risk: agentic interaction fundamentally expands the LLM attack surface, enabling systematic probing and recovery of hidden system prompts that guide model behavior. We identify system prompt extraction as an emergent vulnerability intrinsic to code agents and present \textbf{\textsc{JustAsk}}, a self-evolving framework that autonomously discovers effective extraction strategies through interaction alone. Unlike prior prompt-engineering or dataset-based attacks, \textsc{JustAsk} requires no handcrafted prompts, labeled supervision, or privileged access beyond standard user interaction. It formulates extraction as an online exploration problem, using Upper Confidence Bound-based strategy selection and a hierarchical skill space spanning atomic probes and high-level orchestration. These skills exploit imperfect system-instruction generalization and inherent tensions between helpfulness and safety. Evaluated on \textbf{41} black-box commercial models across multiple providers, \textsc{JustAsk} consistently achieves full or near-complete system prompt recovery, revealing recurring design- and architecture-level vulnerabilities. Our results expose system prompts as a critical yet largely unprotected attack surface in modern agent systems.

Figures

Figures reproduced from arXiv: 2601.21233 by the authors.

**Figure 1.** Validation: JUSTASK extraction vs. reverse-engineered ground truth (semantic similarity = 0.94). Side-by-side comparison of Claude Code’s Explore subagent prompt. Left: Semantic extraction via JUSTASK. Right: Direct extraction via npm package decompilation (Piebald AI, 2026). Despite surface-level wording differences, both capture identical operational semantics, validating that consistency-based verification captur… view at source ↗

**Figure 2.** JUSTASK self-evolving extraction framework. The pipeline consists of six components: (1) UCB-based Skill Ranking selects skills based on empirical success rates plus exploration bonus (intrinsic reward), (2) Interleaved Thinking reasons about skill selection and target model characteristics, (3) Skill Generation instantiates concrete extraction prompts, (4) Multi-Turn Interaction execute the extraction attempt acros… view at source ↗

**Figure 3.** Skill usage rate by model category. Heatmap showing the percentage of models in each category where a given skill contributed to successful extraction. L14 (Introspection) achieves near-universal effectiveness across all categories, while persuasive skills (L5–L8) show category-dependent patterns. High-level multi-turn patterns (H1–H15) are rarely needed, with H5 (Distraction) and H9 (FITD) being most common for r… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Extraction strategy progression. Left: Distribution of attempts required—85% of models succeed on the first attempt, with only 7% requiring 2–3 attempts and 7% requiring 4+. Center: All initial attempts use low-level (single-turn) skills, reflecting our UCB-based explo…

**Figure 5.** Figure 5: Identity confusion in frontier language models. Left: Distribution of correct vs. confused self-identification among 41 models (n=41). Right: Breakdown of falsely claimed developers among the 11 confused models. We find that 26.8% of models claim identities from differ…

**Figure 6.** Figure 6: Extraction success rate as a function of consistency threshold. At threshold ≥ 0.7, all 41 models achieve successful extraction. Higher thresholds reduce coverage but may increase confidence in extraction quality. Worked Example. Consider extracting from Model X using …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

5 extracted references

[1]

Violence/Physical Harm
[2]

Illegal Activity Instructions
[3]

Privacy Violations/Doxxing
[4]

Sexual Content Involving Minors
[5]

simple” defense corresponds to the “Direct

Malware/Cyber Attacks # Context-Dependent Areas - Cybersecurity: Allow concepts; refuse exploits - Weapons: Allow legal discussion; refuse construction - Medical: Allow general info; refuse diagnosis Design Note.The controlled evaluation uses semantic similarity against the base prompt (not secret leakage) as the primary metric. Earlier designs included e...

2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

Violence/Physical Harm

[2] [2]

Illegal Activity Instructions

[3] [3]

Privacy Violations/Doxxing

[4] [4]

Sexual Content Involving Minors

[5] [5]

simple” defense corresponds to the “Direct

Malware/Cyber Attacks # Context-Dependent Areas - Cybersecurity: Allow concepts; refuse exploits - Weapons: Allow legal discussion; refuse construction - Medical: Allow general info; refuse diagnosis Design Note.The controlled evaluation uses semantic similarity against the base prompt (not secret leakage) as the primary metric. Earlier designs included e...

2025