{"id":"53cd5b24-85c4-4b08-be08-8766c9392718","arxiv_id":"2411.12736","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ACING uses off-policy actor-critic RL over continuous latent vectors, decoded by a frozen white-box model, to optimize discrete instructions for black-box LLMs from reward feedback alone.","lead":"ACING is an actor-critic reinforcement learning method that automatically searches for task instructions for black-box language models by sampling continuous latent prompts, decoding them with a frozen white-box model, and keeping the best-scoring ones. It reports gains over several automatic prompt optimizers across 33 tasks, though some headline numbers rely on favorable task subsets and counting ties as wins.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No random-search baseline over the same latent space means the actor-critic mechanism, not just the decoder/projection, is unproven as the driver of ACING's gains.","rationale":"The reader's weakest assumption was that the random projection and frozen decoder create a smooth, informative mapping from actions to rewards at Eq. (3); my concern is the missing control that would make that assumption actionable. Rather than only measuring local smoothness, the decisive question is whether the critic's generalization adds anything beyond stochastic search in the same latent space. The paper's own no-critic ablation already narrows the gap to 0.01 in median score (Table 12), and Section 7 admits that random search can occasionally produce strong prompts. Comparing ACING to random search under identical conditions would settle whether the RL machinery is responsible for the reported gains. A secondary reporting issue independently destabilizes the abstract: the 76% human-overperformance statistic appears to count ties as wins. In Appendix G, Table 19 lists 21 tasks with 9 exact ties; strictly, ACING wins only 7 of 21 (33%), not 76%. This is a reporting/definitional problem rather than a new methodological flaw, and the reader already flagged tie-counting, so it does not change the overall verdict. The paper should remain CONDITIONAL: the core experiments need a same-space random-search baseline and the headline human-overperformance claim needs a strict-count clarification before the central claims are accepted as stated.","tokens_in":32771,"tokens_out":11201,"duration_ms":115247,"concrete_test":"Run a random-search baseline on the same 30 instruction-induction tasks: at each of 165 iterations sample a uniformly from [0,1]^10, decode through the same fixed P and frozen Vicuna-13B with the same 5 exemplars, score on the same validation set, keep the best-scoring prompt, and evaluate on the same test set with the same 3 seeds. Report the median test score and per-task win counts against ACING's Table 3 and Table 9 results. If random search's median is within 0.03 or wins on a comparable number of tasks, the actor-critic component is not load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the off-policy actor-critic, with a critic that generalizes reward in the 10-dimensional action space, is what makes ACING reliable within 165 queries. The paper never compares against random search over the same action space a in [0,1]^10 with the same fixed projection P, the same frozen Vicuna-13B decoder, the same exemplars, and the same validation-based prompt selection. This control is necessary because the random projection plus frozen decoder could already produce good prompts; indeed, Section 7 (Limitations) concedes that 'strong prompts can occasionally arise even from random search or other optimization approaches.' The only critic-free ablation (Appendix F.1, Table 12) shows policy gradient without a critic reaches median 0.70 versus ACING's 0.71 while winning 14 of 33 tasks, so the value added by the critic is visibly small. Without a random-search control, the 10-point median advantage in Table 3 could be an artifact of the continuous latent space and the prompt-selection procedure rather than of the actor-critic learning itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ACING, an actor-critic framework for optimizing natural-language instructions for black-box LLMs. It encodes candidate instructions as continuous latent vectors, obtained by linearly projecting a low-dimensional action vector through a fixed random matrix and decoding to text with a frozen white-box model (Vicuna-13B). ACING trains a stateless off-policy actor with twin critics and entropy regularization, using the validation score of the black-box LLM as reward under a fixed 165-query budget. The paper reports results on 30 instruction-induction tasks, two zero-shot chain-of-thought datasets, and one summarization dataset, claiming that ACING outperforms human prompts in 76% of instruction-induction tasks and improves on automatic baselines by up to 33 points with a 10-point median gain. The appendices contain ablations on the critic, action dimensionality, exemplar count, budget splitting, and decoder choice, plus human and readability evaluations of the generated prompts.","tokens_in":32969,"tokens_out":9722,"duration_ms":91212,"significance":"If its headline claims were supported, ACING would be a useful contribution: it demonstrates a lightweight, API-compatible alternative to LLM-based prompt proposers, provides a broad evaluation across 33 datasets and several black-box models, and ships open source with detailed ablations. The framework itself is clearly specified and the experimental effort is substantial. However, the main quantitative claims as stated are not consistent with the paper's own tables, and the absence of a random-search control leaves the core attribution — that the actor-critic mechanism, rather than the continuous latent space and decoder, drives the improvements — unproven.","major_comments":[{"comment":"The claim that ACING 'outperforms human-written prompts in 76% of instruction-induction tasks' is not supported by the reported comparisons. Table 2, the only direct human-vs-ACING table in the main text, shows ACING winning on 7 of 12 tasks where scores differ (58%). Table 19, the full comparison, shows ACING as best on 16 of 30 tasks (53%) if 'best-performing' is exclusive; no table in the paper yields 76%. The abstract also states 'gains of up to 33 points,' but Table 2 contains a larger 39-point gain (Rhymes, 0.61 to 1.00). Please specify the exact task set, the tie-breaking rule, and the reference baseline for these headline numbers.","section":"Abstract; Section 4.1; Table 2; Table 19"},{"comment":"The '10-point median improvement over the best automatic baseline' is computed on the 23-task subset of Table 3 (median 0.69 vs 0.59 for InstructZero). On the full 30-task instruction-induction set (Table 9), the median gain over the best baseline is 5 points (0.71 vs 0.66), and the CoT (Table 5) and summarization (Table 4) gains are at most 4 points. Thus the 10-point median claim across 33 tasks is not supported by the reported results; the authors should either report a combined 33-task comparison with a single aggregate median or revise the claim to refer only to the 23-task subset.","section":"Abstract; Section 4.2; Table 3; Table 9"},{"comment":"The paper does not include a random-search baseline over the same continuous action space defined by the fixed projection matrix P and the frozen Vicuna decoder. Such a control is necessary to attribute the observed improvements to the off-policy actor-critic mechanism rather than to the latent-space representation and validation-based prompt selection. The paper's own Limitations section concedes that 'strong prompts can occasionally arise even from random search,' and Appendix F.1 (Table 12) shows that a pure policy-gradient without a critic reaches a median of 0.70 vs ACING's 0.71, with ACING winning on 21 of 33 tasks. Adding a random-search or random-action baseline with identical decoder, exemplars, budget, and selection rule would clarify whether the actor-critic learning itself contributes beyond the continuous latent space.","section":"Section 3.3, Eq. (3); Section 4.2; Appendix F.1; Section 7 (Limitations)"}],"minor_comments":[{"comment":"The statement that 'ACING improves the median task score from 0.81 to 0.86' refers only to the 12-task subset of Table 2; please clarify that this is not the median over the full 30-task set.","section":"Section 4.1"},{"comment":"The '# best-performing tasks' counts treat ties as wins (for example, Letters_list is tied between InstructZero and ACING); please state the tie-breaking convention explicitly.","section":"Table 3; Table 9"},{"comment":"The Wilcoxon signed-rank test should state whether it uses per-seed scores or per-task means, whether the test is one-sided or two-sided, and the number of paired observations; currently these details are missing.","section":"Section 4.2; Statistical significance test"},{"comment":"The action-dimensionality results are reported for a fixed seed only; the text should note that these are single-seed observations and that variability across seeds may be substantial, especially given the large standard deviations seen elsewhere.","section":"Appendix F.5; Table 16"},{"comment":"The bottom rows report '# best-performing tasks 14 16' (summing to 30) while the table displays only 21 rows; please clarify whether the counts refer to the full 30-task set or to the displayed subset, and if the latter, explain the discrepancy.","section":"Table 19"},{"comment":"For several tasks the best-instruction test score in Table 21 differs from the mean in Table 9 (e.g., auto_debugging 0.375 vs 0.25, sentence_similarity 0.21 vs 0.13); please explain how the 'best' instruction is selected and why these values differ.","section":"Table 21 vs Table 9"},{"comment":"There is a typo 'Moroever' at the start of Section 4; please correct it.","section":"Section 4; typos"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely problem and the framework is well motivated, but the authors should be asked to reconcile the abstract's headline numbers with their own tables and to add the missing random-search control. The current version overstates both the human-comparison win rate and the median gain; these are fixable with a re-analysis of existing data plus one additional control experiment, so I do not think rejection is warranted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ACING is more credible than its abstract. The method is a real new combination: continuous latent prompt search from InstructZero/INSTINCT, but with the Bayesian optimizer or neural bandit replaced by an off-policy actor-critic with twin critics and entropy-regularized exploration. The experimental setup is mostly fair - same frozen Vicuna-13B decoder, same 165-query budget, same search space as the baselines - and the task coverage is broad: 30 instruction-induction tasks plus CoT and summarization. The ablations on critic count, action dimension, exemplar count, budget splitting, and decoder choice are exactly what a skeptical reader wants. The circularity concern is a non-issue: the optimizer maximizes a validation score and the final prompt is scored on a separate test set.\n\nBut read the tables, not the abstract. The 76% outperform-human figure counts ties as wins: in Table 19, ACING strictly wins 7 of 21 tasks and ties 9, so 16 of 21 are at least as good - that is not outperforming. The 10-point median improvement comes from the 23-task subset in Table 3, not the full 33-task suite; the full 30-task table shows a 5-point median edge. I could not find the up-to-33-points gain anywhere. These are reporting problems, not fatal flaws.\n\nThe more substantive gap is the missing random-search baseline over the same latent action space, with the same projection, decoder, and prompt-selection procedure. The authors even concede in Limitations that strong prompts can arise from random search. The no-critic ablation (Table 12) makes the critic's contribution look small - median 0.70 vs 0.71, winning 14 of 33 tasks - so without a random-search control, we cannot tell how much of the gain comes from the actor-critic mechanism and how much from the latent space itself. That weakens the causal story about the RL components.\n\nMinor issues: the Wilcoxon p-values lack correction for multiple comparisons, and the code availability statement is contradictory (abstract says the implementation is available; appendix says it will be released upon publication).\n\nWho should read this? People building black-box prompt optimizers. The recipe is simple, likely reproducible, and the comparisons are useful. It deserves a serious referee, but the authors should fix the abstract, add a random-search control, and clarify the code release.","headline":"A solid prompt-optimization paper with a useful off-policy actor-critic recipe, but the abstract oversells the headline numbers and the critic's added value over a no-critic ablation looks thin.","tokens_in":818,"tokens_out":1932,"would_cite":true,"duration_ms":63817,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ACING turns prompt discovery for black-box LLMs into a 165-query actor-critic search that beats human-written prompts on most instruction-induction tasks.","keywords":["prompt optimization","black-box LLM","actor-critic","instruction induction","continuous action space","soft prompt","maximum entropy reinforcement learning","chain-of-thought prompting"],"falsifier":"Run ACING on the same 33 tasks with the 165-query budget but replace the critic with random search over the same action space; if random search matches ACING's median, the critic's learned generalization is not what drives the reported gains. A second check: sample pairs of actions within a small radius, decode them, and measure text overlap of the resulting instructions; low correlation would show the projection-decoder map is too discontinuous for the critic to learn from.","tokens_in":32529,"feed_emoji":"🎯","tokens_out":6043,"duration_ms":54538,"temperature":0.7,"pith_summary":"ACING treats instruction optimization for black-box LLMs as a stateless, continuous-action reinforcement-learning problem: an action is a low-dimensional vector, projected into a soft-prompt embedding and decoded by a frozen white-box LLM into a discrete instruction, whose task score from the black-box LLM is the reward. The paper claims that an off-policy actor-critic with twin critics and entropy-regularized exploration discovers instructions that beat human-written prompts on 76% of instruction-induction tasks and outperform four automatic baselines on 33 tasks spanning instruction induction, summarization, and zero-shot chain-of-thought reasoning, all within a fixed 165-query budget. If this holds, prompt engineering for API-only models can be automated with lightweight networks and no gradient access.","feed_headline":"165 black-box queries find prompts that beat human prompts in 76% of tasks","feed_subtitle":"An RL agent searches continuous latent instructions and tops 33-task benchmarks with no per-task tuning.","key_machinery":"The load-bearing object is the projected action vector: an action $a\\in[0,1]^{d'}$ with $d'=10$ is multiplied by a fixed random matrix $P$ with entries from $\\mathrm{Uniform}(-1,1)$ to form a soft prompt $z=Pa$ in the embedding space of a frozen Vicuna-13B decoder, which turns $z$ and five exemplars into a discrete instruction $\\tau$. A stateless off-policy actor-critic, consisting of a Gaussian policy network plus two critics whose value estimate is the minimum of the two, updates on the reward $r=\\frac{1}{m}\\sum_i q(\\hat y_i,y_i)$ from the black-box LLM, with an adaptive entropy temperature to keep exploration alive under the 165-query budget. This machinery converts discrete prompt search into a continuum bandit and lets the method explore an infinite instruction space without model gradients.","core_discovery":"The central claim is that a continuous latent prompt space, searched with soft actor-critic style updates, is a better substrate for black-box instruction optimization than discrete token search or finite candidate pools. ACING reports a median test score of 0.71 across 30 instruction-induction tasks, roughly 10 points above the best baseline median, winning 14 of 30 tasks overall and 13 of the 23 hardest tasks; it also leads on SAMSum summarization and on GSM8K and AQUA-RAT zero-shot reasoning. The discovered instructions remain readable, with a median Flesch Reading Ease of 70.8, and are selected by validation score alone before being evaluated on held-out test sets.","pith_inferences":["Beyond the paper, the same actor-critic loop could optimize not just the instruction text but also the exemplars or the decoding template, since both enter the same decoder pipeline.","If the critic's generalization across actions is the real driver, comparing ACING against a non-parametric baseline such as random search over the same action space would isolate how much of the reported 10-point median gain comes from the learned value function.","The random projection $P$ is a single draw; a testable extension is ensembling multiple projections or learning the projection to reduce the chance that a particular draw makes the action-to-prompt map uninformative."],"forward_implications":["Prompt optimization for API-only LLMs can be driven entirely by task scores, with no token-level access or gradient information.","The same untuned hyperparameters transfer across 33 tasks, suggesting ACING is a general-purpose optimizer rather than a per-task tuner.","Discovered prompts are naturally interpretable and readable, so the optimization output can be deployed or inspected directly.","Because rewards often peak within 60-80 of the 165 calls, the fixed budget can be shortened or split into exploration plus re-ranking to gain about 5 more points in median score.","Swapping the frozen decoder from Vicuna-13B to WizardLM-13B lifts median accuracy by 8 points, so the approach can ride on stronger open decoders as they appear."],"supporting_citations":[{"why":"Supplies the off-policy maximum-entropy actor-critic objective and temperature update that ACING adapts to the stateless setting.","marker":"Haarnoja et al., 2018"},{"why":"Supplies the random-projection soft-prompt decoding pipeline and one of the main black-box baselines ACING is compared against.","marker":"Chen et al., 2024"},{"why":"Supplies a neural-bandit baseline and the fixed 165-query budget convention used across methods.","marker":"Lin et al., 2024b"},{"why":"Provides the 30 instruction-induction tasks, human-written prompts, and human baseline scores ACING is measured against.","marker":"Honovich et al., 2023"},{"why":"Provides the zero-shot chain-of-thought baselines such as 'Let's think step by step' that ACING improves on for GSM8K and AQUA-RAT.","marker":"Kojima et al., 2022"},{"why":"Provides Vicuna-13B, the frozen white-box decoder that turns soft prompts into discrete instructions.","marker":"Chiang et al., 2023"},{"why":"Supplies APE, an LLM-sampling baseline whose results ACING is compared with across all tasks.","marker":"Zhou et al., 2023"}],"fun_headline_variants":["RL agent finds prompts that beat humans in 76% of tasks","Black-box actor-critic tops human prompts across 33 tasks","Continuous instruction search outperforms human-written prompts","ACING: RL discovers better prompts than human-crafted ones","Soft actor-critic beats human prompts in black-box LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that a random 10-dimensional projection into embedding space, decoded by a frozen Vicuna-13B, maps nearby actions to usefully related instructions; if that map is noisy or flat, the critic cannot generalize and the search degrades toward random sampling.","fun_headline_variants_meta":{"raw":{"variants":["RL agent finds prompts that beat humans in 76% of tasks","Black-box actor-critic tops human prompts across 33 tasks","Continuous instruction search outperforms human-written prompts","ACING: RL discovers better prompts than human-crafted ones","Soft actor-critic beats human prompts in black-box LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000589,"raw_usage":{"total_tokens":2719,"prompt_tokens":852,"completion_tokens":1867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":468,"completion_tokens_details":{"reasoning_tokens":1784}},"tokens_in":468,"tokens_out":1867,"duration_ms":13517,"temperature":1.0,"reasoning_tokens":1784,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:11:59.963795+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ACING on the same 33 tasks with the 165-query budget but replace the critic with random search over the same action space; if random search matches ACING's median, the critic's learned generalization is not what drives the reported gains. A second check: sample pairs of actions within a small radius, decode them, and measure text overlap of the resulting instructions; low correlation would show the projection-decoder map is too discontinuous for the critic to learn from.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the off-policy maximum-entropy actor-critic objective and temperature update that ACING adapts to the stateless setting."},{"cited_title":"Bowman, and Omer Levy","cited_arxiv_id":null,"evidence_quote":"Provides the 30 instruction-induction tasks, human-written prompts, and human baseline scores ACING is measured against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the zero-shot chain-of-thought baselines such as 'Let's think step by step' that ACING improves on for GSM8K and AQUA-RAT."},{"cited_title":"Gonzalez, Ion Stoica, and Eric P","cited_arxiv_id":null,"evidence_quote":"Provides Vicuna-13B, the frozen white-box decoder that turns soft prompts into discrete instructions."}],"review_version":1}