{"id":"e1f0eb8e-6d54-4808-8300-bc3779a58d16","arxiv_id":"2506.16389","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RiOT is a tree-structured prompt optimization framework that generates multiple candidate prompts per step, selects by perplexity, and fuses parent and child prompts with a semantic residual connection, outperforming prior methods on most of five reasoning benchmarks.","lead":"This paper introduces RiOT, an automatic prompt optimization method that explores several candidate prompts at each step, picks the one the language model finds most surprising, and merges it with the previous prompt using sentence-level similarity. It reports higher accuracy than prior prompt optimizers on four of five reasoning benchmarks, but the gains are small on some tasks and the method has a few unresolved implementation questions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Candidate diversity at temperature 0 is asserted, not demonstrated; if K=3 outputs are identical, the tree and perplexity selection are vacuous.","rationale":"The reader identified the same weakest assumption, and I agree. The empirical central claim is plausible as reported, but the mechanism producing diversity is load-bearing: it is the paper's claimed novelty ('lack of diversity' challenge) and is supported only by an unsupported assertion of 'inherent variability'. Temperature 0 is normally deterministic; without evidence of distinct candidates, K=3 could be K=1 in effect, making the tree and perplexity selection vacuous. The paper does provide code, which is positive, but the manuscript as submitted does not document any stochasticity source or show example candidate sets. The abstract also overclaims by saying RiOT outperforms all prior prompt optimizers while TextGrad wins on Object Counting, and the evaluation lacks significance tests on small samples. These are fixable, so a CONDITIONAL verdict is appropriate: require the authors to either set temperature > 0 (or otherwise specify sampling), report actual candidate diversity, correct Eq. (6), and soften the overclaim.","tokens_in":16825,"tokens_out":3816,"duration_ms":40956,"concrete_test":"Run the released RiOT code on GSM8K for one parent prompt with the stated settings (temperature 0, K=3, GPT-4o optimizer), logging the raw candidate prompt strings at each of the 15 optimization steps. Compute the exact-match rate and mean embedding cosine similarity among the three candidates. If the candidates are identical or near-identical in every step, the diversity mechanism is vacuous and the Table 3 ablation cannot be attributed to K>1. If the API is non-deterministic at temperature 0, repeat the same call 10 times and quantify edit distance to establish the actual source of variability.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's first contribution is generating K semantically diverse candidates per iteration, and the ablation in Table 3 credits this diversity with a 13-point swing (81.2 vs 68.2). But the Implementation Details state that both target (GPT-3.5-turbo) and optimizer (GPT-4o) run with temperature fixed at 0, and Section 4 simply says 'LLMs exhibit inherent variability in their outputs' without identifying a source of stochasticity. With greedy decoding, K calls with identical inputs and temperature 0 typically return identical or near-identical prompts, so the K nodes collapse to one: perplexity-informed selection would be a constant choice, the tree degenerates to a chain, and the diversity ablation would not measure what it claims. The paper needs to show actual distinct candidate strings, or specify the sampling mechanism (e.g., top_p>1 or API non-determinism) that supplies the claimed variability. A second correctness issue compounds this: Eq. (6) defines perplexity without a logarithm, exp{-(1/J) * sum p_theta}, which is not perplexity and can rank candidates differently than the intended exp{-(1/J) * sum log p_theta}; if the code implements Eq. (6) literally, the selection criterion is not uncertainty-based.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RiOT, a tree-based prompt optimization framework built on the TextGrad backbone. At each optimization step, RiOT generates K candidate prompts from text gradients, selects one child node by a perplexity-based criterion, and applies a text residual connection that fuses parent and child sentences using an embedding model and two similarity thresholds. The method is evaluated on five reasoning benchmarks against manual CoT prompting and four automatic prompt optimization baselines, reporting a weighted average accuracy of 77.2% and additional experiments on generalization, ablations, selection metrics, and computational cost.","tokens_in":16963,"tokens_out":7518,"duration_ms":73962,"significance":"If the mechanism works as described, the contribution is of practical value: it targets two real limitations of black-box prompt optimization, candidate diversity and semantic drift, with a modular tree structure and a lightweight selection scheme. The paper includes a promised code release, held-out evaluation with validation-based prompt selection, and a reasonable range of baselines. The residual fusion idea and the tree formulation are simple and potentially useful for practitioners. However, the current evidence does not yet establish the central claim, because the temperature-0 diversity mechanism is unsupported, the perplexity formula in Eq. (6) is incorrect, and the abstract overstates the per-dataset results relative to Table 1.","major_comments":[{"comment":"The mechanism for generating K diverse candidates assumes stochasticity that the experimental configuration does not provide. Implementation Details set both GPT-3.5-turbo (target) and GPT-4o (optimizer) temperature to 0, yet §4 states that 'LLMs exhibit inherent variability in their outputs' without identifying the source of nondeterminism. With greedy decoding, K calls with identical inputs normally produce identical outputs, which would collapse the tree to a chain and make the perplexity-based selection and the Table 3 diversity ablation vacuous. The paper should either report the actual distinct candidate strings for a representative iteration or specify the sampling mechanism (e.g., API-level nondeterminism, top-p sampling, or varied inputs). This also bears on Table 1: with temperature fixed at 0, the nonzero standard deviations over five runs are unexplained.","section":"§4 and Implementation Details (Section 5)"},{"comment":"Equation (6) defines PPL(p) = exp{-(1/J) * sum p_theta(x_j|x_<j)}, omitting the logarithm that is essential to the standard perplexity definition. As written, the quantity is the exponential of a negative average token probability, not a perplexity; it is at most 1, and maximizing it is equivalent to minimizing the average token probability, which is the opposite of the uncertainty-based interpretation in the text. If the implementation follows Eq. (6) literally, the selection criterion is not perplexity-based. Please correct the formula to exp{-(1/J) * sum log p_theta(x_j|x_<j)} and confirm that the code and reported rankings correspond to the corrected definition.","section":"§4.1, Eq. (6)"},{"comment":"The abstract claims that RiOT 'outperforms both previous prompt optimization methods and manual prompting,' but Table 1 shows that TextGrad reaches 88.3% on Object Counting while RiOT reaches 86.9%, so RiOT does not beat all automatic optimization baselines on every benchmark. The main text correctly describes the Object Counting result as 'suboptimal.' The abstract and the contribution framing should be revised to state that RiOT achieves the best weighted average accuracy and leads on four of five tasks, rather than outperforming all baselines across all five benchmarks.","section":"Abstract and Table 1"},{"comment":"The ablation labeled 'w/o Perplexity-Informed Node Selection' is implemented by setting K=1, which simultaneously removes candidate diversity and the selection rule. This design cannot attribute the 13.0% drop specifically to the perplexity criterion; the drop could equally reflect the loss of multiple candidates or the absence of any selection among them. To isolate the contribution, the authors should compare perplexity-based selection against random or first-candidate selection at the same K (e.g., K=3) and report the resulting accuracy.","section":"Table 3 and §6.2"},{"comment":"Several headline per-task gains are within one standard deviation of the closest baseline. For example, LogiQA 2.0 shows 61.4±1.5 vs. TextGrad's 60.0±1.2, and StrategyQA shows 74.6±1.5 vs. DSPy's 73.4±1.5. Because the central empirical claim depends on small differences, the paper should report paired significance tests or confidence intervals over the five runs, or explicitly limit the claim of per-task superiority to the weighted average and to those tasks where the difference is statistically credible.","section":"Table 1 and §6.1"}],"minor_comments":[{"comment":"In the Date Understanding entry, the sentence 'For our study, For our study, we sample 329 test samples' contains a duplicated phrase; please fix it.","section":"Appendix E"},{"comment":"The rows 'RiOT (Single)' and 'RiOT (Multi)' are not defined in the main text; clarify what the two threading settings are and how the reported runtimes were measured.","section":"Table 6"},{"comment":"The axis label 'Token' in Figure 2 is ambiguous; use 'Prompt length (tokens)' or define the secondary variable explicitly in the caption.","section":"Figure 2"},{"comment":"The caption contains a typo: 'Comparion' should be 'Comparison.'","section":"Table 7 caption"},{"comment":"The definitions of perplexity, entropy, and length do not specify which model supplies the token probabilities p_theta(x_j|x_<j); state whether this is the target model or the optimizer model, and how those probabilities are obtained in the implementation.","section":"§4.1 and Appendix B"},{"comment":"The notation alternates among p_t, p*_t, and p^{(i)}_{t+1} without a formal definition of the tree level and the parent-child relationship; a short formalization of the tree data structure would improve readability.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The temperature-0 diversity concern is the most important issue to weigh: if the released code generates K candidates with identical inputs and no sampling, the tree mechanism and the perplexity-based selection could be vacuous. The editor may wish to verify the GitHub repository's candidate-generation loop before final disposition."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: RiOT is a reasonable engineering contribution to black-box prompt optimization, and the held-out results are mostly honest, but the paper's central diversity mechanism is asserted rather than demonstrated. As written I would not accept; with a few fixes it is a solid paper.\n\nWhat is new and good: combining multi-candidate generation with perplexity-based node selection and sentence-level residual fusion is an easy-to-implement recipe. The best weighted average accuracy is 77.2%, about 2.7 points over DSPy, and the GSM8K ablations show both the residual connection and the candidate selection matter (81.2 vs 68.8 and 68.2). The authors evaluate on held-out test sets, report standard deviations, include a transfer experiment to Gemini-1.5-flash, and release code. That is more than many prompt-optimization papers do.\n\nThe soft spots, roughly in order of seriousness.\n\nFirst, the temperature-0 diversity problem is real. Implementation Details says both the target and optimizer run at temperature 0, and Section 4 says 'LLMs exhibit inherent variability' without identifying what sampling mechanism supplies it. If three calls with the same meta-prompt return identical or near-identical prompts, the tree degenerates, perplexity selection is a no-op, and the 13-point ablation swing for K is not measuring diversity. Fixable: show actual candidate strings, or sample with temperature > 0/top_p. Until then the main mechanism is unsupported.\n\nSecond, Equation (6) is not perplexity; the log is missing. If the code implements what is written, the selection metric ranks prompts by exp(-mean token probability), which is not the standard uncertainty measure. A typo, but it needs correcting because the selection story hangs on it.\n\nThird, the abstract overclaims. Table 1 shows TextGrad beats RiOT on Object Counting (88.3 vs 86.9), so 'outperforms previous prompt optimization methods' is only true on the weighted average. The authors later say 'suboptimal performance' there, but the abstract should say 'highest average' instead.\n\nFourth, the per-task wins are mostly within noise. N ranges from 100 to 329; differences of 1.2-2.2 points with overlapping standard deviations are not significant. The weighted average is the only robust number, and even that lacks a significance test. I would not call the central claim false, but I would call it under-evidenced at per-task granularity.\n\nMinor: the intro says Pryzant et al. and Yang et al. generate one candidate per iteration; APO uses beam search and OPRO samples multiple candidates, so the novelty framing needs adjustment.\n\nBottom line: for practitioners who want a simple black-box optimizer, this is a useful paper. For a reviewer, the candidate-diversity evidence and the corrected formula are needed before the empirical claims can be taken at face value. I would send it to peer review, with major revision.","headline":"A sensible but underspecified prompt-optimization recipe: the diversity mechanism needs evidence before the empirical claims hold.","tokens_in":17584,"tokens_out":4513,"would_cite":true,"duration_ms":45030,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automatic prompt optimization can be made more diverse and drift-resistant by organizing it as a tree that branches into several candidates, keeps the most informative one, and fuses it back with its parent.","keywords":["prompt optimization","residual connections","perplexity","tree search","chain-of-thought prompting","reasoning benchmarks","semantic drift","text gradients"],"falsifier":"Log the K=3 candidate prompts produced at a single optimization step with temperature 0: if repeated invocations return identical text, the tree has no width and the perplexity selection and fusion steps have nothing to act on. A second check is to replace perplexity selection with random selection among the K candidates and compare GSM8K accuracy; if the gap vanishes, the claimed diversity mechanism is not doing the work.","tokens_in":16528,"feed_emoji":"🌳","tokens_out":7478,"duration_ms":72255,"temperature":0.7,"pith_summary":"Automatic prompt optimization has a two-sided failure mode: iterative edits converge on a narrow phrasing, and each rewrite can erase useful pieces of the previous prompt. RiOT claims to fix both at once by treating optimization as a tree: at every step an optimizer LLM proposes several semantically different candidate rewrites, the method keeps the one with the highest perplexity, then merges the survivor with its parent via sentence-level similarity before moving down the tree. On five benchmarks spanning commonsense, math, logic, dates, and object counting, this procedure yields the best weighted-average accuracy of 77.2 percent, beating the strongest prior automatic method by 2.7 points and beating manual chain-of-thought prompting. The implication is that black-box prompt search can be made broader and more stable without parameter access or extra training.","feed_headline":"Tree prompt optimizer wins at 77.2% average accuracy","feed_subtitle":"Branching into diverse candidates and fusing survivors beats the best baseline by 2.7 points.","key_machinery":"The load-bearing object is the optimization tree itself: each node is a prompt, each level is one refinement step, and the tree width K is the number of candidate rewrites per parent. Two mechanisms drive it. Perplexity-informed node selection scores each child by its perplexity under the target LLM and keeps the maximum, treating high uncertainty as a proxy for semantic informativeness. Text residual connection then fuses parent and child: it tokenizes both into sentences, embeds them, keeps parent sentences whose best similarity to a child sentence is at least 1-b1, adds child sentences whose best similarity to a parent sentence is below 1-b2, and unions the two sets into the next prompt. The fusion thresholds b1=0.25 and b2=0.5 control how much old content is preserved and how much new content enters, so each step is a bounded edit instead of a fresh rewrite.","core_discovery":"The paper's central claim is that the two known failure modes of iterative prompt optimization—low diversity and semantic drift—are jointly curable by a tree whose nodes are prompts and whose edges are residual fusions. Starting from a root prompt, each iteration produces K child candidates through text-gradient feedback, selects the child with the highest perplexity as the most informative branch, and then composes the selected child with its parent by embedding both at sentence level and retaining parent sentences that resemble the child and child sentences that are new. This step is repeated down a single path of the tree for 15 iterations with K=3. On the paper's five task benchmarks, the method reaches 81.2 percent on GSM8K, improves over zero-shot CoT on all five tasks, and reaches the highest weighted average of 77.2 percent; ablations attribute 13.0 and 12.4 points on GSM8K to the diversity-based selection and the residual connection respectively.","pith_inferences":["A natural extension the paper does not run is to make K's diversity explicit: sampling with temperature above zero or with varied optimization instructions could make the three candidates genuinely distinct, which would let the perplexity and residual components be tested on their own terms.","The same sentence-level fusion may apply beyond prompts, to instructions, code scaffolding, or reasoning templates, where the parent-child edit is discrete text and drift is the main failure mode.","The paper's own limitations—text-only tasks and uneven per-task gains—suggest the strongest case is for textual reasoning; whether RiOT's fusion generalizes to multimodal instructions is untested and would be the most direct next experiment.","Because K=3 is optimal and larger K hurts, more candidates is not automatically better; the interaction between candidate count and fusion thresholds is worth a dedicated study."],"forward_implications":["If the claim holds, prompt optimization remains fully black-box: only LLM APIs and an embedding model are needed, with no parameter gradients, logit access, or reward-model training.","A single fixed path through the tree yields the gains, so the framework can later be expanded into beam search over multiple lineages without changing the fusion step.","Perplexity-maximizing selection outperforms entropy- or length-based selection on GSM8K, suggesting the diversity signal is semantic rather than statistical verbosity.","The optimized prompts transfer to a different target model in three of five tasks, and optimizing directly for that model improves all five, implying the gains are not tied to one LLM.","On the harder AMC12 set, the method reaches 46.0 percent, 5.1 points over zero-shot CoT, pointing to usefulness where data is scarce."],"supporting_citations":[{"why":"Supplies the TextGrad backbone that the paper's optimization operator builds on, and is the strongest single-candidate iterative baseline.","marker":"Yuksekgonul et al., 2024"},{"why":"Introduced LLM-generated prompt candidates (APE), the parallel-candidate baseline that RiOT extends and compares against.","marker":"Zhou et al., 2023"},{"why":"OPRO baseline that uses a meta-prompt optimizer with one candidate per iteration.","marker":"Yang et al., 2024"},{"why":"DSPy baseline, the best-performing prior automatic method in the weighted-average comparison.","marker":"Khattab et al., 2024"},{"why":"Formalized text gradients for prompt optimization, the mechanism each RiOT candidate generation step invokes.","marker":"Pryzant et al., 2023"},{"why":"Residual learning in deep networks, the inspiration for keeping parent content during optimization.","marker":"He et al., 2016"},{"why":"Supplies the GSM8K dataset used for the main result, hyperparameter analysis, and ablations.","marker":"Cobbe et al., 2021"},{"why":"Supplies the zero-shot chain-of-thought manual prompting baseline and the initial prompts.","marker":"Kojima et al., 2022"},{"why":"Supplies the few-shot chain-of-thought manual prompting baseline.","marker":"Wei et al., 2022"},{"why":"Information-theoretic justification for treating high-perplexity content as more informative in node selection.","marker":"Shannon, 1948"}],"fun_headline_variants":["Tree optimizer prunes prompt drift, hits 77.2%","Prompt optimizer branches out, beats manual by 2.7 pts","RiOT: residuals + diversity boost prompts to 77.2%","Tree-based prompt tuning wins by 2.7 points","Refining prompts with a tree: 77.2% average"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that calling the same optimizer model several times with the same input produces several genuinely different candidate prompts; the paper never explains where that variation comes from, since it runs the model with its randomness-level fixed at zero.","fun_headline_variants_meta":{"raw":{"variants":["Tree optimizer prunes prompt drift, hits 77.2%","Prompt optimizer branches out, beats manual by 2.7 pts","RiOT: residuals + diversity boost prompts to 77.2%","Tree-based prompt tuning wins by 2.7 points","Refining prompts with a tree: 77.2% average"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000512,"raw_usage":{"total_tokens":2466,"prompt_tokens":897,"completion_tokens":1569,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":1479}},"tokens_in":513,"tokens_out":1569,"duration_ms":10209,"temperature":1.0,"reasoning_tokens":1479,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:27:24.056191+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Log the K=3 candidate prompts produced at a single optimization step with temperature 0: if repeated invocations return identical text, the tree has no width and the perplexity selection and fusion steps have nothing to act on. A second check is to replace perplexity selection with random selection among the K candidates and compare GSM8K accuracy; if the gap vanishes, the claimed diversity mechanism is not doing the work.","supporting_citations":[],"review_version":2}