{"id":"1adc5c72-f996-4039-aa5e-cf84c4b50d3b","arxiv_id":"2411.14479","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A knowledge-graph-based reinforcement learning policy selects and orders in-context examples for LLM prompts, reporting modest ROUGE and BLEU improvements.","lead":"The paper introduces GRL-Prompt, a reinforcement learning system that automatically selects and orders in-context examples when building prompts for large language models. It reports small ROUGE and BLEU gains over baseline prompt-selection methods on two instruction-following datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5) is not the probability of the ordered prompts actually sampled in Eq. (6), so the policy-gradient update in Eq. (8) may optimize a different distribution than the one evaluated.","rationale":"The reader's weakest assumption already identified the gap between Eq. (5) and Eq. (6); this stress-test confirms and sharpens it. The problem is not merely underspecification: Eq. (5) lacks non-selection factors and subset normalization, so it is not a distribution over ordered subsets of variable length, and the pairwise-order distribution can produce cycles that make topological sorting ill-defined. This is an internal inconsistency in the formalization, not a disagreement with field consensus. It is load-bearing because the paper's explanation for its gains is RL over a KG-defined policy; if the policy-gradient update optimizes a different distribution than the prompts being evaluated, the claimed mechanism is unsupported. Still, the flaw is fixable by precise definitions and code, and the empirical comparisons may survive once the sampler is correctly specified, so conditional acceptance remains the right verdict rather than rejection. The additional concerns about the inflated 0.10 average (computed against the mean of all baselines rather than the best baseline) and the absence of significance testing reinforce the conditional verdict but are secondary to the sampling-distribution mismatch.","tokens_in":14537,"tokens_out":10171,"duration_ms":95952,"concrete_test":"Obtain the released code or ask the authors for the exact sampling routine. Fix a small candidate pool, say N=4, and a single query. Enumerate all ordered subsets. Compute the probability Eq. (5) assigns to each sequence and the probability assigned by the implemented sampler, including any non-selection factors, subset-size choice, and cycle handling. Run the sampler 100,000 times and compare the empirical frequency of each sequence with the Eq. (5) probability using a chi-square test or maximum absolute deviation. If the two distributions differ by more than sampling error, the log-probability used in Eq. (8) is not the log-probability of the sampled prompt and the policy-gradient update is biased.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the assumption that p(pi) in Eq. (5) is the distribution from which ordered prompts are sampled in Eq. (6) and whose log-probability is used in the REINFORCE update of Eq. (8). As written, this cannot hold. First, Eq. (5) multiplies f_icmn over all N candidates, but a variable-length ordered subset sampler would need non-selection factors (1 - f_icmn) for excluded candidates and a normalization over subset sizes; without these, Eq. (5) is not a probability distribution over ordered subsets. Second, the paper never states how the set of selected examples is formed from f_icmn, how many examples are selected, or how cycles in the pairwise f_pec predictions are resolved; topological sorting in Eq. (6) is undefined when pairwise predictions are cyclic, and such cycles have positive probability under Eq. (3). Third, Eq. (8) uses log pi_theta of the sampled sequence; if the actual sampler differs from Eq. (5), the gradient estimator is biased and the policy does not optimize the distribution of prompts being scored. The claimed RL mechanism is therefore unsupported by the formalization, even if the empirical tables survive.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GRL-Prompt, a reinforcement-learning framework for in-context prompt optimization. It constructs a graph from the user query and candidate examples, applies a heterogeneous graph transformer to embed the graph, and uses a policy network combining a pairwise edge classifier (PEC) and an in-context matching network (ICMN) to select and order in-context examples. Experiments on Alpaca and Dolly with GPT-4, GPT-3, and LLaMA report improved ROUGE and BLEU scores over several baselines, together with ablations and sensitivity analyses.","tokens_in":14854,"tokens_out":4862,"duration_ms":45281,"significance":"If the method works as described, it would add a useful learned alternative to random, retrieval-based, and fixed-length prompt-selection methods, with the claimed advantage of handling order sensitivity through RL. The paper ships ablations, sensitivity analyses, and a stated public code release. However, the formalization of the policy distribution and the RL update is not sound as written, and the empirical claims are weakened by missing variance and significance reporting. The core idea is worth considering, but the manuscript needs substantial technical clarification before the claims can be assessed.","major_comments":[{"comment":"The distribution p(pi) in Eq. (5) is not a probability distribution over ordered subsets of candidate examples. It multiplies f_icmn over all N candidates and f_pec over all i<j pairs, with no normalization over subset sizes and no non-selection factors (1-f_icmn) for excluded examples. Consequently, Eq. (5) cannot be the distribution from which the ordered prompt in Eq. (6) is sampled, and the log-probability used in the REINFORCE estimator in Eq. (8) is not the log-probability of the sampled sequence. Moreover, the mechanism by which pairwise PEC predictions are converted into a global ordering via topological sorting is undefined when the predicted pairwise orders contain cycles, which have positive probability under Eq. (3). Please specify the exact sampling procedure: how many examples K are selected, how the subset size is chosen, and how cyclic pairwise predictions are resolved. Then provide the correct log-probability of the sampled sequence and re-derive the policy-gradient update.","section":"4.2.1, Eqs. (5)-(6) and 4.2.3, Eq. (8)"},{"comment":"The abstract claims an average improvement of 0.10 in ROUGE-1, 0.07 in ROUGE-2, 0.07 in ROUGE-L, and 0.05 in BLEU, but this is not the gain over the best baseline. In Table 1, the margins over the best baseline are at most 0.04, and several cells show no improvement at all (e.g., Dolly/GPT-4 ROUGE-2, ROUGE-L, and BLEU are all +0.00). Please clarify the reference point for the reported average improvement and, given that the experiments were repeated three times, report standard deviations or a statistical significance test so that the reader can judge whether the small margins are reliable.","section":"Abstract and Section 5.2, Table 1"},{"comment":"The number of in-context examples K used by the main method is never defined. The prompt template in the appendix uses →P_ic = [p^1_ic, ..., p^K_ic], and Eq. (6) defines the state space over all possible ordered subsets, but the policy network in Eq. (5) does not specify how many examples are selected or how K is determined. The ablation variant “-w/o RF” sets the number of examples to 2, but the main method’s K is absent. Without this, the action space and the reported results are not fully reproducible.","section":"5.1.2 and 4.2.1"}],"minor_comments":[{"comment":"The text says “We compare GRL-Prompt with three baselines” but then lists five baselines (Random, PromptPG, CoT, Pillow, UDR). Please correct the enumeration.","section":"5.1.2"},{"comment":"The pair score function ps(vi_c, vj_c) is defined via a softmax-like expression, but Eq. (2) then takes the max over the two directions. It is unclear why a max is used rather than a direct probability over the two directions, and the notation ps is easy to confuse with a probability. Please clarify the motivation and notation.","section":"Eq. (3)"},{"comment":"The reward term Rm is called “fuzzy textual similarity” but is never precisely defined; if it is automatically computed by ROUGE/BLEU or a related n-gram overlap measure, that should be stated explicitly, especially because the evaluation metrics are the same family of scores.","section":"4.2.2, Eq. (7)"},{"comment":"The LLaMA citation is listed as “Hugo et al.”; the correct author is Touvron et al. (2023).","section":"References"},{"comment":"The prompt template is called “Figure 1,” but Figure 1 in the main text is the system overview. The cross-referencing between the main text and appendix is confusing; please renumber or refer explicitly.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The central RL formalization is currently not correct as written, and the empirical margins are modest and unsupported by significance tests. That said, the issues are addressable if the authors can clearly specify their sampling procedure and re-derive the policy-gradient update. I would not reject on the current evidence, but I would require a careful revision before a second round of review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: GRL-Prompt is a novel and plausible combination of knowledge-graph state representation and RL for ordered in-context example selection, but the formal RL story as written doesn't hang together, and the headline numbers overstate the gain over the strongest baselines. It deserves a serious referee, but the authors will need to fix the math and release code.\n\nWhat's new: the heterogeneous graph over query and candidate examples, plus the pairwise edge classifier (PEC) and in-context matching network (ICMN) jointly producing an ordered prompt, is not in the cited work. The ablation and sensitivity analyses are thoughtful, and running across GPT-4, GPT-3, and LLaMA on Alpaca and Dolly is reasonable effort. Code is promised.\n\nSoft spots:\n\n1. Eq (5) is not a probability over ordered subsets. It multiplies f_icmn over all N candidates and f_pec over all pairs, with no non-selection factors, no subset-size normalization, and no stated K. Topological sorting of pairwise predictions is undefined when the PEC produces cycles, which has positive probability. So Eq (8)'s policy-gradient estimator is not guaranteed to optimize the distribution actually used to sample prompts. This is load-bearing: the 'RL' claim rests on it. The empirical tables may survive, but the theoretical description needs a rewrite and a precise sampling algorithm.\n\n2. The abstract's average improvement of 0.10 on ROUGE-1 is not the gain over the best baseline; looking at Table 1, the gains over UDR or Pillow are mostly 0.01 to 0.04, sometimes 0.00. That's a meaningful difference in claims.\n\n3. No standard deviations or significance tests are reported, even though runs were repeated three times. With differences that small, we need to know the variance.\n\nMinor: the reward includes text-similarity scores that overlap with the evaluation metrics, so the evaluation is not fully external. Not fatal, but worth stating.\n\nWho's it for: researchers working on prompt optimization and in-context learning, especially those interested in structured state representations. With the math fixed and code available, it would be a useful baseline.\n\nRecommendation: send to peer review, but expect major revision. The core idea is worth engaging; the execution currently undersells itself with an overbroad abstract and an underspecified policy.","headline":"A novel but formally under-specified RL approach to ordered in-context example selection; the core idea is worth engaging, but the policy-gradient math needs major repair and the headline gains are overstated.","tokens_in":15336,"tokens_out":4170,"would_cite":false,"duration_ms":38709,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GRL-Prompt claims that building a knowledge graph from a user query and candidate in-context examples, then training a policy network with reinforcement learning to select and order those examples, improves LLM generation scores by…","keywords":["prompt optimization","in-context learning","reinforcement learning","knowledge graph","heterogeneous graph transformer","large language models","example ordering","policy gradient"],"falsifier":"Instrument the sampler on a fixed query: compute every pairwise order score and matching score, record the ordered prompt the system actually sends to the LLM, and check whether that prompt has the highest probability under Eq. (5); any prompt whose probability is not maximal, or any pair predicted in a cycle, shows the training objective is not the distribution being evaluated.","tokens_in":14373,"feed_emoji":"🧠","tokens_out":7824,"duration_ms":68795,"temperature":0.7,"pith_summary":"GRL-Prompt tries to automate the labor-intensive part of prompt engineering for large language models: choosing which in-context examples to show and in what order. The paper builds a knowledge graph whose nodes are the user query and candidate examples, learns embeddings with a heterogeneous graph transformer, and trains a policy network by reinforcement learning to pick and order examples for the final prompt. The reported result is an average gain over baselines of 0.10 on ROUGE-1, 0.07 on ROUGE-2, 0.07 on ROUGE-L, and 0.05 on BLEU across Alpaca and Dolly with GPT-4, GPT-3, and LLaMA. If true, this makes prompt optimization automatic, order-aware, and independent of the specific LLM being prompted.","feed_headline":"Prompt optimizer with a knowledge graph lifts ROUGE-1 by 0.10","feed_subtitle":"REINFORCEMENT learning selects and orders in-context examples, beating baselines on Alpaca and Dolly across three LLMs.","key_machinery":"The load-bearing object is a heterogeneous knowledge graph $G(q)$ with one query node and one node per candidate example, carrying three edge relations ($r_{cc}$, $r_{qc}$, $r_{cq}$), embedded by a two-layer Heterogeneous Graph Transformer. On top of it, the policy network factors as a product over a pairwise edge classifier (which computes order probabilities between two candidates via a sin-based pair score) and an in-context matching network (which scores query-candidate relevance with a scaled sigmoid attention). The policy-gradient update uses the REINFORCE estimator with an embedding-based reward $R(a,\\hat{a}) = \\lambda R_m(a,\\hat{a}) + (1-\\lambda)R_e(a,\\hat{a})$ that stabilizes training. This machinery matters because it couples relevance and order into one differentiable distribution while keeping the LLM a black box that only supplies rewards.","core_discovery":"The paper's central claim is that prompt optimization for in-context learning can be cast as a structured reinforcement-learning problem over a knowledge graph. The query and candidate examples become nodes; a heterogeneous graph transformer embeds them; a policy network scores each candidate's relevance to the query (in-context matching network) and predicts, for every pair of candidates, which should appear first (pairwise edge classifier). The two scores multiply into one policy distribution, and policy gradient maximizes an embedding-shaped reward that combines fuzzy textual similarity with cosine embedding similarity. Across Alpaca and Dolly with GPT-4, GPT-3, and LLaMA, the resulting ordered example sequences outperform all tested baselines by the reported margins, with ablations attributing the gains to both the knowledge graph and the reward feedback.","pith_inferences":["The paper never fixes the number of in-context examples, so a reader could test whether the gains come from choosing a different-length prompt rather than from ordering; comparing against a variant with K fixed would separate the two effects.","The knowledge graph edges are all pre-defined rather than learned, so the graph construction may be less important than the HGT embedding and RL reward; replacing the graph with a plain encoder in the same training loop would isolate that.","The loose link between the factored policy in Eq. (5) and the topological-sort sampling in Eq. (6) suggests the training objective and the executed prompt distribution may diverge; checking this directly is a concrete falsification experiment.","The same order-sensitive selection view transfers naturally to retrieval-augmented generation, where the order of retrieved passages is known to affect answers, so GRL-Prompt-style pairwise ranking could be adapted to rerank retrieved contexts."],"forward_implications":["The same knowledge-graph-plus-policy design can be re-trained for any black-box LLM, since the generator only supplies rewards, so prompt optimization becomes a transferable service rather than a per-model manual exercise.","Directly modeling pairwise order should make prompt selection less sensitive to permutation effects, which prior retrieval and fixed-length selectors ignore.","The ablations imply that both the knowledge-graph representation and the reward-feedback loop are doing measurable work, so future prompt optimizers should keep structure and feedback rather than relying on embeddings alone.","Because the reward is computed from ROUGE/BLEU scores, the approach is immediately extendable to other text-to-text metrics, and the same pipeline can be pointed at a new dataset without changing the method."],"supporting_citations":[{"why":"Supplies the Heterogeneous Graph Transformer (HGT) used to learn node embeddings in the constructed knowledge graph.","marker":"(Hu et al., 2020)"},{"why":"PromptPG is the main RL-based in-context example selection baseline and supplies the policy-gradient setup GRL-Prompt extends.","marker":"(Lu et al., 2023)"},{"why":"Pillow is a prompt-matching RL baseline that GRL-Prompt must outperform.","marker":"(Qi et al., 2023)"},{"why":"UDR is the strongest baseline, a ranking-based demonstration retriever, and is used for training-loss comparisons.","marker":"(Li et al., 2023)"},{"why":"Chain-of-thought prompting is included as a manual few-shot prompting baseline.","marker":"(Wei et al., 2022b)"},{"why":"Defines the in-context learning paradigm and the random-selection baseline.","marker":"(Brown et al., 2020)"},{"why":"Provides the k-nearest-neighbor calibration method that replaces reward feedback in the w/o RF ablation.","marker":"(Nie et al., 2022)"},{"why":"The REINFORCE policy-gradient algorithm used to update the policy network.","marker":"(Williams, 1992)"},{"why":"Alpaca dataset: one of the two text-to-text instruction-following evaluation sets.","marker":"(Taori et al., 2023)"},{"why":"Dolly dataset: the second evaluation set with human-annotated instruction-response pairs.","marker":"(Conover et al., 2023)"}],"fun_headline_variants":["Knowledge graph + RL picks prompt examples, beats baselines","RL with knowledge graph auto-orders prompt examples","Prompt tuning via RL on a knowledge graph","Knowledge graph boosts RL prompt selection","GRL-Prompt: RL prompt optimizer built on a knowledge graph"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim assumes the prompts actually evaluated are sampled from the distribution the policy gradient optimizes, but the paper never specifies how the per-candidate matching scores decide how many examples are selected or how conflicting pairwise order predictions are resolved into a single ordered prompt.","fun_headline_variants_meta":{"raw":{"variants":["Knowledge graph + RL picks prompt examples, beats baselines","RL with knowledge graph auto-orders prompt examples","Prompt tuning via RL on a knowledge graph","Knowledge graph boosts RL prompt selection","GRL-Prompt: RL prompt optimizer built on a knowledge graph"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000605,"raw_usage":{"total_tokens":2831,"prompt_tokens":966,"completion_tokens":1865,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":1792}},"tokens_in":582,"tokens_out":1865,"duration_ms":12678,"temperature":1.0,"reasoning_tokens":1792,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:34:23.058105+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Instrument the sampler on a fixed query: compute every pairwise order score and matching score, record the ordered prompt the system actually sends to the LLM, and check whether that prompt has the highest probability under Eq. (5); any prompt whose probability is not maximal, or any pair predicted in a cycle, shows the training objective is not the distribution being evaluated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the in-context learning paradigm and the random-selection baseline."}],"review_version":1}