{"id":"3c73c33e-d224-4a99-bc65-8bc3619505b5","arxiv_id":"2502.06487","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On StereoSet and SBIC, a trained DeBERTa encoder that selects per-input prompt compositions from 64 options raises macro F1 above every fixed composition, but on CobraFrames it falls below the best fixed composition.","lead":"An encoder model predicts, for each text, which combination of prompting techniques to feed an LLM when detecting social bias, instead of using one fixed prompt for everything. It beats the best fixed prompt on two of three bias datasets, but fails on the third and shows only tiny gains on three other NLP tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The adaptive selector is trained with per-instance BCE but evaluated on macro F1; on imbalanced CobraFrames this mismatch, not an unlearnable input-composition mapping, likely drives the reported failure, weakening the robustness claim.","rationale":"The reader's weakest assumption is that the DeBERTa encoder can learn, from text alone, which of the 64 prompt compositions will elicit a correct label, and that this learnability fails on CobraFrames. I agree that the CobraFrames failure is the main threat to the robustness claim, but the paper's own Table 5 suggests a more specific and more load-bearing mechanism: the encoder's training objective is per-instance binary cross-entropy over composition correctness, while all headline results are macro F1. On CobraFrames, a 96%/4% class imbalance makes accuracy and macro F1 diverge strongly. The encoder can appear to succeed on its training objective (high ratio of correct selected compositions) while producing lower macro F1 than a fixed composition, exactly as observed. This is not merely an alternative interpretation; it changes what the paper should fix to support the 'robustly' claim. The central positive evidence on StereoSet remains credible: adaptive prompting beats even the test-set-oracle Best on Test for all three LLMs, and the authors provide code and per-seed training. The Shapley analysis is a useful independent contribution, and the limitations section is candid about CobraFrames. However, the claimed robustness is not supported across datasets, and the stated reason (encoder cannot learn CobraFrames connections) is contradicted by the accuracy-level results in Table 5. The appropriate verdict remains CONDITIONAL: the paper should either align the selector objective with macro F1 or soften the robustness claim, and it should report the requested per-class breakdowns and variance estimates.","tokens_in":46501,"tokens_out":7185,"duration_ms":65250,"concrete_test":"Retrain the CobraFrames adaptive selector with a macro-F1-aware objective: replace the per-instance binary cross-entropy targets with per-class weighted BCE (e.g., weight the minority class inversely to its frequency, or define the target composition on the validation split as the one maximizing validation macro F1 rather than validation accuracy), using the same LLM labels, encoder architecture, and random seeds. Then compare test macro F1 for adaptive prompting against Best on Val and Best on Test on CobraFrames for all three LLMs. If the gap closes, the identified mismatch is the cause and the robustness claim needs re-scoping; if the gap persists, the encoder-learnability explanation in Section 5.5 is supported. Additionally report per-class precision and recall of the selected compositions to expose the accuracy/F1 trade-off.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that adaptive prompting \"robustly ensures high detection performance\" depends on the encoder learning to select the composition that maximizes macro F1 per instance. Yet Section 3.2 trains the encoder with a sigmoid output and binary cross-entropy on whether each composition produces the correct binary label, i.e., it optimizes per-instance accuracy, not macro F1. These objectives diverge sharply on CobraFrames, whose test split is 1862 positive versus 77 negative instances (Table 4). The authors attribute the CobraFrames failure to the encoder being unable to learn meaningful text-composition connections (Section 5.5), but their own Table 5 shows the encoder's selected compositions yield correct labels for 84.6% (Mistral), 83.3% (Command-R), and 82.0% (Llama 3) of CobraFrames instances, comparable to or better than the other datasets. Meanwhile, adaptive macro F1 on CobraFrames is below Best on Test for all three LLMs (0.580 vs 0.604, 0.561 vs 0.668, 0.567 vs 0.605; Table 8) and below Best on Val for Command-R and Llama 3. An accuracy-trained selector can be correct on the vast majority of instances while making errors on the 77 minority-class instances that dominate macro F1. Thus the paper's own diagnosis is not the only, or most consistent, explanation; the accuracy-versus-macro-F1 objective mismatch is a load-bearing alternative that directly bears on the claimed robustness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an adaptive prompting approach that trains a DeBERTa-based encoder to predict, for each input text, which of 64 hand-defined prompt compositions will make a target instruction-tuned LLM produce a correct binary social-bias label. The approach is evaluated on three datasets (StereoSet, SBIC, CobraFrames) with three LLMs (Mistral-7B, Command-R-35B, Llama-3-70B) and compared against fixed compositions, an ensemble, a fine-tuned DeBERTa, and Shapley-value-based composition selection. The central empirical claim is that per-instance composition selection can outperform the best fixed composition on the test set: on StereoSet this holds for all three LLMs, on SBIC it holds only for Llama 3, and on CobraFrames adaptive prompting is below the best fixed test composition for all three LLMs. The paper also includes a Shapley interaction analysis of the prompting techniques and additional experiments on three other NLP tasks.","tokens_in":46797,"tokens_out":6648,"duration_ms":53917,"significance":"The idea of selecting prompt compositions per input instance is a useful and relatively underexplored direction, and the experimental setup is extensive, covering three LLMs of different sizes, three datasets, and a large number of compositions. The comparison against Best on Test is appropriately conservative, as it uses oracle knowledge of test-set performance, and the Shapley interaction analysis provides interpretable insight into how prompting techniques interact. The StereoSet results, where adaptive prompting beats the best fixed test composition for all three LLMs, are the strongest evidence for the approach. However, the failure on CobraFrames, together with the statistical reporting limitations, substantially weakens the robustness and generality claims made in the abstract and conclusion. If the objective-mismatch issue is resolved and the significance analysis is made more rigorous, the contribution would be solid for a specialized NLP methods venue.","major_comments":[{"comment":"The encoder is trained with binary cross-entropy on whether each composition produces the correct binary label per instance, which optimizes per-instance accuracy, but it is evaluated with macro F1. On CobraFrames, the test split is highly imbalanced (1862 positive vs 77 negative instances, Table 4), so an accuracy-trained selector can be correct on the vast majority of instances while making errors on the 77 negatives, which dominate macro F1. Table 5 shows that the encoder's selected compositions yield correct labels for 84.6%, 83.3%, and 82.0% of CobraFrames instances for Mistral, Command-R, and Llama 3 respectively, yet Table 8 shows adaptive macro F1 is below Best on Test for all three LLMs. The paper's diagnosis in §5.5 that the encoder \"can likely not learn meaningful connections between the inputs and compositions\" is therefore not the most consistent explanation; the accuracy-vs-macro-F1 objective mismatch is a load-bearing alternative that directly affects the claimed robustness. The authors should either re-analyze the failure with per-class breakdowns or adjust the training objective (e.g., cost-sensitive or metric-aware learning), and revise the explanation accordingly.","section":"§3.2, §5.5, Tables 4, 5, 8"},{"comment":"The significance testing is selective and lacks multiple-comparison correction. As stated in §B.3, a one-sided independent t-test over five seeds is used only when adaptive prompting shows the best overall results, and no correction is applied across the many comparisons (three LLMs, multiple datasets, multiple baselines). Table 1, for instance, reports p<.05 and p<.01 for three LLM comparisons without accounting for the multiple tests, and no confidence intervals are given for the macro F1 scores. This weakens the headline claim that adaptive prompting \"robustly ensures high detection performance\" and \"is best in several settings.\" I recommend reporting all tested comparisons, applying a multiple-comparison correction (e.g., Bonferroni or Holm), and providing per-seed variances or confidence intervals.","section":"§B.3, Tables 1, 7, 8"}],"minor_comments":[{"comment":"The statement \"Our approach robustly ensures high detection performance\" is too strong given that adaptive prompting underperforms Best on Test on CobraFrames for all three LLMs (Table 8). Suggest softening to something like \"can achieve high detection performance in several settings.\"","section":"Abstract and Conclusion"},{"comment":"There are typos in the captions of Tables 9, 10, and 11: \"defintion\" should be \"definition.\"","section":"Tables 9–11 captions"},{"comment":"The Acknowledgments contain a typo: \"ChaptGPT\" should be \"ChatGPT.\"","section":"Acknowledgments"},{"comment":"The rationale for avoiding a multi-class setup, namely \"to avoid a few dominant compositions from possibly being preferred over others consistently,\" would benefit from a more detailed explanation of how this preference would arise and why a sigmoid output avoids it.","section":"§3.2"},{"comment":"The main results tables report adaptive prompting as a single macro F1 value without standard deviations or confidence intervals, even though it is averaged over five seeds; including the variance would aid interpretation of the significance claims.","section":"Tables 1, 7, 8"},{"comment":"The majority baseline value (0.364) shown for CobraFrames in Figure 9 is not clearly defined; given the class imbalance in Table 4, it would be helpful to state how the majority baseline macro F1 is computed (e.g., the macro F1 of always predicting the majority class).","section":"Figure 9 and Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the core idea is interesting. The main concerns are the objective mismatch on CobraFrames, which undermines the paper's own explanation of the failure, and the selective significance testing. I believe these are addressable within a revision, but they require substantial additional analysis or retraining with a macro-F1-aware objective. If the authors can demonstrate that the objective mismatch explains the CobraFrames results, or improve the method accordingly, the paper would be acceptable. I would also recommend asking the authors to provide per-class precision/recall for the adaptive method on CobraFrames and to report all significance tests transparently."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a solid, hard-working empirical paper on per-instance prompt composition selection for social bias detection. The idea is genuinely new in the framing—predicting which of 64 hand-designed discrete prompt compositions will work for each input text, rather than searching for one global prompt. The authors compare against the best composition on the test set (a generous baseline) and their adaptive method wins on StereoSet across all three LLMs and on SBIC for Llama 3. They also ship a code link, report many ablations, and are candid that the method fails on CobraFrames.\n\nWhat the paper does well: the evaluation is extensive—three LLMs (7B to 70B), three datasets, 64 compositions, five seeds, plus a Shapley interaction analysis. The finding that prompt composition performance is highly volatile and that instance-specific selection can beat the best fixed composition on the test set is credible and useful for practitioners. The Shapley interaction analysis also gives real insight into which techniques combine well and which interact negatively.\n\nBut there are real issues. The CobraFrames failure is the main one. The authors attribute it to the encoder not learning meaningful text-composition connections, but the stress-test analysis surfaces a more likely culprit: the encoder is trained with binary cross-entropy on per-instance correctness, while the evaluation metric is macro F1. On CobraFrames, the test set is 1862 positive vs 77 negative. A selector that gets 84% of instances right can still produce a macro F1 near 0.58 because it misses most of the 77 minority-class instances. That is an objective mismatch, not necessarily a learnability failure. The paper's own numbers in Table 5 support this—the encoder's per-instance accuracy is actually comparable to the other datasets. So the 'robustly' claim in the abstract is too strong; at best, the method is robust on reasonably balanced data.\n\nI also wish the significance testing were sturdier: a one-sided t-test over five seeds without multiple-comparison correction, and no confidence intervals for the headline macro F1 numbers. This is a moderate concern, not fatal—the StereoSet gains are large enough that they would likely survive a more conservative test, but the SBIC results (0.790 vs 0.792 with Mistral) would not survive.\n\nWho should read it: anyone working on automatic prompting, few-shot selection, or bias detection with LLMs. The idea is subfield-level, not paradigm-shifting, but it is a real step beyond global prompt search. It deserves a serious referee: I would accept this for review and ask for the objective-mismatch analysis and corrected significance testing as major revisions. The cross-dataset generalization is weak, but they acknowledge that too. Overall, a paper worth engaging with despite the flagged concerns.","headline":"Solid empirical paper on per-instance prompt composition selection, but the CobraFrames failure is likely an objective mismatch (BCE vs macro F1) rather than the unlearnability the authors claim.","tokens_in":47372,"tokens_out":3107,"would_cite":true,"duration_ms":27825,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Per-input prompt selection can beat any single fixed prompt composition in social bias detection.","keywords":["adaptive prompting","prompt composition","social bias detection","large language models","in-context learning","Shapley interactions","encoder-based selection","prompt engineering"],"falsifier":"On a held-out test set, compute the per-instance oracle composition (the one that yields the correct label), then train the encoder on the training split. If the encoder's chosen composition is correct no more often than choosing the single most frequent composition from the training set, the central claim is refuted; CobraFrames with Mistral (0.580 adaptive vs 0.604 best fixed) is a concrete case where this already appears to happen.","tokens_in":46277,"feed_emoji":"🎯","tokens_out":6787,"duration_ms":52812,"temperature":0.7,"pith_summary":"This paper tries to show that the most effective prompt for a large language model is not fixed: it depends on the text being classified, and a learned selector can choose the right combination of prompting techniques for each input. The authors test this on social bias detection with three LLMs and three datasets, building 64 prompt compositions from five techniques and training an encoder to predict which composition will produce a correct label. On StereoSet the per-instance selector beats the best fixed composition for all three LLMs (macro F1 0.809 vs 0.800 for Mistral, 0.781 vs 0.706 for Command-R, 0.853 vs 0.817 for Llama 3); on SBIC it improves over the best fixed composition for Llama 3 and is competitive for Mistral. The approach fails on CobraFrames, where the encoder cannot learn meaningful input-to-composition links, so the paper's central claim is conditional on that learnability.","feed_headline":"Adaptive prompt choice beats best fixed prompt in bias tests","feed_subtitle":"An encoder picks which prompt techniques to combine per text, outperforming the best single composition on StereoSet and SBIC.","key_machinery":"The load-bearing object is the prompt composition: a fixed-order combination of the base task description and input with any subset of five techniques, where the in-context demonstrations technique has three mutually exclusive variants, yielding $2^4 \\cdot (3+1) = 64$ compositions. The mechanism that carries the argument is an encoder-based selection model: a fine-tuned DeBERTa-v3-large with a sigmoid regression head trained by binary cross-entropy to estimate, for each input, the likelihood that each of the 64 compositions will generate a correct label; at inference it applies the highest-likelihood composition. The Shapley value and Shapley interaction analysis over the composition game supplies the second part: it attributes performance to individual techniques and their pairwise synergies, supporting the claim that technique interactions matter and that no single composition is universally best.","core_discovery":"The paper's central claim is that prompt compositions—ordered sets of discrete prompting techniques—can and should be chosen ad hoc per input instance rather than optimized once per dataset or model. The authors define 64 compositions from five techniques (persona, definition, in-context demonstrations in one of three selection modes, directional stimulus, and reasoning steps), collect LLM labels for every composition on training texts, then fine-tune a DeBERTa-v3-large encoder to output a likelihood for each composition and pick the argmax. They report that this adaptive prompting outperforms the best single test-set composition on StereoSet for all three LLMs, and on SBIC for Llama 3. They also find that the performance of compositions is volatile across models and texts, that adding more techniques does not monotonically improve results, and that Shapley interaction analysis reveals both positive and negative pairwise interactions between techniques.","pith_inferences":["A testable extension is to restrict the selector to the small set of compositions that disagree most often on a validation set; the gains should concentrate there, and the computational cost of collecting labels would drop sharply.","The CobraFrames failure suggests a diagnostic for when to trust adaptive prompting: measure the mutual information between the input and the oracle-optimal composition. If it is near zero, a fixed composition is the safer choice.","If the encoder is choosing the most frequent composition on hard datasets, the approach degenerates to best-fixed selection; comparing the selector's chosen frequencies to per-composition correctness on the training split, as the paper does, is a cheap sanity check.","The gains over best fixed compositions are modest in absolute terms; a sympathetic reading is that adaptive prompting is a safety property—it avoids the worst compositions—rather than a large average improvement, and that is what the cross-dataset results suggest."],"forward_implications":["Prompt selection can be moved from human trial-and-error to a learned per-instance decision, reducing the cost of finding a good prompt for a new LLM or dataset.","A fixed best-on-validation composition is not an upper bound; input-dependent selection can beat it, as shown on StereoSet and SBIC.","Compositions that combine more techniques are not automatically better; Shapley interactions show that individually helpful techniques can hurt when combined, so selection must consider interactions.","The approach transfers to other tasks with smaller gains: on sentiment analysis, natural language inference, and question answering it beats the base composition and the best-validation composition in the reported experiments.","On datasets where text content does not predict which prompt will succeed, such as CobraFrames, adaptive prompting falls back toward the level of a fixed or random selector."],"supporting_citations":[{"why":"Supplies the StereoSet corpus and its stereotype definition, the primary dataset where adaptive prompting beats all fixed compositions.","marker":"Nadeem et al., 2021"},{"why":"Supplies the SBIC corpus with implicit-bias labels and annotation questionnaire that shapes the task and reasoning steps.","marker":"Sap et al., 2020"},{"why":"Supplies the CobraFrames dataset and the offensiveness-to-binary conversion used for the third evaluation.","marker":"Zhou et al., 2023a"},{"why":"Provides the reasoning-steps prompting technique that the compositions include as a discrete component.","marker":"Wei et al., 2022"},{"why":"Provides the in-context learning survey that defines the demonstrations technique and its variants used in the compositions.","marker":"Dong et al., 2024"},{"why":"Provides DeBERTa-v3-large, the encoder architecture that the adaptive prompting selector is fine-tuned from.","marker":"He et al., 2023"},{"why":"Provides Mistral-7B-Instruct, one of the three LLMs whose labels and prompt sensitivity are evaluated.","marker":"Jiang et al., 2023"},{"why":"Provides Command-R, the 35B LLM used in the evaluation.","marker":"CohereForAI, 2024"},{"why":"Provides Llama 3 70B, the largest LLM in the evaluation where adaptive prompting gives the largest gains.","marker":"Dubey et al., 2024"},{"why":"Provides the Shapley interaction computation used for the composition analysis.","marker":"Fumagalli et al., 2023"}],"fun_headline_variants":["Adaptive prompts outsmart best fixed composition in bias tests","Per-instance prompt choice beats one-size-fits-all for bias","Ad-hoc prompt mixes beat static prompts for bias detection","Pick prompt per text beats best fixed in bias tests","Adaptive prompt selection per input beats best static"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach rests on the assumption that a text-encoder can learn, from the input text alone, which of many prompt compositions will make the LLM produce a correct label; the paper itself finds this fails on CobraFrames.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive prompts outsmart best fixed composition in bias tests","Per-instance prompt choice beats one-size-fits-all for bias","Ad-hoc prompt mixes beat static prompts for bias detection","Pick prompt per text beats best fixed in bias tests","Adaptive prompt selection per input beats best static"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00089,"raw_usage":{"total_tokens":3810,"prompt_tokens":887,"completion_tokens":2923,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":2854}},"tokens_in":503,"tokens_out":2923,"duration_ms":18833,"temperature":1.0,"reasoning_tokens":2854,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T15:17:42.539285+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out test set, compute the per-instance oracle composition (the one that yields the correct label), then train the encoder on the training split. If the encoder's chosen composition is correct no more often than choosing the single most frequent composition from the training set, the central claim is refuted; CobraFrames with Mistral (0.580 adaptive vs 0.604 best fixed) is a concrete case where this already appears to happen.","supporting_citations":[{"cited_title":"Smith, and Yejin Choi","cited_arxiv_id":null,"evidence_quote":"Supplies the SBIC corpus with implicit-bias labels and annotation questionnaire that shapes the task and reasoning steps."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Command-R, the 35B LLM used in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Shapley interaction computation used for the composition analysis."}],"review_version":1}