On the Stability of Prompt Ranking in Large Language Model Evaluation

Chuanqi Shi; Hang Zhang; Lun Wang; Penghao Liang; Shaoshuai Du; Yixian Shen

arxiv: 2606.24381 · v1 · pith:6KUUS66Nnew · submitted 2026-06-23 · 💻 cs.CL · cs.AI

On the Stability of Prompt Ranking in Large Language Model Evaluation

Shaoshuai Du , Penghao Liang , Yixian Shen , Chuanqi Shi , Hang Zhang , Lun Wang This is my paper

Pith reviewed 2026-06-25 23:55 UTC · model grok-4.3

classification 💻 cs.CL cs.AI

keywords prompt ranking stabilityLLM evaluationprompt selectionlower confidence boundevaluation uncertaintyrank correlationbenchmarking robustness

0 comments

The pith

Prompt rankings in LLM evaluations frequently change with small variations in seeds or data subsets, making top prompt selection unreliable.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether the ordering of candidate prompts stays consistent when evaluation conditions vary slightly through different random seeds or smaller data subsets. It reports that overall rank correlations between runs are often moderate to high, yet the single highest-ranked prompt switches identity in many cases, which undermines the common practice of picking the apparent winner for downstream use. To mitigate this, the authors introduce a selection rule based on a lower confidence bound that factors in both average performance and observed variance across conditions. Experiments across three open-weight models and two tasks show the bound-based rule delivers more stable choices in variable settings without losing ground when rankings are already reliable. The central point is that prompt selection must treat evaluation uncertainty as a first-class factor rather than assuming average scores alone suffice.

Core claim

Across three open-weight LLMs and two benchmark tasks, prompt rankings exhibit instability under variations in random seeds and evaluation subsets, with the top-performing prompt changing frequently despite moderate to high overall rank correlations. A lower confidence bound selection strategy that incorporates variance improves robustness in unstable settings while staying competitive in stable ones.

What carries the argument

Lower confidence bound selection that combines mean performance with variance estimates across repeated evaluation conditions to rank prompts.

If this is right

Selecting the prompt with the highest lower confidence bound reduces the chance of choosing one that only appears best due to a lucky seed or subset.
Reporting only average performance or rank correlation can mask frequent flips in the actual top prompt.
Variance-aware selection remains competitive even when evaluation conditions are stable, so the added computation does not sacrifice performance.
LLM benchmarking workflows that ignore evaluation variance risk publishing or deploying prompts that do not generalize across minor run differences.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Standard practice in prompt engineering and model cards may need to include multiple seeds or subset sizes as routine reporting requirements.
The same lower-confidence-bound logic could apply to other ranking-based decisions such as hyperparameter search or model selection where variance across runs is observable.
Developers might adopt a default of evaluating each prompt at least three times with different seeds before final selection to surface unstable candidates early.

Load-bearing premise

The variability from random seeds and limited evaluation subsets is representative of the uncertainty that matters for real prompt selection decisions.

What would settle it

Run a large-scale held-out evaluation with many more seeds and full data; check whether the lower-confidence-bound prompt consistently underperforms the prompt that actually scores highest on that held-out set.

Figures

Figures reproduced from arXiv: 2606.24381 by Chuanqi Shi, Hang Zhang, Lun Wang, Penghao Liang, Shaoshuai Du, Yixian Shen.

**Figure 1.** Figure 1: Overview of the proposed prompt evaluation and selection framework. The pipeline simulates evaluation variability through multi-seed subsampling before applying the stability-aware selection strategy. 3.1 Problem Formulation Let P = {p1, p2, . . . , pM} denote a fixed set of candidate prompts for a given task, and let D denote the evaluation dataset. Under an evaluation condition c (e.g., a specific rando… view at source ↗

read the original abstract

Prompt-based interaction has become a dominant paradigm for using large language models (LLMs), where multiple candidate prompts are evaluated and the top-ranked one is selected for downstream use. This workflow implicitly assumes that prompt rankings are stable under minor variations in evaluation conditions. In this paper, we systematically study prompt ranking stability under common sources of variability, including random seeds and limited evaluation subsets. Across three open-weight LLMs and two benchmark tasks, we find that while overall rank correlations are often moderate to high, the identity of the top-performing prompt frequently changes, leading to unreliable selection decisions. To address this issue, we propose a simple stability-aware selection strategy based on a lower confidence bound, which accounts for both performance and variance. Our results show that this approach improves robustness in unstable settings while remaining competitive in more stable regimes. These findings highlight the importance of accounting for evaluation uncertainty in prompt selection and LLM benchmarking.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper shows prompt rankings flip often under seed or subset changes and offers a simple LCB fix, but the variance estimates look under-checked for small sample sizes.

read the letter

The main thing to know is that this work documents real instability in which prompt ranks first when you change random seeds or shrink the evaluation set, and it offers a lower-confidence-bound rule to pick prompts that are less sensitive to those changes.

They test three open-weight models on two tasks and report that rank correlations are often decent but the top prompt switches frequently. That observation is the concrete new piece, and it matches what many people see in practice when they try to lock in a prompt.

The paper does a clean job of framing the problem around an actual selection workflow instead of abstract metrics. The LCB approach is just a direct application of a mean-minus-variance bound, which keeps it easy to implement.

The soft spot is exactly the one in the stress-test note. The abstract gives no numbers on how many seeds or subset sizes they used, and no sign they checked whether the variance estimates themselves are stable or compared the LCB decisions to a bootstrap. With small replicate counts the sample variance can be noisy enough to drive the ranking, so the claimed robustness gain rests on an unverified assumption. If the full paper has those checks, the concern shrinks; otherwise it stays material.

This is for people who run prompt sweeps for benchmarks or production and want a slightly more cautious selection rule. A practitioner reader will find the empirical pattern useful even if the fix needs more validation.

It is worth sending to peer review. The core observation is grounded enough to matter, and referees can press on the variance reliability question directly.

Referee Report

2 major / 2 minor

Summary. The paper claims that prompt rankings for LLMs are unstable under common sources of variability such as random seeds and limited evaluation subsets: while overall rank correlations are often moderate to high, the identity of the top prompt frequently changes. It proposes a lower-confidence-bound (LCB) selection rule that trades off observed mean performance against variance across replicates and subsets, and reports that this rule improves robustness in unstable regimes while remaining competitive in stable ones. The study is conducted across three open-weight LLMs and two benchmark tasks.

Significance. If the empirical findings hold, the work is significant for LLM evaluation and prompt engineering practice because it identifies a concrete source of unreliability in the standard mean-based selection workflow and offers a simple, variance-aware alternative. The multi-model, multi-task design is a strength that increases the generality of the instability observation.

major comments (2)

[Abstract and §3] Abstract and §3 (LCB selector definition): the claim that the LCB rule 'improves robustness in unstable settings' rests on variance estimates whose sampling reliability is not verified; with only a handful of seeds or small subsets the sample variance itself has high sampling error, yet no standard errors on the variance estimates, sensitivity analysis to replicate count, or comparison against bootstrap/Bayesian alternatives that propagate variance uncertainty are reported.
[Experimental results (§4–5)] Experimental results (presumably §4–5): the reported rank correlations and top-prompt change frequencies are presented without accompanying statistical tests or confidence intervals on the instability metrics themselves, so it is unclear whether the observed changes exceed what would be expected from finite-sample noise alone.

minor comments (2)

[§3] Notation for the LCB formula should be introduced with an explicit equation number and a clear statement of the quantile or multiplier used (e.g., 95 % one-sided bound).
[§4] The paper should state the exact number of random seeds and subset sizes used in each experiment so that readers can assess the degrees of freedom available for variance estimation.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on our work regarding prompt ranking stability. We address each major comment below.

read point-by-point responses

Referee: [Abstract and §3] Abstract and §3 (LCB selector definition): the claim that the LCB rule 'improves robustness in unstable settings' rests on variance estimates whose sampling reliability is not verified; with only a handful of seeds or small subsets the sample variance itself has high sampling error, yet no standard errors on the variance estimates, sensitivity analysis to replicate count, or comparison against bootstrap/Bayesian alternatives that propagate variance uncertainty are reported.

Authors: We agree that sampling variability in the variance estimates merits attention, particularly with limited replicates. Our LCB formulation is presented as a practical heuristic that incorporates observed variance to improve selection robustness, and the empirical gains are shown consistently across three models and two tasks. To strengthen the analysis, we will add a sensitivity study varying the number of replicates used for variance estimation and include standard errors on key variance-derived quantities where appropriate. revision: partial
Referee: [Experimental results (§4–5)] Experimental results (presumably §4–5): the reported rank correlations and top-prompt change frequencies are presented without accompanying statistical tests or confidence intervals on the instability metrics themselves, so it is unclear whether the observed changes exceed what would be expected from finite-sample noise alone.

Authors: The reported metrics are intended as descriptive evidence of the instability phenomenon. We will incorporate bootstrap confidence intervals around the rank correlations and top-prompt change frequencies in the revised manuscript to better quantify uncertainty and address whether the observed instability exceeds finite-sample expectations. revision: yes

Circularity Check

0 steps flagged

Empirical study with no derivation chain or self-referential reductions

full rationale

The paper is an empirical investigation of prompt ranking stability across random seeds and evaluation subsets on three LLMs and two tasks. It reports observed rank correlations and top-prompt changes, then defines an LCB selection rule directly from sample means and variances. No equations, predictions, or uniqueness claims reduce to fitted inputs by construction; the LCB is a standard statistical construct applied to the observed data rather than derived from prior self-citations or ansatzes. The work is self-contained against external benchmarks with no load-bearing self-citation chains.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities are identifiable from the abstract alone; the work is an empirical measurement study rather than a theoretical derivation.

pith-pipeline@v0.9.1-grok · 5691 in / 1087 out tokens · 14270 ms · 2026-06-25T23:55:05.232881+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

13 extracted references · 6 canonical work pages

[1]

AI, M.: Mistral-7b-instruct-v0.3.https://huggingface.co/mistralai/ Mistral-7B-Instruct-v0.3(2024), accessed: 2026-03-23

2024
[2]

In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H

Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Win- ter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford...

2020
[3]

In: Inui, K., Jiang, J., Ng, V., Wan, X

Dodge, J., Gururangan, S., Card, D., Schwartz, R., Smith, N.A.: Show your work: Improved reporting of experimental results. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Ho...

2019
[4]

Show Your Work: Improved Reporting of Experimental Results

pp. 2185–2194. Association for Computational Linguistics (2019).https: //doi.org/10.18653/V1/D19-1224,https://doi.org/10.18653/v1/D19-1224 12 Authors Suppressed Due to Excessive Length

work page doi:10.18653/v1/d19-1224 2019
[5]

The hitchhiker’s guide to test- ing statistical significance in natural language processing

Dror, R., Baumer, G., Shlomov, S., Reichart, R.: The hitchhiker’s guide to test- ing statistical significance in natural language processing. In: Gurevych, I., Miyao, Y. (eds.) Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Vol- ume 1: Long Papers. pp. 1383–1392....

work page doi:10.18653/v1/p18-1128 2018
[6]

In: Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho, K., Oh, A

Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language mod- els are zero-shot reasoners. In: Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho, K., Oh, A. (eds.) Advances in Neural Information Process- ing Systems 35: Annual Conference on Neural Information Processing Sys- tems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28...

2022
[7]

Microsoft: Phi-3-mini-4k-instruct.https://huggingface.co/microsoft/ Phi-3-mini-4k-instruct(2024), accessed: 2026-03-23

2024
[8]

5-7B-Instruct(2024), accessed: 2026-03-23

Qwen Team, A.G.: Qwen2.5-7b-instruct.https://huggingface.co/Qwen/Qwen2. 5-7B-Instruct(2024), accessed: 2026-03-23

2024
[9]

In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp

Reimers, N., Gurevych, I.: Reporting score distributions makes a difference: Per- formance study of lstm-networks for sequence tagging. In: Palmer, M., Hwa, R., Riedel, S. (eds.) Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017. pp. 338–348. Association for Comput...

work page doi:10.18653/v1/d17-1035 2017
[10]

CoRRabs/2412.20061(2024).https://doi.org/10.48550/ ARXIV.2412.20061,https://doi.org/10.48550/arXiv.2412.20061

Shen, Y., Wang, L., Shi, C., Du, S., Tao, Y., Shen, Y., Zhang, H.: Compara- tive analysis of listwise reranking with large language models in limited-resource language contexts. CoRRabs/2412.20061(2024).https://doi.org/10.48550/ ARXIV.2412.20061,https://doi.org/10.48550/arXiv.2412.20061

work page doi:10.48550/arxiv.2412.20061 2024
[11]

In: Croft, W.B., Moffat, A., van Rijsbergen, C.J., Wilkin- son, R., Zobel, J

Voorhees, E.M.: Variations in relevance judgments and the measurement of re- trieval effectiveness. In: Croft, W.B., Moffat, A., van Rijsbergen, C.J., Wilkin- son, R., Zobel, J. (eds.) SIGIR ’98: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 24-28 1998, Melbourne, Australia. ...

work page doi:10.1145/290941.291017 1998
[12]

CoRRabs/2502.15770 (2025).https://doi.org/10.48550/ARXIV.2502.15770,https://doi.org/10

Wang, L., Shi, C., Du, S., Tao, Y., Shen, Y., Zheng, H., Shen, Y., Qiu, X.: Per- formance review on LLM for solving leetcode problems. CoRRabs/2502.15770 (2025).https://doi.org/10.48550/ARXIV.2502.15770,https://doi.org/10. 48550/arXiv.2502.15770

work page doi:10.48550/arxiv.2502.15770 2025
[13]

In: Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho, K., Oh, A

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E.H., Le, Q.V., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models. In: Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho, K., Oh, A. (eds.) Advances in Neural Information Process- ing Systems 35: Annual Conference on Neural Information Processing ...

2022

[1] [1]

AI, M.: Mistral-7b-instruct-v0.3.https://huggingface.co/mistralai/ Mistral-7B-Instruct-v0.3(2024), accessed: 2026-03-23

2024

[2] [2]

In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H

Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Win- ter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford...

2020

[3] [3]

In: Inui, K., Jiang, J., Ng, V., Wan, X

Dodge, J., Gururangan, S., Card, D., Schwartz, R., Smith, N.A.: Show your work: Improved reporting of experimental results. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Ho...

2019

[4] [4]

Show Your Work: Improved Reporting of Experimental Results

pp. 2185–2194. Association for Computational Linguistics (2019).https: //doi.org/10.18653/V1/D19-1224,https://doi.org/10.18653/v1/D19-1224 12 Authors Suppressed Due to Excessive Length

work page doi:10.18653/v1/d19-1224 2019

[5] [5]

The hitchhiker’s guide to test- ing statistical significance in natural language processing

Dror, R., Baumer, G., Shlomov, S., Reichart, R.: The hitchhiker’s guide to test- ing statistical significance in natural language processing. In: Gurevych, I., Miyao, Y. (eds.) Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Vol- ume 1: Long Papers. pp. 1383–1392....

work page doi:10.18653/v1/p18-1128 2018

[6] [6]

In: Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho, K., Oh, A

Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language mod- els are zero-shot reasoners. In: Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho, K., Oh, A. (eds.) Advances in Neural Information Process- ing Systems 35: Annual Conference on Neural Information Processing Sys- tems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28...

2022

[7] [7]

Microsoft: Phi-3-mini-4k-instruct.https://huggingface.co/microsoft/ Phi-3-mini-4k-instruct(2024), accessed: 2026-03-23

2024

[8] [8]

5-7B-Instruct(2024), accessed: 2026-03-23

Qwen Team, A.G.: Qwen2.5-7b-instruct.https://huggingface.co/Qwen/Qwen2. 5-7B-Instruct(2024), accessed: 2026-03-23

2024

[9] [9]

In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp

Reimers, N., Gurevych, I.: Reporting score distributions makes a difference: Per- formance study of lstm-networks for sequence tagging. In: Palmer, M., Hwa, R., Riedel, S. (eds.) Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017. pp. 338–348. Association for Comput...

work page doi:10.18653/v1/d17-1035 2017

[10] [10]

CoRRabs/2412.20061(2024).https://doi.org/10.48550/ ARXIV.2412.20061,https://doi.org/10.48550/arXiv.2412.20061

Shen, Y., Wang, L., Shi, C., Du, S., Tao, Y., Shen, Y., Zhang, H.: Compara- tive analysis of listwise reranking with large language models in limited-resource language contexts. CoRRabs/2412.20061(2024).https://doi.org/10.48550/ ARXIV.2412.20061,https://doi.org/10.48550/arXiv.2412.20061

work page doi:10.48550/arxiv.2412.20061 2024

[11] [11]

In: Croft, W.B., Moffat, A., van Rijsbergen, C.J., Wilkin- son, R., Zobel, J

Voorhees, E.M.: Variations in relevance judgments and the measurement of re- trieval effectiveness. In: Croft, W.B., Moffat, A., van Rijsbergen, C.J., Wilkin- son, R., Zobel, J. (eds.) SIGIR ’98: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 24-28 1998, Melbourne, Australia. ...

work page doi:10.1145/290941.291017 1998

[12] [12]

CoRRabs/2502.15770 (2025).https://doi.org/10.48550/ARXIV.2502.15770,https://doi.org/10

Wang, L., Shi, C., Du, S., Tao, Y., Shen, Y., Zheng, H., Shen, Y., Qiu, X.: Per- formance review on LLM for solving leetcode problems. CoRRabs/2502.15770 (2025).https://doi.org/10.48550/ARXIV.2502.15770,https://doi.org/10. 48550/arXiv.2502.15770

work page doi:10.48550/arxiv.2502.15770 2025

[13] [13]

In: Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho, K., Oh, A

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E.H., Le, Q.V., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models. In: Koyejo, S., Mohamed, S., Agarwal, A., Bel- grave, D., Cho, K., Oh, A. (eds.) Advances in Neural Information Process- ing Systems 35: Annual Conference on Neural Information Processing ...

2022