Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Benchmarking Prompt Sensitivity in Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Existing methods fail to predict whether a rephrased prompt will be answered correctly by an LLM.

desk verdict PromptSET is a useful new benchmark for prompt answerability, but the unspecified answer-matching and filtering make the gold labels unverifiable. read the letter →

arxiv 2502.06065 v1 pith:3JROBPOH submitted 2025-02-09 cs.CL cs.AIcs.IR

Amirhossein Razavi , Mina Soltangheis , Negar Arabzadeh , Sara Salamat , Morteza Zihayat , Ebrahim Bagheri This is my paper · ORCID
classification cs.CLcs.AIcs.IR
keywords promptsensitivitypredictionSETbenchmarkdatasetlargelanguagemodelsqueryperformancetextclassificationreformulation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a new task, Prompt Sensitivity Prediction: given a question and its rephrased variations that preserve the same information need, predict whether a given LLM will answer the variation correctly. The authors build PromptSET from TriviaQA and HotpotQA, generating nine variations per prompt with LLaMA3.1 and Mistral-nemo, filtering for semantic similarity, and labeling each prompt by matching the LLM's answer to the gold answer. They then benchmark methods from three families: LLM self-evaluation, text classification, and query performance prediction. Their central finding is that none of these existing approaches reliably predicts prompt answerability, with the strongest supervised baselines reaching roughly 0.66 to 0.71 accuracy on TriviaQA while performing much worse on HotpotQA. If this holds, it means users currently have no dependable way to know beforehand whether a reformulated prompt will get a correct answer, highlighting the need for a task-specific predictor.

What carries the argument

The load-bearing artefact is PromptSET, built by generating nine variations of each source question with two open LLMs, filtering variations with a cosine-similarity threshold and an information-need-preservation check, then labeling each prompt as answerable or not by comparing the LLM's response with the gold answer from TriviaQA or HotpotQA. The benchmark baselines span three families: LLM self-evaluation (asking the model to judge its own answerability), text classification (BERT and others trained on the training split), and query performance prediction (including the supervised BERT-PE and unsupervised specificity metrics such as Closeness Centrality, Degree Centrality, PageRank, and Inverse Edge Frequency). The specificity metrics perform worst, which the authors attribute to the near-identical specificity levels of original prompts and their variations.

What would settle it

A human annotation study of a random sample of PromptSET variations: if a non-trivial share of variations are judged to change the information need of the original question, or if the answer-matching accepts responses that do not actually answer the question, then the labels do not measure prompt sensitivity. A second concrete check: re-running the baseline evaluation with a stricter information-need filter or with LLM-judged answer equivalence should materially change accuracy if the current labels are the reason baselines fail.

Watch

Extended reading notes

Core claim

The central claim is that prompt answerability—whether a reformulated version of a question will be correctly answered by an LLM—is a distinct prediction task that existing methods from adjacent tasks do not solve. The paper supports this by constructing PromptSET, a dataset of 11,469 prompts each with nine LLM-generated variations, and showing that LLM self-evaluation, text classifiers, and query performance prediction baselines all fail to achieve reliable accuracy across the two source datasets. The strongest baselines, BERT-PE and a BERT text classifier, reach around 0.66 to 0.71 accuracy on TriviaQA but drop substantially on HotpotQA, while the LLMs' self-assessments are inconsistent. The paper also reports that when both LLMs fail on the original prompt, at least one of the two can often answer a rephrased variation correctly, suggesting that prompt reformulation itself is a viable strategy whose outcomes the baselines cannot forecast.

Load-bearing premise

The dataset labels are only as trustworthy as the filtering that keeps each of the nine variations semantically aligned with the original question and the answer-matching procedure that decides whether the LLM actually fulfilled the information need; if variations drift in meaning or matching is too lax, the benchmark measures paraphrase artifacts rather than prompt sensitivity.

Editorial extensions

If this is right

  • Prompt sensitivity prediction is a real, measurable task: there exists a dataset where answerability is not trivially predictable from prompt text alone.
  • Users of LLMs cannot rely on the model itself, a text classifier, or retrieval-based difficulty estimators to tell them whether a reformulation will succeed.
  • Supervised methods such as BERT-PE and BERT text classification are the strongest current baselines, indicating that learning-based approaches are the right direction even if current ones fall short.
  • Prompt reformulation can rescue unanswerable prompts: for many cases where both evaluated LLMs failed on the original question, at least one LLM answered a variation correctly, so PromptSET doubles as a resource for reformulation research.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A dedicated supervised model trained directly on PromptSET, rather than a method borrowed from query performance prediction or text classification, would likely beat the reported baselines; the paper does not train such a model.
  • Because the answerability labels rely on exact or substring matching to gold answers, a model trained on these labels may be learning quirks of answer-form matching rather than genuine semantic fulfillment; testing with LLM-based answer equivalence would clarify this.
  • The similarity-based analysis suggests LLMs generalize better to formulations close to ones seen in training; a testable extension is to measure answerability against paraphrase distance from training data across many models to see if this bias is systematic.
  • The task could serve as a preprocessing step in prompt optimization pipelines: a reliable predictor would let systems search over reformulations and pick the one most likely to be answered correctly without calling the expensive LLM.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces a new task, Prompt Sensitivity Prediction, and a dataset called PromptSET built from TriviaQA and HotpotQA. For each of 11,469 questions, the authors generate nine paraphrase variations using LLaMA3.1-8B and Mistral, ask the same models to answer each prompt and variation, and label each prompt as answerable or not by comparing the generated answer with the reference answer. They then benchmark several baselines: LLM self-evaluation, text classification (BERT), and query performance prediction methods (BERT-PE, CC, DC, IEF, PageRank). The central claim is that existing methods do not reliably predict prompt answerability, and the paper reports accuracy and F1 results in Table 2. The paper also includes exploratory analyses of the relationship between original-prompt correctness, similarity to the original, and answerability of variations.

Significance. If the PromptSET labels are valid, the task and dataset are a useful contribution: the benchmark covers a practical problem, the baselines are drawn from related areas, and the dataset is publicly released. The main result, that most existing methods perform only modestly, is plausible and could motivate new work on prompt sensitivity prediction. However, the paper's significance is currently limited by under-specification of the label-generation procedure, the absence of human validation, and the lack of statistical rigor; these issues make the dataset's construct validity and the strength of the empirical claims difficult to assess. The release of the dataset and the reproducible baseline setup are explicit strengths.

major comments (4)
  1. [Section 2, 'LLM Response Generation'] The paper states that 'we compared the produced answer against the expected answer in the TriviaQA and HotpotQA datasets' but never specifies the matching procedure. Exact string match, token-level F1, or an LLM-based answer-equivalence check would produce very different labels. If matching is strict, semantically correct answers with different surface forms (synonyms, reordered entities, longer explanations) are labeled as failures, which would make the benchmark measure surface-form robustness rather than prompt sensitivity. Please specify the matching procedure, include examples of matched and mismatched pairs, and report a human-evaluation or label-noise analysis on a sample.
  2. [Section 2, 'Generating Prompt Variations'] The filtering criteria for variations are not operationalized. The similarity threshold tau in the definition P = {p' | Sim(p,p') > tau and I_p == I_p'} is never given, and the procedure for checking that a variation preserves the information need I_p is not described. The only concrete filter stated is 'at least nine valid variations generated.' These unspecified parameters determine which prompts enter the dataset and what counts as a 'slight' variation; without them the dataset cannot be reproduced and the construct validity of the benchmark is unclear.
  3. [Table 2 and Section 3, 'Baseline Performance'] The abstract's claim that existing methods 'struggle to effectively address prompt sensitivity prediction' is only partially supported by the results. BERTPE reaches 0.710 accuracy on PromptSET-HotpotQA and BERT reaches 0.664 on PromptSET-TriviaQA, which are not negligible. The narrative should be qualified to acknowledge that supervised QPP and BERT classification perform reasonably on one dataset but are unstable across datasets. Additionally, no error bars, confidence intervals, significance tests, or multiple-seed results are reported, so it is unclear whether the observed differences between methods are meaningful.
  4. [Section 2, 'Establishing Baselines'] For the QPP baselines, the paper says 'we convert them to binary by classifying values above and below the mean of the data.' If the mean is computed on the test set, this is a transductive setup that may inflate reported performance; if the mean is computed on the training set, this should be stated explicitly. Please clarify which data are used to derive the threshold and, if test-set thresholding was used, re-evaluate with a training-derived threshold.
minor comments (5)
  1. [Abstract] There is a capitalization typo: 'Large language Models' should be 'Large language models'.
  2. [Section 2, 'Generating Prompt Variations'] The text says 'resulting in 114,690K variations,' but the arithmetic appears to be 11,469 prompts times 10 prompts per group (original plus nine variations) equals 114,690 total prompts, or 103,221 variations if only the nine variations are counted.
  3. [Throughout] The dataset name is spelled inconsistently as 'HotpotQA' and 'HotPotQA'; please unify.
  4. [Section 3, first paragraph] The phrase 'We analyzetheperformanceoftheofbaselines' contains a typo ('the of'); it should read 'We analyze the performance of the baselines.'
  5. [References] Reference [20] is cited for 'Mistral-nemo' but the paper title is 'Mistral 7B'; please verify the correct model and citation, as Mistral-Nemo is a different model.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark labels are grounded in external gold answers and the evaluated baselines are tested on a held-out split.

full rationale

The paper's derivation chain is the construction of PromptSET and the evaluation of baselines. Gold labels are produced by running LLaMA and Mistral on original prompts and variations and comparing the generated answers against the human-annotated accepted answers in TriviaQA and HotpotQA; this is an external ground truth, not a quantity defined by the model being tested. The variations are generated by LLMs, and the same LLMs' outputs define labels for their own answerability, but this is a construct-validity concern about the dataset rather than a circular reduction: the label for a prompt is not stipulated to equal any baseline's prediction. Baselines (self-evaluation, text classification, QPP) are evaluated on a held-out 30% split after any training on the 70% split, so their predictions are not forced by construction. The paper's extensive self-citations are to prior QPP metrics used as baselines or as methodological inspiration; they do not justify the central claim that existing methods struggle, and the claim is supported by Table 2 on held-out data. The underspecified answer-matching and similarity-threshold filtering affect label validity and reproducibility, but they do not make the evaluation circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim is an empirical benchmark result, so the ledger is dominated by design choices and domain assumptions rather than fitted mathematical parameters. The hand-chosen values, including N=9, word-length bounds, similarity threshold, and minimum valid variations, shape the dataset and could affect all downstream scores. The labels themselves rely on assumptions about semantic preservation, answer matching, and the determinism of reference answers.

free parameters (4)
  • Number of variations per prompt (N) = 9
    Hand-selected in Section 2; determines the size of PromptSET and the granularity of the sensitivity distribution.
  • Similarity threshold tau
    Referenced in the task definition and filtering step but never given a numeric value; it controls which rephrasings are accepted.
  • Word-length bounds = 4 to 40 words
    Hand-chosen inclusion filter in Section 2 that removes short and long questions before variation generation.
  • Minimum valid variations per prompt = 9
    Post-hoc filter that drops prompts with fewer than nine valid rephrasings; the count and criteria are not reported.
assumptions (4)
  • domain assumption Generated variations that pass the similarity and alignment filters preserve the original information need.
    Defines the set P in Section 2 and is the basis for treating variation failure as prompt sensitivity.
  • domain assumption Exact or normalized matching of the generated answer to the reference answer set is a valid correctness measure.
    Labels are assigned by comparing the generated answer with the expected answer, but the matching rule is not specified.
  • domain assumption TriviaQA and HotpotQA answer sets are complete enough for reliable binary labels.
    The authors selected these datasets because they have deterministic and concise answers; open-generation answers may be correct without appearing in the reference set.
  • domain assumption Cosine similarity in MiniLM embedding space is an adequate proxy for semantic similarity between prompt variations.
    Used in Figure 1 to measure Sim(p,p') and to interpret the effect of similarity on answerability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Prompt Sensitivity in Large Language Models." pith.science (2026). https://pith.science/paper/3JROBPOH

@misc{pith2026250206065,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Prompt Sensitivity in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3JROBPOH}},
  note         = {Machine review of arXiv:2502.06065}
}
read the original abstract

Large language Models (LLMs) are highly sensitive to variations in prompt formulation, which can significantly impact their ability to generate accurate responses. In this paper, we introduce a new task, Prompt Sensitivity Prediction, and a dataset PromptSET designed to investigate the effects of slight prompt variations on LLM performance. Using TriviaQA and HotpotQA datasets as the foundation of our work, we generate prompt variations and evaluate their effectiveness across multiple LLMs. We benchmark the prompt sensitivity prediction task employing state-of-the-art methods from related tasks, including LLM-based self-evaluation, text classification, and query performance prediction techniques. Our findings reveal that existing methods struggle to effectively address prompt sensitivity prediction, underscoring the need to understand how information needs should be phrased for accurate LLM responses.

Figures

Figures reproduced from arXiv: 2502.06065 by the authors.

Figure 1
Figure 1. (a) A histogram of correctly answered variations when the original prompt yields [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Distribution of answerability of variations of the questions in [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 21 canonical work pages

  1. [1]

    Arabzadeh,N., Bigdeli, A., Zihayat, M., Bagheri, E.: Queryperformance prediction through retrievalcoherency.In:AdvancesinInformationRetrieval:43rdEuropeanConferenceonIR Research, ECIR 2021, Virtual Event, March 28–April 1, 2021, Proceedings, Part II 43. pp. 193–200. Springer (2021)

    work page 2021

  2. [2]

    Arabzadeh, N., Hamidi Rad, R., Khodabakhsh, M., Bagheri, E.: Noisy perturbations for estimatingquerydifficultyindenseretrievers.In:Proceedingsofthe32ndACMInternational Conference on Information and Knowledge Management. pp. 3722–3727 (2023)

    work page 2023

  3. [3]

    In: Al- Onaizan,Y.,Bansal,M.,Chen,Y.N.(eds.)Proceedingsofthe2024ConferenceonEmpirical Methods in Natural Language Processing

    Arabzadeh, N., Huo, S., Mehta, N., Wu, Q., Wang, C., Awadallah, A.H., Clarke, C.L.A., Kiseleva, J.: Assessing and verifying task utility in LLM-powered applications. In: Al- Onaizan,Y.,Bansal,M.,Chen,Y.N.(eds.)Proceedingsofthe2024ConferenceonEmpirical Methods in Natural Language Processing. pp. 21868–21888. Association for Computational Linguistics, Miami...

    doi:10.18653/v1/2024 2024

  4. [4]

    In: European Conference on Information Retrieval

    Arabzadeh,N.,Meng,C.,Aliannejadi,M.,Bagheri,E.:Queryperformanceprediction:From fundamentals to advanced techniques. In: European Conference on Information Retrieval. pp. 381–388. Springer (2024)

    work page 2024

  5. [5]

    In: Proceedings of the 2024 AnnualInternationalACMSIGIRConferenceonResearchandDevelopmentinInformation Retrieval in the Asia Pacific Region

    Arabzadeh, N., Meng, C., Aliannejadi, M., Bagheri, E.: Query performance prediction: Techniques and applications in modern information retrieval. In: Proceedings of the 2024 AnnualInternationalACMSIGIRConferenceonResearchandDevelopmentinInformation Retrieval in the Asia Pacific Region. pp. 291–294 (2024)

    work page 2024

  6. [6]

    In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management

    Arabzadeh, N., Seifikar, M., Clarke, C.L.: Unsupervised question clarity prediction through retrieved item coherency. In: Proceedings of the 31st ACM International Conference on Information & Knowledge Management. pp. 3811–3816 (2022)

    work page 2022

  7. [7]

    Arabzadeh,N.,Zarrinkalam,F.,Jovanovic,J.,Al-Obeidat,F.,Bagheri,E.:Neuralembedding- basedspecificitymetricsforpre-retrievalqueryperformanceprediction.InformationProcess- ing & Management57(4), 102248 (2020)

    work page 2020

  8. [8]

    Arabzadeh,N.,Zarrinkalam,F.,Jovanovic,J.,Bagheri,E.:NeuralEmbedding-BasedMetrics for Pre-retrieval Query Performance Prediction, p. 78–85. Springer International Publishing (2020). https://doi.org/10.1007/978-3-030-45442-5_10, http://dx.doi.org/ 10.1007/978-3-030-45442-5_10

    work page doi:10.1007/978-3-030-45442-5_10 2020

Show all 41 references
  1. [9]

    In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management

    Arabzadeh, N., Zarrinkalam, F., Jovanovic, J., Bagheri, E.: Geometric estimation of specificity within embedding spaces. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management. p. 2109–2112. CIKM ’19, ACM (Nov 2019). https://doi.org/10...

    2019

  2. [10]

    ArXivabs/2310.04444 (2023)

    Bhargava, A., Witkowski, C., Shah, M., Thomson, M.W.: What’s the magic word? a control theory of llm prompting. ArXivabs/2310.04444 (2023). https://doi.org/10.48550/ arXiv.2310.04444

    2023 doi arXiv

  3. [11]

    In: European Conference on Information Retrieval

    Bigdeli, A., Arabzadeh, N., Bagheri, E.: Learning to jointly transform and rank difficult queries. In: European Conference on Information Retrieval. pp. 40–48. Springer (2024)

    2024

  4. [12]

    Collins-Thompson, K., Bennett, P.N.: Predicting Query Performance via Classifica- tion, p. 140–152. Springer Berlin Heidelberg (2010). https://doi.org/10.1007/ 978-3-642-12275-0_15, http://dx.doi.org/10.1007/978-3-642-12275-0_15

    2010 doi

  5. [13]

    In: European Conference on Information Retrieval

    Ebrahimi, S., Khodabakhsh, M., Arabzadeh, N., Bagheri, E.: Estimating query perfor- mance through rich contextualized query representations. In: European Conference on Information Retrieval. pp. 49–58. Springer (03 2024). https://doi.org/10.1007/ 978-3-031-56066-8_6

    2024

  6. [14]

    In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Informa- tion Retrieval

    Faggioli, G., Ferro, N., Muntean, C.I., Perego, R., Tonellotto, N.: A geometric frame- work for query performance prediction in conversational search. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Informa- tion Retrieval. p. 1355...

    2023

  7. [15]

    ArXivabs/2405.11891(2024), https://api.semanticscholar

    Feng,Z.,Zhou,H.,Zhu,Z.,Qian,J.,Mao,K.:Unveilingandmanipulatingpromptinfluencein large language models. ArXivabs/2405.11891(2024), https://api.semanticscholar. org/CorpusID:269922034

    2024 arXiv

  8. [16]

    Information13(2), 83 (2022)

    Gasparetto, A., Marcuzzo, M., Zangari, A., Albarelli, A.: A survey on text classification algorithms: From text to predictions. Information13(2), 83 (2022)

    2022

  9. [17]

    IEEE Access11, 76581–76604 (2023)

    Hambarde, K.A., Proença, H.: Information retrieval: Recent advances and beyond. IEEE Access11, 76581–76604 (2023)

    2023

  10. [18]

    In: CIKM (2008)

    Hauff, C., Hiemstra, D., de Jong, F.: A survey of pre-retrieval query performance predictors. In: CIKM (2008)

    2008

  11. [19]

    In: Proceedings of the 33rd ACM International Confer- ence on Information and Knowledge Management

    Hosseini, S.M., Arabzadeh, N., Zihayat, M., Bagheri, E.: Enhanced retrieval effectiveness through selective query generation. In: Proceedings of the 33rd ACM International Confer- ence on Information and Knowledge Management. pp. 3792–3796 (2024)

    2024

  12. [20]

    Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L.R., Lachaux, M.A., Stock, P., Scao, T.L., Lavril, T., Wang, T., Lacroix, T., Sayed, W.E.: Mistral 7b (2023),https: //arxiv.or...

    2023 arXiv

  13. [21]

    In: Barzilay, R., Kan, M.Y

    Joshi, M., Choi, E., Weld, D., Zettlemoyer, L.: TriviaQA: A large scale distantly super- vised challenge dataset for reading comprehension. In: Barzilay, R., Kan, M.Y. (eds.) Proceedings of the 55th Annual Meeting of the Association for Computational Linguis- tics (Volume 1: L...

    doi

  14. [22]

    In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Kamalloo, E., Dziri, N., Clarke, C., Rafiei, D.: Evaluating open-domain question answering in the era of large language models. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Com- putational ...

    2023 doi

  15. [23]

    Khodabakhsh, M., Zarrinkalam, F., Arabzadeh, N.: BertPE: A BERT-Based Pre-retrieval Estimator for Query Performance Prediction, p. 354–363. Springer Nature Switzerland (2024). https://doi.org/10.1007/978-3-031-56063-7_27, http://dx.doi.org/ 10.1007/978-3-031-56063-7_27

    2024 doi

  16. [24]

    Internet Reference Services Quarterly27, 203 – 210 (2023).https://doi.org/10.1080/ 10875301.2023.2227621

    Lo, L.S.: The art and science of prompt engineering: A new literacy in the information age. Internet Reference Services Quarterly27, 203 – 210 (2023).https://doi.org/10.1080/ 10875301.2023.2227621

    2023

  17. [25]

    https://doi.org/10.18653/v1/2023.findings-emnlp.241, http: //dx.doi.org/10.18653/v1/2023.findings-emnlp.241

    Loya, M., Sinha, D., Futrell, R.: Exploring the sensitivity of llms’ decision-making capabili- ties:Insightsfrompromptvariationsandhyperparameters.In:FindingsoftheAssociationfor ComputationalLinguistics:EMNLP2023.p.3711–3716.AssociationforComputationalLin- guistics (2023). htt...

    2023 doi

  18. [26]

    Meng, C., Arabzadeh, N., Aliannejadi, M., de Rijke, M.: Query performance prediction: Fromad-hoctoconversationalsearch.In:Proceedingsofthe46thInternationalACMSIGIR Conference on Research and Development in Information Retrieval. p. 2583–2593. SIGIR ’23,ACM(Jul2023). https://doi...

  19. [27]

    arXiv preprint arXiv:2404.01012 (2024)

    Meng, C., Arabzadeh, N., Askari, A., Aliannejadi, M., de Rijke, M.: Query performance prediction using relevance judgments generated by large language models. arXiv preprint arXiv:2404.01012 (2024)

    2024 arXiv

  20. [28]

    arXiv preprint arXiv:2407.21783 (2024)

    Meta, L.T.A..: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  21. [29]

    ArXivabs/2305.14310 (2023)

    Mu,Y.,Wu,B.,Thorne,W.,Robinson,A.,Aletras,N.,Scarton,C.,Bontcheva,K.,Song,X.: Navigatingpromptcomplexityforzero-shotclassification:Astudyoflargelanguagemodels in computational social science. ArXivabs/2305.14310 (2023). https://doi.org/10. 48550/arXiv.2305.14310

    2023 doi arXiv

  22. [30]

    ArXivabs/2311.07599 (2023)

    Murr, L., Grainger, M., Gao, D.: Testing llms on code generation with varying levels of prompt specificity. ArXivabs/2311.07599 (2023). https://doi.org/10.48550/arXiv. 2311.07599

    2023 doi arXiv

  23. [31]

    Poesina, E., Costache, A.V., Chifu, A.G., Mothe, J., Ionescu, R.T.: Pqpp: A joint benchmark for text-to-image prompt and query performance prediction (2024),https://arxiv.org/ abs/2406.04746

    2024 arXiv

  24. [32]

    ArXivabs/2308.09138 (2023)

    Raj, H., Gupta, V., Rosati, D., Majumdar, S.: Semantic consistency for assuring reliability of large language models. ArXivabs/2308.09138 (2023). https://doi.org/10.48550/ arXiv.2308.09138

    2023 doi arXiv

  25. [33]

    In: Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region

    Rajapakse, T.C., Yates, A., de Rijke, M.: Simple transformers: Open-source for all. In: Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region. p. 209–215. SIGIR-AP 2024, Association for ...

    2024

  26. [34]

    In: Proceedings of the 32nd ACM Inter- national Conference on Information and Knowledge Management

    Salamat, S., Arabzadeh, N., Seyedsalehi, S., Bigdeli, A., Zihayat, M., Bagheri, E.: Neural disentanglement of query difficulty and semantics. In: Proceedings of the 32nd ACM Inter- national Conference on Information and Knowledge Management. pp. 4264–4268 (2023)

    2023

  27. [35]

    In: European Conference on Information Re- trieval.pp.30–39.Springer(2024)

    Saleminezhad, A., Arabzadeh, N., Beheshti, S., Bagheri, E.: Context-aware query term dif- ficulty estimation for performance prediction. In: European Conference on Information Re- trieval.pp.30–39.Springer(2024). https://doi.org/10.1007/978-3-031-56066-8_4

    2024 doi

  28. [36]

    arXiv preprint arXiv:2310.11324 (2023)

    Sclar, M., Choi, Y., Tsvetkov, Y., Suhr, A.: Quantifying language models’ sensitivity to spu- rious featuresin promptdesign or:How ilearned to startworrying aboutprompt formatting. arXiv preprint arXiv:2310.11324 (2023)

    2023 arXiv

  29. [37]

    In: Proceedings of the 34th International Conference on Neural Information Processing Systems

    Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: Minilm: deep self-attention distillation for task-agnostic compression of pre-trained transformers. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. NIPS ’20, Curran Associa...

    2020

  30. [38]

    eugeneyan.com (Aug 2024),https://eugeneyan.com/writing/llm-evaluators/

    Yan, Z.: Evaluating the effectiveness of llm-evaluators (aka llm-as-judge). eugeneyan.com (Aug 2024),https://eugeneyan.com/writing/llm-evaluators/

    2024

  31. [39]

    Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., Manning, C.D.: Hot- potQA:Adatasetfordiverse,explainablemulti-hopquestionanswering.In:Riloff,E.,Chiang, D.,Hockenmaier,J.,Tsujii,J.(eds.)Proceedingsofthe2018ConferenceonEmpiricalMeth- ods in Natural Languag...

    doi

  32. [40]

    ArXivabs/2306.04528(2023)

    Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Gong, N., Zhang, Y., Xie, X.: Promptbench: Towards evaluating the robustness of large language mod- els on adversarial prompts. ArXivabs/2306.04528(2023). https://doi.org/10.48550/ arXiv.2306.04528

    2023 doi arXiv

  33. [41]

    Zhuo,J.,Zhang,S.,Fang,X.,Duan,H.,Lin,D.,Chen,K.:Prosa:Assessingandunderstanding the prompt sensitivity of llms (2024),https://arxiv.org/abs/2410.12405

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.