Pith. sign in

REVIEW 3 major objections 6 minor 65 references

Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training an inverse model to reconstruct a target LLM's training instructions from its outputs enables one-shot generation of evaluation prompts that outperform human-crafted ones.

desk verdict A genuinely new idea—inversion learning for model-specific evaluation prompts—with broad experiments, but the 'single sample suffices' claim rests on single point estimates with no variance or significance testing. read the letter →

arxiv 2504.21117 v3 pith:QXBMCSVW submitted 2025-04-29 cs.CL

Hanhua Hong , Chenghao Xiao , Yang Wang , Yiqi Liu , Wenge Rong , Chenghua Lin This is my paper · ORCID
classification cs.CL
keywords LLMevaluationpromptgenerationinversionlearningmodel-specificpromptsone-shotNLGoptimizationLLM-as-a-judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish a one-shot generative route to prompt engineering for LLM-based NLG evaluation: rather than hand-crafting or searching over prompts, an 'inverse model' is trained to map a target LLM's outputs back to the instructions that produced them, and then given a single (content, human-score) example from a new evaluation task, it writes an evaluation prompt for that task. On summarisation, machine translation, and dialogue benchmarks, these inverse prompts achieve higher Spearman and Pearson correlations with human judgments than human-crafted prompts, prompts written by the forward instruction-tuned model itself, and prompts found by HPSS, a search-based optimiser that needs up to 100 validation samples. The paper argues that LLMs from different families have different interpretive biases, so evaluation prompts should be model-specific rather than one-size-fits-all, and that inversion captures the prompt style each model responds to best. If correct, this replaces manual or data-hungry prompt optimisation for evaluation with a single-sample generative step.

What carries the argument

The central mechanism is inversion learning: a base language model is fine-tuned to reverse the input–output relationship of instruction tuning, so it can reconstruct an instruction from a model-generated response. Because the target model's outputs are distilled first (black-box) or generated from the same SFT data (white-box), the inverse model learns the specific instruction style that the target LLM follows. At prompting time, a single (content, human-score) sample is embedded in a meta-prompt; the inverse model's output becomes a task-specific evaluation prompt, whose content slot is then replaced with a placeholder to obtain a reusable template for scoring arbitrary items with the forward model.

What would settle it

Replace the human score in the one-shot meta-prompt with a random number (or a deliberately wrong value) and regenerate the evaluation prompt; if the resulting prompt still produces the same correlations with human judgments, then the inverse model is not conditioning on the evaluation outcome, and the claimed one-shot mechanism would be overturned. A complementary check: train the inverse model on an instruction dataset containing no evaluation-related tasks and measure whether its prompts still beat forward prompts, which would indicate the gains come from instruction style rather than evaluation understanding.

Watch

Extended reading notes

Core claim

An inverse mapping learned from instruction-tuning data transfers to evaluation prompt generation. In the black-box setting, the authors distill outputs from the target instruct model on a large instruction dataset, then fine-tune a base model on the reversed pairs (output → input), so the inverse model internalises the structure of instructions the target model follows; in the white-box setting, forward and inverse models are trained from the same SFT data by swapping input–output pairs. At inference, the inverse model receives a meta-prompt containing the text to be evaluated, a human score for one sample, and an elicitation instruction, and emits an evaluation prompt; the task-specific content is then replaced by a placeholder to form a general template. Evaluated with the forward model as the judge, these templates outperform human-crafted, forward-model, and HPSS-optimised prompts in both settings across LLaMA-3.1-8B and Qwen-2.5 (3B–14B), with the largest relative gains on abstractive summarisation and with cross-model prompt swapping causing clear performance drops.

Load-bearing premise

The load-bearing premise is that the inverse model's ability to reconstruct the target LLM's own training instructions from its outputs transfers to the ability to generate effective evaluation prompts for entirely new tasks from a single labeled example; the paper asserts this transfer in Section 3.2 rather than isolating it with a control, and it is not supported by an analysis of what the inverse model has actually learned about evaluation quality.

Editorial extensions

If this is right

  • If inverse-generated prompts consistently beat hand-crafted ones, existing evaluator prompts such as G-EVAL and GEMBA are not optimal for a given judge model, and benchmark evaluations should report and use model-specific prompts.
  • Because only a single annotated sample is needed, evaluation becomes practical for new tasks or domains where labelled data is scarce, in contrast to search-based prompt optimisation that requires validation sets of 50–100 items.
  • Prompt swapping across model families degrades evaluator agreement, so evaluation results obtained with different judge models or different prompts are not directly comparable without controlling for prompt–model fit.
  • The gains persist when scaling the inverse model up to 14B parameters and transfer partially to a proprietary judge (GPT-4o-mini), suggesting the effect is not an artifact of one small model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never isolates what the inverse model learns about evaluation quality as opposed to instruction style; a direct test would be to compare prompts generated from a correct versus an incorrect one-shot score, and our prior is that much of the gain comes from matching the judge model's formatting and role-assignment style.
  • A testable extension is cross-architecture inversion: the black-box setting uses an inverse model of the same family as the judge; a cross-family inverse model (e.g., trained on Qwen outputs but used with a LLaMA judge) would reveal whether the learned mapping generalises beyond the model family.
  • If the transfer is real, the same inverse model could generate prompts for other prompt-sensitive applications beyond evaluation, such as instruction-conditioned text generation or refusal-boundary probing; researchers should verify this instead of assuming evaluation-specific learning.
  • Since exact score ranges and decimals matter little (≤5% drop when removed), practitioners could feed coarsely rounded or even synthetic scores, making the one-shot example cheaper to obtain; the paper shows robustness to rounding but does not test fully synthetic scores.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an inversion learning framework for automatically generating model-specific evaluation prompts for LLM-based NLG evaluation. In the inversion modeling stage, an inverse model is trained to reconstruct input instructions from outputs generated by the target instruction-tuned LLM, either in a black-box setting (using distilled outputs from an off-the-shelf model) or white-box setting (using the same SFT dataset for both forward and inverse fine-tuning). In the inversion prompting stage, the inverse model receives a one-shot meta-prompt containing a single evaluation content example and its human judgment, and generates a prompt template that is then used to evaluate the full test set. Experiments on summarization (SummEval, QAGS), conversational response generation (Topical-Chat), and machine translation (WMT-22) with LLaMA-3.1-8B and Qwen-2.5 (3B/7B/14B) report consistent improvements in Spearman and Pearson correlations over human-crafted prompts, forward-model prompts, and HPSS-optimized prompts. The paper also includes prompt-swapping, numerical-sensitivity, and model-scaling analyses.

Significance. If the results hold, the paper introduces a genuinely new generative paradigm for prompt engineering in LLM-based evaluation, reducing the need for manual prompt crafting or large validation sets. The idea of learning an inverse mapping from outputs to instructions and then using it to generate evaluation prompts from a single sample is novel and well-motivated. The experimental design covers two model families, several tasks, and both black-box and white-box settings, and the qualitative case study provides useful insight into why inverse prompts differ from human-crafted ones. The paper is also transparent about hyperparameters, enabling replication. However, the central quantitative claim is not yet statistically established: the headline improvements are point estimates without error bars, confidence intervals, or significance tests, and several comparisons to strong baselines are small. The robustness of the 'single evaluation sample' claim and the model-specificity claim therefore remains unverified.

major comments (3)
  1. [Section 4, Table 2] The central claim that a single randomly drawn one-shot sample suffices to generate a high-quality prompt is not statistically supported. The paper reports one point estimate per condition with no seeds, no repeated draws, no confidence intervals, and no significance tests. Several comparisons to the forward-prompt baseline are small: on Topical-Chat with LLaMA-3.1-8B-Instruct the inverse prompt achieves ρ=0.437 vs. forward ρ=0.419, and on WMT-22 the Pearson correlation is r=0.256 vs. forward r=0.248. The abstract promises that the method 'requires only a single evaluation sample,' which implies robustness to the choice of that sample, but this is not demonstrated. I request repeated sampling (e.g., 5-10 random one-shot examples per task and model) with means and standard deviations or confidence intervals, and a significance test (e.g., paired bootstrap over items or a permutation test) against the forward and human-crafted baselines. The same statistical treatment should be applied to the prompt-swapping results in Table 3, where the reported drops (e.g., 0.423 to 0.384 in average ρ for LLaMA with Qwen prompts) could plausibly be within noise.
  2. [Section 3.2] The transfer from learning to invert SFT instructions to generating effective evaluation prompts is asserted rather than demonstrated. The white-box comparison in Section 3.1.2 controls for the forward versus inverse training objective, showing that a model trained to invert outputs outperforms the same model trained forward, which supports the inversion mechanism. However, there is no control or analysis showing that the inverse model has learned anything specifically about evaluation quality; the gains could in principle arise from the model's instruction-following style or from the particular meta-prompt format. A direct control—for example, training the inverse model on the same data and testing whether generated prompts improve monotonically with the quality of inversion, or comparing against a non-evaluation meta-prompt that asks for a generic task instruction—would strengthen the causal claim that inversion learning, rather than incidental prompt style, is responsible for the improvements. As written, the empirical gains are consistent with the claim but do not isolate it.
  3. [Section 5.2, Table 5] The numerical sensitivity ablation raises a question about what the one-shot sample actually contributes. Removing the human score and score range from the meta-prompt changes average correlations by only about 5% (e.g., average ρ drops from 0.484 in the full setting to 0.469 in the 'w/o Score' condition). If the human judgment is nearly irrelevant, the method is better described as generating prompts from the evaluation content alone, which weakens the paper's framing that a single labeled evaluation sample is what enables the method. The authors should discuss this interpretation and ideally report a control where only the content c_t is provided as the one-shot example, to clarify the mechanism.
minor comments (6)
  1. [Table 2] In the Qwen-2.5-7B-WhiteBox row, Topical-Chat forward prompt ρ is shown as '0..419' with a double period; this should be '0.419'.
  2. [Section 1] The first sentence of the introduction contains a duplicated citation: '(Zhao et al., 2023, 2024) (Zhao et al., 2024)' should be cleaned up.
  3. [Section 4] The discussion of the failed output2prompt baseline (lines beginning 'We also attempt to benchmark...') is placed in the Baselines paragraph but reads like a limitations statement. Consider moving it to a dedicated limitations paragraph, and report the settings used when the baseline was attempted so that the failure is reproducible.
  4. [Figure 1] The caption says 'the bold text ... indicates substituting the specific example with a generic placeholder,' but the bold formatting is not visible in the rendered figure. Please make the substitution explicit in the figure itself or in a more detailed caption.
  5. [Section 5.1] The phrase 'outperforming the human-crafted and forward prompts over 13% and 35% on average, respectively' is ambiguous. State explicitly that these are relative gains over each baseline, and consider reporting absolute differences as well, since the relative gains on QAGS-XSUM exceed 100% due to a small forward baseline.
  6. [General] No code, data, or model checkpoints are provided or mentioned as available. Given the paper's emphasis on a new training paradigm, releasing the inversion-training code and the generated prompts would substantially help reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: inverse-model training, prompt generation, and human-correlation evaluation are independently grounded, with no fitted target or self-citation carrying the derivation.

full rationale

I examined the full derivation chain. The inverse model is trained by Eq. (2) (black-box) or Eq. (5) (white-box) to reconstruct input instructions x from model-generated outputs y or from swapped SFT pairs; no human evaluation scores, no benchmark labels, and no target correlation values enter this training. Evaluation prompts are produced in Eq. (7) from a single (content, human-score) pair, and the reported Spearman and Pearson correlations are computed by running the prompted evaluator in Eq. (8) against externally collected human judgments. None of these steps fits a parameter to the quantity being claimed as a prediction, and no test-label information is used to tune anything. The one-shot sample contains a human score, but the paper's own ablation (Table 5) shows that removing all numerical information changes performance only marginally, so that score is not the mechanism driving the result. The comparisons against forward prompts, human-crafted prompts, and HPSS are genuine external baselines. Self-citations in the paper (e.g., Liu et al. 2024 on self-preference, Loakman et al. 2023 on human evaluator variation, Zhao et al. 2023/2024) are background context and are not load-bearing for the central claim. The main risks are empirical rather than circular: the one-shot sample is randomly drawn without reported seeds or confidence intervals, and the transfer from instruction reconstruction to evaluation-prompt generation is asserted in Section 3.2 rather than isolated with a control. These are correctness or robustness concerns, not evidence that the derivation reduces to its own inputs. Therefore no circular step is identified and the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The framework rests on a small set of domain assumptions, mostly about transfer from instruction inversion to evaluation-prompt generation. No new physical or theoretical entities are introduced. The free parameters are the manually designed elicitation instruction and the uncontrolled random one-shot sample.

free parameters (2)
  • Response elicitation instruction E_T
    The hand-designed meta-prompt that tells the inverse model what kind of evaluation prompt to output; it is not ablated and may affect prompt quality.
  • One-shot example (c_t, r_t)
    A single content-score pair is randomly sampled from the target task's training set; no seed or variance is reported, so the exact sample is an uncontrolled parameter of the reported results.
assumptions (4)
  • domain assumption Outputs generated by MInstruct on Infinity-Instruct inputs, paired with the original instructions, are sufficient for training an inverse model that reconstructs a general output-to-instruction mapping.
    The black-box training objective in Eq. (2) assumes the distribution of distilled outputs from Infinity-Instruct inputs captures the latent correspondence that will be useful for evaluation-prompt generation on unrelated tasks.
  • ad hoc to paper The transfer from reconstructing SFT instructions to generating effective evaluation prompts from a meta-prompt with one annotated example is valid.
    Section 3.2 asserts this transfer without a control study; the empirical results, if not confounded, are the only evidence.
  • domain assumption Different LLM families have distinct interpretive biases, so model-specific prompts are needed.
    Motivated in Section 1 and used to justify the approach; the swapping experiment in Table 3 provides supporting but not decisive evidence.
  • standard math Standard supervised fine-tuning objectives and LoRA are assumed to train the inverse model correctly.
    Equations (2), (3), and (5) use standard SFT losses without formal verification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts." pith.science (2026). https://pith.science/paper/QXBMCSVW

@misc{pith2026250421117,
  author       = {Pith},
  title        = {Pith review of: Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QXBMCSVW}},
  note         = {Machine review of arXiv:2504.21117}
}
read the original abstract

Evaluating natural language generation systems is challenging due to the diversity of valid outputs. While human evaluation is the gold standard, it suffers from inconsistencies, lack of standardisation, and demographic biases, limiting reproducibility. LLM-based evaluators offer a scalable alternative but are highly sensitive to prompt design, where small variations can lead to significant discrepancies. In this work, we propose an inversion learning method that learns effective reverse mappings from model outputs back to their input instructions, enabling the automatic generation of highly effective, model-specific evaluation prompts. Our method requires only a single evaluation sample and eliminates the need for time-consuming manual prompt engineering, thereby improving both efficiency and robustness. Our work contributes toward a new direction for more robust and efficient LLM-based evaluation.

Figures

Figures reproduced from arXiv: 2504.21117 by the authors.

Figure 1
Figure 1. Illustration of the inverse prompt generation process. The bold text in "Prompt Template" [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Example of a meta-prompt for the inverse [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Numerical sensitivity ablation examples from QAGS-X: (a) the original meta-prompt for [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Prompts for Qwen on QAGS dataset. Green texts denote the [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Prompts for LLaMA on QAGS dataset [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Prompts for Qwen on Summeval dataset [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Prompts for Qwen on Topical-Chat dataset. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Prompts for Qwen on WMT22-EN-DE dataset. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Prompts for the numerical sensitivity study using Qwen. [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 59 canonical work pages

  1. [1]

    **Comprehensive Coverage**: Ensure the summary captures the main points and key details from the article without omit- ting crucial information

    work page

  2. [2]

    InFindings of the Association for Computational Linguistics: NAACL 2025, pages 8397–8437, Albuquerque, New Mexico

    Do LLMs have distinct and consistent per- sonality? TRAIT: Personality testset designed for LLMs with psychometrics. InFindings of the Association for Computational Linguistics: NAACL 2025, pages 8397–8437, Albuquerque, New Mexico. Association for Computational Linguistics. Christoph Leiter and Steffen Eger. 2024. PrExMe! large scale prompt exploration of...

    work page 2025

  3. [3]

    **Relevance**: Confirm that the sum- mary is relevant to the article’s topic and does not include irrelevant information

    work page

  4. [4]

    **Precision**: Check that the summary uses precise language that aligns with the article’s tone and style

    work page

  5. [5]

    ‘json {

    **Brevity**: Ensure the summary is concise and does not include unnecessary details or elongated phrases that detract from its clarity. By adhering to these criteria, summaries will be evaluated for their consistency with the original article. Now please evaluate the following sum- mary to the article based on the above guideline criteria: Article: {artic...

    work page

  6. [6]

    **Language Clarity**: The summary uses clear and concise language, avoiding ambiguity and jargon

    work page

  7. [7]

    InProceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers)

    Evaluating open-domain dialogues in la- tent space with next sentence prediction and mu- tual information. InProceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers). Kun Zhao, Bohao Yang, Chen Tang, Chenghua Lin, and Liang Zhan. 2024. SLIDE: A framework integrating small and large language models f...

    work page 2024

  8. [8]

    **Comprehensive Coverage**: The summary provides a comprehensive overview of the article’s main points, without omitting important details

    work page

Show all 65 references
  1. [9]

    **Accuracy**: Verify that the summary accurately reflects the content of the article, maintaining the correct facts and figures

  2. [10]

    ‘json {

    **Contextual Understanding**: The summary demonstrates an understanding of the article’s context, including any relevant background information or relationships between events. **Scoring** By following these criteria, models can sys- tematically assess the consistency of sum- ...

  3. [15]

    ‘json {

    Assign a score for consistency based on the Evaluation Criteria. Please evaluate the following summary: Source Text: {article} Summary: {summary} Please just directly output the consistency score in a json format. For example: “‘json { "article": "content of the article", "sum...

  4. [16]

    Examine each sentence in the summary in relation to the article’s content

  5. [17]

    Identify any factual inconsistencies, such as misrepresentations, contradictions, or omitted key details

  6. [18]

    ‘json {

    Assign a factual consistency score to the summary on a scale of 0 to 1, where 1 indicates perfect factual consistency and 0 indicates complete factual inconsistency. For your reference, here is a detailed evalu- ation guideline and format requirement: **Evaluation Guideline:**...

  7. [19]

    **Factual Accuracy**: The summary accurately reflects the main facts and events described in the article

  8. [20]

    **Main Idea Representation**: The summary effectively conveys the main idea or key takeaway from the article

  9. [21]

    **Relevance**: The summary only includes information that is relevant to the article’s main topic and events

  10. [22]

    **Omission of Irrelevant Details**: The summary does not include unnecessary or tangential details that are not essential to understanding the article’s main point

  11. [23]

    **Cohesion**: The summary is logically organized and easy to follow, with a clear structure and flow of ideas

  12. [25]

    **Accuracy of Quotes and State- ments**: If the summary includes quotes or statements from individuals, they are accurately represented and not taken out of context

  13. [27]

    **Avoidance of Misrepresentation**: The summary does not misrepresent or distort the information presented in the article

  14. [29]

    Read the news article carefully and iden- tify the main facts and details it presents

  15. [30]

    Check if the summary contains any factual errors that are not supported by the article

    Read the summary and compare it to the article. Check if the summary contains any factual errors that are not supported by the article

  16. [31]

    ‘json {

    Assign a score for consistency based on the Evaluation Criteria. Please evaluate the following summary: Source Text: {article} Summary: {summary} Please just directly output the consistency score in a json format. For example: “‘json { "article": "content of the article", "sum...

  17. [32]

    ‘json {

    Consistency Score: [consistency score] “‘json { "article": {article}, "summary": {summary}, "consistency_score": <a score between 0 and 1> } “‘ ”’ (c) Inverse Prompt Figure 5: Prompts for LLaMA on QAGS dataset. To evaluate summaries, consider the fol- lowing criteria:

  18. [33]

    Ensure that the summary presents a clear and consistent narrative without contradictions or abrupt shifts

    **Coherence Score**: Assess whether the summary logically flows and connects the key points of the article. Ensure that the summary presents a clear and consistent narrative without contradictions or abrupt shifts

  19. [34]

    The sum- mary should maintain the same stance and perspective as the original text

    **Consistency Score**: Verify that the summary accurately reflects the content of the article without introducing new infor- mation or omitting crucial details. The sum- mary should maintain the same stance and perspective as the original text

  20. [35]

    The summary should be grammatically cor- rect, well-structured, and easy to under- stand

    **Fluency Score**: Evaluate the read- ability and smoothness of the summary. The summary should be grammatically cor- rect, well-structured, and easy to under- stand

  21. [36]

    ‘json {

    **Relevance Score**: Determine whether the summary captures the essential information and main points of the article. Ensure that every sentence in the summary is pertinent to the article’s content and does not include irrelevant details. Each criterion should be scored on a s...

  22. [37]

    Read the news article carefully and iden- tify the main topic and key points

  23. [38]

    Check if the summary covers the main topic and key points of the news article, and if it presents them in a clear and logical order

    Read the summary and compare it to the news article. Check if the summary covers the main topic and key points of the news article, and if it presents them in a clear and logical order

  24. [39]

    article": <article>,

    Assign scores on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria. Please evaluate the following summary: Source Text: {article} Summary: {summary} Evaluation Form (scores ONLY): - Coherence: - Consistency: - Fluency: - Relevance: ...

  25. [40]

    Responses should flow smoothly and be easily understood by humans

    **Naturalness Score**: Assess whether the response sounds natural and fluent, without awkward phrasing or forced con- nections. Responses should flow smoothly and be easily understood by humans

  26. [41]

    The response should logically follow from the previous exchanges and integrate the given information meaningfully

    **Coherence Score**: Evaluate how well the response aligns with the conver- sation history and the provided fact. The response should logically follow from the previous exchanges and integrate the given information meaningfully

  27. [42]

    The re- sponse should add value to the dialogue, provide relevant information, or provoke further discussion

    **Engagingness Score**: Determine whether the response keeps the conver- sation interesting and engaging. The re- sponse should add value to the dialogue, provide relevant information, or provoke further discussion

  28. [43]

    ‘json {

    **Groundedness Score**: Assess whether the response is grounded in the provided conversation history and fact. The response should be relevant and not intro- duce unrelated or irrelevant information. Now please evaluate the following model’s response according to the conversat...

  29. [44]

    Read the conversation history and the corresponding facts carefully and identify the main topic and key points

  30. [45]

    Check if the conversa- tion covers the main topic and key points of the corresponding facts, and if it presents them in a clear and logical order

    Read the conversation history and the corresponding facts. Check if the conversa- tion covers the main topic and key points of the corresponding facts, and if it presents them in a clear and logical order

  31. [46]

    ‘ { "conversation

    Assign scores on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria. Example: Conversation: {conversation} Corresponding Facts: {fact} Evaluation Form (scores ONLY): - Naturalness: - Coherence: - Engagingness: - Groundedness: (b) Hum...

  32. [47]

    **Accuracy**: The translation should accurately convey the meaning of the orig- inal sentence without adding, omitting, or altering information

  33. [48]

    **Fluency**: The translated sentence should read naturally in the target language, maintaining proper grammar, syntax, and word order

  34. [49]

    **Consistency**: The translation should be consistent in terms of tense, per- son, and number across the sentence

  35. [50]

    **Relevance**: The translation should be relevant to the context and purpose of the original sentence

  36. [51]

    **Preservation of Style**: The transla- tion should maintain the style and tone of the original sentence, including any formal or informal elements

  37. [52]

    **Proper Names and Terms**: Names, titles, and technical terms should be cor- rectly transliterated or translated as per the reference

  38. [53]

    **Punctuation and Spacing**: The use of punctuation and spacing should be cor- rect and consistent with the target language standards

  39. [54]

    ‘json {

    **Coherence**: The translation should be coherent and logically connected, en- suring that the sentence makes sense as a whole. Assign a score from 0 to 100 based on the overall quality of the translation, consider- ing the above criteria. Now please evaluate the following tra...

  40. [55]

    Do not respond with anything other than the required JSON

  41. [56]

    The quality_score must be a numerical value between 0 and 100

  42. [57]

    Do not provide any additional information or explanations

  43. [58]

    Compare the machine translation with the reference translation and assess the quality of the translation

  44. [59]

    Consider the meaning, grammar, and fluency of the translation when evaluating the quality

  45. [60]

    Original: {original} Reference: {reference} Translation: {translation} (c) Inverse Prompt Figure 8: Prompts for Qwen on WMT22-EN-DE dataset

    Ensure that the evaluation is profes- sional and objective, reflecting the quality of the translation accurately. Original: {original} Reference: {reference} Translation: {translation} (c) Inverse Prompt Figure 8: Prompts for Qwen on WMT22-EN-DE dataset. You are a highly exper...

  46. [61]

    Every detail mentioned in the summary must be explicitly stated in the news article

    A summary is considered to be factually consistent if it does not contain any halluci- nated or factually incorrect details. Every detail mentioned in the summary must be explicitly stated in the news article

  47. [62]

    Assign a score between 0 and 1, where a higher score indicates a higher degree of factual consistency

  48. [63]

    article: {article} summary: {summary} score: (a) 1 Decimal Place Please undertake the evaluation of factual consistency of the summaries to the article

    The format of the input will be as fol- lows: {{article}}, {{summary}}, and the output should be a single score reflecting the factual consistency. article: {article} summary: {summary} score: (a) 1 Decimal Place Please undertake the evaluation of factual consistency of the su...

  49. [64]

    Summaries must be factually consistent with the article. A summary is deemed fac- tually consistent if all the numbers in the summary are presented the same way as in the article and if the facts presented align accurately with those in the article

  50. [65]

    You should score the factual consistency of the summary on a scale of 1 to 10, where 1 indicates completely inconsistent and 10 indicates completely consistent

  51. [66]

    The input will follow the format pro- vided below, with the article, summary, and a placeholder for the score

  52. [67]

    consistency_score

    Evaluate the factual consistency strictly based on the information provided in the article. Do not incorporate any external in- formation. Here is the format of the input: { "consistency_score": score } The article: {article} The summary: {summary} What is the factual consiste...

    2003

  53. [2020]

    InPro- ceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 5008–5020, Online

    Asking and answering questions to evalu- ate the factual consistency of summaries. InPro- ceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 5008–5020, Online. Association for Computa- tional Linguistics. Jiaan Wang, Yunlong Liang, Fa...

    2023 arXiv

  54. [2021]

    InAdvances in Neural Infor- mation Processing Systems, volume 34, pages 27263–27277

    Bartscore: Evaluating generated text as text generation. InAdvances in Neural Infor- mation Processing Systems, volume 34, pages 27263–27277. Curran Associates, Inc. Collin Zhang, John X. Morris, and Vitaly Shmatikov. 2024. Extracting prompts by invert- ing llm outputs. Tianyi...

    2024

  55. [2023]

    Mingjie Sun, Yida Yin, Zhiqiu Xu, J Zico Kolter, and Zhuang Liu

    Autohint: Automatic prompt optimiza- tion with hint generation.arXiv preprint arXiv:2307.07415. Mingjie Sun, Yida Yin, Zhiqiu Xu, J Zico Kolter, and Zhuang Liu. 2025. Idiosyncrasies in large language models. InForty-second International Conference on Machine Learning. Alex Wan...

    2025 arXiv

  56. [2024]

    gradient descent

    Llms-as-judges: A comprehensive survey on llm-based evaluation methods. Jijie Li, Li Du, Hanyu Zhao, Bo wen Zhang, Liangdong Wang, Boyan Gao, Guang Liu, and Yonghua Lin. 2025. Infinity instruct: Scaling instruction selection and synthesis to enhance language models. Yang Liu, ...

    2025 arXiv

  57. [2025]

    Karthik Gopalakrishnan, Behnam Hedayatnia, Qin- lang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Z

    Analyzing and evaluating correlation mea- sures in nlg meta-evaluation. Karthik Gopalakrishnan, Behnam Hedayatnia, Qin- lang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Z. Hakkani-Tür. 2019. Topical-chat: Towards knowledge-grounded open-domain...

    2019 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.