*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation

Arnaud Delhay; Daisy Munson; Damien Lolive; Jonathan Chevelu; L\'eane Jourdan; Pierre Alain; Quentin Lemesle

REVIEW 3 major objections 4 minor 4 references

Task-specific prompts make a perplexity-based LLM judge more human-aligned without the cost of text generation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 22:44 UTC pith:RKRL6VD5

load-bearing objection Useful prompt-adaptation study with one real win; the abstract oversells the consistency. the 3 major comments →

arxiv 2602.15778 v2 pith:RKRL6VD5 submitted 2026-02-17 cs.CL

*-PLUIE: Personalisable metric with Llm Used for Improved Evaluation

Quentin Lemesle , L\'eane Jourdan , Daisy Munson , Pierre Alain , Jonathan Chevelu , Arnaud Delhay , Damien Lolive This is my paper

classification cs.CL

keywords LLM-as-a-judgeperplexity-based metricprompt personalisationevaluation of generated textparaphrase classificationscientific text revisionnetwork intent translationhuman alignment

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether the prompt template of a perplexity-based LLM judge can be personalised to a task without paying the usual generation cost. It claims that three task-specific variants—French paraphrase detection, network-policy translation, and scientific revision—improve alignment with human ratings compared with the generic template, and that they stay far cheaper than LLM judges that write out answers. The central bet is that the log-ratio of a single 'Yes' over a single 'No' token, computed from the model's own next-token probabilities, is a faithful confidence signal. If the bet holds, automatic evaluation of generated text can be made both more human-aligned and roughly eight times faster, with no output parsing. The strongest reported gains are on scientific text revision, where the personalised prompt with a gold reference outperforms all compared metrics; on paraphrase classification the improvement over the original prompt is marginal.

Core claim

The paper's central claim is that personalising the prompt inside the ParaPLUIE formula—whose score is the log-ratio of one 'Yes' token's probability to one 'No' token's probability—improves agreement with human judgement compared with the generic template, while keeping the computation to roughly two token probabilities. Three variants are tested: French paraphrase detection, network-intent translation, and scientific-text revision. Across preference evaluations, the task-specific variants reach the strongest or second-strongest alignment scores, and the best variant on scientific revision outperforms all output-based LLM judges compared. Because no text is generated, the paper reports the

What carries the argument

The load-bearing object is the ParaPLUIE score: the log ratio of the probability the model assigns to 'Yes' versus 'No' when those tokens are appended to a prompt containing the two texts being compared. The work it does is to convert the model's internal next-token distribution into a continuous signed confidence value—positive for 'Yes', negative for 'No'—without generating a word of output. The task-specific prompts change only the question being asked; the score remains a single-token log-likelihood ratio, so the metric can be re-aimed at any binary semantic judgment by editing the prompt. The paper's contribution is to show that this prompt-edit is the dial that improves human alignment

Load-bearing premise

The load-bearing premise is that the ratio of the model's probabilities for a single 'Yes' token versus a single 'No' token is a faithful numerical expression of the model's semantic judgement; this is inherited from the original ParaPLUIE and is not independently validated here.

What would settle it

Compare the sign of the *-PLUIE score with the model's own sampled answer on the same items: if the model writes 'Yes' while the score is negative (or vice versa) often enough that the sign predicts the generated answer no better than chance, the perplexity ratio is not tracking the model's judgement. This can be checked on any of the three datasets without new annotations.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Evaluation can be run inside training or hyperparameter loops: the score is two token probabilities, so an 8x speedup over text-generating judges is a direct corollary.
The zero threshold is a usable decision boundary for classification without dataset-specific calibration; the paper shows this on French paraphrase and Nile translation.
Prompt personalisation becomes a controlled dial: any binary semantic question can be turned into a *-PLUIE variant by editing the prompt, and the paper demonstrates this on three quite different tasks.
When a gold revision exists, including it as a one-shot example is beneficial; Rev-PLUIE with reference beats all compared LLM judges on scientific revision.
Net-PLUIE's near-transitive behaviour (98% of the critical triplets) makes automatic alignment checks between natural language and a formal intent language plausible enough to replace manual annotation in parts of the pipeline.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Because the metric returns a signed confidence score rather than a label, it can be reused as a cheap probe for how much a prompt rephrasing shifts an LLM's decision boundary—something the paper does not explore.
The single-token 'Yes'/'No' requirement ties the score to tokenizer behaviour; across morphologically rich languages the two answers may not be single tokens, so the formula's equal-length assumption could distort cross-lingual scores—the paper leaves this open.
The reported kappa values around 0.2–0.4 show the metric captures only part of human agreement; averaging several *-PLUIE prompts or models, in the spirit of a jury, is a natural way to push alignment higher than any single prompt achieves.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful prompt-adaptation study with one real win; the abstract oversells the consistency.

read the letter

Quick take: this is a solid empirical extension of the ParaPLUIE metric, but the advertised 'stronger correlations' only really show up on one of the three tasks. The efficiency result is real and worth having.

The paper does three things: it designs task-specific prompts (French paraphrase, Nile translation, scientific revision), evaluates them against the original Para-PLUIE and a range of baselines, and adds a short analysis of using Net-PLUIE as an alignment function between natural language and Nile intents. The best contribution is Rev-PLUIE: on scientific text revision it clearly beats the original prompt (κ ≈ 0.33 vs 0.15 for Phi, similar for Llama), and the gain holds whether or not the gold revision is included as a one-shot example. The runtime comparison is also the most concrete thing in the paper: PLUIE-style scoring is up to ~8x cheaper than output-based LLM-judges, which is a genuine practical advantage.

Where I'd push back: the abstract and conclusion say 'consistently achieves stronger correlations,' but the tables don't support that across tasks. On Nile translation, Para-PLUIE is at least as good as Net-PLUIE on every reported measure (e.g., Llama κ 0.43 vs 0.42). On French paraphrase, the F1 deltas are within noise. Only the revision task shows a clear win. Also, there are no significance tests or confidence intervals anywhere, so even the revision gain is not quantified against chance. That should be added or the claims softened.

One concrete error: Appendix B lists all six transformation-pair types as 'paraphrase,' which would give 100% positives, yet the paragraph says the dataset is 33.60% positive. The intended labeling is probably two positive and four negative types (s vs S(s), P(s) vs S(P(s)) positive; the rest negative). As printed, the dataset construction is not reproducible as written.

Minor: the Net-PLUIE alignment-function analysis reports 15% of triplets failing condition 5, then calls it the best available solution; the wording is a bit generous, but the numbers are transparent.

Nothing here is fatal. The underlying formula is self-cited but parameter-free and independently computable, so I don't see a circularity problem. The paper is worth a serious referee: the revision result is useful, the efficiency comparison is practical, and the main fixes are adding significance testing, correcting Appendix B, and aligning the claims with the actual results.

Referee Report

3 major / 4 minor

Summary. This paper presents *-PLUIE, a prompt-personalisation wrapper around the ParaPLUIE perplexity ratio metric. Three task-specific prompts (Fr-PLUIE for French paraphrase classification, Net-PLUIE for Nile translation evaluation, Rev-PLUIE for scientific text revision) are compared with the original Para-PLUIE prompt, similarity-based metrics, and output-based LLM judges, using Phi-4 and Llama-3 as perplexity models. The central claim is that task-specific prompting gives stronger correlations with human ratings while preserving low computational cost. The reported results support this only for the revision task; on Nile and French paraphrase the personalised variants are at best tied with Para-PLUIE, and sometimes slightly worse.

Significance. The proposal is pragmatic, and the parameter-free derivation in Appendix A is a genuine, self-contained identity rather than a circular reuse of the earlier method. The efficiency advantage over output-based judges is credible and useful: the runtime numbers in Table 3 are consistently lower. The strongest concrete contribution is Rev-PLUIE, which clearly improves on Para-PLUIE for scientific revision (Table 2). If confirmed with significance testing, this is a useful low-cost evaluation tool. However, as written the paper overclaims the general 'consistently stronger' result; the overall contribution is therefore a specialised prompt-engineering result plus an efficiency benchmark rather than a demonstrated general improvement.

major comments (3)

[§3.2, Table 2 and Conclusion] The conclusion states that *-PLUIE 'consistently achieves stronger correlations with human judgement', but this is contradicted by the paper's own numbers. In Table 2, for Nile preference, Net-PLUIE Phi has pair acc./V/κ of 0.69/0.42/0.40 while Para-PLUIE Phi gives 0.70/0.44/0.42; with Llama the comparison is 0.70/0.43/0.42 vs 0.72/0.46/0.43. In Table 1, French paraphrase optimal-threshold F1 is 0.63 vs 0.64 (Phi) and tied at 0.59 (Llama); Nile classification F1 is identical at 0.85 for Net-PLUIE and Para-PLUIE under both models. Only the scientific-revision row shows a clear advantage (κ 0.33 vs 0.15). The 'consistently stronger' claim must be withdrawn or substantially narrowed.
[§3.1, Table 1] The classification comparison is vulnerable to overfitting. Baselines are 'calibrate[d] by determining an optimal decision threshold that maximises the F1 score' on the test data, and *-PLUIE is also reported at an a posteriori optimal threshold. No validation split or cross-validation is described. Under this protocol the reported F1 gaps of 0.01–0.03 are likely within noise, and no significance tests, confidence intervals, or bootstrap estimates are given for Tables 1 and 2. I would require a threshold-selection protocol that does not use test labels, plus paired resampling CIs for the differences that support the main claims.
[Appendix I and §4 Conclusion] The sentence in Section 4 that Net-PLUIE 'can be used as SOTA alignment function' is not supported by Appendix I. That appendix reports failure counts (56/430 for condition (5), 7/430 for condition (6)) but provides no baseline or competing alignment method, no confidence interval around the failure rates, and no analysis of the failure cases except a few examples. Without a comparative baseline, 'SOTA' is an overclaim; the authors should either add a comparison or remove the SOTA wording.

minor comments (4)

[Table 3] Several numbers are hard to parse, e.g. 'A100 x108 min' and '0.610.61 w.g.'. Use proper column separation and expand *-PLUIE into Fr-PLUIE, Net-PLUIE, and Rev-PLUIE in the row labels.
[§2.3] The citation group 'Prior work (Rios and Kavuluru, 2018; Brown et al., 2020; Chung et al., 2024), Lemesle et al. (2025)' is grammatically awkward and should be rephrased; the sentence structure makes it unclear which works support the claim about few-shot examples.
[Appendix A, Eq. (4)] Please check the notation in the multi-token generalisation. The formula appears to have unbalanced parentheses and inconsistent indices (N1...Nj vs Y1...Ni); define the token lengths explicitly.
[§2.2] 'Modern BertScore' is not defined. State whether it is BERTScore computed with ModernBERT embeddings and, if so, which code/library was used, so the baseline is reproducible.

Circularity Check

1 steps flagged

Rev-PLUIE's reported advantage is conditioned on the gold reference for the same test instance; the core metric formula is a harmless algebraic identity.

specific steps

other [Section 2.3 (*-PLUIE Metrics) and Appendix E, Prompt segment 8]
"Rev-PLUIE: a prompt designed to assess whether a generated revision follows its associated instruction. This variant uses the gold reference as a one-shot example to ground the model’s task understanding. [Prompt segment 8:] user="""P1: "{reference}"; I: "{instruction}"; P2: "{gold}"""" assistant="""Yes""""

For each test instance, the gold revised paragraph (the human-approved answer for that same P1/I) is inserted into the prompt as a 'Yes' few-shot example immediately before the candidate is scored. The human judgement being predicted — whether the revision addressed the instruction and which revision is preferred — is therefore partly encoded in the metric's input. The paper's headline gain of Rev-PLUIE over Para-PLUIE on scientific text revision is obtained under this test-time conditioning (Table 2, 'w.g.'), so the apparent improvement is not an independent prediction but a target-derived prompt effect. This is not a fitted parameter in the usual sense, but it has the same self-confirming structure: the answer key for the instance is part of the scorer's input.

full rationale

The central ParaPLUIE formula is not circular: Appendix A derives Eq. 3 from Eq. 1 by algebraic manipulation of the log-likelihood ratio and perplexity, so the self-citation to Lemesle et al. (2025) is not load-bearing — the present paper re-derives the identity and evaluates it on new tasks. Fr-PLUIE and Net-PLUIE provide genuine, if mixed, external evaluations; the abstract's claim of 'consistently stronger correlations' is contradicted by Tables 1 and 2 for French paraphrase and Nile translation, but that is an evidence/overclaim problem, not a definitional circularity. The only place where a 'prediction' reduces to its own inputs is Rev-PLUIE, where the gold reference for the same instance is used as a one-shot example, and that is the only task where the personalised variant clearly beats Para-PLUIE. No uniqueness theorem or ansatz is imported from the authors' prior work. Score 3 reflects one target-derived input in the single task that supports the central improvement claim; the underlying metric formula itself remains an independent algebraic construction.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

No new physical entities; the paper inherits the ParaPLUIE formula from prior work and introduces only prompt templates. The main fitted numbers are decision thresholds for classification; the human labels and the perplexity-as-confidence premise are taken as axioms.

free parameters (2)

classification decision threshold (baselines) = per-metric optimal on test set (e.g., BERTScore 0.80 on paraphrase)
Each baseline metric is assigned a threshold that maximizes F1 on the test set (§3.1, Table 1), a fitted value.
classification decision threshold (*-PLUIE optimal) = e.g., -7.63 for Fr-PLUIE Phi
The 'optimal' threshold rows in Table 1 are fitted to the test set, though the natural threshold of 0 is also reported.

axioms (4)

domain assumption The log-likelihood ratio of the single tokens 'Yes' vs 'No' following the prompt measures the model's semantic judgment confidence (Eq. 1–3).
The entire metric inherits this from ParaPLUIE (Lemesle et al. 2025); no independent validation is given beyond correlation with human labels.
domain assumption Human annotations (Mushra binarization, ParaReval preferences) are reliable ground truth.
Used as the gold standard in all alignment computations; inter-annotator agreement is not reported.
standard math The two answer tokens are single tokens under the model's tokenizer.
Required by the equivalence in Eq. 2; the paper notes this restriction (Appendix A).
domain assumption LLM loss equals log-perplexity and is directly comparable across different prompts.
Used to convert the perplexity ratio into a loss difference (Eq. 3).

pith-pipeline@v1.3.0-alltime-deepseek · 20666 in / 13729 out tokens · 124938 ms · 2026-08-02T22:44:42.086479+00:00 · methodology

0 comments

read the original abstract

Evaluating the quality of automatically generated text often relies on LLM-as-a-judge (LLM-judge) methods. While effective, these approaches are computationally expensive and require post-processing. To address these limitations, we build upon ParaPLUIE, a perplexity-based LLM-judge metric that estimates confidence over ``Yes/No'' answers without generating text. We introduce *-PLUIE, task specific prompting variants of ParaPLUIE and evaluate their alignment with human judgement. Our experiments show that personalised *-PLUIE achieves stronger correlations with human ratings while maintaining low computational cost.

Figures

Figures reproduced from arXiv: 2602.15778 by Arnaud Delhay, Daisy Munson, Damien Lolive, Jonathan Chevelu, L\'eane Jourdan, Pierre Alain, Quentin Lemesle.

**Figure 1.** Figure 1: *-PLUIE workflow. More formally, given: • S the source sentence, • H the hypothetical sentence, • Prompt(S, H) the prompt filled with the sentences S and H, • T: length in tokens of the Prompt(S, H), • ◦: the concatenation operator of 2 sequences of tokens. ParaPLUIE(S, H) = log p (Yes|Prompt(S, H)) p (No|Prompt(S, H)) (1) It can be expanded, using the Bayes formula, to make appear the perplexity of th… view at source ↗

**Figure 2.** Figure 2: Example of data in the ParaReval dataset [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

**Figure 3.** Figure 3: Score distribution of Modern BertScore (a) and Fr-PLUIE (b). The [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗

**Figure 4.** Figure 4: Accuracy, recall, precision and F1-score score distribution over different threshold values for Modern BertScore (a) and Net-PLUIE (b). Emphasis is placed on the maximum F1-score obtained by the metric. Figures 5, 6, 7, 8, 9, 10, 11 show the different classification results with all the combination of metrics. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗

**Figure 5.** Figure 5: Classification with Levenshtein. (a) Llama translations (b) English source intents (c) Mistral translations [PITH_FULL_IMAGE:figures/full_fig_p021_5.png] view at source ↗

**Figure 6.** Figure 6: Classification with METEOR. (a) Llama translations (b) English source intents (c) Mistral translations [PITH_FULL_IMAGE:figures/full_fig_p021_6.png] view at source ↗

**Figure 7.** Figure 7: Classification with BERTScore. (a) Llama translations (b) English source intents (c) Mistral translations [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗

**Figure 8.** Figure 8: Classification with BLEU. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗

**Figure 9.** Figure 9: Classification with Modern BertScore. (a) Llama translations (b) English source intents (c) Mistral translations [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗

**Figure 10.** Figure 10: Classification with metric Para-PLUIE with Phi-4 14B. [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗

**Figure 11.** Figure 11: Classification with metric Net-PLUIE with Phi-4 14B. [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

4 extracted references · 4 linked inside Pith

[4]

Yes” against “No

Neat: A nile-english aligned translation cor- pus based on a robust methodology for intent based networking.Computer Networks, 271:111519. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov...

Pith/arXiv arXiv 2024
[2023]

InPro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 12914– 12929

Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. InPro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 12914– 12929. Ehsan Doostmohammadi, Oskar Holmström, and Marco Kuhlmann. 2024. How reliable are automatic eval- uation methods for instruction-tuned LLMs? In Findings of...

Pith/arXiv arXiv 2023
[2024]

Arthur Selle Jacobs, Ricardo José Pfitscher, Ronaldo Alves Ferreira, and Lisandro Zam- benedetti Granville

A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594. Arthur Selle Jacobs, Ricardo José Pfitscher, Ronaldo Alves Ferreira, and Lisandro Zam- benedetti Granville. 2018. Refining network intents for self-driving networks. InProceedings of the Afternoon Workshop on Self-Driving Networks, SelfDN 2018, page 15–21, New York, NY , USA. ACM. Albert Q. Jiang...

Pith/arXiv arXiv 2018
[2025]

In Proceedings of the 31st International Conference on Computational Linguistics, pages 8057–8087, Abu Dhabi, UAE

Paraphrase generation evaluation powered by an LLM: A semantic metric, not a lexical one. In Proceedings of the 31st International Conference on Computational Linguistics, pages 8057–8087, Abu Dhabi, UAE. Association for Computational Linguis- tics. Jingjing Li, Zichao Li, Tao Ge, Irwin King, and Michael Lyu. 2022. Text revision by on-the-fly representa- ...

Pith/arXiv arXiv 2022

[1] [4]

Yes” against “No

Neat: A nile-english aligned translation cor- pus based on a robust methodology for intent based networking.Computer Networks, 271:111519. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov...

Pith/arXiv arXiv 2024

[2] [2023]

InPro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 12914– 12929

Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. InPro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 12914– 12929. Ehsan Doostmohammadi, Oskar Holmström, and Marco Kuhlmann. 2024. How reliable are automatic eval- uation methods for instruction-tuned LLMs? In Findings of...

Pith/arXiv arXiv 2023

[3] [2024]

Arthur Selle Jacobs, Ricardo José Pfitscher, Ronaldo Alves Ferreira, and Lisandro Zam- benedetti Granville

A survey on llm-as-a-judge.arXiv preprint arXiv:2411.15594. Arthur Selle Jacobs, Ricardo José Pfitscher, Ronaldo Alves Ferreira, and Lisandro Zam- benedetti Granville. 2018. Refining network intents for self-driving networks. InProceedings of the Afternoon Workshop on Self-Driving Networks, SelfDN 2018, page 15–21, New York, NY , USA. ACM. Albert Q. Jiang...

Pith/arXiv arXiv 2018

[4] [2025]

In Proceedings of the 31st International Conference on Computational Linguistics, pages 8057–8087, Abu Dhabi, UAE

Paraphrase generation evaluation powered by an LLM: A semantic metric, not a lexical one. In Proceedings of the 31st International Conference on Computational Linguistics, pages 8057–8087, Abu Dhabi, UAE. Association for Computational Linguis- tics. Jingjing Li, Zichao Li, Tao Ge, Irwin King, and Michael Lyu. 2022. Text revision by on-the-fly representa- ...

Pith/arXiv arXiv 2022