{"id":"353b0555-2959-42d6-946e-6a338cacfde6","arxiv_id":"2501.14956","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ExPerT is a reference-based LLM evaluation metric that extracts and matches atomic aspects, scores content and style, and reports 0.74 human alignment on LongLaMP, a 7.2% relative gain over GEMBA and G-Eval.","lead":"This paper introduces ExPerT, an explainable evaluation metric that judges personalized long-form text by splitting it into atomic aspects and comparing content and writing style against a user-written reference. It reports higher agreement with human judgment than existing metrics on the LongLaMP benchmark, and human raters gave its explanations 4.7 out of 5.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 7.2% improvement is not yet established: the Table 1 variant was selected after inspecting Figure 3 on the same 100 disagreement-filtered examples, with no multiple-comparison control or confidence intervals.","rationale":"The reader's weakest assumption concerns the one-to-one vs many-to-many matching semantics (Section 2.2 vs Appendix D/Figure 9). I read the equations as standard max-based recall/precision: each reference aspect gets credit from its best generated match, and each generated aspect gets credit from its best reference match. This permits many-to-one matching (and, jointly, many-to-many), which is what Appendix D describes; the 'at most one' sentence is a loose description, not a scoring defect. The prompt's single-best-match choice per aspect is consistent with the max equations. I therefore do not think the matching inconsistency changes the headline. The load-bearing weakness is empirical. The 0.74 in Table 1 is the maximum over five aggregation choices evaluated on the same 100 examples, and the choice was made after seeing Figure 3. This is a selection-on-test-set problem. The disagreement-filtered sample makes the estimate conditional on a hard subset, and with n=100 the 0.74 vs 0.69 gap has no reported uncertainty. These are fixable with a pre-registered variant, a fresh sample, and paired significance tests; the explainability result (4.7/5) and robustness analyses are independent and survive. Hence the reader's CONDITIONAL verdict remains appropriate, but for the empirical-selection reason rather than the matching semantics.","tokens_in":17079,"tokens_out":6800,"duration_ms":60585,"concrete_test":"Run a permutation/max-t test on the released code and the 100 human judgments: under the null that all five ExPerT aggregation variants have the same true alignment as GEMBA/G-Eval (0.69), resample the 100 paired judgments, recompute all five variants, and record the maximum alignment; if the observed maximum (0.74) is not beyond the 95th percentile of the permutation distribution, the reported gain is explained by variant selection. As a robustness follow-up, compute alignment for a pre-registered variant (e.g., CONTENT OR STYLE) on a fresh unfiltered random sample of LongLaMP test instances.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ExPerT aligns with human judgments better than GEMBA/G-Eval (0.74 vs 0.69, Table 1). The value 0.74 is not a pre-specified metric: it is the best of five aggregation variants (CONTENT, STYLE, CONTENT AND STYLE, CONTENT OR STYLE, CONTENT/STYLE AVERAGE) evaluated on the same 100 human-annotated examples (Figure 3), and the paper reports no pre-registration, held-out variant selection, confidence interval, or multiple-comparison correction. Selecting the maximum over five correlated variants on the test set inflates the expected alignment relative to a fixed rule; with n=100, a 95% CI for 0.74 spans roughly 0.65-0.82, so the 5-point gap over 0.69 is within sampling noise. In addition, the 100 examples were deliberately filtered to contain at least one metric disagreement (Section 3.1), so 0.74 is a conditional estimate on a hard subset, not a random-sample alignment estimate. Unless the AVERAGE variant remains significantly ahead of GEMBA/G-Eval after accounting for the five-way selection, the headline 7.2% relative improvement is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ExPerT proposes an explainable, reference-based evaluation metric for personalized long-form text generation. It decomposes both the reference (expected) and generated outputs into atomic aspects with supporting evidence, matches aspects via an LLM in a recall- and precision-style procedure, and scores matched evidence on content and writing-style alignment using one of five aggregation variants. The final score is the F-measure of the per-aspect precision and recall. Experiments on the LongLaMP benchmark report the highest alignment with human judgments among compared metrics (0.74 versus 0.69 for GEMBA and G-Eval), a 4.7/5 human rating for explanation usefulness, robustness to a simple prompt-injection trick, and sensitivity to profile replacement. The authors release code for the metric.","tokens_in":17197,"tokens_out":2522,"duration_ms":24434,"significance":"If the empirical claims hold, ExPerT is a meaningful contribution to the underexplored problem of evaluating personalized long-form text generation. Its design has several strengths: it is reference-based and thus grounded in the user's own expected output; it provides fine-grained, per-step rationales; it explicitly separates content and style dimensions; and the authors publicly release the code. The robustness analysis against prompt-injection and the linear sensitivity to profile corruption are useful, falsifiable checks. However, the central quantitative claim—that ExPerT outperforms GEMBA and G-Eval by 7.2% relative alignment—is currently under-supported because the winning aggregation variant was selected on the same 100 human-annotated examples used for the final comparison, and because the reported point estimates are not accompanied by confidence intervals or significance tests.","major_comments":[{"comment":"The headline result of Table 1 (0.74 vs. 0.69) is obtained after selecting the CONTENT/STYLE AVERAGE aggregation from the five variants presented in Figure 3, all evaluated on the same 100 human-annotated examples. Selecting the variant with the maximum observed alignment on the test set inflates the expected alignment relative to a fixed, pre-specified rule, and with n=100 the 0.05 gap over GEMBA/G-Eval is within a plausible sampling error (a 95% CI for 0.74 spans roughly 0.65–0.82). The paper reports no confidence intervals, significance tests, multiple-comparison correction, or a held-out variant selection. To support the claimed superiority, the authors should either pre-register the aggregation rule, use a separate validation set for variant selection, or report corrected (e.g., Bonferroni or bootstrap) intervals and pairwise tests.","section":"Section 3.1 and Figure 3 / Table 1"},{"comment":"The matching step assumes that each aspect from either text can be matched to at most one aspect from the other set, and the LLM prompt implements this by selecting a single best match or 'none'. However, Appendix D and Figure 9 explicitly state that 'an aspect from the generated output or the expected output can be matched to multiple aspects from the other set' and illustrate this behavior. This is an internal contradiction: if the actual extraction and matching allow many-to-many relations, the linear-time assumption and the single-inference prompt are not consistent with the claims, and the recall/precision scores in Equations (1)–(3) undercount aligned aspects. The authors must either restrict the method (and Figure 9 caption) to one-to-one matching or revise the matching procedure and the definitions of R, P, and F to account for many-to-many alignment.","section":"Section 2.2 and Appendix D (Figure 9)"},{"comment":"The 100 samples are not a random sample from the LongLaMP test sets; they are deliberately filtered so that at least one metric selects a different response as better than the others. This means the reported alignment values are conditional estimates on a disagreement-filtered subset, not estimates of alignment on the original test distribution. Consequently, the 0.74 figure cannot be directly interpreted as 'alignment with human judgment on the benchmark' as implied in Table 1 and the abstract. The authors should report the alignment on an unfiltered random sample as well, or clearly frame the numbers as hard-subset discriminative power.","section":"Section 3.1, Human Annotation"},{"comment":"The claim that ExPerT 'achieves the highest alignment with human judgments' is based solely on point estimates with no statistical testing. Given the small sample (100 examples, 3 annotators each, majority vote) and the post-hoc variant selection, the 0.74 vs. 0.69 difference could well be noise. A bootstrap or permutation test over examples, or over annotators, is needed to establish that the difference is not attributable to sampling variation.","section":"Section 3.2, first finding"}],"minor_comments":[{"comment":"The terminology 'recall- and precision-based manner' is used before the formal definitions; for clarity, the formal definitions of R and P should be moved closer to the matching description or introduced earlier.","section":"Section 2.2"},{"comment":"The cost comparison uses 18.6 LLM calls per instance for ExPerT as measured on the 100 filtered human-annotated examples. Since the number of aspects may differ on a random sample, reporting the range or a formula for the call count in terms of the number of extracted aspects would be more informative.","section":"Section 3.2, 'How efficient is ExPerT...'"},{"comment":"The human explanation study reports that annotators 'correctly identified the output with the higher ExPerT score in 94% of cases.' Because the annotators see the aspects, matching, and rationales but not the final score, this is a reasonable protocol; however, the paper does not report inter-annotator agreement for this task, which would be useful given the 4.7/5 rating.","section":"Section 3.2, 'How explainable is ExPerT...'"},{"comment":"The profile-replacement curve (Figure 5) would benefit from error bars or a repeat with multiple random profile permutations; the current plot cannot indicate whether the downward trend is statistically reliable.","section":"Section 3.2, 'How sensitive is ExPerT...'"},{"comment":"There are several typographical issues, e.g., 'G-E VAL' in the introduction (should be 'G-Eval'), 'A VERAGE' in Section 2.2, and 'Strick' for 'Stricked' in Figure 6; these should be corrected in a final pass.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is candid about its limitations and the code release is commendable. The main risk is the statistical validity of the headline improvement: the aggregation variant is selected on the same test examples, and no uncertainty quantification is provided. Even after fixing that, the contradiction between Section 2.2's one-to-one matching assumption and Appendix D's many-to-many example should be resolved before the paper can be accepted. I would ask the authors to either provide a held-out variant selection or corrected confidence intervals, and to align the matching description with the actual implementation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. ExPerT is a fresh combination: atomic aspect/evidence extraction, LLM aspect matching, content/style agreement, and recall/precision/F, with rationales at every step. That is a real contribution to personalized long-form evaluation. The second thing: the headline number, 0.74 vs 0.69 for GEMBA/G-Eval, is not yet reliable. The 0.74 is the best of five aggregation variants from Figure 3, selected after inspecting the same 100 human-annotated examples that produced Table 1. With n=100 and no confidence intervals or significance tests, a 5-point gap is within sampling noise. The examples were also deliberately filtered to contain at least one metric disagreement, so 0.74 is a conditional estimate on a hard subset; the paper calls this 'without any side effects,' which is not right. These issues are fixable: pre-register the aggregation or hold it out, report CIs, and ideally annotate a random sample.\n\nWhat's genuinely good: the robustness tests are useful. The profile-replacement curve is a nice check that the metric tracks personalization; the injection attack shows GEMBA is badly fooled and ExPerT penalizes the trick; the position-bias result is also in ExPerT's favor. The human explainability study (94% correct, 4.7/5) is a solid selling point. Code is released.\n\nThe soft spots beyond the selection issue: the matching description is internally inconsistent. Section 2.2 states one-to-one matching, but Appendix D and Figure 9 describe many-to-many matches. Actually the max-based equations allow multiple aspects on one side to pick the same best partner, so the math is softer than the prose. The authors need to reconcile the description with the implementation, or the metric's scoring semantics are unclear. Also, small sample size is okay for a pilot, but the paper overstates what it establishes.\n\nWho it's for: NLG evaluation and personalization researchers who want an explainable reference-based metric. It deserves a serious referee, but my own verdict is conditional: the framework is worth engaging with, the empirical claim needs another pass. I'd send it out and ask for the aggregation selection and matching clarity to be fixed.","headline":"ExPerT is a real and explainable evaluation framework, but the headline 7.2% gain is a post-hoc selection on 100 filtered examples and is not yet established.","tokens_in":17844,"tokens_out":5028,"would_cite":true,"duration_ms":46556,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ExPerT evaluates personalized long-form text by decomposing it into atomic aspects, matching those aspects against a user-written reference, and scoring content and style alignment per aspect, reporting the highest human-agreement of the…","keywords":["personalized text generation","long-form text generation","LLM evaluation","explainable evaluation","aspect extraction","reference-based metrics","F-measure","prompt-injection robustness"],"falsifier":"Design a human-annotated test set in which one reference aspect is deliberately mapped to two distinct generated aspects, run both ExPerT's single-best-match prompt and a many-to-many prompt on it, and check whether the many-to-many variant recovers the human-aligned matches and raises agreement; if it does not, the one-to-one matching assumption is not the limiting factor.","tokens_in":16733,"feed_emoji":"📏","tokens_out":7999,"duration_ms":66081,"temperature":0.7,"pith_summary":"The paper tries to establish that personalized text generation can be evaluated automatically and transparently by an LLM that works at the level of atomic aspects rather than whole documents. Because only the person who wrote the prompt can truly judge a personalized output, ExPerT uses that person's own reference text and asks whether the generated text covers the same aspects and whether the evidence for each matched aspect agrees in content and writing style. The reported result is that this aspect-level score agrees with human judges 74% of the time, beating GEMBA and G-Eval by 7.2% relative, while also producing human-readable rationales rated 4.7 out of 5. If correct, this gives the field an evaluation signal that is more reliable than holistic LLM scoring, harder to cheat with a simple confidence phrase, and explainable enough for a user to see why one output was preferred.","feed_headline":"Aspect-by-aspect evaluator beats LLM judges by 7.2%","feed_subtitle":"ExPerT scores content and style at the aspect level, explains each step, and resists simple score-inflating tricks.","key_machinery":"The central object is the atomic aspect: a single claim or topic unit in a text, together with its evidence, the sentences or phrases that support it. ExPerT uses an off-the-shelf instruction-tuned LLM to extract these aspect-evidence pairs from the reference and generated texts, then runs a one-pass matching step in which each aspect is paired with at most one aspect from the other text or with 'none'. For each matched pair, the LLM makes binary content-alignment and style-alignment decisions on the evidence and writes a rationale; the two decisions are averaged, the default CONTENT/STYLE AVERAGE aggregation, to score evidence similarity. Recall averages the best matched evidence score over reference aspects, precision over generated aspects, and the final score is their harmonic mean, the F-measure. The matching and binary-decision structure is what converts an LLM judgment from an opaque holistic score into a transparent, decomposable calculation.","core_discovery":"The paper's central claim is that personalized long-form generation is best evaluated not by comparing whole texts, but by comparing the atomic aspects each text contains and the evidence attached to those aspects. On the LongLaMP benchmark, ExPerT agrees with human pairwise judgments at 0.74 alignment, a 7.2% relative improvement over the previous best whole-text LLM evaluators GEMBA and G-Eval, both at 0.69. The paper also claims that this decomposition makes the evaluation explainable: human annotators rated its per-step rationales 4.7 out of 5 and could infer which output ExPerT preferred in 94% of cases from the explanations alone. The same experiments show ExPerT resists a simple prompt-injection phrase that inflates GEMBA scores by up to 24.3% relative, and its scores decrease linearly as user profiles are randomly replaced, evidence that it tracks personalization rather than surface text. In short, the paper is trying to establish that a reference-based, aspect-level, content-and-style F-measure is a more effective, more robust, and more transparent way to evaluate personalized text than holistic LLM judging.","pith_inferences":["A many-to-many aspect matching variant, allowing one reference aspect to split across several generated aspects, would likely raise recall and precision on texts where a single idea branches into multiple sub-points; the paper's one-to-one matching may undercount those alignments.","If a reference-free model can synthesize a plausible user reference, ExPerT could be applied where no user-written reference exists, turning the metric into a hybrid reference-based and reference-free evaluator.","The per-aspect rationales ExPerT produces could be mined as training feedback for personalized generators, giving models a finer-grained error signal than a scalar score.","The linear sensitivity to profile substitution suggests ExPerT could serve as a diagnostic for how much of a system's output actually depends on the user profile, not just as a final quality score."],"forward_implications":["Personalized generation systems can be ranked and iterated on without re-recruiting the original prompt author each time, since the user's reference text supplies the personal ground truth.","Evaluation reports become inspectable: recall and precision identify which aspects of the reference were missed, which extra aspects were invented, and where content or style diverged, not just a single score.","The prompt-injection and position-bias results suggest that LLM-based evaluation of personalized text need not be abandoned for lack of robustness, provided matching is structured rather than holistic.","Model-size experiments indicate that a large open model reaches the same human-agreement level as GPT-4o under ExPerT, so reproducible evaluation does not require a proprietary API.","Because the same aspect, content, and style machinery applies to any reference-based long-form generation, the metric should transfer to summarization or translation evaluation tasks, though the paper does not test those settings."],"supporting_citations":[{"why":"supplies the LongLaMP benchmark's three personalized generation tasks and the reference outputs used in every experiment.","marker":"(Kumar et al., 2024)"},{"why":"defines GEMBA, the whole-text LLM baseline whose 0.69 human agreement ExPerT must beat and which the trick experiment shows is inflated by a confidence phrase.","marker":"(Kocmi and Federmann, 2023)"},{"why":"defines G-Eval, the other 0.69 LLM baseline and the 20-call per-instance cost comparison point.","marker":"(Liu et al., 2023)"},{"why":"gives BERTScore, the semantic overlap baseline and the source of the one-to-one matching simplification ExPerT adopts.","marker":"(Zhang et al., 2020)"},{"why":"introduces the atomic decomposition of long-form text into fine-grained units that ExPerT adapts into aspect and evidence extraction.","marker":"(Min et al., 2023)"},{"why":"provides the personalized retrieval-augmented setup that generates the outputs ExPerT evaluates.","marker":"(Salemi et al., 2024b)"},{"why":"supplies the finding that reference-based evaluation is more reliable than reference-free personalized judging, motivating ExPerT's design.","marker":"(Dong et al., 2024)"}],"fun_headline_variants":["Aspect-level evaluator beats LLM judges by 7.2%","ExPerT: explainable evaluation for personalized text","Personalized text evaluation: aspect matching beats whole-text","Explainable eval for personalized text: 7.2% better","Aspect-based evaluation: 94% explanations match human preference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each aspect in one text matches at most one aspect in the other, so the LLM is instructed to pick a single best match or 'none'; if real aspects connect many-to-many, the recall and precision scores undercount aligned content and the reported F-measure no longer measures the alignment the equations describe.","fun_headline_variants_meta":{"raw":{"variants":["Aspect-level evaluator beats LLM judges by 7.2%","ExPerT: explainable evaluation for personalized text","Personalized text evaluation: aspect matching beats whole-text","Explainable eval for personalized text: 7.2% better","Aspect-based evaluation: 94% explanations match human preference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000532,"raw_usage":{"total_tokens":2574,"prompt_tokens":975,"completion_tokens":1599,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":1513}},"tokens_in":591,"tokens_out":1599,"duration_ms":33193,"temperature":1.0,"reasoning_tokens":1513,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:45:57.449595+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Design a human-annotated test set in which one reference aspect is deliberately mapped to two distinct generated aspects, run both ExPerT's single-best-match prompt and a many-to-many prompt on it, and check whether the many-to-many variant recovers the human-aligned matches and raises agreement; if it does not, the one-to-one matching assumption is not the limiting factor.","supporting_citations":[],"review_version":1}