Pith. sign in

REVIEW 5 major objections 6 minor 51 references

Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewards

T0 review · 5 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A principle-guided LLM judge, refined by adversarial training with a reflection step, produces higher-quality Chinese greetings from a 7B model than a reward model trained on multi-agent-filtered preferences, with excellence rates of 92.4%,

desk verdict A useful, honest comparison of two RLAIF reward pipelines for a narrow task; the broad ranking is probably right, but 'demonstrably superior' overstates what the evidence shows. read the letter →

arxiv 2508.21476 v1 pith:P55SUY5Y submitted 2025-08-29 cs.CL cs.AI

Xiaolong Wei , Bo Lu , Xingyu Zhang , Zhejun Zhao , Dongdong Shen , Long Xia , Dawei Yin This is my paper
classification cs.CLcs.AI
keywords RLAIFLLM-as-a-JudgesmalllanguagemodelscreativewritingChinesegreetingsrewardmodeladversarialtrainingmulti-agentevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a 7B-parameter language model can be trained to write creative Chinese greetings without expensive human preference labels. It compares two AI-generated reward signals inside a reinforcement-learning-from-AI-feedback loop: a reward model trained on preference pairs filtered by a multi-agent debate system, and a direct LLM judge guided by explicit creative-writing principles and refined through adversarial training with a reflection step. The central claim is that the LLM judge produces clearly better generation quality, with excellence rates of 92.4%, 96.6%, and 95.0% on three evaluation metrics, while also being simpler to train and less dependent on human annotation. If true, this suggests a scalable route to creative small language models: use a strong model's judgment as the reward, not a separately trained reward model.

What carries the argument

Two reward-generation mechanisms carry the argument. The first is a multi-agent rejection sampling framework: a retrieval agent supplies high-quality exemplars, positive and negative debate agents argue for a response's strengths and weaknesses, a judge agent synthesizes an initial verdict, and a reflect agent ratifies or overrides it, yielding preference pairs used to train a scalar reward model. The second is an adversarially optimized LLM-as-a-Judge: a generator tries to produce bad greetings that fool a detector, the detector learns to separate good from bad, and a reflector feeds the detector diagnostic feedback on its mistakes, converging on a prompt that encodes ten evaluation princip

What would settle it

Run GRPO twice on the same 4,000-query training set and the same 2,000-item evaluation set, once with the hand-written principles prompt and once with the supposedly adversarially optimized prompt, holding all other hyperparameters fixed. If the two runs produce statistically indistinguishable excellence rates, the adversarial optimization is not the source of the reported gain.

Watch

Extended reading notes

Core claim

The paper's central claim is that, inside a reinforcement-learning-from-AI-feedback loop, the choice of reward signal decides how much creative ability a small model can gain. For a 7B Qwen2.5 model generating Chinese greetings, it compares two AI-built rewards: a scalar reward model trained on preference pairs produced by a multi-agent debate-and-reflection pipeline, and a binary judge reward from a strong LLM prompted with ten explicit creative-writing principles and refined by an adversarial generator–detector loop plus a reflection step. The paper reports that the judge-based reward yields the best generation quality, with excellence rates of 92.4%, 96.6%, and 95.0% on three evaluation m

Load-bearing premise

The central claim rests on the assumption that the adversarial training loop actually improves the judge's prompt during training; the paper does not show the prompt changing, report the detector's accuracy over time, or compare against the same judge prompt without adversarial training.

Editorial extensions

If this is right

  • LLM-as-a-Judge + RL reaches state-of-the-art excellence rates of 92.4%, 96.6%, and 95.0% on the high-frequency greeting set, surpassing both the RM-based approach and strong general-purpose LLMs.
  • Because the judge reward is binary and prompt-based, it avoids training a separate reward model, cutting pipeline complexity and human preference annotation.
  • Both AI-feedback strategies improve over SFT alone; SFT followed by RM+RL adds gains of 11.5%, 6.3%, and 5.8% on ordinary queries across the three evaluation dimensions.
  • Automated evaluation with either framework agrees with human experts above 70%, with the multi-agent framework reaching 80–87%, supporting the use of these evaluators as proxies for human annotation.
  • A discrete 0/1 reward can support stable GRPO training and high-quality creative output, indicating that continuous rewards are not necessary for this task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the judge-based reward works because it encodes explicit principles, the same recipe should transfer to other short-form creative domains such as festival copy, product taglines, or celebration messages where rubrics can be written; long-form narrative would need richer principles.
  • The paper's comparison leaves the RM pipeline under a handicap: the RM+RL run did not converge, so a well-tuned converged RM might narrow or even reverse the reported gap.
  • A cheap testable extension is to blend the binary judge reward with the continuous RM reward, or to anneal from one to the other, to see whether the judge's strong filtering combines usefully with the RM's fine-grained gradients.
  • The multi-agent framework's high agreement with humans (80–87%) suggests it could serve as a low-cost labeler for other subjective text-quality tasks, not just Chinese greetings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper studies RLAIF for a 7B Chinese-greeting generator. It compares two reward signals: a reward model trained on preference data produced by a multi-agent rejection-sampling framework (Positive/Negative/Judge/Reflect agents), and a principle-guided LLM-as-a-Judge whose scoring prompt is allegedly optimized by an adversarial Generator–Detector loop with a Reflector. Both signals are used with GRPO to train Qwen2.5-7B-Instruct. The authors report that LLM-as-a-Judge + RL reaches excellence rates of 92.4/96.6/95.0 on high-frequency greetings and 91.0/93.4/92.4 on ordinary greetings under Signal-1, Signal-2, and human evaluation, outperforming SFT+RM+RL and external LLMs. Ablations quantify the contribution of each agent in the two evaluation frameworks.

Significance. If fully supported, the paper would make a useful practical contribution: it provides a largely AI-driven pipeline for improving creative generation in modest-size models, with released code/data and a human-evaluation protocol. The multi-agent preference-data curation and the idea of a reflection-augmented adversarial judge are interesting, and the ablation study is informative. However, the headline 'demonstrably superior' conclusion is not yet established: the reported margins are within evaluation noise, the RM+RL condition is missing, and the adversarial training is not shown to have actually produced the final judge prompt. The significance is therefore conditional on additional evidence.

major comments (5)
  1. [§5.2–5.3, Tables 2–3] The central claim that LLM-as-a-Judge + RL 'demonstrably yields superior generation quality' is not supported by the reported statistics. All comparisons are point estimates from a 2,000-item set; for a binary rate near 95%, the standard error is roughly ±0.9 percentage points, so head-to-head margins of 0.2–0.6 points (Table 2) are within evaluation noise. The automated judges themselves have ~13–15% label error (Table 1), and no confidence intervals, significance tests, or inter-annotator agreement are reported for the human column. Please add uncertainty quantification (e.g., bootstrap CIs, McNemar tests for paired comparisons) and report annotation reliability.
  2. [§3.2, Appendix A.4, Figs. 14–15] The paper's second, 'more novel' contribution is the adversarial optimization of the judge prompt, but no evidence is given that adversarial training actually changes the prompt. Appendix A.4 describes strategy updates, yet the final prompt in Figs. 14–15 is a hand-authored list of 10 principles; there is no training curve for the detector/generator, no accuracy trajectory, and no comparison of the final prompt to the initial strategy. As written, the reported gains could be due entirely to the hand-written principles, not to the adversarial+reflection loop. Please include prompt-evolution traces, detector accuracy over training, and a non-adversarial baseline using the same final principles.
  3. [§4.2, Tables 2–3] Reward Model + RL is excluded because 'training not converging', leaving SFT+RM+RL as the only RM-based comparator. Since the paper's core comparison is between two reward signals, omitting the direct RM+RL condition—without training curves or failure analysis—makes the efficiency and superiority claims incomplete. Please report the convergence failure in detail or include a stabilized RM+RL run.
  4. [§3.3, Table 2] The Signal-2 evaluation uses the same detector family that supplies the RL reward, and Signal-1 is produced by the same multi-agent framework that generated the RM's preference data. This creates a training–evaluation loop that can inflate apparent gains through reward overfitting. Human evaluation is the only fully external check, but it is reported only as a point estimate. Please evaluate with a held-out judge variant or quantify the risk by measuring agreement between the training judge and an independent judge on the final set.
  5. [§4.3, Tables 2–3] The binary 'excellence' label is defined by a weighted-score threshold of ≥2.0 (§4.3). All automated and human evaluation numbers, and the reward labels used for training, depend on this threshold, but no sensitivity analysis is provided. The 0.2–0.6pp differences in Table 2 may be threshold artifacts. Please report results across a range of thresholds or use continuous scores.
minor comments (6)
  1. [Figure 2] The agreement-rate figure would benefit from numeric values, sample sizes, and confidence intervals; currently the visual comparison lacks the precision needed to support the 80–87% range claimed in §5.1.
  2. [§4.2] The 'high-quality' and 'low-quality' labels for the final evaluation set are heuristically derived from click-through and replication rates. This is a weak gold standard for creative quality; please provide validation or acknowledge the limitation more explicitly.
  3. [Abstract and §1] The abstract and introduction refer only to 'Github'; the full URL should appear in the main text, not only in the abstract.
  4. [Table 1] The Adversarial Framework has precision 78.54% and recall 97.70%. This asymmetry should be discussed, since it suggests the Signal-2 judge may be systematically lenient, which has direct implications for the reported excellence rates.
  5. [Figure 1] The 'Signal-1' and 'Signal-2' labels in the figure are not defined in the caption; please define them so the figure is self-contained.
  6. [Appendix A.6] The entropy loss is reported to increase during GRPO training. This is unusual and should be explained, since increasing entropy is not obviously consistent with policy convergence.

Circularity Check

1 steps flagged · score 6.0 of 10

Signal-2 evaluation reuses the training reward, making one reported metric self-confirming; human evaluation gives partial independent support.

  1. fitted input called prediction [Section 3.3 (reward definition) and Table 2 caption / Section 5.3 (Signal-2 evaluation claim)]
    "The output score sD = fD(p, r; θD) from the adversarially trained and reflector-enhanced Detector (detailed in Section 3.2) is used directly as a reward signal, RD(p, r) = fD(p, r; θD). ... Here, Signal-1 refers to Section 3.1, Signal-2 refers to Section 3.2, and Human refers to the evaluation by human experts."

    The Signal-2 evaluation column is produced by the same adversarial detector fD that defines the training reward for LLM-as-a-Judge + RL. Since GRPO maximizes expected reward RD = fD, the reported Signal-2 excellence rate for that model is the training objective itself, not an independent measurement. Comparing this score to models trained on a different reward (RM) makes the apparent advantage on Signal-2 partly forced by construction. Human and Signal-1 columns are external to this loop and provide independent evidence, so the paper is only partially circular.

full rationale

The main reduction is visible from the paper's own equations: RD(p,r)=fD(p,r;θD) in Section 3.3, and the Signal-2 evaluation column is the same adversarial detector (Table 2 caption: 'Signal-2 refers to Section 3.2'). For LLM-as-a-Judge + RL, the 96.6% Signal-2 excellence rate is therefore the reward that GRPO was trained to maximize, so that particular metric is partly self-confirming. The comparison to RM-trained models on this axis is inflated because those models were optimized for a different reward. The paper is not wholly circular: the Human column (95.0% vs 94.6%) and the Signal-1 column are external to the Signal-2 training loop and point in the same direction, so the central claim retains independent content. The multi-agent Signal-1 evaluator is also the source of the RM's training preferences, creating a similar but less direct same-source issue for the RM-based method. The Limitations section concedes that the reflection mechanism's depth and impact are unverified, but that is a missing-evidence issue rather than circularity. No load-bearing self-citation or imported uniqueness theorem was found. Score 6 reflects partial circularity of the Signal-2 metric, not a fully forced derivation.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The central empirical comparison rests on hand-chosen rubric weights, an arbitrary binary threshold, and hand-written initial strategies for the adversarial generator and detector. The reward model is trained on preference data labeled by the multi-agent judge, and the evaluator frameworks are built from the same principles, so the evaluation is partly self-referential. The 'excellence' metric is a constructed composite, not an independent measurement.

free parameters (5)
  • Evaluation rubric weights = Language 30%, Creativity 30%, Emotional Resonance 15%, Cultural Appropriateness 15%, Content Richness 10%
    Chosen by hand in Section 4.3, not derived; the 'excellence rate' depends directly on these weights.
  • Excellence threshold = weighted score >= 2.0
    Set ad hoc in Section 4.3; a response with 2/3 on every dimension passes, so the label is closer to 'acceptable' than 'excellent'.
  • LLM-as-a-Judge principles = 10 hand-written strategies (Appendix A.4, Figures 14-15)
    Initial strategies for the detector; the paper does not show that adversarial training changed these principles.
  • Generator initial strategy = e.g., generate a greeting using at least one greeting phrase that sounds slightly archaic
    Hand-chosen in Appendix A.4; influences the adversarial training trajectory.
  • Detector initial strategy = e.g., assess whether the greeting conveys sincere emotion rather than being a stock phrase
    Hand-chosen in Appendix A.4; determines the reward signal before any training.
assumptions (4)
  • domain assumption Click-through rate and replication rate are valid proxies for greeting quality
    Used to construct the 'high-quality' and 'low-quality' halves of the evaluation set (Section 4.2). If engagement is not aesthetic quality, the benchmark is invalid.
  • domain assumption The five rubric dimensions with the given weights capture creative writing quality
    Adopted in Section 4.3 without empirical justification; all 'excellence' claims are conditional on this rubric.
  • domain assumption Multi-agent debate reduces evaluation bias and improves accuracy over single-agent judgment
    Basis for the multi-agent rejection sampling framework (Section 3.1); supported only by the paper's own ablation on its own evaluation set.
  • domain assumption Human evaluator judgments are ground truth for greeting excellence
    Standard for this type of paper; human evaluation is used to validate the AI frameworks, but no inter-annotator agreement is reported (Section 5.1).
invented entities (2)
  • Multi-agent debate agents (Positive, Negative, Judge, Reflect)
    purpose: Generate and filter preference data for the reward model (Section 3.1); also used as the Signal-1 evaluation framework.
    Software components introduced by the paper. The ablation in Table 4 shows they affect classification on the paper's own evaluation set, but there is no independent evidence of their general validity.
  • Reflector module in adversarial reward optimization
    purpose: Provide structured feedback to the detector when it misclassifies a response (Section 3.2).
    A new component in the adversarial pipeline; only evidence is the ablation in Table 4 showing a drop in F1 when removed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewards." pith.science (2026). https://pith.science/paper/P55SUY5Y

@misc{pith2026250821476,
  author       = {Pith},
  title        = {Pith review of: Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewards},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P55SUY5Y}},
  note         = {Machine review of arXiv:2508.21476}
}
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable creative writing capabilities, yet their substantial computational demands hinder widespread use. Enhancing Small Language Models (SLMs) offers a promising alternative, but current methods like Supervised Fine-Tuning (SFT) struggle with novelty, and Reinforcement Learning from Human Feedback (RLHF) is costly. This paper explores two distinct AI-driven reward strategies within a Reinforcement Learning from AI Feedback (RLAIF) framework to ignite the creative writing of a 7B-parameter SLM, specifically for generating Chinese greetings. The first strategy employs a RM trained on high-quality preference data curated by a novel multi-agent rejection sampling framework designed for creative tasks. The second, more novel strategy utilizes a principle-guided LLM-as-a-Judge, whose reward function is optimized via an adversarial training scheme with a reflection mechanism, to directly provide reward signals. Comprehensive experiments reveal that while both approaches significantly enhance creative output over baselines, the principle-guided LLM-as-a-Judge demonstrably yields superior generation quality. Furthermore, it offers notable advantages in training efficiency and reduced dependency on human-annotated data, presenting a more scalable and effective path towards creative SLMs. Our automated evaluation methods also exhibit strong alignment with human judgments. Our code and data are publicly available at https://github.com/weixiaolong94-hub/Igniting-Creative-Writing-in-Small-Language-Models.

Figures

Figures reproduced from arXiv: 2508.21476 by the authors.

Figure 1
Figure 1. The figure depicts two distinct reward signals. Signal 1 is derived from a multi-agent system, yielding a [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Comparison of agreement rate between dif [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Training metrics of LLM-as-a-Judge + RL. [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: An example of positive agent and negative agent. Given query and response, they generate advantages and [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: An example of positive agent and negative agent. Given query and response, they generate advantages and [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Prompt for the Positive Agent [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Prompt for the Positive Agent [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Prompt for the Negative Agent [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Prompt for the Negative Agent [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Prompt for the Judge Agent [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Prompt for the Judge Agent [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Prompt for the Reflect Agent [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Prompt for the Reflect Agent [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: Prompt for the LLM-As-Judge principles [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: Prompt for the LLM-As-Judge principles [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 47 canonical work pages

  1. [1]

    Language Accuracy and Elegance: Assess whether the language used is standard and precise; evaluate the wording for its elegance and vi vidness; check if appropriate and expressive words, similes, or metaphors are used to enhance the infectiousness of the greeting

    work page

  2. [2]

    Expression Fluency and Naturalness: Assess whether the sentences are smooth, fluent, and natural; whether they are pleasant to read or h ear; whether the transitions between sentences are tight and logical

    work page

  3. [3]

    Hellobench: Evaluating long text generation capabilities of large language models. CoRR. Chris Richardson, Yao Zhang, Kellen Gillespie, Sudipta Kar, Arshdeep Singh, Zeynab Raeesy, Omar Zia Khan, and Abhinav Sethy. 2023. Integrating sum- marization and retrieval for enhanced personaliza- tion via large language models. arXiv preprint arXiv:2310.20081. Chri...

    arXiv 2023

  4. [4]

    Creativity and Uniqueness: Assess whether the content is novel and creative; whether it uses unique ways of expression, perspectives, or st ructures that differ from common f, leaving a deep impression on the recipient

    work page

  5. [5]

    Adversarial Reward Sig- nal Optimization with Reflection

    Self-rewarding language models. arXiv preprint arXiv:2401.10020, 3. Shuang Zeng, Xinyuan Chang, Mengwei Xie, Xin- ran Liu, Yifan Bai, Zheng Pan, Mu Xu, and Xing Wei. 2025. Futuresightdrive: Thinking visually with spatio-temporal cot for autonomous driving. arXiv preprint arXiv:2505.17685. Yiming Zhang, Harshita Diddee, Susan Holm, Hanchen Liu, Xinyue Liu,...

    arXiv 2025

  6. [6]

    Appropriateness and Suitability: The content, tone, and style of the greeting should perfectly match the current context, occasion, the recipie nt, and your relationship with them, making it feel very considerate and suitable

    work page

  7. [7]

    Positive Wording: Assess whether the wording is positive and full of positive energy, effectively conveying encouragement, support, praise, o r beautiful wishes, while completely avoiding any negative or potentially uncomfortable implications

    work page

  8. [8]

    Sincere and Full Emotion: Assess whether the emotion expressed is genuine and from the heart; whether it can effectively convey warmth, c are, joy, and other positive emotions; whether it can move people and make them feel the sincerity of the well-wisher

    work page

Show all 51 references
  1. [9]

    Merit 1":

    Other Highlights: Any other positive aspect that cannot be categorized into the above but clearly enhances the quality of the greeting and ma kes it more valuable. [[The Output Format]] Present in JSON format: ```json { "Merit 1": "Reason 1", "Merit 2": "Reason 2", ... } ``` [...

  2. [10]

    tailor-made for them

    Content Specificity and Targetedness: A greeting has strong personalization when it includes specific details about the recipient's personal c haracteristics, achievements, experiences, or shared memories, making the recipient feel that this greeting is "tailor-made for them"

  3. [13]

    Pure Intention: The greeting purely expresses well-wishes, without being mixed with any other objectives, requests, or promotional content

  4. [15]

    Text/Grammar Errors: Including typos, missing words, extra words, inappropriate word choice, punctuation errors, grammatical structure erro rs, etc

  5. [16]

    Fluency of Expression: Whether the sentences are awkward or unnatural, whether the transitions between sentences are abrupt, and wheth er there is unnecessary repetition or wordiness

  6. [17]

    Emotional Depth/Sincerity: Whether the emotional expression is bland or hollow, lacking genuine feeling, overly formulaic like a template, an d missing personalized warmth

  7. [18]

    Creativity/Novelty: Whether the content is cliché and lacks creativity, highly similar to other common greetings, and without unique ideas or ways of expression

  8. [19]

    Whether there are logical inconsistencies, inappropriate wording, or ill-timed remarks

    Logic/Appropriateness: Whether the content of the greetings is suitable for the context, occasion (e.g., birthday, holiday, promotion, recovery ), the recipient, and your relationship with them. Whether there are logical inconsistencies, inappropriate wording, or ill-timed remarks

  9. [20]

    Vagueness/Lack of Specificity: Whether the greetings are too generic, feeling like a template that could be sent to anyone, and lacking speci fic details about the recipient's personal traits, achievements, or current situation

  10. [21]

    Inappropriate Wording/Negative Implications: Whether it uses words or expressions that could cause misunderstanding, discomfort, contain negative connotations, create pressure (e.g., urging marriage, childbearing, excessive expectations), or sound sarcastic

  11. [22]

    Cultural or Customary Taboos: Whether it violates any cultural taboos or customary improprieties relevant to the recipient or the current cont ext

  12. [23]

    Flaw 1":

    Other Quality-Impacting Issues: Any other aspects that might diminish the expressive effect, sincerity, and quality of the greetings. [[The Output Format]] Present in JSON format: ```json { "Flaw 1": "Reason 1", "Flaw 2": "Reason 2", ... } ``` [[Evaluation Example]] {"query": ...

  13. [24]

    Based entirely on the provided input information, comprehensively weigh the strengths (positive) and weaknesses (negative) of the greeting to make a final quality judgment (0 for bad, 1 for good)

  14. [25]

    Generate a concise reason explaining why you made this judgment

  15. [26]

    New Year's greeting for the lover in the Year of the Snake

    Output the judgment result (0 or 1) and the reason in the specified JSON format. Note: The standard for judging a greeting as 1 is very high; it can only be rated as 1 if it is excellent in all aspects. [[Input Information]] query: A string describing the application scenario ...

  16. [27]

    Based entirely on the provided input information, re-examine the initial judgment (initial_judge) and reason (initial_reason)

  17. [28]

    Considering the greeting's strengths (positive) and weaknesses (negative), independently think about whether the initial judgment is reason able and if there are any biases

  18. [29]

    Make your final quality judgment (0 for bad, 1 for good)

  19. [30]

    Generate a concise final reason, explaining why you made the final judgment

  20. [31]

    judge": 0 | 1,

    Output the final judgment result (judge) and the final reason (reason) in the specified JSON format. Note: The standard for you to change the initial_judge is very strict. You can only change the judgment if there are clear and sufficient reasons to believe the initial judgmen...

  21. [32]

    **评估情感⽀持效果：** 区分'社交礼仪性祝福'和'深度情感⽀持'需求，前者只需满⾜基本⿎励即可。

  22. [33]

    **评估情境贴合度：** 商务场景中，简洁、正⾯且符合基本礼仪的祝福语视为合格，⽆需⾼度个性化。

  23. [34]

    **评估情感真诚度：** 判断祝福语是否⾃然传递真实情感，复杂场景⿎励具体事例或回忆增强可信度。

  24. [35]

    **评估语⾔得体性：** 检查⽤词和语⽓是否匹配双⽅关系及场合，基础场景需符合社会礼仪标准。

  25. [36]

    **评估⽂化适宜性：** 避免违背⽂化禁忌，确保祝福语符合通⽤⽂化习惯。

  26. [37]

    **评估流畅度：** 排除语法错误和逻辑断层，保证语⾔⾃然流畅。

  27. [38]

    **评估个性化程度：** 亲密关系或特殊场合需包含接收者细节并设定更⾼权重，基础场景接受通⽤表达不扣分。

  28. [39]

    **评估原创性：** 复杂场景⿎励新颖⽐喻并设定更⾼权重，基础场景避免陈词滥调即可。

  29. [40]

    **评估内容丰富度：** 确保祝福语在简洁的同时能传达更丰富的情感和个性化信息。

  30. [41]

    judge":1|0,

    **评估综合表现：** 若祝福语过于通⽤或缺乏原创性，即使其他⽅⾯表现良好，也应考虑降低其评级。 [[评分输出格式]] ⽤JSON格式呈现： ```json { "judge":1|0, "reason":给出判断理由 } ``` [[评估对象]] content Figure 14: Prompt for the LLM-As-Judge principles. LLM-as-a-Judge principles [[Task Definition]] You are a professional "Greetings Evaluation Ma...

  31. [42]

    **Evaluate Emotional Support Effect:** Differentiate between 'social etiquette greetings' and 'deep emotional support' needs; the former only needs to meet basic encouragement standards

  32. [43]

    **Evaluate Contextual Fit:** In business contexts, a greeting that is concise, positive, and adheres to basic etiquette is considered compete nt, without needing high levels of personalization

  33. [44]

    **Evaluate Emotional Sincerity:** Judge whether the greeting naturally conveys authentic emotions; complex scenarios are encouraged to u se specific examples or memories to enhance credibility

  34. [45]

    **Evaluate Linguistic Appropriateness:** Check if the word choice and tone match the relationship between the parties and the occasion; fo undational scenarios must meet social etiquette standards

  35. [46]

    **Evaluate Cultural Appropriateness:** Avoid violating cultural taboos and ensure the greeting aligns with common cultural customs

  36. [47]

    **Evaluate Fluency:** Eliminate grammatical errors and logical gaps to ensure the language is natural and fluent

  37. [48]

    **Evaluate Personalization Level:** Close relationships or special occasions require including details about the recipient and are weighted h igher; accepting generic expressions in basic scenarios will not result in a penalty

  38. [49]

    **Evaluate Originality:** Complex scenarios are encouraged to use novel metaphors and are weighted higher; foundational scenarios simpl y need to avoid clichés

  39. [50]

    **Evaluate Content Richness:** Ensure the greeting, while concise, can convey rich emotions and personalized information

  40. [51]

    judge": 1 | 0,

    **Evaluate Overall Performance:** If a greeting is too generic or lacks originality, even if it performs well in other aspects, consider lowering its rating. [[The Output Format]] Present the output in JSON format: ```json { "judge": 1 | 0, "reason": "Provide the reason for th...

  41. [2022]

    Advances in neural in- formation processing systems, 35:27730–27744

    Training language models to follow instruc- tions with human feedback. Advances in neural in- formation processing systems, 35:27730–27744. Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isab...

    2024 arXiv

  42. [2023]

    arXiv preprint arXiv:2310.15123

    Branch-solve-merge improves large language model evaluation and generation. arXiv preprint arXiv:2310.15123. Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Mult...

    2021 arXiv

  43. [2024]

    In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 16366–16393

    Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 16366–16393. Tarek Naous and Wei Xu. 2025. On the origin of cultural biases i...

    2025 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.