Pith. sign in

REVIEW 4 major objections 5 minor 35 references

LCTG Bench: LLM Controlled Text Generation Benchmark

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LCTG Bench is a Japanese benchmark that tests whether LLMs obey format, length, keyword, and banned-word constraints across three tasks; its evaluation finds GPT-4 on top and Japanese-specific models trailing.

desk verdict Useful first Japanese controllability benchmark; rule-based controllability results likely hold, but GPT-4-as-judge quality scores are the weak link. read the letter →

arxiv 2501.15875 v1 pith:YYXW5YZU submitted 2025-01-27 cs.CL

Kentaro Kurihara , Masato Mita , Peinan Zhang , Shota Sasaki , Ryosuke Ishigami , Naoaki Okazaki This is my paper · ORCID
classification cs.CL
keywords JapaneseLLMbenchmarkcontrollabilityevaluationinstructionfollowingrule-basedcontrolledtextgenerationcharactercountconstraintmultilingualvsmodelsquality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LCTG Bench is the first Japanese benchmark designed to measure how well LLMs follow concrete output constraints, not just how fluent or knowledgeable their text is. It uses three business-style tasks—summarization, ad text generation, and pros & cons generation—and scores every model on the same four rule-based checks: format, character count, keyword use, and banned-word avoidance. Using this framework, the paper finds GPT-4 leading nine evaluated models and Japanese-specific models trailing multilingual ones by a wide margin. If the framework is sound, it gives developers a reusable tool for choosing Japanese LLMs according to the specific constraints their application needs to enforce, and it identifies character-count control as a weak point for all current models.

What carries the argument

The load-bearing mechanism is a unified set of four rule-based constraints—Format, Character count, Keyword, and Prohibited word—applied identically to all three generative tasks. Each constraint is checked by surface rules: boundary matching for format, character counting for length, string search for keyword inclusion, and string search for banned-word exclusion. Because the same checks run across tasks, scores are directly comparable and do not depend on gold reference outputs. A GPT-4-based cleaning step removes extraneous explanatory text before scoring, and a separate GPT-4-based quality gate verifies that condition-satisfying outputs still meet the task's content requirements.

What would settle it

Re-annotate the same generated outputs with trained human judges who do not know which model produced them; if Japanese-specific models are judged appropriate as often as GPT-4, the claimed performance gap is refuted.

Watch

Extended reading notes

Core claim

The paper introduces LCTG Bench, which it argues is the first Japanese benchmark for evaluating LLM controllability. The benchmark contains three generative tasks—summarization, ad text generation, and pros & cons generation—each scored under the same four verifiable rule-based perspectives: Format, Character count, Keyword, and Prohibited word. Across nine evaluated models, including GPT-3.5, GPT-4, Gemini, and several Japanese-specific models, GPT-4 achieves the highest average controllability and quality scores in almost every task. Japanese-specific models show a large performance gap compared to multilingual models, even when their output quality scores are relatively high. Character-count control is low for all models, which the paper attributes to token-level tokenizers struggling with character-level constraints.

Load-bearing premise

The evaluation assumes GPT-4 is a neutral, accurate judge of output quality and a reliable cleaner of extraneous text, even when judging GPT-4's own outputs; the paper's own agreement checks show only moderate agreement with human judges for two of the three tasks.

Editorial extensions

If this is right

  • Developers can select a Japanese LLM based on which constraint matters most: format, character count, keyword use, or banned-word avoidance.
  • Because the same four checks apply to every task, the scores can transfer as rough guidance to other constrained text-generation use cases.
  • Character-count control is a common failure mode across all evaluated models, so downstream applications should post-process length or treat it as a separate capability.
  • Quality-only evaluation is insufficient: models with high quality scores can still fail controllability, so both dimensions need to be reported.
  • The gap between multilingual and Japanese-specific models marks instruction-following, not raw generation quality, as the main target for improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication not pursued in the paper: the same condition-template collection process could be reused to build controllability benchmarks for other under-resourced languages, since no gold outputs are required.
  • The aligned keyword and prohibited-word scores within Japanese models suggest a shared difficulty with negative instructions; a focused probe on negation could isolate that cause without running the full benchmark.
  • Because the quality evaluator is GPT-4, the reported quality gap may partially reflect judge bias; a blinded human re-judgment of the same outputs would test whether the gap is real.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces LCTG Bench, a Japanese-language benchmark for evaluating the controllability of large language models. It defines three generation tasks (summarization, ad text generation, and pros & cons generation) and four rule-based controllability dimensions (format, character count, keyword inclusion, and prohibited-word exclusion). Nine models are evaluated, including Japanese-specific models and multilingual models such as GPT-4, GPT-3.5, and Gemini-Pro. The authors report that GPT-4 achieves the highest average scores on most tasks and that Japanese-specific models generally show a significant performance gap, particularly on character-count control. They also use GPT-4 as a judge of generation quality and as a tool for removing extraneous explanatory text, and they validate both procedures with human annotations.

Significance. If the benchmark is sound, it fills a genuine gap: there is no established Japanese benchmark focused specifically on controllability. The four controllability metrics are objective, rule-based, and task-uniform, which makes the benchmark reusable for model selection in business applications. The release of the dataset and the crowdsourced condition templates are concrete assets. The paper also provides a useful validation of the text-cleaning step via human comparison. However, the paper's stronger claim of a 'significant gap' between GPT-4 and Japanese models in both controllability and output quality depends on the GPT-4-based quality scores, which are only moderately validated (Cohen's kappa 0.410 for summarization and 0.308 for ad text generation, with low human-human agreement). The controllability-only results are more robust and stand as the main contribution. The overall significance therefore hinges on whether the quality-evaluation methodology can be strengthened or its claims appropriately qualified.

major comments (4)
  1. [§4.1, §5, Table 5] The quality evaluation uses GPT-4 as the judge for all models, including GPT-4 itself, and the validation in Table 5 shows only moderate agreement between GPT-4 and human evaluators for summarization (kappa = 0.410) and ad text generation (kappa = 0.308), with low human-human agreement as well (Fleiss kappa 0.259 and 0.277). The abstract and Section 4.2 conclude that GPT-4 shows superior performance 'both in terms of controllability and quality' and that there is a 'significant gap' between GPT-4 and Japanese models. The quality component of this claim is load-bearing and is not robustly supported. I recommend reporting controllability-only average scores, adding a self-preference analysis (e.g., comparing GPT-4's judgments of its own outputs vs. other models' outputs), and either using an additional independent judge or substantially tempering the quality-based conclusions.
  2. [§4.1, FORMAT metric] The FORMAT score is computed as the exact match of the first and last ten (or five) characters of the raw output and the GPT-4-cleaned output. This is an indirect proxy: it assumes that any extraneous explanatory text appears only at the beginning or end and that GPT-4 always removes exactly those parts. The validation in Table 6 reports high Levenshtein similarity overall, but that does not directly measure whether the FORMAT decision is correct for each sample, especially for models with low FORMAT scores. I request a per-sample error analysis of the FORMAT metric, or at least a discussion of how errors in GPT-4-based cleaning could affect the FORMAT scores across models.
  3. [§3.2, KEYWORD/P-WORD construction] Keywords are extracted from summaries generated by GPT-4, and prohibited words are selected from those same high-importance words. Since GPT-4 is also the highest-scoring model in the evaluation, this creates a potential advantage for GPT-4 in the KEYWORD and P-WORD dimensions: the test words are derived from GPT-4's own output distribution. The paper does not test the sensitivity of the results to the choice of keyword-extraction model. I recommend either validating with keywords extracted by another method or demonstrating that the relative ranking is unchanged when keywords are selected from a non-GPT-4 source.
  4. [§5, quality evaluation validation] The human validation of the quality evaluation uses a majority vote of five annotators as the gold standard, but the Fleiss kappa values in Table 5 are low (0.259 for summarization, 0.277 for ad text generation). With human agreement this low, the majority vote is a weak gold standard, and the reported GPT-4-human kappa values should be interpreted with caution. The paper does acknowledge this in the discussion, but the abstract and conclusion do not carry the same caveat. Please make the limitations of the quality component explicit in the abstract and conclusion, or strengthen the validation.
minor comments (5)
  1. [Appendix F, Figure 11] The caption of Figure 11 reads 'Prompt used for removing unnecessary explanatory phrases in pros & cons generation,' but the prompt shown is the quality-evaluation prompt (it asks for an 'appropriate/inappropriate' judgment of whether both pros and cons are mentioned). Please correct the caption.
  2. [§4.1] The sentence describing that controllability and quality evaluations are performed on GPT-4-cleaned outputs is long and can be misread as applying to FORMAT as well. Please rephrase to clarify that FORMAT is measured by comparing pre- and post-cleaning outputs, while the other three controllability dimensions and quality are measured on the cleaned text.
  3. [Table 1] The row 'Avg.char count' should specify whether the character count includes whitespace, line breaks, or punctuation. This matters for interpreting the C-COUNT results, since models may count characters differently.
  4. [§3.2] The paper does not define how 'characters' are counted for the C-COUNT conditions (e.g., Unicode code points, full-width vs. half-width forms). A precise definition would improve reproducibility, especially for Japanese text with mixed-width characters.
  5. [Introduction and Appendix C] The text 'LCTG B ENCH' in Section 1 has an unintended space. Also, footnote markers 12–17 in Appendix C appear as superscripts on model names; check that the rendering is consistent with the journal style.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark's controllability metrics are rule-based, and the GPT-4-based quality judgment is a validated operationalization rather than a derivation from its own outputs.

full rationale

LCTG Bench is an evaluation study, not a derivation, and its claimed results do not reduce to their inputs by construction. The controllability metrics (FORMAT, C-COUNT, KEYWORD, P-WORD) are defined by external rule-based checks (Section 3.3) and are independent of any fitted parameter or model output; the cleaning step is validated against manual removal (Table 6). The headline gap between GPT-4 and Japanese models is visible in these objective controllability scores and does not depend solely on the quality judgment. The quality scores are explicitly operationalized as GPT-4 classifications (Section 4.1: 'we use GPT-4 as the evaluator'), so the observation that GPT-4 scores highest on quality is a report of that metric; the paper's own Table 5 shows moderate GPT-4-human agreement for summarization (kappa=0.410) and ad text generation (kappa=0.308), which is a validity and limitation concern, not a circularity. The keyword extraction from GPT-4 summaries (Section 3.2) may favor GPT-4 in the KEYWORD dimension, but it is a dataset-construction choice, not an equation or fitted-parameter prediction. Self-citations (Kurihara et al. 2022; Mita et al. 2023) are used as related work or as a data source and are not load-bearing. No step in the paper is equivalent to its inputs by definition, and no fitted input is renamed as a prediction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The benchmark introduces no new theoretical entities or fitted parameters in the physical-science sense. The free parameters listed here are design choices in the evaluation protocol. The axioms are the unstated or partially validated assumptions that the evaluation pipeline (GPT-4-based cleaning, keyword extraction, and quality judging) works as intended.

free parameters (4)
  • FORMAT match length = 10 characters (5 for ad text)
    Chosen by hand in Section 4.1 to compare original and cleaned outputs. This threshold directly determines whether a FORMAT violation is detected.
  • Character count bounds per task = 50-200 (summarization), 20-50 (ad text), 100-400 (pros/cons)
    Set in Section 3.2 as design choices for condition templates. These bounds define C-COUNT difficulty but are not fitted to any external target.
  • Number of GPT-4 summaries for keyword extraction = 5
    Chosen in Section 3.2 when extracting high-importance keywords. A different count could change the keyword sets and consequently KEYWORD/P-WORD scores.
  • Generation repetitions per prompt = 3
    Chosen in Section 4.1 to average over output variability. The reported scores are means over three generations with no variance reported.
assumptions (4)
  • domain assumption Words appearing commonly in GPT-4-generated summaries are high-importance keywords for the document.
    Stated in Section 3.2 as the basis for KEYWORD and P-WORD construction. No independent evidence is given that these words are indeed the most important for controllability testing.
  • domain assumption GPT-4 can reliably remove unnecessary explanatory text from any model's output without altering the task-relevant content.
    Assumed in Section 4.1 and used for the FORMAT comparison and for cleaning before other controllability checks. The paper validates this against manual removal in Table 6, with high similarity, but the validation is limited to a 10% sample.
  • domain assumption GPT-4's binary 'appropriate/inappropriate' quality judgment is a valid proxy for human quality judgment.
    Assumed in Section 4.1 and Appendix F. Table 5 shows only moderate Cohen's kappa for summarization and ad text, so this assumption is only partially supported.
  • domain assumption Exact matching of the first and last ten characters before and after cleaning is sufficient to detect unwanted explanatory text.
    Used in Section 4.1 to score FORMAT. This is a coarse proxy that ignores possible explanatory text in the middle of outputs and depends on the cleaning step being perfect.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LCTG Bench: LLM Controlled Text Generation Benchmark." pith.science (2026). https://pith.science/paper/YYXW5YZU

@misc{pith2026250115875,
  author       = {Pith},
  title        = {Pith review of: LCTG Bench: LLM Controlled Text Generation Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YYXW5YZU}},
  note         = {Machine review of arXiv:2501.15875}
}
read the original abstract

The rise of large language models (LLMs) has led to more diverse and higher-quality machine-generated text. However, their high expressive power makes it difficult to control outputs based on specific business instructions. In response, benchmarks focusing on the controllability of LLMs have been developed, but several issues remain: (1) They primarily cover major languages like English and Chinese, neglecting low-resource languages like Japanese; (2) Current benchmarks employ task-specific evaluation metrics, lacking a unified framework for selecting models based on controllability across different use cases. To address these challenges, this research introduces LCTG Bench, the first Japanese benchmark for evaluating the controllability of LLMs. LCTG Bench provides a unified framework for assessing control performance, enabling users to select the most suitable model for their use cases based on controllability. By evaluating nine diverse Japanese-specific and multilingual LLMs like GPT-4, we highlight the current state and challenges of controllability in Japanese LLMs and reveal the significant gap between multilingual models and Japanese-specific models.

Figures

Figures reproduced from arXiv: 2501.15875 by the authors.

Figure 1
Figure 1. Overview of LCTG Bench. bility. For evaluating controllability in this study, we also evaluated the generated text using GPT-4 to confirm that the generated text meets the task requirements to some extent. The evaluation of var￾ious LLMs using LCTG Bench shows the current status and issues in the controlled text generation of Japanese LLMs, including the large gap between GPT-4 and Japanese LLMs. 2 Related Work In E… view at source ↗
Figure 2
Figure 2. An example of summarization prompt (Character count). [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An example of ad text generation prompt (Keyword). [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: An example of pros & cons generation prompt (Prohibited word). In this example, “ [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: An Example of LLM output in ad text generation that includes irrelevant explanations at the beginning [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Prompt used for removing unnecessary explanatory phrases in summarization. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Prompt used for removing unnecessary explanatory phrases in ad text generation. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Prompt used for removing unnecessary explanatory phrases in pros & cons Generation. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Prompt used for evaluating the quality of generation in summarization. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Prompt used for evaluating the quality of generation in ad text generation. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Prompt used for removing unnecessary explanatory phrases in pros & cons generation. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 5 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  4. [4]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  5. [5]

    1972", volume =

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 1972

  6. [6]

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. https://arxiv.org/abs/2204.06745 GPT - N eo X - 20B : An open-source autoregressive language m...

    arXiv 2022

  7. [7]

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. https://arxiv.org/abs/2308.07201 Chat E val: Towards better LLM -based evaluators through multi-agent debate . Preprint, arXiv:2308.07201

    arXiv 2023

  8. [8]

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. GPTS core: Evaluate as you desire. arXiv:2302.04166

    arXiv 2023

Show all 35 references
  1. [9]

    Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. https://doi.org/10.5281/zenodo.53...

    2021 doi

  2. [10]

    Gemini-Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, and Anja Hauth et al. 2024. https://arxiv.org/abs/2312.11805 Gemini: A family of highly capable multimodal models . Abs/2312.11805

    2024 arXiv

  3. [11]

    Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M

    Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. https://doi.org/10.18653/v1/2021.findings-acl.413 XL - S um: Large-scale multilingual abstractive summarization for 44 languages . In F...

    2021 doi

  4. [12]

    Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Lida Chen, Xintao Wang, Yuncheng Huang, Haoning Ye, Zihan Li, Shisong Chen, Yikai Zhang, Zhouhong Gu, Jiaqing Liang, and Yanghua Xiao. 2024. Can large language models understand real-world complex ...

    2024

  5. [13]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . Abs/2009.03300

    2021 arXiv

  6. [14]

    Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2023. https://arxiv.org/abs/2310.20410 F ollow B ench: A multi-level fine-grained constraints following benchmark for large language models . Abs/2310.20410

    2023 arXiv

  7. [15]

    Yimin Jing, Renren Jin, Jiahao Hu, Huishi Qiu, Xiaohua Wang, Peng Wang, and Deyi Xiong. 2023. https://arxiv.org/abs/2311.09829 F ollow E val: A multi-dimensional benchmark for assessing the instruction-following capability of large language models . Abs/2311.09829

    2023 arXiv

  8. [16]

    Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. 2022. https://aclanthology.org/2022.lrec-1.317 JGLUE : J apanese general language understanding evaluation . In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2957--2966, Marseille, Fr...

    2022

  9. [17]

    Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan

    Yixin Liu, Alexander R. Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan. 2023. https://arxiv.org/abs/2311.09184 Benchmarking generation and evaluation capabilities of large language models for instruction c...

    2023 arXiv

  10. [18]

    Joshua Maynez, Priyanka Agrawal, and Sebastian Gehrmann. 2023. https://doi.org/10.18653/v1/2023.acl-long.511 Benchmarking large language model capabilities for conditional generation . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (...

    2023 doi

  11. [19]

    Masato Mita, Soichiro Murakami, Akihiko Kato, and Peinan Zhang. 2023. https://arxiv.org/abs/2309.12030 CAMERA : A multimodal dataset and benchmark for ad text generation . Abs/2309.12030

    2023 arXiv

  12. [20]

    Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2023. https://arxiv.org/abs/2401.00595 State of what art? A call for multi-prompt LLM evaluation . Abs/2401.00595

    2023 arXiv

  13. [21]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 GPT -4 technical report . ArXiv, abs/2303.08774

    2023 arXiv

  14. [22]

    Zhang, Mark Harman, and Meng Wang

    Shuyin Ouyang, Jie M. Zhang, Mark Harman, and Meng Wang. 2023. https://arxiv.org/abs/2308.02828 LLM is like a box of chocolates: the non-determinism of chatgpt in code generation . Abs/2308.02828

    2023 arXiv

  15. [23]

    Hassan Shakil, Atqiya Munawara Mahi, Phuoc Nguyen, Zeydy Ortiz, and Mamoun T. Mardini. 2024. https://arxiv.org/abs/2405.04053 Evaluating text summaries generated by large language models using O pen AI 's GPT . Abs/2405.04053

    2024 arXiv

  16. [24]

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022. https://arxiv.org/abs/2210.03057 Language models are multilingual chain-of-thought reasoners . Abs/2210.03057

    2022 arXiv

  17. [25]

    Taiga Someya, Yushi Sugimoto, and Yohei Oseki. 2023. https://arxiv.org/abs/2309.12676 JC o LA : Japanese corpus of linguistic acceptability

    2023 arXiv

  18. [26]

    Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. 2024. https://aclanthology.org/2024.acl-long.51 F ine S ur E : Fine-grained summarization evaluation using LLM s . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V...

    2024

  19. [27]

    Jiao Sun, Yufei Tian, Wangchunshu Zhou, Nan Xu, Qian Hu, Rahul Gupta, John Frederick Wieting, Nanyun Peng, and Xuezhe Ma. 2023 a . https://arxiv.org/abs/2310.14542 Evaluating large language models on controlled generation tasks . Abs/2310.14542

    2023 arXiv

  20. [28]

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023 b . Is chatgpt good at search? investigating large language models as re-ranking agent. arXiv preprint arXiv:2304.09542

    2023 arXiv

  21. [29]

    Mirac Suzgun, Nathan Scales, Nathanael Sch \"a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. https://doi.org/10.18653/v1/2023.findings-acl.824 Challenging BIG -bench tasks and whether chain-of-thought c...

    2023 doi

  22. [30]

    Masatoshi Suzuki, Koji Matsuda, Naoaki Okazaki, and Kentaro Inui. 2020. JAQKET : Quiz wo daizai ni shita nihongo qa dataset no kochiku. In NLP2020. In Japanese

    2020

  23. [31]

    Alexey Tikhonov and Max Ryabinin. 2021. https://doi.org/10.18653/v1/2021.findings-acl.310 I t ' s A ll in the H eads: U sing A ttention H eads as a B aseline for C ross- L ingual T ransfer in C ommonsense R easoning . In Findings of the Association for Computational Linguistic...

    2021 doi

  24. [32]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 LL a M...

    2023 arXiv

  25. [33]

    Hanjie, Runzhe Yang, and Karthik Narasimhan

    Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang, and Karthik Narasimhan. 2023. https://arxiv.org/abs/2307.08689 COLLIE : Systematic construction of constrained text generation tasks . Abs/2307.08689

    2023 arXiv

  26. [34]

    P Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging LLM -as-a- J udge with mt-bench and chatbot arena ...

    2023 arXiv

  27. [35]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. https://arxiv.org/abs/2311.07911 Instruction-following evaluation for large language models . Abs/2311.07911

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.