Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability

Hidetaka Kamigaito; Taro Watanabe; Yusuke Sakai

REVIEW 3 major objections 6 minor 1 cited by

Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A benchmark that permutes four words shows LLMs understand order instructions but rarely obey them fully; the best model reaches only about 75% ordered coverage.

desk verdict A useful benchmark for order-constrained generation with solid within-model findings, but the 75% 'deficit' rests on an unmeasured human ceiling and the appendix has a real internal inconsistency. read the letter →

arxiv 2506.15629 v1 pith:HR5W6PUJ submitted 2025-06-18 cs.CL cs.AI

Yusuke Sakai , Hidetaka Kamigaito , Taro Watanabe This is my paper

classification cs.CLcs.AI

keywords OrderedCommonGencompositionalgeneralizationinstructionfollowingcommonsensereasoninglargelanguagemodelscoverageconceptpermutationoutputdiversity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Ordered CommonGen, a benchmark that takes the 192 four-concept sets of CommonGen-lite, generates all 24 permutations of each, and adds the phrase “in the specified order” to the instruction. It tests whether LLMs can compose a sentence containing all concepts in the given order, thereby measuring compositional generalization and instruction following at once. Across 36 instruction-tuned LLMs, the central finding is that models do understand the order instruction—adding it raises ordered coverage—yet even the best model, Llama3.1-405B, reaches only about 75% ordered coverage. Models also default to their preferred natural concept order, producing identical sentences across permutations a substantial fraction of the time. The authors claim this gap matters because humans, via the “infinite use of finite means,” can compose sentences in any specified concept order.

What carries the argument

The Ordered Concept Set is the load-bearing object: all 4! = 24 permutations of each of the 192 CommonGen-lite concept sets, yielding 4,608 concept sets and 27,648 instances after pairing with six FLAN-derived instruction templates modified by appending “in the specified order.” The central metric is Ordered Rate (and Coverage w/ order), which checks whether lemmatized concepts appear in the specified sequential order; comparing scores with and without the order phrase isolates the instruction-following effect, while pairwise similarity and Diverse Rate measure whether the model actually changes its output when the concept order changes.

What would settle it

Run the same 27,648 Ordered CommonGen instances, or a representative sample, with human participants and measure their ordered coverage; if humans also fall well below 100% on unnatural or verb-heavy orders such as throw-catch-jump-run, the premise that humans can compose sentences in any specified order fails and the measured LLM gap needs reinterpretation. A smaller version would ask a few dozen native speakers to write sentences for the 24 permutations of a set like (dog, catch, throw, frisbee) and compare their ordered coverage with Llama3.1-405B’s.

Watch

Extended reading notes

Core claim

The paper argues that existing generative commonsense reasoning benchmarks overstate compositional ability because they ignore whether concepts appear in the instructed order. Ordered CommonGen permutes four concepts and explicitly instructs the model to write a sentence using them “in the specified order,” then measures ordered coverage: whether the lemmatized concepts appear in the specified sequence. The results show that LLMs generally understand the instruction’s intent, since ordered coverage improves when the phrase is added, but their execution is unreliable: the best model, Llama3.1-405B, achieves an Ordered Rate of about 75%, GPT-4o about 55%, and most models far less. The paper also finds that verb-only (VVVV) concept sets are the hardest to compose, that models often produce identical sentences across different permutations, and that a one-shot “ideal example” improves instruction adherence for strong models while sharply reducing output diversity. Appendix C.3 shows that every concept set is successfully composed by at least one of the 36 LLMs, so the task is feasible, yet no model comes close to consistent adherence.

Load-bearing premise

The benchmark’s reading of 75% as a deficit rests on the untested claim, stated in Section 2.1, that humans can always compose natural sentences following any specified order of concepts; no human evaluation was run to verify this ceiling.

Editorial extensions

If this is right

If the 75% ceiling holds, current instruction-tuned LLMs are not reliable for tasks where concept or event order is semantically load-bearing, such as chronological narration, action planning, lyrics, and story composition.
The benchmark provides a single jointly defined score for instruction following and compositional generalization, making it usable as a regression test when new model releases appear.
Because adding the order phrase improves coverage for most models, the failure is partly execution rather than comprehension, pointing to training or decoding methods that enforce ordered output without collapsing diversity.
Verb-only concept sets expose a specific compositional weakness that standard instruction tuning has not fixed, suggesting a targeted target for future training data.
The one-shot priming result implies that strong instruction followers can be steered toward order compliance, but only by sacrificing the diversity that natural text generation requires.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not test its human ceiling, but a direct human experiment would sharpen the claim: if humans also fail on some unnatural verb-heavy orders, the measured LLM gap would narrow and the benchmark would need recalibration.
The identical-sentences-across-permutations result suggests a testable extension: train or fine-tune on reordered concept sets and measure whether Diverse Rate rises together with ordered coverage, which would indicate whether the bias is learned frequency rather than architectural limitation.
The monotonic one-shot template that improves adherence is linguistically degenerate; a more informative extension would use few-shot examples with varied natural paraphrases, testing whether ordered coverage can improve without collapsing diversity.
The benchmark’s logic transfers beyond text: the same permutation-plus-order-instruction design could evaluate instruction following in image or video generation, where a specified sequence of frames or scenes plays the role of the ordered concepts.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: Ordered CommonGen is a genuinely useful extension of CommonGen—permute the four concepts, add 'in the specified order', measure ordered coverage across 36 models. The within-model findings are solid: adding the order phrase improves ordered coverage for most models, the best model (Llama3.1-405B) reaches about 75%, and several models emit identical sentences across permutations. The POS analysis is a nice addition. This part deserves a real referee.

The soft spots are real but concentrated. The human ceiling in Section 2.1 is asserted, not measured. The claim that humans can compose sentences following any specified order is load-bearing for reading 75% as a deficit; without human norming, we don't know how many of the 24 permutations are actually feasible, especially verb-heavy ones. Table 2's VVVV row (w/o-order coverage 37%) and Appendix C.3's report that 33 concept sets had zero ordered successes across all 36 models make the infeasibility worry concrete, not hypothetical. A small human calibration study would settle it. That said, this doesn't kill the benchmark—the within-model contrasts stand alone.

One problem the stress-test didn't flag: the text says both that every ordered concept set was composed by at least one LLM (Figure 3) and that for 33 concept sets no LLM composed any permutation (Figure 4). Both cannot be true as stated unless the criteria differ; the paper doesn't explain. That needs fixing.

Smaller issues: the ordered-matching algorithm is underspecified (can intervening words appear? are partial orders counted?), and there are no significance tests or error bars. Minor for a benchmark, but the missing artifacts are not minor: no data or code is released, and the evaluation is the benchmark. That should be a condition for acceptance.

A serious referee should engage. Acceptance should be conditional on releasing the benchmark and evaluation code, adding at least a small human norming study, and fixing the appendix contradiction. I'd cite this for instruction-following evaluation and bring it to reading group.

Referee Report

3 major / 6 minor

Summary. The paper introduces Ordered CommonGen, a benchmark that takes the 192 four-concept sets of CommonGen-lite, generates all 24 permutations of each set, and appends the phrase "in the specified order" to six FLAN-derived instruction templates, producing 27,648 prompt instances. It evaluates 36 instruction-tuned LLMs with metrics for unordered and ordered concept coverage, ordered rate, pairwise BLEU/BLEURT similarity, distinct-n, diverse rate, and GPT-2 perplexity. The main findings are that adding the order phrase improves ordered coverage for most models, that the best model (Llama3.1-405B) reaches only about 75% ordered rate, that outputs become more diverse when order is specified, and that identical outputs across permutations are common for several models. A one-shot "ideal example" priming experiment raises ordered adherence for the strongest models while reducing output diversity.

Significance. If the measurements are taken at face value, Ordered CommonGen is a useful diagnostic that separates ordinary concept coverage from adherence to an explicitly specified concept order, and the paper's broad model coverage (36 models, 6 templates) makes the comparative table informative. The within-model contrasts are direct text statistics and are largely supported by the reported tables. However, the headline interpretation that the 75% ordered rate represents a shortfall relative to human ability is not supported by any human measurement, and an internal inconsistency in Appendix C.3 about item-level feasibility weakens the associated "statistical evidence" claim. The paper's contribution is therefore conditional: the benchmark and descriptive measurements are valuable, but the human-deficit framing and the feasibility analysis need revision before the central claim is established.

major comments (3)

[Section 2.1, Section 4 (Finding 3), Section 6, Appendix C.3] The claim that Llama3.1-405B's 75.26 Ordered Rate (Table 1) represents a deficit relative to humans rests on the Section 2.1 assertion that humans "can compose sentences following any specified order of concepts." This assertion is cited to generative-grammar literature but is never tested with human participants. Appendix C.3 reports that for 33 concept sets no LLM produced an ordered sentence for any permutation, which at minimum shows that some items are extremely difficult for all 36 models; without a human norming study on those items, including the VVVV patterns shown in Table 2, the 75% figure conflates instruction-following failure with item feasibility. A small human study on a stratified sample of permutations, or a reanalysis restricted to permutations independently judged feasible, is needed before interpreting the 75% figure as a human-relative shortfall.
[Appendix C.3, Figures 3-4, Section 4 (Finding 3)] The feasibility claims are internally inconsistent as written. Finding 3 states that "at least one of the 36 LLMs we tested is capable of composing each sentence correctly," and Appendix C.3 first says "every concept set was composed by at least one LLM," but then states "for 33 concept sets, no LLM could successfully compose sentences for any of their permutations." These statements cannot both be true if "concept set" refers to the same objects. The authors need to disambiguate the 192 original concept sets from the 4,608 ordered instances and reconcile the numbers, because the "statistical evidence" for per-item feasibility is used to support the paper's conclusions about compositional generalization instability.
[Section 3.1] The definitions of Coverage w/ order and Ordered Rate are under-specified. The text says the check proceeds "sequentially from the beginning," but it does not state whether the ordered concepts must appear as a subsequence, whether intervening words are allowed, how repeated concept occurrences are handled, or how lemmatization interacts with morphological variants (e.g., "thrown" versus "throws" in the Figure 2 example). Because Ordered Rate is the paper's central dependent variable, precise pseudocode and the exact lemmatization and matching procedure are required for reproducibility.

minor comments (6)

[Table 1] All values in Table 1 are single point estimates with no confidence intervals or significance tests. While the largest differences (e.g., +55 points for Llama3.1-405B) are unlikely to be noise, several comparisons in the "most LLMs" claims are small, and per-model variance across the six templates is not reported; Table 3 reports only aggregate variance across models.
[Table 5] Table 5 lists "OMLo2-7B" and "OMLo2-13B," but the text uses "OLMo2"; the spelling should be unified.
[Appendix B, Table 6] Table 6 contains formatting typos: "keywordsin" and "thingsin" should be "keywords in" and "things in."
[Figures 3-4] Figures 3 and 4 are difficult to read, and the axis definitions are ambiguous (for example, whether the x-axis bins are "at least X models" or "exactly X models"). Please clarify the captions and the bin labels.
[Section 5.3] The "ideal example" "My favorite words are A, B, C, and D" is a list-like template rather than a grammatical sentence that integrates the given verbs; calling it a universal ideal example should be qualified so readers do not mistake it for a natural sentence under the CommonGen definition.
[General] No URL or repository is given for the Ordered CommonGen dataset or the evaluation code. Since the dataset is derived from CommonGen-lite and FLAN templates, providing the exact data and checker code would substantially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Ordered CommonGen is built from external resources and the headline results are direct measurements, not predictions derived from the benchmark's own definitions.

full rationale

The paper does not contain a derivation chain that reduces to its inputs. Ordered CommonGen is constructed from external resources (CommonGen-lite concept sets and FLAN instruction templates), and the evaluation metrics (coverage with/without order, ordered rate, pBLEU/BLEURT, distinct, diverse rate, perplexity) are computed directly from the generated text with no fitted parameter that is later renamed as a prediction. The central finding that the most instruction-compliant LLM reaches only about 75% ordered coverage is an empirical measurement across 36 models, not a consequence of the benchmark definition. The only self-citations (Sakai et al. 2024a for decoding settings and Sakai et al. 2024c for averaging across templates) concern evaluation conventions and are not load-bearing for the main claim. The Section 2.1 assertion that humans can compose sentences following any specified concept order is an untested theoretical premise and therefore a validity risk when interpreting the 75% figure as a human-relative deficit, but it is not circular: it is not derived from, nor equivalent to, the measured scores. The within-model contrasts in Table 1 (ordered instruction improves ordered coverage for most models) are likewise direct statistics, and Appendix C.3's report that some concept sets are never fully composed is an empirical observation rather than an artifact of metric construction. No circular step can be quoted and exhibited from the paper.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on no fitted parameters. The benchmark's validity depends on domain assumptions about human performance ceilings, surface-order as the right measurement level, and perplexity as a naturalness proxy. No new theoretical entities are introduced.

assumptions (3)

domain assumption Humans can compose natural sentences following any specified order of concepts, providing a near-100% performance ceiling for Ordered CommonGen.
Section 2.1 asserts humans can use 'infinite use of finite means' to follow arbitrary concept orders, citing Chomsky (1965) and related cognitive science work. This untested ceiling justifies interpreting the 75% LLM score as a deficit.
domain assumption Surface word order, after lemmatization, is the correct level for judging whether a model followed the instructed order.
The ordered coverage metric checks concept tokens in sequence order in the generated sentence; this operationalization is assumed, not validated against human judgments of instruction compliance.
domain assumption GPT2-XL perplexity is a reasonable relative proxy for sentence naturalness in this task.
Section 3.1 uses GPT2-XL LMPPL for quality since reference-based metrics are unavailable; this scoring-model assumption is stated but not justified with correlations.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability." pith.science (2026). https://pith.science/paper/HR5W6PUJ

@misc{pith2026250615629,
  author       = {Pith},
  title        = {Pith review of: Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HR5W6PUJ}},
  note         = {Machine review of arXiv:2506.15629}
}

read the original abstract

In generative commonsense reasoning tasks such as CommonGen, generative large language models (LLMs) compose sentences that include all given concepts. However, when focusing on instruction-following capabilities, if a prompt specifies a concept order, LLMs must generate sentences that adhere to the specified order. To address this, we propose Ordered CommonGen, a benchmark designed to evaluate the compositional generalization and instruction-following abilities of LLMs. This benchmark measures ordered coverage to assess whether concepts are generated in the specified order, enabling a simultaneous evaluation of both abilities. We conducted a comprehensive analysis using 36 LLMs and found that, while LLMs generally understand the intent of instructions, biases toward specific concept order patterns often lead to low-diversity outputs or identical results even when the concept order is altered. Moreover, even the most instruction-compliant LLM achieved only about 75% ordered coverage, highlighting the need for improvements in both instruction-following and compositional generalization capabilities.

Figures

Figures reproduced from arXiv: 2506.15629 by the authors.

**Figure 1.** Overview of our proposed Ordered CommonGen. Unlike CommonGen (Lin et al., 2020), we evaluate whether the composed sentences include the concepts in the specified order. To create the Ordered Concept Sets, we use CommonGen’s Concept Sets containing four concepts and generate all permutations, resulting in a total of 24 permutations per set. et al., 2024b; Makinae et al., 2024), as well as crafting catchy slogans (… view at source ↗

**Figure 2.** Our evaluation methodology for Ordered CommonGen. The left side shows the instruction template for [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** The distribution of all 4,608 concept sets successfully composed by all 36 LLMs using any of the six [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: The distribution of successfully composed permutations of concept sets by all 36 LLMs using any of the [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning
cs.CL 2026-08 conditional novelty 7.0 of 10

Skill entropy, a reference-model-based measure of skill-switching difficulty, calibrates a new cross-skill benchmark and serves as an RL reward, more than doubling small models' scores.

Reference graph

Works this paper leans on

14 extracted references · 7 canonical work pages · cited by 1 Pith paper

[4]

Preprint, arXiv:2501.12948

Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning. Preprint, arXiv:2501.12948. Hiroyuki Deguchi, Yusuke Sakai, Hidetaka Kami- gaito, and Taro Watanabe. 2024a. mbrs: A li- brary for minimum Bayes risk decoding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations...

arXiv 2024
[6]

Preprint, arXiv:2301.13688

The flan collection: Designing data and methods for effective instruction tuning. Preprint, arXiv:2301.13688. Zhicong Lu, Li Jin, Guangluan Xu, Linmei Hu, Nayu Liu, Xiaoyu Li, Xian Sun, Zequn Zhang, and Kai- wen Wei. 2023. Narrative order aware story gener- ation via bidirectional pretraining model with opti- mal transport reward. In Findings of the Assoc...

arXiv 2023
[8]

In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 650–669, Miami, Florida, USA

MalayMMLU: A multitask benchmark for the low-resource Malay language. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 650–669, Miami, Florida, USA. Associ- ation for Computational Linguistics. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Researc...

arXiv 2024
[9]

In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 294– 305, Abu Dhabi, UAE

Controlled language generation for lan- guage learning items. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 294– 305, Abu Dhabi, UAE. Association for Computa- tional Linguistics. Mirac Suzgun, Luke Melas-Kyriazi, and Dan Jurafsky

work page 2022
[10]

In Findings of the Association for Computational Linguistics: ACL 2023, pages 4265–4293, Toronto, Canada

Follow the wisdom of the crowd: Effective text generation via minimum Bayes risk decoding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4265–4293, Toronto, Canada. Association for Computational Linguistics. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Man- sooreh Karami, Jundong Li, ...

arXiv 2023
[11]

Learning to predict concept ordering for common sense generation. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (V olume 2: Short Papers), pages 10–19, Nusa Dua, Bali. Association for Computational Linguistics. Tia...

work page
[12]

In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9226–9242, Mi- ami, Florida, USA

Improving diversity of commonsense genera- tion by large language models via in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9226–9242, Mi- ami, Florida, USA. Association for Computational Linguistics. Chao Zhao, Faeze Brahman, Tenghao Huang, and Snigdha Chaturvedi. 2022. Revisiting generative commonsen...

arXiv 2024
[14]

Due to variations in CoT reasoning, unified evaluation is challenging; therefore, we evaluated using reasoning models

have emerged that achieve comparable capa- bilities through training rather than prompt-based control. Due to variations in CoT reasoning, unified evaluation is challenging; therefore, we evaluated using reasoning models. Table 7 compares the eval- uation results for Qwen2-7B (Yang et al., 2024)8 and its reasoning model Marco-o1 (Zhao et al., 2024b)9, as ...

work page 2024

Show all 14 references

[2004]

in the specified order

Computational modeling of numerical cogni- tion. The Handbook of Mathematical Cognition. A Detailed LLMs Information Table 5 shows the source information of each LLM. We used the Transformers (Wolf et al., 2020) and bitsandbytes (Dettmers et al., 2022) libraries for the infere...

2023
[2021]

All that’s ‘human’ is not gold: Eval- uating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V olume 1: Long Papers), page...

2024
[2022]

In Proceedings of the 29th International Conference on Computational Linguistics, pages 3578–3588, Gyeongju, Republic of Korea

IMPARA: Impact-based metric for GEC using parallel data. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3578–3588, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. Mana Makinae, Yusuke Sakai, Hidetaka K...

2024 arXiv
[2023]

In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM ’23, page 3998–4002, New York, NY , USA

Effective slogan generation with noise perturbation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM ’23, page 3998–4002, New York, NY , USA. Association for Computing Machinery. Fajri Koto, Nurul Aisyah, Haonan Li, and Tim...

2023 arXiv
[2024]

Preprint, arXiv:2410.12924

Interpreting token compositionality in llms: A robustness analysis. Preprint, arXiv:2410.12924. Nura Aljaafari, Danilo S. Carvalho, and André Freitas

arXiv
[2025]

Preprint, arXiv:2502.11066

Carma: Enhanced compositionality in llms via advanced regularisation and mutual information alignment. Preprint, arXiv:2502.11066. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic proposi- tional image caption evaluation. In Computer Visi...

2016 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [4]

Preprint, arXiv:2501.12948

Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning. Preprint, arXiv:2501.12948. Hiroyuki Deguchi, Yusuke Sakai, Hidetaka Kami- gaito, and Taro Watanabe. 2024a. mbrs: A li- brary for minimum Bayes risk decoding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations...

arXiv 2024

[2] [6]

Preprint, arXiv:2301.13688

The flan collection: Designing data and methods for effective instruction tuning. Preprint, arXiv:2301.13688. Zhicong Lu, Li Jin, Guangluan Xu, Linmei Hu, Nayu Liu, Xiaoyu Li, Xian Sun, Zequn Zhang, and Kai- wen Wei. 2023. Narrative order aware story gener- ation via bidirectional pretraining model with opti- mal transport reward. In Findings of the Assoc...

arXiv 2023

[3] [8]

In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 650–669, Miami, Florida, USA

MalayMMLU: A multitask benchmark for the low-resource Malay language. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 650–669, Miami, Florida, USA. Associ- ation for Computational Linguistics. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Researc...

arXiv 2024

[4] [9]

In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 294– 305, Abu Dhabi, UAE

Controlled language generation for lan- guage learning items. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 294– 305, Abu Dhabi, UAE. Association for Computa- tional Linguistics. Mirac Suzgun, Luke Melas-Kyriazi, and Dan Jurafsky

work page 2022

[5] [10]

In Findings of the Association for Computational Linguistics: ACL 2023, pages 4265–4293, Toronto, Canada

Follow the wisdom of the crowd: Effective text generation via minimum Bayes risk decoding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4265–4293, Toronto, Canada. Association for Computational Linguistics. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Man- sooreh Karami, Jundong Li, ...

arXiv 2023

[6] [11]

Learning to predict concept ordering for common sense generation. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (V olume 2: Short Papers), pages 10–19, Nusa Dua, Bali. Association for Computational Linguistics. Tia...

work page

[7] [12]

In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9226–9242, Mi- ami, Florida, USA

Improving diversity of commonsense genera- tion by large language models via in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9226–9242, Mi- ami, Florida, USA. Association for Computational Linguistics. Chao Zhao, Faeze Brahman, Tenghao Huang, and Snigdha Chaturvedi. 2022. Revisiting generative commonsen...

arXiv 2024

[8] [14]

Due to variations in CoT reasoning, unified evaluation is challenging; therefore, we evaluated using reasoning models

have emerged that achieve comparable capa- bilities through training rather than prompt-based control. Due to variations in CoT reasoning, unified evaluation is challenging; therefore, we evaluated using reasoning models. Table 7 compares the eval- uation results for Qwen2-7B (Yang et al., 2024)8 and its reasoning model Marco-o1 (Zhao et al., 2024b)9, as ...

work page 2024