{"id":"a551435a-3d24-466c-98f7-78ad9cfee180","arxiv_id":"2506.15629","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new benchmark permutes CommonGen concept sets and asks LLMs to compose sentences in the specified order; even the best model achieves only about 75% ordered coverage.","lead":"The paper introduces Ordered CommonGen, a test where language models must write sentences using given words in an exact specified order. Across 36 models, the best one still followed the order only about 75% of the time, showing a real gap in instruction-following.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 75% ceiling is only interpretable as an LLM deficit if humans can actually compose natural sentences for every permutation; no human baseline is measured, and the appendix hints that some concept sets may be infeasible.","rationale":"The reader's weakest assumption and my concern coincide: the human ceiling in Section 2.1. I agree this is the most load-bearing unvalidated step. The headline deficit interpretation requires it; absent human data, the benchmark still shows LLMs fail an explicit instruction some fraction of the time, but the normative claim that this is a deficit rather than a property of the item set is unsupported. The paper's own appendix enriches the concern: as written, 33 concept sets had no successful ordered sentence from any of 36 LLMs, which is exactly what one would expect if some permutations are infeasible or near-infeasible. I would not reject the paper: the within-model effects are plausible and measured, the model coverage is broad, and the missing piece is a calibration study, which is a standard condition of acceptance rather than a fatal flaw. Therefore I keep the reader's CONDITIONAL verdict and propose a concrete human-baseline test. This is not a consensus dispute or an ad hominem; it is a falsifiable missing control directly tied to the central claim.","tokens_in":32296,"tokens_out":9398,"duration_ms":121693,"concrete_test":"Recruit at least 3 native English speakers and present a stratified sample of, say, 5 concept sets × 24 permutations (or all 192 sets if budget allows), asking each participant to write a grammatical, natural sentence containing the four lemmas in the specified surface order; score with the same ordered-matching rule used in Section 3.1 and report per-permutation human success plus inter-annotator agreement. If human ordered coverage is near 100% on all sampled permutations, the concern is resolved; if humans also fail on substantial subsets, especially VVVV patterns and the 33 concept sets flagged in Appendix C.3, the benchmark should exclude or reweight those items and the 75% ceiling should be re-reported as a feasibility-normalized score.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result, Finding 3, reads Llama3.1-405B's ~75% ordered coverage as a shortfall against a human ceiling. That ceiling is asserted, not measured: Section 2.1 states that humans 'can compose sentences following any specified order of concepts,' citing generative grammar, but the paper collects no human responses. The assertion is load-bearing because the benchmark deliberately includes all 24 permutations of every four-concept set, including POS patterns such as VVVV that Table 2 shows are hard for all models (w/o order 37.38±28.54). If some permutations require list-like or imperative constructions that do not count as natural, grammatical 'commonsense' sentences in the sense the task requires, then those items are not probes of instruction-following but infeasible tasks, and the 75% figure partly reflects item feasibility rather than compliance. As written, Appendix C.3 adds weight to this worry by reporting that for 33 concept sets, no LLM produced an ordered sentence for any permutation. The within-model contrasts (the order phrase helps, outputs become more diverse, identical strings occur) are robust, but the specific claim that LLMs underperform a human-level ability depends on the unvalidated human ceiling. A small human norming study would settle it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Ordered CommonGen, a benchmark that takes the 192 four-concept sets of CommonGen-lite, generates all 24 permutations of each set, and appends the phrase \"in the specified order\" to six FLAN-derived instruction templates, producing 27,648 prompt instances. It evaluates 36 instruction-tuned LLMs with metrics for unordered and ordered concept coverage, ordered rate, pairwise BLEU/BLEURT similarity, distinct-n, diverse rate, and GPT-2 perplexity. The main findings are that adding the order phrase improves ordered coverage for most models, that the best model (Llama3.1-405B) reaches only about 75% ordered rate, that outputs become more diverse when order is specified, and that identical outputs across permutations are common for several models. A one-shot \"ideal example\" priming experiment raises ordered adherence for the strongest models while reducing output diversity.","tokens_in":32487,"tokens_out":8140,"duration_ms":87181,"significance":"If the measurements are taken at face value, Ordered CommonGen is a useful diagnostic that separates ordinary concept coverage from adherence to an explicitly specified concept order, and the paper's broad model coverage (36 models, 6 templates) makes the comparative table informative. The within-model contrasts are direct text statistics and are largely supported by the reported tables. However, the headline interpretation that the 75% ordered rate represents a shortfall relative to human ability is not supported by any human measurement, and an internal inconsistency in Appendix C.3 about item-level feasibility weakens the associated \"statistical evidence\" claim. The paper's contribution is therefore conditional: the benchmark and descriptive measurements are valuable, but the human-deficit framing and the feasibility analysis need revision before the central claim is established.","major_comments":[{"comment":"The claim that Llama3.1-405B's 75.26 Ordered Rate (Table 1) represents a deficit relative to humans rests on the Section 2.1 assertion that humans \"can compose sentences following any specified order of concepts.\" This assertion is cited to generative-grammar literature but is never tested with human participants. Appendix C.3 reports that for 33 concept sets no LLM produced an ordered sentence for any permutation, which at minimum shows that some items are extremely difficult for all 36 models; without a human norming study on those items, including the VVVV patterns shown in Table 2, the 75% figure conflates instruction-following failure with item feasibility. A small human study on a stratified sample of permutations, or a reanalysis restricted to permutations independently judged feasible, is needed before interpreting the 75% figure as a human-relative shortfall.","section":"Section 2.1, Section 4 (Finding 3), Section 6, Appendix C.3"},{"comment":"The feasibility claims are internally inconsistent as written. Finding 3 states that \"at least one of the 36 LLMs we tested is capable of composing each sentence correctly,\" and Appendix C.3 first says \"every concept set was composed by at least one LLM,\" but then states \"for 33 concept sets, no LLM could successfully compose sentences for any of their permutations.\" These statements cannot both be true if \"concept set\" refers to the same objects. The authors need to disambiguate the 192 original concept sets from the 4,608 ordered instances and reconcile the numbers, because the \"statistical evidence\" for per-item feasibility is used to support the paper's conclusions about compositional generalization instability.","section":"Appendix C.3, Figures 3-4, Section 4 (Finding 3)"},{"comment":"The definitions of Coverage w/ order and Ordered Rate are under-specified. The text says the check proceeds \"sequentially from the beginning,\" but it does not state whether the ordered concepts must appear as a subsequence, whether intervening words are allowed, how repeated concept occurrences are handled, or how lemmatization interacts with morphological variants (e.g., \"thrown\" versus \"throws\" in the Figure 2 example). Because Ordered Rate is the paper's central dependent variable, precise pseudocode and the exact lemmatization and matching procedure are required for reproducibility.","section":"Section 3.1"}],"minor_comments":[{"comment":"All values in Table 1 are single point estimates with no confidence intervals or significance tests. While the largest differences (e.g., +55 points for Llama3.1-405B) are unlikely to be noise, several comparisons in the \"most LLMs\" claims are small, and per-model variance across the six templates is not reported; Table 3 reports only aggregate variance across models.","section":"Table 1"},{"comment":"Table 5 lists \"OMLo2-7B\" and \"OMLo2-13B,\" but the text uses \"OLMo2\"; the spelling should be unified.","section":"Table 5"},{"comment":"Table 6 contains formatting typos: \"keywordsin\" and \"thingsin\" should be \"keywords in\" and \"things in.\"","section":"Appendix B, Table 6"},{"comment":"Figures 3 and 4 are difficult to read, and the axis definitions are ambiguous (for example, whether the x-axis bins are \"at least X models\" or \"exactly X models\"). Please clarify the captions and the bin labels.","section":"Figures 3-4"},{"comment":"The \"ideal example\" \"My favorite words are A, B, C, and D\" is a list-like template rather than a grammatical sentence that integrates the given verbs; calling it a universal ideal example should be qualified so readers do not mistake it for a natural sentence under the CommonGen definition.","section":"Section 5.3"},{"comment":"No URL or repository is given for the Ordered CommonGen dataset or the evaluation code. Since the dataset is derived from CommonGen-lite and FLAN templates, providing the exact data and checker code would substantially improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The empirical core of this paper is a useful benchmark and a broad, largely reproducible model comparison. My recommendation is driven by two issues: the human-ceiling assumption that frames the headline 75% figure as a deficit, and the internal inconsistency in Appendix C.3 about item feasibility. Both are fixable within the scope of the current manuscript, either by adding a small human norming study or by reframing the paper's claims as measurements of LLM order-adherence without the human-relative gap assertion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: Ordered CommonGen is a genuinely useful extension of CommonGen—permute the four concepts, add 'in the specified order', measure ordered coverage across 36 models. The within-model findings are solid: adding the order phrase improves ordered coverage for most models, the best model (Llama3.1-405B) reaches about 75%, and several models emit identical sentences across permutations. The POS analysis is a nice addition. This part deserves a real referee.\n\nThe soft spots are real but concentrated. The human ceiling in Section 2.1 is asserted, not measured. The claim that humans can compose sentences following any specified order is load-bearing for reading 75% as a deficit; without human norming, we don't know how many of the 24 permutations are actually feasible, especially verb-heavy ones. Table 2's VVVV row (w/o-order coverage 37%) and Appendix C.3's report that 33 concept sets had zero ordered successes across all 36 models make the infeasibility worry concrete, not hypothetical. A small human calibration study would settle it. That said, this doesn't kill the benchmark—the within-model contrasts stand alone.\n\nOne problem the stress-test didn't flag: the text says both that every ordered concept set was composed by at least one LLM (Figure 3) and that for 33 concept sets no LLM composed any permutation (Figure 4). Both cannot be true as stated unless the criteria differ; the paper doesn't explain. That needs fixing.\n\nSmaller issues: the ordered-matching algorithm is underspecified (can intervening words appear? are partial orders counted?), and there are no significance tests or error bars. Minor for a benchmark, but the missing artifacts are not minor: no data or code is released, and the evaluation is the benchmark. That should be a condition for acceptance.\n\nA serious referee should engage. Acceptance should be conditional on releasing the benchmark and evaluation code, adding at least a small human norming study, and fixing the appendix contradiction. I'd cite this for instruction-following evaluation and bring it to reading group.","headline":"A useful benchmark for order-constrained generation with solid within-model findings, but the 75% 'deficit' rests on an unmeasured human ceiling and the appendix has a real internal inconsistency.","tokens_in":33071,"tokens_out":4532,"would_cite":true,"duration_ms":48185,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark that permutes four words shows LLMs understand order instructions but rarely obey them fully; the best model reaches only about 75% ordered coverage.","keywords":["Ordered CommonGen","compositional generalization","instruction following","commonsense reasoning","large language models","ordered coverage","concept permutation","output diversity"],"falsifier":"Run the same 27,648 Ordered CommonGen instances, or a representative sample, with human participants and measure their ordered coverage; if humans also fall well below 100% on unnatural or verb-heavy orders such as throw-catch-jump-run, the premise that humans can compose sentences in any specified order fails and the measured LLM gap needs reinterpretation. A smaller version would ask a few dozen native speakers to write sentences for the 24 permutations of a set like (dog, catch, throw, frisbee) and compare their ordered coverage with Llama3.1-405B’s.","tokens_in":32057,"feed_emoji":"🧩","tokens_out":5083,"duration_ms":55961,"temperature":0.7,"pith_summary":"The paper proposes Ordered CommonGen, a benchmark that takes the 192 four-concept sets of CommonGen-lite, generates all 24 permutations of each, and adds the phrase “in the specified order” to the instruction. It tests whether LLMs can compose a sentence containing all concepts in the given order, thereby measuring compositional generalization and instruction following at once. Across 36 instruction-tuned LLMs, the central finding is that models do understand the order instruction—adding it raises ordered coverage—yet even the best model, Llama3.1-405B, reaches only about 75% ordered coverage. Models also default to their preferred natural concept order, producing identical sentences across permutations a substantial fraction of the time. The authors claim this gap matters because humans, via the “infinite use of finite means,” can compose sentences in any specified concept order.","feed_headline":"Even the best LLM obeys word order only 75–percent of the time","feed_subtitle":"New Ordered CommonGen benchmark finds models understand “in the specified order” yet default to natural word orders.","key_machinery":"The Ordered Concept Set is the load-bearing object: all 4! = 24 permutations of each of the 192 CommonGen-lite concept sets, yielding 4,608 concept sets and 27,648 instances after pairing with six FLAN-derived instruction templates modified by appending “in the specified order.” The central metric is Ordered Rate (and Coverage w/ order), which checks whether lemmatized concepts appear in the specified sequential order; comparing scores with and without the order phrase isolates the instruction-following effect, while pairwise similarity and Diverse Rate measure whether the model actually changes its output when the concept order changes.","core_discovery":"The paper argues that existing generative commonsense reasoning benchmarks overstate compositional ability because they ignore whether concepts appear in the instructed order. Ordered CommonGen permutes four concepts and explicitly instructs the model to write a sentence using them “in the specified order,” then measures ordered coverage: whether the lemmatized concepts appear in the specified sequence. The results show that LLMs generally understand the instruction’s intent, since ordered coverage improves when the phrase is added, but their execution is unreliable: the best model, Llama3.1-405B, achieves an Ordered Rate of about 75%, GPT-4o about 55%, and most models far less. The paper also finds that verb-only (VVVV) concept sets are the hardest to compose, that models often produce identical sentences across different permutations, and that a one-shot “ideal example” improves instruction adherence for strong models while sharply reducing output diversity. Appendix C.3 shows that every concept set is successfully composed by at least one of the 36 LLMs, so the task is feasible, yet no model comes close to consistent adherence.","pith_inferences":["The paper does not test its human ceiling, but a direct human experiment would sharpen the claim: if humans also fail on some unnatural verb-heavy orders, the measured LLM gap would narrow and the benchmark would need recalibration.","The identical-sentences-across-permutations result suggests a testable extension: train or fine-tune on reordered concept sets and measure whether Diverse Rate rises together with ordered coverage, which would indicate whether the bias is learned frequency rather than architectural limitation.","The monotonic one-shot template that improves adherence is linguistically degenerate; a more informative extension would use few-shot examples with varied natural paraphrases, testing whether ordered coverage can improve without collapsing diversity.","The benchmark’s logic transfers beyond text: the same permutation-plus-order-instruction design could evaluate instruction following in image or video generation, where a specified sequence of frames or scenes plays the role of the ordered concepts."],"forward_implications":["If the 75% ceiling holds, current instruction-tuned LLMs are not reliable for tasks where concept or event order is semantically load-bearing, such as chronological narration, action planning, lyrics, and story composition.","The benchmark provides a single jointly defined score for instruction following and compositional generalization, making it usable as a regression test when new model releases appear.","Because adding the order phrase improves coverage for most models, the failure is partly execution rather than comprehension, pointing to training or decoding methods that enforce ordered output without collapsing diversity.","Verb-only concept sets expose a specific compositional weakness that standard instruction tuning has not fixed, suggesting a targeted target for future training data.","The one-shot priming result implies that strong instruction followers can be steered toward order compliance, but only by sacrificing the diversity that natural text generation requires."],"supporting_citations":[{"why":"Defines CommonGen, the source task and seed dataset whose four-concept sets Ordered CommonGen permutes.","marker":"Lin et al., 2020"},{"why":"Supply the FLAN instruction templates that Ordered CommonGen modifies by appending “in the specified order.”","marker":"Wei et al., 2022a; Longpre et al., 2023"},{"why":"Grounds the human ceiling with the “infinite use of finite means” premise that humans can follow any concept order.","marker":"Chomsky, 1965"},{"why":"Provides evidence that models bias toward frequent concept-order patterns, used to explain identical outputs across permutations.","marker":"Zhang et al., 2023"},{"why":"Supplies the Pairwise-BLEU idea adapted to measure sentence-level similarity across the 24 permutations.","marker":"Shen et al., 2019"},{"why":"Motivates averaging across the six templates, since prompt phrasing shifts LLM performance.","marker":"Sakai et al., 2024c"},{"why":"Defines the Llama 3 model family, including Llama3.1-405B, the best-performing model on ordered coverage.","marker":"Dubey et al., 2024"}],"fun_headline_variants":["New benchmark: LLMs follow word order only 75% of the time","LLMs grasp instruction intent but fail to obey concept order","Ordered CommonGen: best LLM only 75% order-compliant","Compositional generalization worse than thought when order matters","LLMs default to natural order despite explicit 'specified order' prompts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark’s reading of 75% as a deficit rests on the untested claim, stated in Section 2.1, that humans can always compose natural sentences following any specified order of concepts; no human evaluation was run to verify this ceiling.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark: LLMs follow word order only 75% of the time","LLMs grasp instruction intent but fail to obey concept order","Ordered CommonGen: best LLM only 75% order-compliant","Compositional generalization worse than thought when order matters","LLMs default to natural order despite explicit 'specified order' prompts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1431,"prompt_tokens":935,"completion_tokens":496,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":404}},"tokens_in":551,"tokens_out":496,"duration_ms":5461,"temperature":1.0,"reasoning_tokens":404,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:51:41.056378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 27,648 Ordered CommonGen instances, or a representative sample, with human participants and measure their ordered coverage; if humans also fall well below 100% on unnatural or verb-heavy orders such as throw-catch-jump-run, the premise that humans can compose sentences in any specified order fails and the measured LLM gap needs reinterpretation. A smaller version would ask a few dozen native speakers to write sentences for the 24 permutations of a set like (dog, catch, throw, frisbee) and compare their ordered coverage with Llama3.1-405B’s.","supporting_citations":[],"review_version":1}