{"id":"c2b426ea-5366-4df8-b366-2bf2950b5532","arxiv_id":"2501.14649","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs show measurable deficiencies in both decomposition and composition during natural-to-formal conversion, with decomposition errors dominating, under the new DEDC evaluation framework.","lead":"This paper introduces DEDC, a framework that measures how well large language models decompose and compose primitives when translating natural language into an unfamiliar formal language. It finds that current LLMs are weaker at decomposition than composition, and that odd symbol names and unseen combinations degrade performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Dd = Pc - Pdc difference is not validated as a pure measure of decomposition: Pc and Pdc differ in demonstration quantity and format, not only in decomposition demand, so the headline Dd > Dc rests on an untested identification assumption.","rationale":"The reader's weakest-assumption analysis identifies the unvalidated additivity of Dd = Pc - Pdc, and I agree that this is the load-bearing point. My formulation sharpens it: the Pc condition changes the demonstration set in multiple observable ways, so even an additive decomposition of error rates cannot identify a 'decomposition effect' unless the two conditions are matched on every other factor. The proposed control is cheap and directly tests whether the Pc - Pdc gap is specific to supplying the test primitives' meanings. Because the paper's framework is transparent, the dataset and code are released, and the error taxonomy is independently useful, this concern does not by itself warrant rejection; it does mean the headline conclusion should be treated as conditional pending a cleaner decoupling check. I therefore keep the verdict at CONDITIONAL, which is unchanged from the reader's assessment.","tokens_in":20078,"tokens_out":9884,"duration_ms":97923,"concrete_test":"Run a matched control on the same 323 test samples, preserving the exact Pc construction from Section 2.4 but replacing the primitive demonstrations with the same number of one-expression demonstrations drawn from primitives and parameters not used in the test sample, so that the extra examples provide matched quantity and format without revealing the needed primitive meanings. Let P_control be the accuracy in this condition and compute D_control = P_control - Pdc. Under the paper's decoupling assumption, D_control should be near zero. If D_control exceeds roughly 30% of the reported Dd, then the original Dd is partly an artifact of demonstration quantity or format rather than decomposition, and the Dd > Dc claim would need to be re-examined; if D_control is close to zero, the decomposition interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLMs are more deficient in decomposition than composition (Section 3.2, Table 1) rests on the metric Dd = Pc - Pdc defined in Section 2.4. The two measurement conditions are not matched. In Pdc the model receives three full multi-step demonstration samples, from which it must infer the meaning and format of the primitives. In Pc it receives a simplest one-expression sample for each primitive used in the test sample, which directly labels those primitives. Thus Pc and Pdc differ not only in whether primitive meanings must be inferred, but also in the number of demonstrations (up to four extra exemplars), in the length and syntactic form of the demonstrations, and in whether multi-step composition is demonstrated before the test question. The paper asserts that Pc - Pdc is the additional error caused by decomposition but provides no ablation, filler-example control, or human baseline to show that the other differences are negligible. Since the reported Dd values are only 7.74-19.20 points and Dc values are 1.24-13.31 points, a few points of bias from demonstration-set mismatch could change the ordering for several models and would propagate to the Section 4 estimates of compositional-gap and counter-intuitive-name effects, which reuse the same decoupling.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DEDC, a semi-automatic framework for constructing samples and tasks that aim to evaluate LLMs' decomposition and composition capabilities in natural-to-formal-language (N2F) conversion under in-context learning. Using a simple tabular-reasoning formal language with ten primitives, the framework creates 323 test samples from six base graphs, and for each test sample builds a task in which the model sees either three full demonstration samples (Pdc condition) or one simplest sample per primitive used in the test sample (Pc condition). The authors define composition deficiency as Dc = 100 - Pc and decomposition deficiency as Dd = Pc - Pdc. They evaluate five LLMs and report that all of them are deficient in both capabilities, that decomposition deficiency is systematically larger than composition deficiency, and that compositional gaps and counter-intuitive symbolic names degrade both capabilities. The paper also provides an error-type taxonomy and a detailed analysis of error distributions across the five models.","tokens_in":20318,"tokens_out":4430,"duration_ms":42107,"significance":"If the decoupling assumption is valid, the paper makes a useful empirical contribution: it is one of the first attempts to separately measure decomposition and composition abilities for N2F in an unfamiliar formal language, and its finding that decomposition is the weaker capability across five diverse models is striking and practically relevant. The framework is reusable, the dataset and code are released, and the error taxonomy (primitive confusion, primitive fiction, variable misuse, redundancy, omission, incorrect meaning) is a sensible qualitative contribution. The additional settings for compositional gaps and counter-intuitive names are also well-motivated. The main value of the paper is therefore as an evaluation methodology plus a set of reproducible headline findings; both currently hinge on an identification assumption that the paper does not validate.","major_comments":[{"comment":"The decoupling metric Dd = Pc - Pdc rests on an unvalidated identification assumption. The Pc condition and the Pdc condition differ not only in whether the model must infer primitive meanings and formats, but also in the number of demonstrations (up to four additional exemplars in Pc), in the length and syntactic form of the demonstrations (single-step versus multi-step expressions), and in whether the model ever sees compositional use of the primitives before the test item. The paper asserts that Pc - Pdc isolates the additional error caused by decomposition, but provides no ablation, filler-example control, or human baseline to show that the other differences are negligible. Since the margins Dd - Dc are as small as 4.65 points (DeepSeek-2.5) and 6.50 points (Claude-3.5 and Llama-3.1), a small confound could change the ordering for several models. I would like to see a control that matches demonstration count and format across the two conditions, for example by adding the primitive demonstrations to the three full demonstrations, or by replacing the three full demonstrations with an equal number of single-step demonstrations in the Pdc condition.","section":"Section 2.4, Table 1"},{"comment":"The results are point estimates with no reported variance, even though demonstration samples are randomly selected. Setting temperature to 0 removes decoding stochasticity but does not remove the variance induced by the random choice of demonstrations. With 323 test samples and Dd-Dc margins as low as 4.65 points, the claim that all five models show Dd > Dc would be more convincing if the authors reported repeated runs with different random demonstration draws, or at least a bootstrap confidence interval. This concern also propagates to the Section 4 setting-effect estimates, which compare point estimates across conditions.","section":"Section 2.2, Section 3.2"},{"comment":"The setting-effect estimates reuse the same additive decoupling assumption without checking for interactions between decomposition and composition. Specifically, Delta_c^s = Pc^s - Pc and Delta_d^s = (Pdc^s - Pdc) - Delta_c^s assume that the total change in Pdc decomposes into independent composition and decomposition components. If the demonstrations in the 0% gap or 100% gap settings also change the difficulty of primitive inference (for example, by altering which primitives are demonstrated or how they are used in context), then the reported Delta_d^s values are biased. The paper should either validate the additive assumption with targeted controls or present the raw Pdc^s and Pc^s values with a sensitivity analysis that shows the qualitative conclusions are robust to alternative decompositions.","section":"Section 4, Eqs. (1)-(2)"}],"minor_comments":[{"comment":"The error-type counts are small in several cells (e.g., Claude-3.5 has only two errors in the Pc condition), so statements about the 'wide coverage' of error types should be phrased as descriptive observations rather than as strong generalizations, or accompanied by confidence intervals.","section":"Section 3.3, Table 3"},{"comment":"The paper should state how many random demonstration selections were used for each test sample and whether a fixed seed was used; this would help readers interpret the reproducibility of the point estimates.","section":"Section 2.2"},{"comment":"The reference to Kim and Linzen (2020) is duplicated in the third sentence; one of the two occurrences should be removed.","section":"Section 4.1.1"},{"comment":"The caption says 'the sample construction of the STD framework,' but the framework is called DEDC; this appears to be a typo.","section":"Figure 2 caption"},{"comment":"The grammatical-divergence re-experiment is run only on DeepSeek and changes both Pc and Pdc by 0.31 points; this is a useful sanity check but should be reported as a single anecdotal data point rather than as evidence that grammatical divergence has no effect on other models.","section":"Appendix B.1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a sound empirical core and a useful released artifact, but the central claim depends on an identification assumption that is currently unvalidated. The proposed control experiments are feasible within the scope of the manuscript, so I do not recommend rejection; I would support acceptance if the authors can show that the Dd > Dc ordering is robust to matched demonstration count and format."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the good news: this paper actually does something new. It separates decomposition (inferring primitive meanings from demonstrations) from composition (assembling primitives into expressions) in natural-to-formal conversion, and it ships a benchmark, code, and data to support that separation. The framework is carefully built: sample graphs are enumerated with validity filters, answer checking accounts for topological order and swappable arguments, and the error taxonomy (six types with examples) is concrete and useful. The central empirical finding—all five tested models are more deficient at decomposition than composition—is consistent and directionally plausible. I expect that qualitative conclusion to survive, and the error analysis is a real contribution.\n\nThe soft spot is the headline metric. Dd = Pc − Pdc is interpreted as the error attributable to decomposition, but Pc and Pdc are not otherwise matched: Pc gives a simplest one-expression sample for each primitive, while Pdc gives three multi-step demonstrations. The conditions differ not only in decomposition demand but in demonstration count, length, and whether composed examples appear. The paper asserts that the difference isolates decomposition, but it never validates that with a filler-example control or an ablation (e.g., adding a primitive glossary to the Pdc condition). Since Dd ranges from 7.7 to 19.2 and Dc from 1.2 to 13.3, a few points of bias could change the ordering for some models, especially DeepSeek, whose margin is only 4.65. This is not fatal, but it means the precise Dd/Dc numbers are upper/lower bounds rather than clean measurements. The same caveat applies to the Δd estimates in Section 4. A smaller issue is that every number is a point estimate from a single run despite stochastic demonstration selection; no variance or repeated sampling is reported.\n\nThe citation pattern is clean; the single self-citation is background. The limitations section is honest about scope (one synthetic language, no human baseline). This paper is for evaluation-minded NLP researchers working on compositional generalization and N2F. It deserves a serious referee; the framework and data will be reused even if the metrics need tightening. I would recommend major revision: add a demonstration-format control, report variance, and soften the causal claims around Dd. Those are doable fixes, and the result would be a solid contribution.","headline":"Genuinely new N2F evaluation framework and a plausible empirical finding, but the headline decomposition metric rests on an unvalidated matching assumption.","tokens_in":20811,"tokens_out":4872,"would_cite":false,"duration_ms":42639,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models are deficient in both decomposition and composition when converting natural language to a formal language, and the decomposition deficiency is consistently larger across all five models tested.","keywords":["decomposition","composition","natural-to-formal conversion","in-context learning","compositional generalization","symbolic names","large language models","evaluation framework"],"falsifier":"A model evaluated on the DEDC benchmark that produces $P_{dc} \\geq P_c$ — that is, $D_d \\leq 0$ — would refute the claim that decomposition deficiencies are consistently larger than composition deficiencies.","tokens_in":19862,"feed_emoji":"🧩","tokens_out":13939,"duration_ms":126728,"temperature":0.7,"pith_summary":"This paper asks whether large language models can learn an unfamiliar formal language from a few examples, a prerequisite for reliable natural-to-formal conversion. To answer it, the authors propose a framework that measures two abilities separately: decomposition, working out what each symbol means from examples, and composition, combining the known symbols into new expressions. In tests with five leading LLMs on a tabular-reasoning formal language, every model made more errors from decomposition than from composition, and all models produced a wide range of error types. The paper further shows that compositional gaps between examples and test questions and misleading symbol names worsen both abilities. A sympathetic reader would care because these two abilities are what an LLM must possess to convert natural language into any formal language it has not been trained on.","feed_headline":"Decomposition, not composition, is the LLM bottleneck","feed_subtitle":"Five LLMs all fail more on decomposition than on composition when learning new formal languages.","key_machinery":"The DEDC framework carries the argument. It semi-automatically builds 323 samples over a fixed formal language of 10 tabular-reasoning primitives; each sample is a directed acyclic graph of expressions plus a natural-language question. For each test sample it selects three demonstration samples that cover the primitives, then measures an LLM in two conditions: $P_{dc}$, with only demonstrations, and $P_c$, with demonstrations plus a simplest worked example for each primitive. The difference defines decomposition deficiency $D_d = P_c - P_{dc}$ and composition deficiency $D_c = 100 - P_c$, making the two capabilities comparable on the same scale. Additional settings (0% vs 100% compositional gap, anomalous vs cross-mapped names) are scored by the same protocol to yield delta measures for each ability.","core_discovery":"The paper's core discovery is that LLMs' weakness in natural-to-formal conversion is concentrated in decomposition rather than composition. In the base evaluation, every one of the five models showed $D_d > D_c$: Claude-3.5 had $D_d = 7.74$ vs $D_c = 1.24$, GPT-4o had $D_d = 13.31$ vs $D_c = 5.26$, and the open-weight models ranged from $D_d = 15.79$ to $19.20$. The paper also finds that compositional gaps between demonstration samples and the test question degrade both abilities, and that counter-intuitive symbolic names impose an even larger penalty, adding more than 20 points to decomposition error for every model. Error analysis shows six recurring error types, with primitive confusion most frequent, pointing to deficiencies in both natural language understanding and in learning and using a symbolic system.","pith_inferences":["Editorial inference: the paper's decoupling assumes additive, non-interacting capabilities; a direct test is to present the primitive demonstrations without the three full demonstration samples, which, if $P_c$ drops, would show $D_d$ overstates the decomposition-only burden.","Editorial inference: the benchmark uses one template-generated formal language, so the finding that decomposition dominates composition may not transfer to messier real-world formalisms like SQL; replicating DEDC across other N2F tasks would reveal whether the bottleneck is general.","Editorial inference: because evaluation is at temperature zero, the error-type distribution may be a property of greedy decoding; sampling could shift the balance between primitive confusion and omission, which the paper does not explore.","Editorial inference: the authors' next-token-probability hypothesis for counter-intuitive names suggests a concrete intervention—explicit name-to-meaning mapping prompts or lightweight fine-tuning should shrink the more-than-20-point decomposition penalty, which the paper leaves untested."],"forward_implications":["Eliminating the decomposition deficiency alone would raise accuracy by 7.7 to 19.2 percentage points depending on the model, making decomposition the highest-leverage target for improvement.","Demonstration selection matters: replacing demonstrations so that every test sample has zero compositional gap with them improves both $P_{dc}$ and $P_c$ for all five models, so graph-matching demonstration samplers are a concrete fix.","Counter-intuitive symbolic names are a severe robustness threat; any deployment that renames primitives without retraining should expect decomposition error to rise by more than 20 points for every model.","The DEDC framework is reusable: for any N2F task with a defined primitive set and question templates, the same two-condition protocol yields directly comparable $D_d$ and $D_c$ scores.","Primitive confusion, the most common error, indicates that symbolic-system learning should be a focal point alongside natural-language understanding in future N2F work."],"supporting_citations":[{"why":"provides the directed-acyclic-graph representation and compositional-gap concept used to build samples and the 0/100% gap settings","marker":"Keysers et al., 2020"},{"why":"supplies the decomposition/compositionality taxonomy that motivates separating the two abilities","marker":"Hupkes et al., 2020"},{"why":"defines the tabular-reasoning semantic parsing setting whose formal language the DEDC primitives emulate","marker":"Chen et al., 2020b"},{"why":"establishes the composition-without-systematicity benchmark that this work extends to LLM N2F","marker":"Lake and Baroni, 2018"},{"why":"the COGS challenge defines compositional gaps as a core generalization test that DEDC adapts","marker":"Kim and Linzen, 2020"},{"why":"shows demonstration diversity affects in-context compositional generalization, grounding the demonstration-sample design","marker":"Levy et al., 2023"},{"why":"motivates the N2F task via logical-expression generation for symbolic reasoning","marker":"Xu et al., 2024"}],"fun_headline_variants":["LLMs fail at decomposition, not composition, in formal language tasks","Decomposition errors dominate when LLMs learn new formal languages","Compositional gaps and odd names trip up LLM formal conversion","LLM weak spot: breaking natural language into formal steps","Decomposition deficit is the true LLM bottleneck in N2F"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers assume that seeing a simplest sample of each primitive fully removes the need for decomposition, leaving the entire gap between the two conditions ($P_c - P_{dc}$) to measure decomposition alone, with no interaction between decomposition and composition.","fun_headline_variants_meta":{"raw":{"variants":["LLMs fail at decomposition, not composition, in formal language tasks","Decomposition errors dominate when LLMs learn new formal languages","Compositional gaps and odd names trip up LLM formal conversion","LLM weak spot: breaking natural language into formal steps","Decomposition deficit is the true LLM bottleneck in N2F"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000584,"raw_usage":{"total_tokens":2748,"prompt_tokens":948,"completion_tokens":1800,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":1713}},"tokens_in":564,"tokens_out":1800,"duration_ms":11928,"temperature":1.0,"reasoning_tokens":1713,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:56:08.961987+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A model evaluated on the DEDC benchmark that produces $P_{dc} \\geq P_c$ — that is, $D_d \\leq 0$ — would refute the claim that decomposition deficiencies are consistently larger than composition deficiencies.","supporting_citations":[],"review_version":1}