{"id":"9320b10e-5b75-4e41-8469-67c6568407be","arxiv_id":"2502.06086","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLMs, including o1, struggle to generate noun phrases that exhibit emergent properties, and a new dataset and spreading-activation prompting method only partially close the gap.","lead":"The paper introduces CCPT, a dataset of 12,315 human-annotated triples linking noun phrases, properties, and property types (emergent, component, canceled), and evaluates six LLMs on three tasks. It finds that current LLMs generate emergent-property-possessing noun phrases poorly, and proposes a spreading-activation retrieval method that improves most but not all generative scores.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Comparative-sentence sampling in CCPT may make the noun-phrase-completion deficit specific to figurative/extreme comparisons rather than conceptual combination generally.","rationale":"The reader's weakest assumption is the same concern, and I agree with it. This is the most load-bearing threat because it targets the scope of the central claim rather than secondary method details. The paper's own Limitations section confirms the risk of skewness in comparative-sentence sampling. The existing human evaluation (Table 4) and judge correlation (r=0.85) mitigate concerns about metric reliability, so the dataset-construction bias is the strongest remaining threat to the abstract's unqualified claim about LLMs' conceptual-combination ability. I do not see an internal inconsistency that would invalidate the measurement on the benchmark as defined; the issue is external validity. The reader already conditioned the verdict on this limitation and other addressable issues, so my analysis does not change the overall CONDITIONAL verdict.","tokens_in":23072,"tokens_out":12337,"duration_ms":117021,"concrete_test":"Construct a held-out evaluation set of noun-phrase-completion instances from non-comparative conceptual-combination sources: emergent-property items from BIG-bench-CC, FakeReef-style adjective-noun pairs, and 50-100 manually authored adjective-noun or noun-noun compounds with verified emergent properties, using the same head-noun/property/type format as CCPT. Run the same noun phrase completion prompt on the same six models and score with the same LLM-as-a-judge. Compare the E distribution to Table 3-(3). If E on this non-comparative set approaches the CCPT gold range (about 60), the comparative-frame sampling is the cause of the apparent LLM deficit; if E remains in the 20-35 range, the central claim generalizes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLMs, including o1, struggle to generate noun phrases that possess given emergent properties is supported primarily by the noun-phrase-completion emergence scores in Table 3-(3). Those test instances are drawn from CCPT, whose Step 1 (Section 3.2) extracts noun phrases only from comparative sentences containing 'like' or 'as' (e.g., 'as unstable as an apple on a toothpick'). This sampling scheme systematically selects noun phrases unusual enough to appear in a simile and properties salient enough to be the explicit point of comparison. The authors' Limitations section concedes: 'the inherent nature of comparative sentences may introduce skewness, favoring certain types of properties over others.' If that skew is severe, the emergent-property distribution in CCPT may be dominated by figurative, extreme, or metaphor-driven combinations, so the low E scores could reflect difficulty with comparative-style creativity rather than with conceptual combination as a general cognitive ability. Because the abstract and title state the claim about conceptual combination in general ('Is a Peeled Apple Still Red?'), this distributional restriction is load-bearing for the headline finding: without evidence that the same deficit appears on non-comparative noun phrases (e.g., adjective-noun or noun-noun compounds), the claim is only established for the comparative-frame subset.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CCPT, a dataset of 12,315 annotated (noun phrase, property, property type) triplets built from comparative sentences, and defines three tasks to probe LLM conceptual combination: property induction, noun phrase completion, and property type prediction. It proposes emergence (E) and cancellation (C) metrics computed from component and whole-phrase relevance scores, validates an LLM-as-a-judge (GPT-4o) against human relevance ratings, and evaluates six LLMs including o1. The main findings are that the LLM judge correlates strongly with human ratings, that all evaluated models score far below humans and the gold dataset on noun-phrase completion with emergent properties (E <= 33 vs. gold 59.9), and that a spreading-activation prompting method improves some generative scores. The paper also reports a classification task where GPT-4o lags behind humans in property-type prediction.","tokens_in":23261,"tokens_out":6225,"duration_ms":52449,"significance":"If the central claims hold, CCPT fills a real gap: prior work on conceptual combination in NLP has addressed either property verification or noun-compound interpretation, but not the full component/emergent/canceled typology with generative tasks. The public release of data and code, the use of an externally validated LLM judge, and the explicit cognitive-science framing (spreading activation) are strengths. The headline finding that even strong models like o1 default to component-property solutions is interesting and falsifiable. However, the scope of the claim is narrower than the title and abstract suggest, because the test instances are drawn entirely from comparative sentences; the metric-validation evidence is on raw relevance rather than the derived E/C scores; and the spread-activation improvement is not consistently present across every model-task cell. These issues require qualification but are addressable within the manuscript's scope.","major_comments":[{"comment":"The CCPT noun phrases are extracted exclusively from comparative sentences containing 'like' or 'as' (Section 3.2, Step 1), and the noun-phrase completion test in Table 3-(3) uses gold answers drawn from this pool. Comparative frames select for combinations that are striking enough to be worth a simile (e.g., 'as unstable as an apple on a toothpick') and for properties that are the explicit point of comparison. As the authors concede in the Limitations, 'the inherent nature of comparative sentences may introduce skewness, favoring certain types of properties over others.' Because the abstract and title state the conclusion about conceptual combination in general ('Is a Peeled Apple Still Red?'), this sampling restriction is load-bearing. The paper should either demonstrate that the emergent-property deficit reproduces on non-comparative constructions (e.g., adjective-noun compounds such as 'peeled apple' or noun-noun compounds such as 'dog house') or explicitly qualify the claim as applying to comparative-frame combinations.","section":"§3.2 Step 1; Limitations (3)"},{"comment":"The abstract's first claim is that the automatic metric 'grading property emergence and cancellation closely corresponds with human judgments.' However, Section 6.1 reports only the correlation between LLM-as-a-judge and human ratings on the raw relevance scores (R_H,P, R_M,P, R_N,P), with Pearson r=0.85. The emergence and cancellation scores E and C are nonlinear functions — max and max-minus — of those raw scores (Equations (1)-(3)), so a strong raw correlation does not by itself establish that the derived E and C values match human-derived E and C values. The authors should report the correlation or agreement between E and C computed from LLM and human relevance ratings on the same 300 sampled pairs, or provide an explicit derivation that raw-score correlation transfers to the max-based metrics.","section":"§6.1, Equations (1)-(3)"},{"comment":"The abstract and Section 5.1 state that the proposed spread activation method 'improves performances in all generative tasks' and that it 'achieves the highest performance among the evaluated approaches.' These statements are not consistently supported by Table 3. For example, on NOUN PHRASE COMPLETION-EMERGENT, Qwen2.5-72B with S.A.w/LLM has E=23.3, worse than Base (24.5) and CoT (26.5); on the same task, LLaMa3.1-70B with S.A.w/Both has E=24.3, worse than S.A.w/LLM (26.6). Given that only three seeds are averaged and error bars overlap, the improvement claim needs significance testing or an explicit statement that the positive effect holds on aggregate but not for every model-task combination.","section":"Abstract; §5.1; Table 3"},{"comment":"The inter-annotator agreement for property type annotation is low: Fleiss' kappa is 0.312 for emergent property data and 0.410 for canceled property data during annotation, and the final binary classification kappas are 0.498 and 0.505. These values indicate only moderate agreement, which raises questions about the reliability of the gold labels used to score LLM performance. The paper should discuss how much of the measured LLM deficit could be attributable to label noise, and ideally report the upper bound on achievable accuracy given this agreement.","section":"Appendix A.3"}],"minor_comments":[{"comment":"In the paragraph comparing the two generative tasks, the second reference to 'Table 3-(1)' should refer to Table 3-(3) for noun phrase completion.","section":"§5.1"},{"comment":"The model name 'LLaMa' should be written as 'LLaMA' or 'Llama' for consistency with the official naming.","section":"Throughout"},{"comment":"The test instance counts (N=200, 167, 167) are not justified; a short description of how the representative sample was drawn and whether it preserves the dataset's property-type balance would strengthen the evaluation.","section":"§4"},{"comment":"The description of Multi-Oracle as 'selecting the best score among the multiple solutions' is ambiguous: clarify whether the best score is chosen per instance or per entire run, and how ties are broken.","section":"Table 3 caption"},{"comment":"Figure 3's caption says 'PMI score' on a log-2 scale; consider labeling the axis as 'log2 PMI' to avoid confusion with the formula in the text.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution with a novel dataset and tasks, but the scope of the headline claim is narrower than the title suggests due to the comparative-sentence sampling. The reported correlation for the metric is on raw relevance, not on the derived E/C scores, which is the metric that the abstract advertises. The spread-activation improvement is inconsistently present across model-task cells. I recommend major revision: the authors should add a non-comparative control or qualify the claims, report E/C-level metric validation, and temper or statistically support the improvement claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on the CCPT paper. It's a genuinely useful contribution. The dataset is the first to cover component, emergent, and canceled properties in one place, with 12.3K human-annotated triplets and a new pair of generative tasks (property induction and noun phrase completion) plus a classification task. The LLM-as-judge metric correlates at r=0.85 with human relevance ratings, which gives me confidence that the headline result—LLMs including o1 score far below gold on generating noun phrases with truly emergent properties—is measuring something real, not just a weird artifact. The spreading-activation prompting is a reasonable application of a classic cognitive model, and the ablation shows iteration and filtering help.\n\nThe soft spots are real but mostly addressable. First, the dataset is built exclusively from comparative sentences containing 'like' or 'as'. That means the noun phrases and properties are skewed toward figurative, extreme, or simile-worthy combinations. The authors concede this in the limitations, but the abstract and title state the claim about conceptual combination in general. The benchmark is still valuable, but the generality of the deficit to, say, adjective-noun compounds is not established. Second, the abstract's claim that spreading activation improves all generative tasks is too strong: for Qwen2.5-72B in noun phrase completion, S.A. variants slightly underperform base. That should be qualified. Third, the 'Human' row uses best-of-1-to-3 solutions, which inflates the human baseline; fine as an upper bound, but it shouldn't be read as typical human performance. Fourth, the classification result in Table 5 has no error bars; a single run of GPT-4o on 1000 samples.\n\nNone of this kills the core contribution. The benchmark is reusable, the judge validation is solid, and the failure mode of LLMs defaulting to component properties is a clear, falsifiable finding. This paper deserves a serious referee. I'd recommend conditional accept, with revisions to qualify the generality claims and fix the overstatements.","headline":"A solid new benchmark that shows LLMs are bad at generating noun phrases with truly emergent properties, with a real caveat that the benchmark is built only from comparative sentences.","tokens_in":23852,"tokens_out":2922,"would_cite":true,"duration_ms":25632,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that current LLMs, including o1, fail at the generative side of conceptual combination: when asked to build a noun phrase whose modifier does not itself carry the target property, they default to combinations where one…","keywords":["conceptual combination","emergent property","property type","large language models","benchmark dataset","noun phrase generation","spread activation","LLM-as-a-judge"],"falsifier":"Compute the same noun-phrase-completion emergence scores on a held-out set of noun phrases not derived from comparative sentences (e.g., adjective–noun and noun–noun compounds sampled from ordinary text, with the same annotation and grading); if LLM emergence scores rise to near-gold and human levels on that set, the deficit is an artifact of the comparative-sentence sampling rather than a general inability to generate emergent combinations.","tokens_in":22820,"feed_emoji":"🍎","tokens_out":7272,"duration_ms":56440,"temperature":0.7,"pith_summary":"Conceptual combination is how minds and language make new meanings by fusing simpler concepts, and the properties of the result can be inherited, newly emergent, or canceled—a peeled apple is white (emergent) though neither 'peeled' nor 'apple' alone means white. The paper builds CCPT, a 12.3K-instance dataset of noun phrases, properties, and property types drawn from comparative sentences, and defines three tasks: generating a property for a given combination, generating a combination for a given property, and classifying a property's type. Its central finding is that all six tested LLMs, including o1, score far below gold and below humans on generating noun phrases that exhibit a specified emergent property; they fall back on modifiers that already carry the property. It also reports that an automatic LLM-as-a-judge metric for emergence and cancellation agrees with human ratings (Pearson r=0.85), and that a spread-activation method inspired by cognitive psychology improves all generative tasks.","feed_headline":"LLMs can't coin noun phrases with truly new properties","feed_subtitle":"A 12.3K-item benchmark shows even OpenAI's o1 settles for modifiers that already carry the target property.","key_machinery":"The load-bearing machinery is the CCPT dataset together with the emergence and cancellation scores: CCPT labels each noun phrase–property pair as component, emergent, or canceled following the cognitive taxonomy of Springer and Murphy (1992), and the scores are defined as E = max(R_{N,P} − max(R_{H,P}, R_{M,P}), 0) and C = max(max(R_{H,P}, R_{M,P}) − R_{N,P}, 0), where R_{X,P} is a relevance rating of how strongly concept X possesses property P, gathered either from human judges or an LLM-as-a-judge. The second piece is the proposed Spreading Activation method, an iterative loop that expands a concept set through LLM or ConceptNet queries and filters by relatedness to the original concepts, used to give the model bridge concepts before final generation.","core_discovery":"The paper's core claim is that modern LLMs can recognize properties of combinations but cannot reliably create them: they do poorly specifically on the generative task of producing a noun phrase whose modifier does not itself possess the target property, so that the property only emerges from the combination. In the noun-phrase-completion task, every model's emergence score E is 33 or below while the gold score is 59.9, and humans score 38.0; the same models do better at property induction (identifying an emergent property from a given phrase) and much better at property type prediction, suggesting the bottleneck is constructive rather than interpretive. The paper attributes the repeated failures to the models' tendency to choose modifiers that directly entail the property (e.g., 'burned banknote' for 'useless'), and offers a spreading-activation method that iteratively gathers related concepts and filters distractors, improving all three generative settings.","pith_inferences":["The comparative-sentence pipeline may be why emergent properties cluster around a few semantic frames (e.g., precarity, instability); a dataset built from other constructions could reveal whether the generation deficit is frame-specific.","The same E/C geometry could be applied to adjective–noun or verb–object combinations, and one could test whether LLMs trained with multimodal data show smaller deficits, since some emergent properties are perceptual.","The spread-activation idea suggests a testable extension: replacing ConceptNet with a learned concept graph might yield further gains, and one could measure whether the method helps humans as well by presenting the same activated concept sets.","If property type prediction is the bottleneck, then training or prompting models to explicitly reason about the three-way taxonomy before generation may improve the generative tasks as well."],"forward_implications":["If the finding holds, LLM-based creative writing and ideation tools will systematically produce combinations that are actually familiar in disguise, because the models fall back on component properties.","The emergence metric E provides a cheap, human-aligned way to grade open-ended conceptual-combination outputs, enabling scalable evaluation of future models.","Property type prediction accuracy lags (56.4% for GPT-4o vs 81% humans), so classification of property origins is an open capability gap.","Spreading activation is a practical, training-free way to push generation toward genuine emergence, and its gains grow with iterations and with the filtering step.","The dataset's low PMI combinations (average -1.03 vs 5.78 for ConceptNet) mean it tests genuinely novel combinations rather than memorized co-occurrences."],"supporting_citations":[{"why":"Supplies the three-way property taxonomy (component/emergent/canceled) that the dataset and tasks are built on.","marker":"Springer and Murphy (1992)"},{"why":"Supplies the uni-gram concept set and HasProperty relations used to build noun phrases and canceled-property candidates.","marker":"Speer et al. (2017)"},{"why":"GPT-4o-mini performs property extraction and candidate selection in the pipeline.","marker":"OpenAI (2024a)"},{"why":"VERA-T5-XXL filters extracted and head-noun properties by plausibility score.","marker":"Liu et al. (2023)"},{"why":"Provides the LLM-as-a-judge method whose relevance scores define the E and C metrics and match human ratings.","marker":"Zheng et al. (2023)"},{"why":"Provides the spreading-activation theory that motivates the proposed method's iterative concept expansion.","marker":"Anderson (1983); Collins and Loftus (1975)"},{"why":"Motivates the noun-phrase-completion task as a creative generation test.","marker":"Kohn et al. (2011)"},{"why":"Prior emergent-property benchmark that the work extends by adding property types and generative tasks.","marker":"BIG-bench authors (2023)"}],"fun_headline_variants":["LLMs can't create noun phrases with truly emergent properties","Even o1 picks modifiers that already carry the property","New benchmark: LLMs struggle to generate emergent properties","Cognitive-psychology trick boosts LLM generative property tasks","12.3K benchmark exposes LLM limitation in conceptual combination"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole benchmark is harvested from comparative sentences containing 'like' or 'as', and the authors concede this may skew the property types found; if that framing bias is severe, the measured LLM failures may not generalize to other ways of combining concepts.","fun_headline_variants_meta":{"raw":{"variants":["LLMs can't create noun phrases with truly emergent properties","Even o1 picks modifiers that already carry the property","New benchmark: LLMs struggle to generate emergent properties","Cognitive-psychology trick boosts LLM generative property tasks","12.3K benchmark exposes LLM limitation in conceptual combination"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000633,"raw_usage":{"total_tokens":2913,"prompt_tokens":931,"completion_tokens":1982,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":1902}},"tokens_in":547,"tokens_out":1982,"duration_ms":13926,"temperature":1.0,"reasoning_tokens":1902,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:48:25.039162+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the same noun-phrase-completion emergence scores on a held-out set of noun phrases not derived from comparative sentences (e.g., adjective–noun and noun–noun compounds sampled from ordinary text, with the same annotation and grading); if LLM emergence scores rise to near-gold and human levels on that set, the deficit is an artifact of the comparative-sentence sampling rather than a general inability to generate emergent combinations.","supporting_citations":[],"review_version":1}