REVIEW 4 major objections 5 minor 34 references
Is a Peeled Apple Still Red? Evaluating LLMs' Ability for Conceptual Combination with Property Type
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This paper claims that current LLMs, including o1, fail at the generative side of conceptual combination: when asked to build a noun phrase whose modifier does not itself carry the target property, they default to combinations where one…
desk verdict A solid new benchmark that shows LLMs are bad at generating noun phrases with truly emergent properties, with a real caveat that the benchmark is built only from comparative sentences. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the CCPT dataset together with the emergence and cancellation scores: CCPT labels each noun phrase–property pair as component, emergent, or canceled following the cognitive taxonomy of Springer and Murphy (1992), and the scores are defined as E = max(R_{N,P} − max(R_{H,P}, R_{M,P}), 0) and C = max(max(R_{H,P}, R_{M,P}) − R_{N,P}, 0), where R_{X,P} is a relevance rating of how strongly concept X possesses property P, gathered either from human judges or an LLM-as-a-judge. The second piece is the proposed Spreading Activation method, an iterative loop that expands a concept set through LLM or ConceptNet queries and filters by relatedness to the original concepts, used to give the model bridge concepts before final generation.
What would settle it
Compute the same noun-phrase-completion emergence scores on a held-out set of noun phrases not derived from comparative sentences (e.g., adjective–noun and noun–noun compounds sampled from ordinary text, with the same annotation and grading); if LLM emergence scores rise to near-gold and human levels on that set, the deficit is an artifact of the comparative-sentence sampling rather than a general inability to generate emergent combinations.
Extended reading notes
Core claim
The paper's core claim is that modern LLMs can recognize properties of combinations but cannot reliably create them: they do poorly specifically on the generative task of producing a noun phrase whose modifier does not itself possess the target property, so that the property only emerges from the combination. In the noun-phrase-completion task, every model's emergence score E is 33 or below while the gold score is 59.9, and humans score 38.0; the same models do better at property induction (identifying an emergent property from a given phrase) and much better at property type prediction, suggesting the bottleneck is constructive rather than interpretive. The paper attributes the repeated failures to the models' tendency to choose modifiers that directly entail the property (e.g., 'burned banknote' for 'useless'), and offers a spreading-activation method that iteratively gathers related concepts and filters distractors, improving all three generative settings.
Load-bearing premise
The whole benchmark is harvested from comparative sentences containing 'like' or 'as', and the authors concede this may skew the property types found; if that framing bias is severe, the measured LLM failures may not generalize to other ways of combining concepts.
Editorial extensions
If this is right
- If the finding holds, LLM-based creative writing and ideation tools will systematically produce combinations that are actually familiar in disguise, because the models fall back on component properties.
- The emergence metric E provides a cheap, human-aligned way to grade open-ended conceptual-combination outputs, enabling scalable evaluation of future models.
- Property type prediction accuracy lags (56.4% for GPT-4o vs 81% humans), so classification of property origins is an open capability gap.
- Spreading activation is a practical, training-free way to push generation toward genuine emergence, and its gains grow with iterations and with the filtering step.
- The dataset's low PMI combinations (average -1.03 vs 5.78 for ConceptNet) mean it tests genuinely novel combinations rather than memorized co-occurrences.
Reading between the lines
- The comparative-sentence pipeline may be why emergent properties cluster around a few semantic frames (e.g., precarity, instability); a dataset built from other constructions could reveal whether the generation deficit is frame-specific.
- The same E/C geometry could be applied to adjective–noun or verb–object combinations, and one could test whether LLMs trained with multimodal data show smaller deficits, since some emergent properties are perceptual.
- The spread-activation idea suggests a testable extension: replacing ConceptNet with a learned concept graph might yield further gains, and one could measure whether the method helps humans as well by presenting the same activated concept sets.
- If property type prediction is the bottleneck, then training or prompting models to explicitly reason about the three-way taxonomy before generation may improve the generative tasks as well.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CCPT, a dataset of 12,315 annotated (noun phrase, property, property type) triplets built from comparative sentences, and defines three tasks to probe LLM conceptual combination: property induction, noun phrase completion, and property type prediction. It proposes emergence (E) and cancellation (C) metrics computed from component and whole-phrase relevance scores, validates an LLM-as-a-judge (GPT-4o) against human relevance ratings, and evaluates six LLMs including o1. The main findings are that the LLM judge correlates strongly with human ratings, that all evaluated models score far below humans and the gold dataset on noun-phrase completion with emergent properties (E <= 33 vs. gold 59.9), and that a spreading-activation prompting method improves some generative scores. The paper also reports a classification task where GPT-4o lags behind humans in property-type prediction.
Significance. If the central claims hold, CCPT fills a real gap: prior work on conceptual combination in NLP has addressed either property verification or noun-compound interpretation, but not the full component/emergent/canceled typology with generative tasks. The public release of data and code, the use of an externally validated LLM judge, and the explicit cognitive-science framing (spreading activation) are strengths. The headline finding that even strong models like o1 default to component-property solutions is interesting and falsifiable. However, the scope of the claim is narrower than the title and abstract suggest, because the test instances are drawn entirely from comparative sentences; the metric-validation evidence is on raw relevance rather than the derived E/C scores; and the spread-activation improvement is not consistently present across every model-task cell. These issues require qualification but are addressable within the manuscript's scope.
major comments (4)
- [§3.2 Step 1; Limitations (3)] The CCPT noun phrases are extracted exclusively from comparative sentences containing 'like' or 'as' (Section 3.2, Step 1), and the noun-phrase completion test in Table 3-(3) uses gold answers drawn from this pool. Comparative frames select for combinations that are striking enough to be worth a simile (e.g., 'as unstable as an apple on a toothpick') and for properties that are the explicit point of comparison. As the authors concede in the Limitations, 'the inherent nature of comparative sentences may introduce skewness, favoring certain types of properties over others.' Because the abstract and title state the conclusion about conceptual combination in general ('Is a Peeled Apple Still Red?'), this sampling restriction is load-bearing. The paper should either demonstrate that the emergent-property deficit reproduces on non-comparative constructions (e.g., adjective-noun compounds such as 'peeled apple' or noun-noun compounds such as 'dog house') or explicitly qualify the claim as applying to comparative-frame combinations.
- [§6.1, Equations (1)-(3)] The abstract's first claim is that the automatic metric 'grading property emergence and cancellation closely corresponds with human judgments.' However, Section 6.1 reports only the correlation between LLM-as-a-judge and human ratings on the raw relevance scores (R_H,P, R_M,P, R_N,P), with Pearson r=0.85. The emergence and cancellation scores E and C are nonlinear functions — max and max-minus — of those raw scores (Equations (1)-(3)), so a strong raw correlation does not by itself establish that the derived E and C values match human-derived E and C values. The authors should report the correlation or agreement between E and C computed from LLM and human relevance ratings on the same 300 sampled pairs, or provide an explicit derivation that raw-score correlation transfers to the max-based metrics.
- [Abstract; §5.1; Table 3] The abstract and Section 5.1 state that the proposed spread activation method 'improves performances in all generative tasks' and that it 'achieves the highest performance among the evaluated approaches.' These statements are not consistently supported by Table 3. For example, on NOUN PHRASE COMPLETION-EMERGENT, Qwen2.5-72B with S.A.w/LLM has E=23.3, worse than Base (24.5) and CoT (26.5); on the same task, LLaMa3.1-70B with S.A.w/Both has E=24.3, worse than S.A.w/LLM (26.6). Given that only three seeds are averaged and error bars overlap, the improvement claim needs significance testing or an explicit statement that the positive effect holds on aggregate but not for every model-task combination.
- [Appendix A.3] The inter-annotator agreement for property type annotation is low: Fleiss' kappa is 0.312 for emergent property data and 0.410 for canceled property data during annotation, and the final binary classification kappas are 0.498 and 0.505. These values indicate only moderate agreement, which raises questions about the reliability of the gold labels used to score LLM performance. The paper should discuss how much of the measured LLM deficit could be attributable to label noise, and ideally report the upper bound on achievable accuracy given this agreement.
minor comments (5)
- [§5.1] In the paragraph comparing the two generative tasks, the second reference to 'Table 3-(1)' should refer to Table 3-(3) for noun phrase completion.
- [Throughout] The model name 'LLaMa' should be written as 'LLaMA' or 'Llama' for consistency with the official naming.
- [§4] The test instance counts (N=200, 167, 167) are not justified; a short description of how the representative sample was drawn and whether it preserves the dataset's property-type balance would strengthen the evaluation.
- [Table 3 caption] The description of Multi-Oracle as 'selecting the best score among the multiple solutions' is ambiguous: clarify whether the best score is chosen per instance or per entire run, and how ties are broken.
- [§3.3] Figure 3's caption says 'PMI score' on a log-2 scale; consider labeling the axis as 'log2 PMI' to avoid confusion with the formula in the text.
Circularity Check
No significant circularity: the benchmark, metrics, and empirical findings are self-contained and do not reduce to their own inputs.
full rationale
The paper's central claims are empirical evaluations of LLMs on a new benchmark, not derivations from assumed conclusions. The emergence and cancellation metrics are defined directly from relevance judgments via E = max(R_N,P - R_{H,M,P}, 0) and C = max(R_{H,M,P} - R_N,P, 0), with R_{H,M,P} = max(R_{H,P}, R_{M,P}); no parameter is fitted to optimize a target score, and the LLM-as-a-judge is validated against external human ratings (Pearson r = 0.85, Spearman = 0.83). The dataset construction steps, including extraction from comparative sentences and GPT-4o-mini-based property filtering, feed the test instances but do not define the metric or the task outcomes. The finding that LLMs score low on noun phrase completion for emergent properties is an empirical result measured on held-out test samples, not an artifact of the metric being equivalent to the data selection criterion. The spread activation method's hyperparameters are hand-set heuristics and are not fitted to test performance, and its improvements are also confirmed by human evaluation on a subsample. The acknowledged limitation that comparative-sentence sampling may introduce skewness is a generalizability concern, not a circularity concern. There are no load-bearing self-citations, no imported uniqueness theorems, and no renamed known results presented as predictions. The chain from data to metric to result is transparent and externally checkable.
Assumptions & free parameters
free parameters (4)
- VERA-T5 alignment threshold =
0.7
- Spread activation maximum iterations T =
5
- Spread activation convergence threshold epsilon =
0.1
- Max candidates per noun phrase =
5
assumptions (4)
- domain assumption Property types (component, emergent, canceled) exhaust the property changes produced by conceptual combination.
- domain assumption Comparative sentences containing 'like' or 'as' yield noun phrases that exhibit emergent and canceled property phenomena.
- domain assumption LLM-as-a-judge relevance ratings (GPT-4o) are a valid proxy for human relevance ratings on generated responses.
- domain assumption MTurk annotations from English-speaking countries with high qualification thresholds represent general human commonsense about property types.
Cite this review
Pith. "Pith review of Is a Peeled Apple Still Red? Evaluating LLMs' Ability for Conceptual Combination with Property Type." pith.science (2026). https://pith.science/paper/SXUWRA6C
@misc{pith2026250206086,
author = {Pith},
title = {Pith review of: Is a Peeled Apple Still Red? Evaluating LLMs' Ability for Conceptual Combination with Property Type},
year = {2026},
howpublished = {\url{https://pith.science/paper/SXUWRA6C}},
note = {Machine review of arXiv:2502.06086}
}
read the original abstract
Conceptual combination is a cognitive process that merges basic concepts, enabling the creation of complex expressions. During this process, the properties of combination (e.g., the whiteness of a peeled apple) can be inherited from basic concepts, newly emerge, or be canceled. However, previous studies have evaluated a limited set of properties and have not examined the generative process. To address this gap, we introduce the Conceptual Combination with Property Type dataset (CCPT), which consists of 12.3K annotated triplets of noun phrases, properties, and property types. Using CCPT, we establish three types of tasks to evaluate LLMs for conceptual combination thoroughly. Our key findings are threefold: (1) Our automatic metric grading property emergence and cancellation closely corresponds with human judgments. (2) LLMs, including OpenAI's o1, struggle to generate noun phrases which possess given emergent properties. (3) Our proposed method, inspired by cognitive psychology model that explains how relationships between concepts are formed, improves performances in all generative tasks. The dataset and experimental code are available at https://github.com/seokwon99/CCPT.git.
Figures
Reference graph
Works this paper leans on
-
[1]
Head Noun: The original concept in the conceptual combination
-
[2]
Modifier: The word that modify head noun to create a new conceptual combination
-
[3]
Component Property: A property inherent to individual concepts (head noun or modifier)
-
[4]
Emergent Property: A new property that arises from the combination of the head noun and the modifier. This property does not exist in either concept individually (head noun or modifier) but emerge in conceptual combination
-
[5]
Table 9: System prompt for background of conceptual combination
Canceled Property: A property that is inherent to individual concept (head noun or modifier) and negated due to the combination. Table 9: System prompt for background of conceptual combination. Task Instruction for Base Method in Noun Phrase Completion Instructions:
-
[6]
Answer in dictionary format: {{"combination": "{{generated_combination}}", "modifier": "{{generated_modifier}}"}}. Do not include other formatting. <Example 1> - Head noun: apple - Emergent property: unappetizing - Correct answer: Let’s think step-by-step. A typical apple is fresh and appetizing, but certain modifications can make it unappetizing. Factors...
-
[13]
Answer in dictionary format: {{"combination": "{{generated_combination}}", "modifier": "{{generated_modifier}}"}}. Do not include other formatting. <Example 1> - Head noun: apple - Emergent property: unappetizing - Correct answer: {{"combination": "brown apple", "modifier": "brown"}} Above answer is correct because each component "brown" and "apple" do no...
-
[14]
Your task is to generate a conceptual combination by adding one modifier
You are given a head noun and emergent property. Your task is to generate a conceptual combination by adding one modifier
Show all 34 references
-
[20]
Your task is to generate a conceptual combination by adding one modifier
You are given a head noun, emergent property and a set of relevant concepts to solve a task. Your task is to generate a conceptual combination by adding one modifier
-
[21]
You can use function word without any constraint
-
[22]
Modifier should not have the given emergent property on its own, but the combination exhibits the emergent property
-
[24]
combination
Final answer should follow dictionary format: {{"combination": "{{generated_combination}}", "modifier": "{{generated_modifier}}"}}. Do not use other formatting. <Example 1> - Head noun: apple - Emergent property: unappetizing - Relevant concepts: [’bland’, ’bitter’, ’inedible’...
-
[28]
property
Answer in dictionary format: {{"property": "{{generated_property}}"}}. Do not include other formatting. <Example 1> - Combination: Brown apple - Correct answer: {{"property": "unappetizing"}} Above answer is correct because property "unappetizing" does not belong to either "br...
-
[29]
Your task is to generate emergent property of a combination
You are given a combination of concepts. Your task is to generate emergent property of a combination
-
[31]
Use the previous examples to learn how to do this
-
[33]
property
Answer in dictionary format: {{"property": "{{generated_property}}"}}. Do not include other formatting. <Example 1> - Combination: Brown apple - Correct answer: Let’s think step-by-step. "Brown" signifies a color change due to aging, oxidation, or decay, indicating the apple i...
-
[34]
Your task is to generate emergent property of a combination
You are given a combination of concepts and a set of relevant concepts to solve a task. Your task is to generate emergent property of a combination
-
[35]
Find a property that does not belong to any of the individual component in the combination but emerges when the words are combined
-
[36]
Come up with your reasoning process before giving your final answer
-
[37]
property
Final answer should follow dictionary format: {{"property": "{{generated_property}}"}}. Do not include other formatting. <Example 1> - Combination: Brown apple - Relevant concepts: [’fruit’, ’apple’, ’core’, ’cider’] - Correct answer: The goal is to find an emergent property o...
-
[38]
Your task is to predict a type of property
You are given a combination and property. Your task is to predict a type of property
-
[39]
- Component: The property is inherited by component of the combination
Definition of each property type is as follows: - Emergent: The property emerges from the combination of components. - Component: The property is inherited by component of the combination. - Canceled: The property is canceled out by the combination of components. - Others: The...
-
[40]
Use the previous examples to learn the task
-
[41]
property_type
Answer in dictionary format: {{"property_type": "{{property_type}}"}}. Do not include other formatting. <Example 1> - Combination: peeled apple - Property: round - Correct answer: {{"property_type": "component"}} Above answer is correct because property "round" is inherited by...
-
[44]
${property}
Fully Relevant Q2. Does the property emerge? (Your answer should be consistent with the 1-5 scaling above) Emergent property. The property "${property}" newly emerges by the combination. Component property. The property "${property}" is already in "${root}" or "${modifier}". O...
1971
-
[45]
${modifier}
Fully Relevant Q1-2. How relevant is "${modifier}" to "${property}"?
-
[46]
${combination}
Fully Relevant Q1-3. How relevant is "${combination}" to "${property}"?
-
[47]
${property}
Fully Relevant Q2. Is the property negated? (Your answer should be consistent with the 1-5 scaling above!!) Canceled property. The property "${property}" disappears (is negated) in "${combination}", but the property "${property}" was in "${root}" or "${modifier}". (High releva...
-
[48]
peeled" and
Property: White A peeled apple is usually white, even if "peeled" and "apple" aren’t directly related to being white. The property "white" emerges, thus "white" is an emergent property
-
[49]
round" remains, thus
Property: Round A peeled apple is still round, because apples are round. The property "round" remains, thus "round" is a component property. (Our property data may have mixed passive voice and active voice. Please understand in a way that makes sense.)
-
[50]
red" disappears, thus
Property: Red An apple is usually red, but when peeled, it's not red anymore. The property "red" disappears, thus "red" is a canceled property. Here are some already-solved examples for you. Important Use your own knowledge. Don't look up answers or use AI. We want your person...
-
[2011]
Creativity Research Journal, 23(3):203–210
Conceptual combinations and subsequent cre- ativity. Creativity Research Journal, 23(3):203–210. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. Advances in neural information processin...
2022 arXiv
-
[2021]
# of data
MediaSum: A large-scale media interview dataset for dialogue summarization. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5927–5934, Online. Association for Computation...
2021
-
[2024]
In Proceed- ings of the 2nd GenBench Workshop on Generalisa- tion (Benchmarking) in NLP, pages 131–153, Miami, Florida, USA
Is artificial intelligence still intelligence? LLMs generalize to novel adjective-noun pairs, but don‘t mimic the full human distribution. In Proceed- ings of the 2nd GenBench Workshop on Generalisa- tion (Benchmarking) in NLP, pages 131–153, Miami, Florida, USA. Association f...
2019 arXiv
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.