{"id":"f29545bd-4e42-4052-9baa-0da247be2499","arxiv_id":"2502.01901","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"CMT-based prompting produced higher average auto-evaluator scores on a small metaphor-heavy benchmark, but the evaluation lacks statistical tests and unbiased controls.","lead":"This paper tests whether telling large language models to think using conceptual metaphors, such as comparing time to money, improves their answers. The authors report higher auto-scored averages on a small custom benchmark, but the evaluation has serious methodological weaknesses that make the main claim unsupported.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation is unblinded and metaphor-rewarding, so the reported CMT advantage may be an evaluator artifact rather than a real reasoning improvement.","rationale":"The reader's weakest assumption correctly identifies the evaluation protocol as the load-bearing point. The paper's contribution is empirical: the CMT prompting strategy is claimed to improve reasoning, and the only quantitative evidence is the Llama3.3 70B judge's scores. The judge is shown the treatment labels and is asked to score metaphor effectiveness, which directly aligns with what the CMT prompt instructs the model to produce. This creates a clear confound: the judge may reward style and label rather than underlying reasoning quality. The absence of statistical testing compounds this, making the reported differences in Figure 4 hard to interpret. The paper has strengths: the CMT framing is coherent, the benchmark categories are diverse, and the system prompt in Figure 1 is a concrete, reproducible artifact. But no formal verification, no released code, and no blind human evaluation compensate for the non-blind automatic evaluation. The proposed label-swap test is a cheap, decisive way to determine whether the effect is real or an artifact of the evaluation prompt. Since the reader already recommended REJECT and this concern supports that conclusion, no verdict adjustment is needed.","tokens_in":7879,"tokens_out":2423,"duration_ms":25850,"concrete_test":"Swap the labels 'Baseline' and 'CMT' for the two responses in Figure 3's evaluator prompt while keeping the response texts identical, and measure whether the judge's preferred response flips. If it flips in the direction of the label, the evaluation is confounded by treatment disclosure and the empirical headline is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim depends entirely on the Llama3.3 70B auto-evaluator protocol described in Section VI.A and Figure 3. The evaluator is explicitly told which response comes from the CMT-prompted model and which from the baseline, and the scoring criteria reward 'effectiveness of analogy or metaphor' (Section V.C), while the CMT system prompt instructs the model to reason through source-target mappings. A judge primed with the treatment label and metaphor-oriented criteria can prefer the CMT response simply because it matches the requested rhetorical style, not because it is more accurate. The paper reports mean score differences without variance, significance tests, or blind human rating, so the abstract's claim that CMT prompting 'significantly enhances reasoning accuracy, clarity, and metaphorical coherence' is not established by the evidence as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Conceptual Metaphor Theory (CMT) as a prompting paradigm for large language models. The intervention is a system message (Figure 1) that instructs the model to reason by mapping abstract target domains onto concrete source domains, with illustrative examples (Figure 2). The authors compare four native models (Llama3.2, Phi3, Gemma2, Mistral) against their CMT-configured versions on 100 tasks in four categories: Metaphor Identification and Mapping, Domain-Specific Reasoning, Explanation and Teaching Tasks, and Reading Comprehension of Metaphors. Responses are scored by Llama3.3 70B using an evaluation prompt (Figure 3), and the paper reports average scores in Figure 4, concluding that CMT prompting 'significantly enhances reasoning accuracy, clarity, and metaphorical coherence.'","tokens_in":8018,"tokens_out":2426,"duration_ms":24151,"significance":"The idea of using cognitive-linguistic metaphor theory to structure LLM prompting is interesting and connects to a live research area on metaphor understanding and reasoning. The paper gives a clear statement of the intervention and a benchmark that covers several relevant task types. If the empirical claims were supported by sound, blinded, statistically grounded evaluation, the result would be a useful practical contribution to prompt engineering. However, the current evidence does not establish the central claim: the automated evaluator is unblinded, the scoring criteria explicitly reward metaphor use, no significance tests or variance estimates are reported, and at least one displayed result contradicts the text. The paper would need substantial additional empirical work to support its abstract-level conclusion.","major_comments":[{"comment":"The evaluation protocol is unblinded and confounded with the intervention. The evaluation prompt in Figure 3 explicitly labels the two responses as 'Baseline Model Response' and 'CMT-prompted Model Response,' and the criteria include 'effectiveness of analogy or metaphor' (Section V.C). Because the CMT system prompt instructs the model to produce source-target metaphorical mappings, the judge can prefer CMT responses simply because they match the requested rhetorical style, not because they are more accurate. The paper's central claim that CMT 'significantly enhances reasoning accuracy' therefore rests on an outcome measure that overlaps with the intervention. A blinded evaluation, or at minimum a judge that does not know which response is CMT, is necessary to support the claim.","section":"Section VI.A and Figure 3"},{"comment":"No statistical evidence is provided for the word 'significantly' in the abstract. Figure 4 reports only average scores without error bars, standard deviations, per-task counts, or significance tests, and the text in Section VI.A claims inter-rater agreement was assessed but gives no values. Without variance or raw data, the reader cannot distinguish systematic improvement from noise, especially on a 100-task benchmark with small per-category samples. The manuscript should report confidence intervals or significance tests, and ideally provide the response-level data or a public artifact.","section":"Section VI.C and Figure 4"},{"comment":"There is a direct internal contradiction in the RCM results. In Figure 4 (right), the llama3.2 baseline RCM score is 3.81 while the CMT score is 1.24, a large negative difference, yet Section VI.C states that 'Llama3.2, Gemma2, and Mistral showed substantial gains' in RCM, and Section VII concludes that 'Llama3.2 benefits the most from CMT.' Either the figure labels are misaligned or the textual summary is wrong; this must be corrected and reconciled because it affects the paper's per-model conclusions.","section":"Section VI.C, Figure 4 (right), and Section VII"},{"comment":"The comparison lacks an appropriate control for Chain-of-Thought (CoT). The CMT prompt explicitly uses a 'CoT-like approach' with stepwise inference (Section III.B), but the baseline models receive no CoT instruction. Consequently, any observed improvement could be attributed to the presence of step-by-step reasoning rather than to conceptual metaphor mappings. A CoT-only control condition, matched for prompt length and reasoning steps, is needed to isolate the CMT contribution.","section":"Section III.B and Section VI.A"},{"comment":"The benchmark tasks and evaluation criteria appear to reward the very behavior the intervention is designed to elicit, which raises a circularity concern. Section V states that task instructions deliberately avoid prompting metaphorical reasoning, but the evaluation criteria in Sections V.C and V.D explicitly reward 'effectiveness of analogy or metaphor' and 'depth of interpretive insight.' The paper should either use outcome measures that are independent of metaphor use or justify why metaphor-richness is an acceptable proxy for reasoning quality on these tasks.","section":"Section V.C and Figure 3"}],"minor_comments":[{"comment":"The figure caption contains a typo: 'CMT-inspred CoT' should read 'CMT-inspired CoT.'","section":"Figure 2"},{"comment":"The claim that 'Independent evaluations by multiple annotators ensured consistency, with inter-rater agreement assessed for reliability' is not supported by any reported data, and it is unclear how these human annotations relate to the Llama3.3 evaluations that are actually presented.","section":"Section VI.A"},{"comment":"The phrase 'Minor anomalies, such as unusually high CMT-enhanced scores, suggest potential refinements to evaluation methods' is vague and does not identify which models or tasks are anomalous; the RCM llama3.2 result identified above is more than a minor anomaly and needs explicit discussion.","section":"Section VI.C"},{"comment":"Reference [4] has a formatting error: the journal/proceedings volume information appears to be duplicated from reference [3], and the venue should be verified.","section":"References"},{"comment":"Temperature is the only free parameter mentioned, and it is fixed at 0.7 without any sensitivity analysis; a brief justification or ablation would increase confidence that the results are not sensitive to this choice.","section":"Section III.C"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is a modest prompt-engineering paper with a plausible idea and an evaluation that cannot support its headline claim. The CMT vs. baseline comparison is not trustworthy because the judge knows which response is which and the scoring criteria reward metaphor use—the exact behavior the CMT prompt induces. The abstract's 'significantly enhances' should be read as 'preferred by a label-aware judge using metaphor-weighted criteria.'\n\nWhat is actually new: the benchmark of 100 tasks across four categories (MIM, DSR, ETT, RCM) is thoughtfully designed, and the qualitative observations in Section VI.C give a useful sense of where metaphor prompting might help, especially technical explanations and teaching-oriented output. The idea of using CMT as a system prompt is a natural extension of the author's prior cognitive prompting work. The paper is readable and honest enough to flag 'unusually high CMT-enhanced scores' as a potential evaluation artifact.\n\nThe soft spots are in the evidence, not the idea. The evaluator prompt in Figure 3 labels responses as 'Baseline' and 'CMT'; the ETT criteria explicitly include 'Effectiveness of Analogy or Metaphor'; and the CMT system prompt instructs the model to map source to target domains. A judge primed with the treatment label and metaphor-oriented criteria can prefer the CMT response for stylistic reasons, not because it is more accurate. There are no significance tests, no error bars, and no raw data. The paper mentions inter-rater agreement but never reports it. There is no control for plain chain-of-thought prompting, so we cannot attribute gains to CMT specifically rather than to longer, structured instructions. The models are small (2B–7B) and the judge is larger (70B), which is fine, but it does not fix the labeling problem.\n\nThe central argument doesn't hold as stated. The idea is worth a careful follow-up with blind evaluation, a CoT control, and some statistical reporting. For a workshop this would be a reasonable discussion piece; as a citable empirical claim it needs another pass.\n\nFor peer review: I would send it to review only with the expectation of a major revision. The benchmark and idea deserve referee time, but the current evaluation section would need to be redone. If you are considering it for a reading group, it is a good example of LLM-as-judge bias in action.","headline":"Plausible prompt idea, but the evaluation is unblinded and metaphor-rewarding, so the reported CMT advantage is likely an artifact.","tokens_in":8482,"tokens_out":3794,"would_cite":false,"duration_ms":34600,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding one Conceptual Metaphor Theory system message, which tells the model to map abstract target domains onto concrete source domains and draw stepwise inferences, improves LLM reasoning accuracy, clarity, and metaphorical coherence…","keywords":["Conceptual Metaphor Theory","prompt engineering","chain-of-thought prompting","source-target mapping","large language models","automatic evaluation","reasoning benchmarks","metaphor interpretation"],"falsifier":"Run the same 100-task benchmark with the condition labels removed from the judge prompt—or with independent human raters applying the same criteria—and check whether CMT responses still win systematically; if the advantage shrinks to noise, the paper's central empirical claim collapses.","tokens_in":7671,"feed_emoji":"🧠","tokens_out":8083,"duration_ms":70855,"temperature":0.7,"pith_summary":"The paper reports that a single system-message change—instructing the model to reason through Conceptual Metaphor Theory, mapping abstract target domains onto concrete source domains and drawing step-by-step inferences—improves LLM performance on metaphor-heavy reasoning and explanation tasks. Four small models (Llama3.2, Phi3, Gemma2, Mistral) were each run with and without this CMT prompt on a 100-task benchmark covering metaphor identification, domain-specific reasoning, explanation and teaching, and metaphor reading comprehension. The central claim is that the CMT-prompted versions outperform their baselines across all four categories, as scored by a Llama3.3 70B judge on accuracy, coherence, and metaphorical alignment. If true, the result matters because it offers a cheap, task-independent way to make LLM reasoning more structured and its explanations more intuitive, with no fine-tuning and no per-task example crafting.","feed_headline":"Metaphor prompts beat plain prompts on all evaluated tasks","feed_subtitle":"Four small models scored higher on reasoning, clarity, and metaphor quality after one system-prompt change.","key_machinery":"The load-bearing object is the CMT system prompt: a fixed instruction telling the model it is a cognitive agent that interprets abstract concepts (target domains) through concrete experiences (source domains), followed by a mapping-and-inference template and three worked metaphors (time is money, heart of stone, world is a stage). The prompt is applied uniformly across tasks at temperature 0.7, so the only difference between baseline and CMT conditions is the presence of this metaphor-reasoning instruction and its examples. On the evaluation side, the machinery is an LLM judge that receives both responses plus the task description and scoring criteria, assigns 1–5 scores on three criteria per category, and selects the superior response; the CMT condition wins in the reported comparisons.","core_discovery":"The discovery, on the paper's own terms, is that conceptual-metaphor structure is a usable reasoning scaffold for LLMs. By encoding source-domain-to-target-domain mappings once in the system prompt—with three worked examples showing how to infer properties of an abstract concept from a concrete experience—the authors turn metaphor from a linguistic decoration into an inference procedure, in the same spirit as chain-of-thought turns arithmetic into a sequence of steps. The resulting responses are judged by Llama3.3 70B to be more accurate, clearer, and more metaphorically coherent than baseline responses, with gains largest where explanations must transfer structure from one domain to another. The paper interprets this as evidence that CMT-based prompting provides a generalizable cognitive framework rather than a set of task-specific tricks.","pith_inferences":["Because the judge in Figure 3 is told which response is the CMT one and is instructed to reward the 'effectiveness of analogy or metaphor,' part of the reported margin may reflect style preference rather than reasoning quality; a blind judge or human raters would test this.","CMT prompting may primarily raise rhetorical coherence rather than factual correctness; separating correctness from communicative quality would clarify what the gains are made of.","The benchmark is small—100 tasks with selected instances per category and no reported variance or significance tests—so the word 'significant' in the abstract should be read as descriptive, not statistical, until error bars are supplied.","Using an evaluator from the same model family as one of the tested models (Llama3.2 vs Llama3.3 judge) invites self-preference effects; evaluating with a different judge family would strengthen the comparison."],"forward_implications":["A single CMT system message could replace per-task prompt engineering for explanation-heavy applications, because no task-specific examples are required.","Small models—including a 2B-parameter model—show gains, so the recipe may transfer to low-resource or on-device settings where fine-tuning is impractical.","If the score gap reflects genuine improvement, metaphor-guided prompting is a default option for teaching, tutoring, and explainability outputs where analogies carry the communication.","The uneven model-by-model results imply that practitioners should expect CMT prompting to help some architectures and task types more than others, so conditional use is a reasonable deployment strategy."],"supporting_citations":[{"why":"Foundational statement of Conceptual Metaphor Theory; supplies the source-domain/target-domain structure the prompts encode.","marker":"[1]"},{"why":"Extends CMT as a contemporary theory of metaphor; supports treating metaphor as conceptual mapping rather than linguistic decoration.","marker":"[2]"},{"why":"Introduces chain-of-thought prompting; the CMT prompts adopt its stepwise reasoning format.","marker":"[4]"},{"why":"Introduces cognitive prompting, the prior approach this work extends by adding metaphor-based instructions.","marker":"[10]"},{"why":"Earlier work using chain-of-thought prompts for metaphor understanding, which this paper contrasts with its structured-reasoning evaluation.","marker":"[11]"}],"fun_headline_variants":["Metaphor prompts sharpen LLM reasoning across tasks","CMT prompting lifts reasoning, clarity, metaphor quality","Metaphor as scaffold: CMT prompts beat baselines","One prompt change: metaphor framing improves LLM scores","Metaphor-aware prompts outperform plain prompts in tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result rests on trusting that the Llama3.3 70B judge gives unbiased quality scores even though it can see which response came from the CMT-prompted model and is explicitly asked to reward metaphor effectiveness.","fun_headline_variants_meta":{"raw":{"variants":["Metaphor prompts sharpen LLM reasoning across tasks","CMT prompting lifts reasoning, clarity, metaphor quality","Metaphor as scaffold: CMT prompts beat baselines","One prompt change: metaphor framing improves LLM scores","Metaphor-aware prompts outperform plain prompts in tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000115,"raw_usage":{"total_tokens":1017,"prompt_tokens":833,"completion_tokens":184,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":106}},"tokens_in":449,"tokens_out":184,"duration_ms":3023,"temperature":1.0,"reasoning_tokens":106,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T14:02:48.734947+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 100-task benchmark with the condition labels removed from the judge prompt—or with independent human raters applying the same criteria—and check whether CMT responses still win systematically; if the advantage shrinks to noise, the paper's central empirical claim collapses.","supporting_citations":[{"cited_title":"Metaphors We Live By","cited_arxiv_id":null,"evidence_quote":"Foundational statement of Conceptual Metaphor Theory; supplies the source-domain/target-domain structure the prompts encode."},{"cited_title":"The contemporary theory of metaphor","cited_arxiv_id":null,"evidence_quote":"Extends CMT as a contemporary theory of metaphor; supports treating metaphor as conceptual mapping rather than linguistic decoration."},{"cited_title":"Chi, Quoc V","cited_arxiv_id":null,"evidence_quote":"Introduces chain-of-thought prompting; the CMT prompts adopt its stepwise reasoning format."},{"cited_title":"Unlocking structured thinking in language models with cognitive prompting","cited_arxiv_id":null,"evidence_quote":"Introduces cognitive prompting, the prior approach this work extends by adding metaphor-based instructions."},{"cited_title":"Good- man","cited_arxiv_id":null,"evidence_quote":"Earlier work using chain-of-thought prompts for metaphor understanding, which this paper contrasts with its structured-reasoning evaluation."}],"review_version":1}