{"id":"2245f270-bd0e-4377-ada1-474aa56a2136","arxiv_id":"2412.00869","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On a new 15,000-item proportional analogy MCQ benchmark, GPT-3.5-Turbo reaches 55.25% accuracy, and prompts that inject the target semantic relation outperform exemplar and structured-knowledge prompts.","lead":"The authors built a 15,000-question multiple-choice benchmark for proportional analogies and tested nine language models under zero-shot, few-shot, structured-knowledge, and targeted-knowledge prompts. The best model, GPT-3.5-Turbo, reached 55% accuracy, and telling the model the semantic relation between the question pair helped more than giving examples or graph-derived knowledge.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline accuracy and TKP advantage rest on unverified auto-generated AnalogyKB labels, because the winning TKP prompt injects the same relation label used to construct the gold answer.","rationale":"The reader's weakest assumption is exactly the correctness of the 14K auto-generated MCQs inherited from AnalogyKB, and I agree this is the most load-bearing point. The paper's strongest empirical claim—GPT-3.5-Turbo reaching 55.25% with TKP and TKP outperforming zero-shot, few-shot, and structured-knowledge prompts—depends on the validity of the gold answers and on the relation labels supplied by TKP. Since both come from the same automatically generated source without reported human validation, a systematic label error rate would simultaneously corrupt the dataset and inflate or deflate the TKP result. The reader's conditional verdict is appropriate: the benchmark and the main comparison are plausible and reproducible in principle, but they need external validation before the accuracy numbers can be taken at face value. I would not move the verdict to reject because the concern is addressable and the 1K manually annotated subset plus the internal consistency of the method provide some support. A targeted human audit of the auto-generated labels is the single check that would settle whether the concern lands.","tokens_in":21848,"tokens_out":5546,"duration_ms":55638,"concrete_test":"Human-verify a stratified random sample of 400 MCQs from the 14K auto-generated portion, oversampling rare relation types and the five most frequent relations: two independent annotators judge (a) whether the stated relation actually holds between the question pair, and (b) whether exactly one answer choice shares that relation. Then recompute GPT-3.5-Turbo zero-shot and TKP exact-match accuracy on the subset where both annotators agree the gold label is correct and unique. If the verified-subset TKP-minus-zero-shot gap is materially smaller than 9.55 points, or if more than 5% of sampled labels are wrong or ambiguous, the headline comparison is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 builds 14K of the 15K MCQs by directly adopting AnalogyKB (Yuan et al., 2023) without human verification, while only the 1K borrowed examples get manual relation annotation. The TKP prompt (Section 3.3.4, Appendix D) then supplies the question pair's relation label, and the gold answer was constructed as the pair sharing exactly that relation, with distractors chosen to have different relations. Consequently, the same unvalidated automatic pipeline determines both the correctness key and the decisive information in the winning prompt condition. If AnalogyKB's KG-derived relation labels are noisy, or if a distractor pair also satisfies the stated relation (no uniqueness check is described), the reported 55.25% TKP score and the 9.55-point TKP-over-zero-shot gap reflect label artifacts rather than model analogy ability. The absence of a chance baseline and confidence intervals further prevents separating genuine reasoning gains from the effect of revealing the relation. This is not an objection to knowledge-enhanced prompting in general; it is a claim that the evaluation's ground truth and the targeted-knowledge cue come from the same unverified source, so the central comparison is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a 15K multiple-choice proportional-analogy dataset, of which 14K items are generated from AnalogyKB and 1K are borrowed from existing collections with manual relation annotation. It evaluates nine LLMs under six prompting conditions: zero-shot, one-shot, five-shot, structured knowledge prompting with random or semantic filtering, and targeted knowledge prompting. The central reported result is that GPT-3.5-Turbo with Targeted Knowledge Prompting achieves 55.25% exact-match accuracy, the best score in the study, and that targeted knowledge outperforms exemplar and structured knowledge, while structured knowledge often hurts performance relative to zero-shot prompting.","tokens_in":22081,"tokens_out":3849,"duration_ms":37251,"significance":"If the empirical claims are valid, the paper would provide a substantially larger and more relation-diverse benchmark for proportional analogies than prior datasets, and its comparison of knowledge-enhanced prompting strategies would be informative for LLM evaluation. The authors ship code and data, evaluate nine models, and report per-relation results, which are useful contributions. However, the headline TKP advantage is not currently established because the TKP prompt receives the same relation label that was used to construct the correct answer for the large auto-generated subset. The lack of chance baselines, confidence intervals, and significance tests further weakens the central comparison. With added controls, the dataset and evaluation could still be valuable.","major_comments":[{"comment":"The TKP prompt states the semantic relation shared by the question pair, e.g., 'The implicit relation shared by lens and glass is made of.' For the 14K auto-generated items, Section 3.1 constructs the correct answer as the pair sharing exactly that relation and the distractors as pairs with different relations. Therefore, TKP does not measure proportional analogy completion so much as relation matching, and the relation label is derived from the same unvalidated pipeline that produced the answer key. The 9.55-point gap between TKP and zero-shot prompting (55.25% vs. 45.7%) is confounded by this label leakage. The authors should evaluate TKP on items whose relation labels were not used in distractor construction, or add a control prompt that injects an incorrect or held-out relation label, to separate genuine reasoning gains from answer-revealing information.","section":"Section 3.3.4 and Figure 11 vs. Section 3.1"},{"comment":"The paper reports point estimates of exact-match accuracy without chance baselines, confidence intervals, or significance tests. Given the dataset mix of 14,386 five-way, 610 four-way, and 4 three-way MCQs, random guessing yields roughly 20% accuracy, so several reported values are close to or below chance (e.g., CodeParrot 0% in zero-shot, CodeT5 0% in five-shot, GPT-2 6.29% in SKP[random]). Moreover, some of the central claims rest on small absolute differences, such as Falcon's zero-shot 24.17% vs. TKP 25.40% and Mistral's zero-shot 26.59% vs. TKP 27.37%. The authors should report expected chance accuracy, per-condition confidence intervals, and pairwise significance tests or effect sizes before claiming that targeted knowledge outperforms exemplar knowledge and that structured knowledge underperforms zero-shot prompting.","section":"Table 2 and Section 5.1"},{"comment":"The 14K auto-generated MCQs are adopted from AnalogyKB without human verification of the relation labels, and the paper does not describe any check that the correct answer pair is the unique distractor pair satisfying the stated relation. If AnalogyKB's automatically extracted relations are noisy, or if a distractor pair also satisfies the same relation, then both the correctness key and the TKP prompt are corrupted, because the TKP prompt uses that same relation label. The authors should provide a human-annotated or human-audited subset of the 14K items, report label-agreement statistics, and describe a uniqueness check for the correct answer against distractors. Without this, the dataset's ground truth is not independently established.","section":"Section 3.1 and Table 1"}],"minor_comments":[{"comment":"There are several typos and formatting issues: 'Perfromance' in the Figure 4 caption, 'Abalation' in Figure 4, 'ConcepNet' in Section 3.1, 'wwe use' in Appendix A, and '1.5 parameters' in Appendix A should be '1.5B parameters'.","section":"Throughout"},{"comment":"The figure text in the PDF appears garbled or duplicated, and labels such as 'zeroshotcota' and 'cota-zeroshot' seem to be leftovers from a previous version; these should be cleaned up for a publication version.","section":"Figures 4 and 5"},{"comment":"The phrase 'path finding approaches oppose to subgraph extraction approaches' should read 'as opposed to'.","section":"Section 3.3.3"},{"comment":"The text says the ablation is run on '20% of the 15K dataset,' and Table 4 reports 2995 data points; the relationship between these numbers should be stated clearly, and the subset selection procedure should be described.","section":"Section 5.2 and Table 4"},{"comment":"The five-shot prompt example contains placeholders such as 'Example 3: ......................' instead of the actual exemplars; include the full exemplar set for reproducibility.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The dataset release and broad model comparison are potentially useful to the community, but the central TKP advantage is not yet defensible because the relation label in the TKP prompt is the same information used to construct the answer key. The authors should be encouraged to add a control condition and uncertainty quantification rather than merely expanding the evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful dataset contribution, but the central comparison is confounded. The 15K MCQ set with 236 relation types is real progress over the sub-1K sets typically used. The systematic comparison of zero-shot, few-shot, structured-knowledge, and targeted-knowledge prompting across nine models is worth having, and the code and data are released.\n\nThe soft spot is not subtle. TKP supplies the relation label for the question pair, and that same relation label was used to construct the correct answer and to pick distractors with different relations. So the model is being handed the answer key. For the 14K auto-generated items, the label comes from AnalogyKB with no human verification. If those labels are noisy, or if a distractor happens to share the stated relation (no uniqueness check is described), the reported 55.25% and the 9.55-point gap over zero-shot reflect dataset construction rather than analogy ability. The stress-test note is right: Section 3.1 and Figure 11 confirm the same unvalidated pipeline supplies both the gold answer and the decisive prompt cue.\n\nThere are smaller issues too: no confidence intervals, no significance tests, no chance or human baseline. For a 15K MCQ benchmark, chance alone is nontrivial (3-, 4-, 5-way), and a human baseline would help calibrate what 55% means. The paper does acknowledge some limitations—manual prompt brittleness, possible knowledge leakage in SKP—but not the core circularity.\n\nI still think the paper deserves a serious referee. The dataset is large and potentially reusable, and the confound is fixable: evaluate on the 1K human-verified items, or run TKP without revealing the relation, or add a baseline where the relation label is chosen randomly. The authors have done the work of building a resource; the claims just need to be recalibrated to what the evidence can support. I'd send it to review with a request for major revision rather than desk-reject.","headline":"A genuinely larger analogy benchmark, but the headline TKP win is partly built into the data, so the paper needs revision before its main claim can be trusted.","tokens_in":22611,"tokens_out":2242,"would_cite":true,"duration_ms":20680,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Providing LLMs the specific semantic relation between terms lifts proportional-analogy accuracy, but the task remains largely unsolved.","keywords":["proportional analogies","large language models","knowledge-enhanced prompting","targeted knowledge prompting","multiple-choice question answering","semantic relations","chain-of-thought prompting","analogy reasoning"],"falsifier":"Audit a random sample of the 14K automatically generated questions for label correctness and answer uniqueness; if a substantial fraction of labels are wrong or multiple choices are defensible, the reported 55.25% and the TKP advantage are confounded. A second decisive test is to run TKP with a deliberately wrong relation label—if accuracy does not drop, the targeted relation is not what drives the gain.","tokens_in":21690,"feed_emoji":"🧩","tokens_out":8620,"duration_ms":66135,"temperature":0.7,"pith_summary":"This paper tries to establish that completing proportional analogies—'A is to B as C is to D'—remains hard for current large language models, and that the most effective prompt enhancement is not more examples or more facts, but telling the model the specific semantic relation shared by the question pair. The authors build a 15,000-question multiple-choice dataset spanning 236 relation types and evaluate nine LLMs under six prompt variants. Their best result, 55.25% exact-match accuracy, comes from GPT-3.5-Turbo with Targeted Knowledge Prompting, which gives the model the relation label (such as 'made of') plus a one-sentence reasoning instruction. The claim matters because analogy completion is a core reasoning benchmark, and the result suggests that targeted, minimal cues help LLMs more than dumping structured knowledge into the prompt.","feed_headline":"Giving LLMs the relation label lifts analogy accuracy to 55 percent","feed_subtitle":"Targeted knowledge beats examples and graphs on 15K analogy questions, yet the 55% ceiling shows the task is unsolved.","key_machinery":"The load-bearing object is the Targeted Knowledge Prompt (TKP), built by taking the zero-shot MCQ prompt and inserting (i) the implicit semantic relation shared by the question pair—e.g., 'made of' for 'lens' and 'glass'—and (ii) a sentence describing the human cognitive process of choosing the answer pair with that same relation. The TKP relation label comes from the dataset's ground-truth relation, which for the 14K automatically generated questions is inherited from the AnalogyKB construction process. The comparison arms are the same prompt with exemplar knowledge (one- and five-shot) and with structured knowledge (filtered paths from Wikidata, ConceptNet, and WordNet). The 15K dataset itself, with 236 relation types, is the test bed that makes the comparison possible.","core_discovery":"The paper's central claim is that targeted knowledge—the name of the semantic relation between the two question terms, plus the instruction to choose the answer pair sharing that relation—is the most effective way to help LLMs solve proportional analogies. On the new 15K MCQ dataset, Targeted Knowledge Prompting gives the best result for four of nine models, including the overall peak of 55.25% for GPT-3.5-Turbo; it beats zero-shot prompting (45.7% for the same model), one- and five-shot exemplar prompting, and Structured Knowledge Prompting, which appends filtered paths from WordNet, ConceptNet, and Wikidata. The authors also find that structured knowledge can hurt: GPT-3.5-Turbo drops to roughly 38% with structured-knowledge prompts, below its zero-shot 45.7%. They conclude that proportional analogies remain far from solved and that relation-focused prompting outperforms knowledge dumping.","pith_inferences":["A natural next test would be to have the model predict the relation label for the question pair before applying TKP; if accuracy stays near 55%, the gain comes from applying the label rather than from the label being given.","Because the relation label in TKP is the same label used to construct the correct answer, the reported 55.25% measures accuracy when relation identification is bypassed; a stricter test would withhold the label until after the model commits to an answer.","The underperformance of SKP suggests that knowledge-graph paths are too noisy for relation-sensitive reasoning; a testable fix is to filter paths by their relation type matching the question relation, rather than by sentence similarity.","The cost-benefit discussion implies that automated relation-label prediction, not bigger retrieval-augmented pipelines, would be the higher-leverage investment for analogy benchmarks."],"forward_implications":["If the TKP result holds, the most efficient prompt upgrade for analogy-style multiple-choice tasks is a single accurate relation label, not a retrieval pipeline or many exemplars.","The SKP drop below zero-shot shows that adding loosely relevant structured knowledge can mislead models, so knowledge-enhanced prompting needs precise relevance filtering rather than raw volume.","The 55.25% ceiling across nine models means proportional analogy completion remains an open reasoning benchmark for LLM development.","The fact that only four of nine models peak under TKP, while others peak under zero-shot or few-shot, indicates the benefit of targeted knowledge is real but model-dependent."],"supporting_citations":[{"why":"Source of the 14K automatically generated analogies and of the semantic relation labels that both the dataset and the TKP prompt use.","marker":"Yuan et al. (2023)"},{"why":"Provides the MCQ prompt phrasing that all six prompting variants build on.","marker":"Robinson et al. (2023)"},{"why":"Prior claim that GPT-3 can surpass humans on small proportional-analogy sets; this paper scales the evaluation up and reaches a different conclusion about difficulty.","marker":"Webb et al. (2023)"},{"why":"Supplies part of the 1K borrowed questions and the earlier proportional-analogy benchmark tradition.","marker":"Ushio et al. (2021)"},{"why":"Chain-of-Thought prompting, which TKP modifies by adding the targeted relation instead of an exemplar.","marker":"Wei et al. (2022)"},{"why":"The path-finding retrieval method used to pull ConceptNet and Wikidata knowledge for the SKP comparison.","marker":"Lin et al. (2019)"},{"why":"ConceptNet, one of the three structured-knowledge sources in SKP.","marker":"Speer et al. (2017)"},{"why":"Wikidata, the world-knowledge source in SKP.","marker":"Vrandečić and Krötzsch (2014)"},{"why":"English WordNet, the lexical-knowledge source in SKP.","marker":"McCrae et al. (2019)"}],"fun_headline_variants":["Relation label pushes LLM analogy score to 55%","Targeted knowledge beats examples and graphs for analogies","Exemplars and graphs fail: relation labels lift LLM analogies","Structured knowledge can hurt: targeted prompts best for analogies","LLMs still fail analogies: best prompt hits 55%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 14,000 automatically generated multiple-choice questions correctly label each pair's semantic relation and have exactly one defensible answer, so both the accuracy numbers and the TKP prompt depend on those labels being trustworthy.","fun_headline_variants_meta":{"raw":{"variants":["Relation label pushes LLM analogy score to 55%","Targeted knowledge beats examples and graphs for analogies","Exemplars and graphs fail: relation labels lift LLM analogies","Structured knowledge can hurt: targeted prompts best for analogies","LLMs still fail analogies: best prompt hits 55%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1309,"prompt_tokens":967,"completion_tokens":342,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":270}},"tokens_in":583,"tokens_out":342,"duration_ms":3510,"temperature":1.0,"reasoning_tokens":270,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:54:28.152606+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit a random sample of the 14K automatically generated questions for label correctness and answer uniqueness; if a substantial fraction of labels are wrong or multiple choices are defensible, the reported 55.25% and the TKP advantage are confounded. A second decisive test is to run TKP with a deliberately wrong relation label—if accuracy does not drop, the targeted relation is not what drives the gain.","supporting_citations":[],"review_version":1}