{"id":"153ee7a4-f8ad-4962-aacb-581fd4942d1c","arxiv_id":"2505.13282","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LORex combines a trained candidate ranker with batched LLM reasoning to insert new concepts into taxonomies, outperforming twelve baselines on most benchmarks while overstating the average gain.","lead":"This paper introduces LORex, a pipeline that ranks candidate parent terms with a trained BERT scorer and then uses a LLaMA model to filter, retrieve, and verify hypernyms in small batches. It reports gains on four taxonomy benchmarks, but the headline improvement percentages are not fully supported by the tables.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 12% accuracy gain is not reproducible from Table 2: LORex8B-3.1I is worse than FLAME on Food accuracy (55.3 vs 58.7), and the mean relative gain across the four datasets is about 3%, not 12%.","rationale":"I read the paper as an engineering contribution that combines a BERT-based ranker with LLaMA-based filtering, retrieval, and verification. The pipeline is described in enough detail to be reimplementable, the pseudocode and prompts are included, and the ablations isolate several components. The central question for the paper's headline is whether LORex actually achieves the stated 12% accuracy and 5% Wu&P gains. The paper's own Table 2 is the most direct evidence for that claim, and it does not support the 12% figure under standard aggregation. The Food benchmark is especially problematic because LORex8B-3.1I is 3.4 accuracy points below FLAME, so the broad statement 'surpasses ... FLAME' is false for that benchmark. This is not a dispute about whether the method is useful; it is a dispute about whether the published numbers license the paper's strongest empirical conclusion. The reader's verdict is already CONDITIONAL, and their rationale mentions the 12% mismatch, but their designated weakest assumption is the top-15 ranker truncation. I agree that the ranker dependency is real and clearly acknowledged in the Limitations, but the more load-bearing issue for the central claim is that the headline accuracy improvement cannot be reproduced from Table 2. My recommended verdict is therefore unchanged: CONDITIONAL, with the condition that the authors correct or precisely qualify the claimed aggregate gains and report the exact aggregation used.","tokens_in":20225,"tokens_out":7743,"duration_ms":73054,"concrete_test":"Recompute the aggregate improvements from Table 2 using the published rows for LORex8B-3.1I and FLAME: compute per-dataset relative and absolute differences, then average over the four benchmarks. If the mean relative accuracy gain is about 3.1% rather than 12%, the abstract and Section 5.2 overstate the result and must be revised. To test whether the 12% figure arises from a different baseline set or aggregation, the authors should report the exact computation used. As a complementary check, run the Food benchmark with all candidates instead of the top-15 truncation; if the accuracy margin vs FLAME remains negative, the claim of surpassing FLAME on this benchmark is untenable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim, stated in the Abstract and Section 1 and repeated in Section 5.2 as an improvement over FLAME, is that LORex improves accuracy by 12% and Wu&P by 5%. The paper's own Table 2 does not support those numbers. Comparing LORex8B-3.1I with FLAME: Env accuracy 67.3 vs 63.4 (+3.9 pp), Sci 64.7 vs 63.2 (+1.5 pp), Food 55.3 vs 58.7 (-3.4 pp), WordNet 49.5 vs 45.2 (+4.3 pp). The mean absolute gain is 1.6 percentage points; the mean relative gain is about 3.1%; the largest single-dataset relative gain is 9.5% (WordNet). No standard aggregation yields 12%. Wu&P is also mixed: the absolute mean is near 5%, but Environment decreases (82.9 vs 85.1), so a blanket '5% gain' is misleading. Because the 12%/5% numbers are the quantitative basis for the 'state-of-the-art without fine-tuning' conclusion, the strongest claim fails unless the aggregation method is disclosed and reproduced. A related but secondary tension: the contribution 'without discarding relevant candidates' is contradicted by the top-15 truncation in Appendix F and by Table 5, where the semantic filter skips 45 of 719 batches containing the true parent. The ranker-dependency issue is acknowledged in the Limitations, but the unreproducible headline accuracy gain is not acknowledged at all.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LORex, a taxonomy-expansion framework that combines a discriminative candidate ranker (TEMPORA) with an LLM pipeline of semantic filtering, parent retrieval, and path-based verification. Candidates are ranked, chunked into batches, and iteratively processed; the claimed contribution is a plug-and-play method that avoids LLM fine-tuning and does not discard relevant candidates. The method is evaluated on four benchmarks (SemEval-2016 Env, Sci, Food, and WordNet) against twelve baselines, and the abstract and introduction claim a 12% accuracy improvement and a 5% Wu & Palmer (Wu&P) gain over the best baseline. The paper also includes ablations on chunk size, verifier type, ranker contribution, and semantic filtering, plus case studies and a limitations section.","tokens_in":20504,"tokens_out":4319,"duration_ms":39740,"significance":"If the headline results were reproducible, LORex would be a practically valuable contribution: it offers a training-light integration of discriminative ranking with LLM reasoning, and the TEMPORA ranker appears to improve over prior path-based rankers such as TEMP. The manuscript has several concrete strengths: it publicly releases code, evaluates on four benchmarks, reports ablations for each module, and provides failure-case analyses. However, the central quantitative claim is not supported by the paper's own Table 2, and the Appendix F top-15 truncation materially qualifies the 'no discarding relevant candidates' claim. These issues affect the paper's main empirical conclusion and require correction before the stated state-of-the-art claim is defensible.","major_comments":[{"comment":"The repeated claim that LORex improves accuracy by 12% over FLAME is not reproducible from Table 2. For LORex8B-3.1I versus FLAME, the accuracy differences are: Env 67.3 vs 63.4 (+3.9 pp), Sci 64.7 vs 63.2 (+1.5 pp), Food 55.3 vs 58.7 (-3.4 pp), WordNet 49.5 vs 45.2 (+4.3 pp). The mean absolute gain is 1.6 pp, the mean relative gain is about 3.1%, and the largest per-dataset relative gain is 9.5% (WordNet); no standard aggregation yields 12%. The Wu&P claim is also mixed: Environment decreases from 85.1 to 82.9, while the other datasets improve. Please state the exact aggregation rule used for the 12% and 5% figures and revise the abstract, introduction, and Section 5.2 accordingly.","section":"Abstract, Section 1, Section 5.2, Table 2"},{"comment":"The claim that LORex expands taxonomies 'without discarding relevant candidates' is contradicted by the inference protocol in Appendix F. Inference is performed only on three chunks of five candidates, i.e., the top 15 ranked candidates, because Hit@15 exceeds 0.9 for most benchmarks; Fig. 12 shows Hit@15 is below 0.9 for Food, so a nontrivial fraction of Food queries have the true parent outside the examined window. Since the semantic filter, retriever, and verifier operate only on these chunks, the top-15 truncation is a hard upper bound on accuracy that is not disclosed in the main text. Moreover, Table 5 shows the semantic filter itself skips 45 batches containing the true parent (3 Env, 8 Sci, 13 Food, 21 WordNet). Please disclose the truncation prominently and either justify it as a design choice or remove the 'no discarding' claim.","section":"Appendix F, Fig. 12, Table 5"},{"comment":"CodeTaxo (Zeng et al., 2024b), which the paper cites as a third-generation LLM-based taxonomy expansion method and whose evaluation protocol it adopts in Section 5.1 and Appendix I, is not included among the baselines in Table 2. Without a comparison to CodeTaxo, the claim that LORex surpasses state-of-the-art third-generation prompting methods is not fully supported. Please add CodeTaxo to the comparison or clearly justify its exclusion (e.g., API cost) and qualify the state-of-the-art claim accordingly.","section":"Section 5.1, Table 2, Section 2"}],"minor_comments":[{"comment":"The introduction says LORex 'surpasses eight baselines across four benchmarks,' but Table 2 reports twelve baselines; please align these numbers.","section":"Section 1"},{"comment":"The notation for the losses, especially L⇕ and the summation over P and P′, is underspecified. Please define the positive and negative path sets and the scope of the summation explicitly for reproducibility.","section":"Section 4.1, Eqs. (3)-(7)"},{"comment":"The diagram has several small labels that are difficult to read, including the text 'See d T a xon o m y' and the Euler-tour notation. Please clean up the figure so that the pipeline stages are visually clear.","section":"Figure 2"},{"comment":"The claim in Section 4.1 that TEMPORA needs only a 'minimal number of training epochs' is not fully supported by Table 7, where Food uses 50 epochs while other datasets use 20. Please explain the choice or soften the claim.","section":"Appendix F, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The paper's own Table 2 undercuts the headline accuracy number, and the top-15 truncation in Appendix F is material to the method's claimed advantage. In addition, the authors' prior work (FLAME and Quantaxo) is cited prominently, while a directly relevant third-generation baseline, CodeTaxo, is discussed but not evaluated; I would ask the editor to verify that the baseline selection and the aggregation of the headline numbers are handled transparently in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. First, the architecture is genuinely interesting: a BERT-based ranker (TEMPORA) feeds chunks of candidates to an LLM that filters, retrieves, and verifies parent terms using verbalized taxonomy paths. That pipeline is new as a combination, and it avoids fine-tuning LLMs, which matters for cost. Second, the paper's headline numbers—12% accuracy and 5% Wu&P improvement over FLAME—do not survive contact with Table 2. The best relative accuracy gain on any single dataset is about 9.5% (WordNet), Food accuracy actually drops below FLAME, and the mean relative gain across the four datasets is roughly 3%. No aggregation I tried produces 12%. That is a real problem, because the paper's conclusion rests on it.\n\nCredit where it's due. The ranker is a sensible extension of TEMP—adding sibling and child context and verbalizing edges—and it shows consistent Hit@1/Hit@10 gains over TEMP on all four benchmarks. The ablations are reasonably thorough: they test chunk sizes, filter impact, verifier types, and the ranker's contribution, and the appendix includes prompts, pseudocode, and failure cases. Code is released. This is not a toy evaluation.\n\nSoft spots, in proportion. The 'without discarding relevant candidates' claim is contradicted by the top-15 truncation disclosed in Appendix F and by Table 5, where the semantic filter skips 45 batches that contain the true parent. That is an overstatement, not a fatal flaw. More concerning is that the 12% claim is simply not reproducible from the reported tables, and the paper never discloses how it was computed. Also, lambda_1 and lambda_2 in Eq. 6 are never reported in the hyperparameter table, which is sloppy for reproducibility. The Limitations section does acknowledge the method is only as good as its ranker—good—but it does not acknowledge the headline mismatch.\n\nVerdict: the central pipeline is plausible and probably useful; the weakness is in the claims, not the architecture. With the aggregation disclosed and the headline recalibrated to what Table 2 actually shows, this would be a solid practical contribution. It does not reshape the field, but it gives the community a cheaper alternative to fine-tuning LLMs for taxonomy expansion.\n\nWho it is for: practitioners building taxonomy expansion systems, and people working on retrieve-then-reason LLM pipelines. It deserves a serious referee, but the referee should ask for major revisions—specifically, correct the headline numbers, report the missing hyperparameters, and soften the 'no discarding' language. I would send it out, expecting a revise-and-resubmit.","headline":"Solid retrieve-then-reason pipeline for taxonomy expansion, but the headline accuracy gain is overclaimed and doesn't match the paper's own Table 2.","tokens_in":21118,"tokens_out":2827,"would_cite":false,"duration_ms":26037,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Taxonomy expansion can be done without fine-tuning by ranking candidate parents into small chunks and prompting an LLM to reason over verbalized lineage paths; the paper reports 12% higher accuracy and 5% higher Wu & Palmer similarity…","keywords":["taxonomy expansion","large language models","hypernymy","hierarchical reasoning","discriminative ranking","prompt engineering","Euler tour path verbalization","SemEval 2016 Task 13"],"falsifier":"On the Food benchmark, where TEMPORA's Hit@15 is below 0.9, split the test queries into those whose true parent falls inside versus outside the top-15 window and compute LORex's accuracy separately on each group. The paper's limitation predicts near-zero accuracy on the outside group; if accuracy there is not near zero, the chunking and verification stages are recovering from ranker misses, contradicting the claim that the method is only as good as its ranker.","tokens_in":19941,"feed_emoji":"🌳","tokens_out":10939,"duration_ms":101289,"temperature":0.7,"pith_summary":"Taxonomy expansion asks where a new concept should hang in an existing is-a hierarchy. LORex answers by splitting the problem into a cheap ranking step and an expensive reasoning step: a discriminative ranker orders every candidate parent by how well its verbalized lineage path matches the query, and an LLM then works through those candidates in small chunks. The generative stage filters irrelevant chunks, retrieves the most granular plausible parent, and verifies that choice against candidate paths by their average token log-probabilities. On four benchmark taxonomies the paper reports roughly 12% higher accuracy and 5% higher Wu & Palmer similarity, a structural closeness measure based on the least common ancestor, than the strongest fine-tuned baseline. The central claim is that generative hierarchy reasoning becomes reliable only after discriminative ranking has narrowed the candidate window.","feed_headline":"Ranking before prompting lifts taxonomy expansion accuracy 12%","feed_subtitle":"Chunked lineage prompts with a discriminative ranker beat fine-tuned models on four taxonomies.","key_machinery":"The load-bearing mechanism is TEMPORA's verbalized lineage path: a modified Euler tour flattens each candidate's local taxonomy into a string that visits the anchor, climbs to the root, returns through siblings, and passes through children, with every edge written as a relational phrase. Feeding that path to a pretrained encoder with a dual-path margin loss produces a ranking score for every candidate; ranking is what lets LORex chunk candidates into batches of five without losing the true parent in most cases. The second mechanism is the iterative LLM loop over chunks, consisting of a semantic filter, a parent retriever, and a path verifier that scores candidate paths by mean token log-probability, which together accept, reject, or refine a parent decision.","core_discovery":"The paper's discovery is a decomposition: taxonomy expansion need not be either pure retrieval or pure text generation. LORex uses TEMPORA to turn local hierarchy into a verbalized Euler-tour path, covering the anchor, ancestors to the root, siblings, and children with edges labeled is parent of and is child of, and scores those paths with a dual-path margin loss to rank all candidates. The ranked list is chunked, and an instruction-tuned LLM filters chunks, retrieves the best parent, and verifies it path by path; a rejected parent is removed and retrieval repeats on the same chunk. With three open LLMs and no fine-tuning, the system beats all comparison baselines on four benchmarks, and ablations show the ranker, the paths, and the verifier each contribute a large share of the final accuracy.","pith_inferences":["The same rank-chunk-verify loop is a candidate recipe for other structure-placement tasks, such as taxonomy completion or knowledge-graph completion, where one correct position must be chosen from a large candidate set; the paper only frames taxonomy expansion as the testbed.","The mean token-log-probability path scorer could be reused as a general stabilizer for small LLMs on multiple-choice reasoning, since it replaces unstable free-form path generation with a ranking over the model's own continuations.","A direct stress test the paper does not run is to place the true parent at positions 16 through 20 for the Food queries that fall outside TEMPORA's top-15 window; LORex's accuracy on exactly that subset would settle how much of the reported gain is inherited from the ranker.","In practice the limitation suggests monitoring TEMPORA's Hit@15 per taxonomy before deploying LORex, and replacing or retraining the ranker whenever that rate drops; the framework itself is agnostic to which ranker supplies the chunks."],"forward_implications":["A taxonomy can be expanded by prompting an 8B instruction-tuned model, so teams without access to paid or fine-tuned large models can still update real-world hierarchies.","Path-based verification, not bigger models, supplies most of the generative gain: switching from the path verifier to a candidate-only verifier drops accuracy by roughly 47% in the paper's ablations.","Semantic filtering makes the loop practical, cutting the number of retrieval-verification rounds per query from 3.1 to 1.7 while almost never skipping the batch that contains the true parent.","Because the main runs examine only the top-15 candidates, any improvement in the ranker's Hit@15 should translate directly into higher LORex accuracy; the paper's random-shuffle ablation shows accuracy collapsing from roughly 67% to 6% on the Environment benchmark when ranking is removed."],"supporting_citations":[{"why":"Supplies TEMP, the path-based ranker and dynamic margin loss that TEMPORA extends with Euler-tour verbalization and dual-path training.","marker":"Liu et al., 2021"},{"why":"FLAME is the strongest fine-tuned LLM baseline; LORex's reported 12% accuracy and 5% Wu & Palmer gains are measured against it.","marker":"Mishra et al., 2024"},{"why":"CodeTaxo defines the Accuracy and Wu & Palmer evaluation protocol and represents the GPT-4 prompting baseline for taxonomy expansion.","marker":"Zeng et al., 2024b"},{"why":"Provides three of the four benchmark taxonomies used in the experiments: Environment, Science, and Food.","marker":"Bordea et al., 2016b"},{"why":"Provides the 114 WordNet sub-taxonomies that form the fourth benchmark.","marker":"Bansal et al., 2014"},{"why":"TacoPrompt is one of the strongest prompting baselines; comparing against it supports the claim that ranking plus chunking adds value over prompt-only methods.","marker":"Xu et al., 2023"}],"fun_headline_variants":["Rank, chunk, reason: taxonomy expansion +12% with no fine-tuning","Chunked lineage prompts beat fine-tuned models on taxonomy expansion","LORex ranks then reasons, boosting taxonomy accuracy by 12%","Discriminative ranker plus chunked LLM reasoning lifts taxonomy expansion","Lineage-oriented reasoning: rank, chunk, expand for 12% accuracy gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"LORex's central assumption is that the true parent appears in the top-15 ranked candidates, because the LLM only ever sees three chunks of five; the paper's limitation section concedes that the method is only as good as its ranker.","fun_headline_variants_meta":{"raw":{"variants":["Rank, chunk, reason: taxonomy expansion +12% with no fine-tuning","Chunked lineage prompts beat fine-tuned models on taxonomy expansion","LORex ranks then reasons, boosting taxonomy accuracy by 12%","Discriminative ranker plus chunked LLM reasoning lifts taxonomy expansion","Lineage-oriented reasoning: rank, chunk, expand for 12% accuracy gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1449,"prompt_tokens":860,"completion_tokens":589,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":502}},"tokens_in":476,"tokens_out":589,"duration_ms":5440,"temperature":1.0,"reasoning_tokens":502,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:15:35.954931+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the Food benchmark, where TEMPORA's Hit@15 is below 0.9, split the test queries into those whose true parent falls inside versus outside the top-15 window and compute LORex's accuracy separately on each group. The paper's limitation predicts near-zero accuracy on the outside group; if accuracy there is not near zero, the chunking and verification stages are recovering from ranker misses, contradicting the claim that the method is only as good as its ranker.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FLAME is the strongest fine-tuned LLM baseline; LORex's reported 12% accuracy and 5% Wu & Palmer gains are measured against it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the 114 WordNet sub-taxonomies that form the fourth benchmark."}],"review_version":1}