{"id":"ebb9fcb4-2aaa-47e3-93e6-68a6860d3717","arxiv_id":"2507.03829","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RELRaE, a hybrid rule-based and LLM pipeline, generates relationship labels for XML schemas and, on the AnIML chemistry schema, produces labels closer to human expert labels than rules alone or LLMs alone.","lead":"This study introduces RELRaE, a four-stage pipeline that uses rules and large language models to name the relationships inside XML schemas and then converts them into a basic ontology. It reports that the hybrid rule-plus-LLM version produces labels more similar to human expert labels than either approach alone on one chemistry lab schema.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Phrase-BERT similarity is not validated as a proxy for semantic correctness; thresholds are tuned on the same benchmark and no inter-annotator agreement is reported, so the claimed Refined superiority could reflect lexical style rather than accuracy.","rationale":"Read in good faith, RELRaE is a plausible pipeline and the paper is transparent about its limitations, with a reproducibility artifact on GitHub. The Refined method's advantage is consistent across temperatures, so it is not a one-off stochastic effect. However, the load-bearing support for the central claim is the evaluation metric and the gold standard. The authors explicitly flag the lexical-similarity pitfall in Section 5.1, and the reader's weakest assumption identifies exactly this issue. Without a human-validation study of the Phrase-BERT proxy, or at least an inter-annotator agreement measure for the gold labels, the evaluation cannot distinguish semantic correctness from lexical overlap. This does not refute the paper; it means the claim is conditional on metric validity. The reader's CONDITIONAL verdict is therefore appropriate, and the proposed human-rating check is the natural condition to satisfy. No change to the reader's verdict is needed.","tokens_in":13275,"tokens_out":3658,"duration_ms":42748,"concrete_test":"Select a stratified random sample of roughly 120 generated labels covering RuBREx, LLM-only, and Refined across temperature settings. Have at least two additional domain experts, blind to method and to the gold label, rate each label for semantic correctness on a Likert scale. Report inter-annotator agreement (e.g., Krippendorff's alpha) and compute (a) the Spearman correlation between mean expert ratings and Phrase-BERT cosine similarity to the gold reference, and (b) the method rankings under expert ratings. If the correlation is low (<0.5) or the Refined advantage disappears under expert ratings, the headline claim is unsupported. Also recompute Table 7's acceptance percentages using thresholds derived from a held-out set rather than the evaluation set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—that the Refined pipeline 'significantly enhances label accuracy'—rests entirely on Phrase-BERT cosine similarity between generated labels and gold-standard labels in Section 5.1 (Tables 6 and 7). The paper itself acknowledges the weakness: a label can be penalized for lexical difference or rewarded for lexical similarity without representing the relationship correctly. No evidence is provided that Phrase-BERT scores correlate with human semantic quality for this task. The gold standard was produced by only three domain experts, with no inter-annotator agreement reported; if experts disagree, matching any single reference label is not clearly equivalent to being correct. In addition, the acceptance thresholds (0.6 and 0.85) were determined empirically on a small sample from the same AnIML benchmark, so the percentages in Table 7 may be optimistically biased. Because the Refined method anchors to rule-generated 'hasX' labels, it may coincide more often with the surface style of the gold labels without being semantically better; conversely, LLM-only labels may be semantically valid but lexically distant. The 0.88 vs 0.81 gap is therefore not yet established as a difference in accuracy. The absence of confidence intervals or significance tests further weakens the comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RELRaE, a four-stage framework that converts an XML schema into a skeleton ontology by extracting hierarchical relationships, generating rule-based labels (RuBREx), refining those labels with an LLM (LLMR), and evaluating the refined labels with a different LLM acting as a judge (LLME). The authors evaluate three label-generation approaches—rule-based only, LLM-only, and the refined hybrid—on the AnIML analytical-chemistry schema, comparing generated labels against a gold-standard set produced by three domain experts using Phrase-BERT cosine similarity. They also compare LLME's Likert-scale acceptability judgments against a domain expert on a random sample of 60 labels per approach. The central reported finding is that the Refined approach consistently outperforms both baselines, with mean Phrase-BERT similarity around 0.88 versus about 0.81, and with roughly 95% of refined labels above a loose similarity threshold versus roughly 86-91% for baselines.","tokens_in":13485,"tokens_out":2637,"duration_ms":30482,"significance":"If the central claim holds, the paper is a useful applied contribution to semi-automatic ontology engineering: it demonstrates a concrete, reproducible pipeline that combines rule-based structure with LLM refinement, and it provides an empirical comparison grounded in an external human reference set rather than relying solely on intrinsic LLM judgment. The paper is also careful in several respects: experiments are reported across multiple temperature settings, the two LLM roles use different model families to reduce bias, the evaluation setup is described in enough detail to be replicable, and the authors openly acknowledge the subjectivity of label quality and the limitations of their ground truth. The main significance, however, is conditional on the validity of the semantic-similarity proxy used for the headline comparison; the current evidence does not fully support the strong claim that the refined pipeline 'significantly enhances label accuracy.'","major_comments":[{"comment":"The load-bearing quantitative claim—that the Refined approach 'consistently generates superior relationship labels'—rests entirely on Phrase-BERT cosine similarity between generated labels and the gold-standard labels. The paper itself acknowledges (end of §5.1) that a label can be penalized for lexical difference or rewarded for lexical similarity without representing the relationship correctly. No evidence is provided that Phrase-BERT cosine similarity correlates with human semantic quality for these short, domain-specific relationship labels, and no inter-annotator agreement is reported for the three domain experts who produced the gold standard. Without such validation, the 0.88 versus 0.81 gap could partly reflect lexical style or the anchoring of Refined labels to the rule-generated 'hasX' prefix rather than semantic accuracy. I recommend adding a human-evaluation subset or an inter-annotator agreement measure, and/or rephrasing the claim to 'lexical-semantic similarity to the reference labels' rather than 'accuracy.'","section":"§5.1, Tables 6 and 7"},{"comment":"The acceptance thresholds (τ ≥ 0.6 and τ ≥ 0.85) were 'determined empirically prior to the main evaluation using a small sample set of the generated labels' from the same AnIML benchmark. This makes the percentages in Table 7 potentially optimistically biased, since the thresholds and the evaluation set are not independent. In addition, Tables 6 and 7 report no confidence intervals, standard deviations, or significance tests, so the claimed superiority of the Refined approach over RuBREx and LLM-only is not statistically established. I recommend reporting bootstrap confidence intervals or paired significance tests (e.g., Wilcoxon signed-rank across relationship pairs), and ideally evaluating threshold robustness on a held-out set or by sweeping τ and showing the ordering is stable.","section":"§5.1, thresholds and statistical inference"},{"comment":"The RQ2 evaluation is under-specified in ways that affect the conclusion. The acceptance criterion of 'Likely' or 'Yes' is stated to be based on a preliminary study, yet the reported domain-expert agreement with LLME is only 41.7% for that criterion, rising to 93.3% when 'Possible' is included. This large gap suggests that the LLME's moderate-confidence responses may often mean 'not clearly wrong' rather than 'genuinely appropriate,' as the authors themselves speculate in §6. The random selection of 60 labels per approach is reported without a random seed or a description of the sampling distribution, and no chance-level baseline, Cohen's kappa, or per-label agreement breakdown is given. I recommend reporting the full Likert-score distribution, a chance baseline, and a chance-corrected agreement metric, and tempering the claim that LLME can successfully evaluate labels.","section":"§5.2, Table 8 and footnote 6"},{"comment":"The abstract states that RELRaE 'significantly enhances label accuracy compared to other methods,' but the evaluation does not include a significance test, and the proxy used is similarity rather than accuracy. Given the acknowledged limitations of the Phrase-BERT proxy and the small, expert-produced reference set, 'significantly' is not supported. Please either provide appropriate statistical evidence or soften the claim throughout the paper.","section":"Abstract and §5.1"}],"minor_comments":[{"comment":"There is a typo: 'examples of the type of labels expected are given ... that cab improve the quality' should read 'can improve the quality.'","section":"§3.2"},{"comment":"The framework name is spelled inconsistently: 'ReLRAE' appears in §6 and the abstract uses 'RELRaE'; please standardize throughout.","section":"§6"},{"comment":"The sentence 'A few-shot approach was also used, by including the the examples listed in Table 5' contains a duplicated 'the.'","section":"§4.2"},{"comment":"Several entries in Tables 6 and 7 are rendered as '...', which makes the tables incomplete in the manuscript text; all values should be filled in, and the temperature settings should be clearly aligned across rows.","section":"Tables 6 and 7"},{"comment":"Pattern IDs jump from 7 to 9 and 10, with no row for ID 8; either renumber the patterns or explain the omission.","section":"Table 1"},{"comment":"The sentence 'LLME scores the confident with which it generates the relationships' should read 'the confidence with which it evaluates the relationships.'","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern raised in the reader's report is legitimate and lands on the central claim: the headline comparison in Section 5.1 is not yet established as a semantic-accuracy difference because the Phrase-BERT proxy is unvalidated and the thresholds are tuned on the same benchmark. This is fixable within the manuscript's scope by adding a small human validation study, reporting inter-annotator agreement, and adding appropriate statistical inference, so I do not see it as grounds for rejection. The paper would also benefit from a more careful framing of the RQ2 result, since the 41.7% vs 93.3% agreement split is currently under-interpreted. Fit with the journal is appropriate for an applied AI/ISWC-style venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi — quick take on RELRaE. The useful core is the four-stage hybrid: rules give you a deterministic first label, an LLM refines it with schema context, and a separate LLM judges the result. That combination is genuinely not in the cited literature, and the paper ships code and data, which is more than most. The evaluation on the AnIML schema also follows a consistent trend: the Refined pipeline beats both RuBREx and LLM-only across all temperature settings, and the human gold standard grounds the comparison. I buy that this is a sensible way to cut manual work in one step of XSD-to-ontology building.\n\nThe soft spots are concentrated in the evaluation. The headline metric is Phrase-BERT cosine similarity to the gold-standard label, and the paper itself flags the exact problem: a good label that is lexically different gets penalized, a bad label that is lexically similar gets rewarded. That is not a minor footnote; it is the load-bearing assumption of the main result. The acceptance thresholds (0.6/0.85) were set empirically on a small sample from the same benchmark, so the percentages in Table 7 are optimistic in an unknown degree. There are no confidence intervals or significance tests, and the gold standard comes from three experts with no inter-annotator agreement reported. The authors acknowledge the limited expert ground truth, which is honest, but it does not fix the problem.\n\nThe secondary LLM-as-judge evaluation is even thinner: 60 randomly selected labels per approach, one independent human expert, agreement at the 'Possible' threshold is 93%, but that collapses to 41.7% at 'Likely/Yes'. That does not support the claim that LLMs can reliably stand in for domain experts.\n\nSo the central claim — 'significantly enhances label accuracy' — is plausible but not yet demonstrated. The pipeline design and the domain motivation are real, and there is enough here to warrant a serious referee. My recommendation: send it to peer review, but ask for (1) at least one additional schema, (2) validation of Phrase-BERT against human judgment on a sample, (3) robustness analysis across thresholds, and (4) significance testing or confidence intervals. Also, I'd want the authors to soften the 'significantly enhances' phrasing until those are in place. Who's this for? Ontology engineers and people building LLM pipelines for lab automation; a reading group on LLM evaluation pitfalls could also find the measurement issues instructive.","headline":"A sensible hybrid pipeline for XML-to-ontology labeling that is worth refereeing, but the evaluation leans too hard on an unvalidated similarity metric and needs multi-schema, statistically grounded support.","tokens_in":14066,"tokens_out":2620,"would_cite":false,"duration_ms":27357,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Refining simple rule-based relationship labels with a large language model produces more accurate ontology labels than either approach used alone.","keywords":["LLM","XML schema","ontology engineering","relationship extraction","LLM-as-a-judge","knowledge graph","AnIML","prompt engineering"],"falsifier":"Ask a set of experts to judge, blind, whether each generated label correctly describes its domain-range pair without seeing the reference label. If the refined labels do not receive higher correctness ratings than the rule-based and LLM-only labels, the reported advantage is an artifact of the cosine-similarity measure.","tokens_in":13063,"feed_emoji":"🧪","tokens_out":5863,"duration_ms":55871,"temperature":0.7,"pith_summary":"Large language models can reliably upgrade the simple relationship labels that rules generate from XML schemas, producing labels closer to what a domain expert would write. The paper builds RELRaE, a four-stage pipeline that extracts hierarchical relationships from an XML schema, labels them with a rule-based module, refines those labels using an LLM with a few-shot prompt, and evaluates the results with a second LLM acting as a judge. On the AnIML analytical-chemistry schema, the refined labels consistently beat both the rule-based and LLM-only baselines by a wide margin on a Phrase-BERT cosine-similarity measure against a gold-standard set created by three experts. The findings suggest that LLMs can ease the knowledge-acquisition bottleneck in semi-automatic ontology engineering, and that LLM-as-a-judge evaluation is promising though imperfect.","feed_headline":"Hybrid rule-plus-LLM labels beat either method alone","feed_subtitle":"On a lab-chemistry XML schema, the RELRaE pipeline closes most of the gap to expert-written labels.","key_machinery":"The central mechanism is the RELRaE pipeline, which combines a rule-based module (RuBREx) with two LLM stages. RuBREx extracts parent-child, grandparent-grandchild, element-attribute and boolean-type relationships from the XML schema, and generates initial labels using structural patterns such as 'has<r>', 'is<r>', and '<r> subclassOf <d>' (Table 1). The refinement LLM (GPT-4o) is then prompted, with role, domain, use-case and a few-shot set of generic and domain-specific examples, to accept or refine the rule-based label. A different LLM (Gemini-2.0-flash) finally evaluates the refined label on a five-point Likert scale, mapping scores to 'Yes/Likely' acceptance; labels not accepted revert to the RuBREx label. Cosine similarity between generated labels and the reference set, computed via Phrase-BERT embeddings, is the metric used to compare methods.","core_discovery":"The central claim is that combining a rule-based labelling pass with an LLM refinement pass yields significantly better relationship labels than either pass alone. The paper's comparative evaluation on the AnIML schema shows that the refined labels achieve a mean Phrase-BERT cosine similarity of about 0.88 to the expert-generated reference labels, versus roughly 0.81 for both the rule-based RuBREx labels and the LLM-only labels, and that the refined approach also produces the highest share of labels meeting both a loose (≥0.6) and a strict (≥0.85) similarity threshold. The authors interpret this as evidence that the rule-based labels anchor the LLM to a consistent interpretation, while the LLM injects domain knowledge to adjust wording to match expert intuition. They also claim, on the basis of a separate comparison against a domain expert, that an LLM evaluator can serve as a useful proxy for judging label acceptability, especially when 'Possible' is treated as an acceptable score.","pith_inferences":["The evaluation's reliance on Phrase-BERT cosine similarity may underweight semantically correct but lexically distant labels; a direct human-judged semantic-accuracy study could change the relative ranking between the methods.","The strict-threshold results (about 57.6% of refined labels above 0.85) suggest that a large fraction of labels still need human review; the pipeline's practical benefit may be in highlighting which labels to review, not in replacing review entirely.","The finding that the LLM-only baseline performs no better than the rule-based baseline, while the refined version clearly outperforms both, suggests that the rule-based label acts as a strong prior; one could test this by giving the LLM a random or deliberately poor initial label and checking whether refinement still helps.","The framework's modular design means the two LLMs can be swapped independently; the paper does not test model variation, so the reported gains may be specific to the GPT-4o/Gemini pair."],"forward_implications":["The RELRaE pipeline can be applied to other XML schemata beyond AnIML to produce skeleton ontologies with minimal human effort.","The hybrid approach reduces the number of labels a domain expert must hand-correct, since under a loose threshold about 95% of refined labels resemble the expert reference set.","The LLM-as-a-judge stage, accepting labels rated 'Possible' or above, can filter obviously wrong labels automatically, cutting the expert review workload.","If the refinement step's anchoring hypothesis is correct, similar pipelines could be built for other semi-structured formats such as JSON Schema or CSV by swapping the rule-based starter module."],"supporting_citations":[{"why":"Supplies the structural patterns and rule-based mapping that RuBREx extends to seed initial relationship labels.","marker":"[6]"},{"why":"Defines the default 'has<r>' labelling policy that RuBREx falls back on when no pattern applies.","marker":"[15]"},{"why":"Motivates the few-shot prompting strategy used in the label refinement stage.","marker":"[24]"},{"why":"The Gemini-2.0-flash model used as LLME, the judge that scores label acceptability.","marker":"[29]"},{"why":"Supports the LLM-as-a-judge approach as a proxy for human evaluation.","marker":"[32]"},{"why":"Provides the Phrase-BERT embeddings used to compute cosine similarity between generated and gold-standard labels.","marker":"[39]"}],"fun_headline_variants":["Rules plus LLM refinement beats either alone","Hybrid rule-LLM labeling outperforms single passes","RELRaE: anchor with rules, refine with LLM","LLM evaluator matches expert label judgment","Combined rule and LLM labels hit 0.88 similarity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central comparison assumes that how close a label is to an expert-written reference label, as measured by Phrase-BERT cosine similarity after thresholds chosen on a small sample, correctly reflects how good the label is.","fun_headline_variants_meta":{"raw":{"variants":["Rules plus LLM refinement beats either alone","Hybrid rule-LLM labeling outperforms single passes","RELRaE: anchor with rules, refine with LLM","LLM evaluator matches expert label judgment","Combined rule and LLM labels hit 0.88 similarity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000316,"raw_usage":{"total_tokens":1755,"prompt_tokens":879,"completion_tokens":876,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":797}},"tokens_in":495,"tokens_out":876,"duration_ms":10167,"temperature":1.0,"reasoning_tokens":797,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:01:23.731362+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask a set of experts to judge, blind, whether each generated label correctly describes its domain-range pair without seeing the reference label. If the refined labels do not receive higher correctness ratings than the rule-based and LLM-only labels, the reported advantage is an artifact of the cosine-similarity measure.","supporting_citations":[{"cited_title":"Bohring and S","cited_arxiv_id":null,"evidence_quote":"Supplies the structural patterns and rule-based mapping that RuBREx extends to seed initial relationship labels."},{"cited_title":"Hannah, T","cited_arxiv_id":null,"evidence_quote":"Defines the default 'has<r>' labelling policy that RuBREx falls back on when no pattern applies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the few-shot prompting strategy used in the label refinement stage."},{"cited_title":"Shankar, J","cited_arxiv_id":null,"evidence_quote":"Supports the LLM-as-a-judge approach as a proxy for human evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Phrase-BERT embeddings used to compute cosine similarity between generated and gold-standard labels."}],"review_version":1}