{"id":"4d16e08b-0033-4ef3-84d2-62305d4099c5","arxiv_id":"2506.14569","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Adding a similar/2 predicate over pretrained embeddings and fine-tuning those embeddings with a Logic Tensor Network improves TILDE F1 on hate speech, spam, and drug response tasks.","lead":"A team from Mainz augments the TILDE logic-based learner with neural word and gene embeddings, letting its rules test semantic similarity instead of exact matches, then fine-tunes the embeddings with a Logic Tensor Network. Across three benchmarks the hybrid reaches higher F1 than symbolic-only variants, but the paper provides no code, error bars, or strong external baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The F1-superiority claim rests on single-split margins over internal variants only; without variance estimates, significance tests, or external baselines, 'outperforms all baselines' is not established.","rationale":"The reader's verdict is CONDITIONAL, and my concern reinforces that condition rather than overturning it. The paper's core contribution is an empirical claim about F1 superiority, and that claim is only as strong as the experimental evidence. Table 1 reports one number per cell, with no indication of variance, so the small differences on hate speech (0.576 vs. 0.569 vs. 0.561) and even the larger differences elsewhere cannot be assessed for reliability. The absence of standard baselines is also important because the abstract says 'all other baseline methods' when the compared methods are all variations of the same symbolic pipeline plus hand-crafted rules. This is not an accusation of dishonesty; it is a straightforward gap between the strength of the claim and the evidence provided. I do not find an internal inconsistency in the method description, and the three-domain consistency is encouraging, but the central claim needs the additional statistical and comparative support that the reader already requested. The reader's identified weakest assumption about the semantic meaningfulness of similar/2 is a secondary mechanism-level concern: even if the embeddings are meaningful, the reported evidence does not yet establish the superiority claim as stated.","tokens_in":8537,"tokens_out":5269,"duration_ms":62582,"concrete_test":"Run the full pipeline on 10 random train/validation/test splits (or 5-fold cross-validation) for each of the three datasets, keeping the hyperparameters fixed, and report mean and standard deviation of F1 for every row of Table 1. Apply a paired McNemar or bootstrap test comparing TILDE+LTN all against TILDE+LTN constants and Hand-crafted+LTN all within each domain. If the maximal variant is not significantly better (e.g., p<0.05) in all three domains, the abstract's 'outperforms all other baseline methods' should be weakened to a claim about the specific single splits, or the comparison should be expanded to include non-symbolic baselines.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, stated in the abstract and Section 1, is that the maximal variant (TILDE with LTN-revised embeddings) 'outperforms all other variants' and 'all other baseline methods' in terms of F1. The evidence in Table 1 consists of a single accuracy/F1 value per method per dataset, with no error bars, no repeated runs, and no significance tests. This is load-bearing because the claimed margins are small in at least one domain: on hate speech, TILDE+LTN all achieves F1 0.576 versus 0.569 for TILDE+LTN constants and 0.561 for Hand-crafted+LTN all. With a test set of roughly 500 instances, a 0.007 F1 difference can easily arise from noise. Additionally, the 'baselines' are only other TILDE variants and hand-crafted rules; there is no comparison to standard classifiers such as logistic regression, SVM, or a simple neural model. Therefore the headline claim, as literally stated, is not supported by the reported experimental protocol, even though the ablations show a plausible and interesting trend.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a neuro-symbolic classification pipeline that augments the ILP system TILDE with a similar/2 predicate computed from cosine similarity over pretrained embeddings (GloVe for text, GenePT for genes). TILDE induces a logical decision tree that can use this predicate to generalize beyond exact lexical matches. The tree is then converted into fuzzy rules for a Logic Tensor Network, which fine-tunes the embeddings to increase satisfaction of the rules; the refined embeddings are fed back into the original TILDE tree for classification. The approach is evaluated on hate-speech, SMS spam, and drug-response datasets. Table 1 reports accuracy and F1 for TILDE without embeddings, TILDE with similar/2, TILDE with LTN fine-tuning of constants or all embeddings, and for hand-crafted rules with and without LTN fine-tuning. The authors claim that the maximal variant, TILDE with LTN-revised all embeddings, outperforms all other variants and baseline methods in terms of F1 score across all three domains.","tokens_in":8739,"tokens_out":8997,"duration_ms":92364,"significance":"If the empirical claim were established, the paper would make a useful and modular contribution: a lightweight way to inject subsymbolic information into symbolic rule learning while retaining interpretable decision trees. The paper's strengths include a clear pipeline, the use of public datasets and standard pretrained embeddings, and a useful ablation that separates the contribution of the similar/2 predicate from the LTN fine-tuning step. The explicit discussion of data provenance is also a positive feature. However, the headline superiority claim is not currently supported by the reported evaluation: the evidence consists of single-split F1 values with no variance estimates, no significance tests, and no comparison to standard classifiers. The central idea is plausible and interesting, but the experimental protocol needs substantial strengthening before the claim can be accepted.","major_comments":[{"comment":"The headline claim that TILDE+LTN all 'outperforms all other variants' and 'all other baseline methods' is not supported by the reported experimental protocol. The results come from a single train/test split, with no standard deviations, confidence intervals, or significance tests. On the hate-speech task, the margin over TILDE+LTN constants is only 0.576 versus 0.569, and over Hand-crafted+LTN all it is 0.576 versus 0.561, on a test set of roughly 500 instances; these differences are within the range of sampling noise. Please add repeated runs with different seeds, bootstrap confidence intervals, and/or a paired significance test such as McNemar's test, and report the main effect sizes with uncertainty.","section":"Section 4.2, Table 1"},{"comment":"The comparison set contains only TILDE variants and hand-crafted rules. The abstract's statement that the approach 'outperforms all other baseline methods' is therefore an overstatement: no standard supervised baselines such as logistic regression, SVM, random forest, or a simple neural model are included. Such baselines are needed to calibrate whether the reported F1 values are competitive in absolute terms. Please add at least one or two standard classifiers per dataset, ideally with the same or comparable input features.","section":"Section 4.1, Table 1"},{"comment":"The right-branch rule construction appears to negate the entire accumulated conjunction rather than only the current node test. Specifically, rule+ is defined as rulecomp union Conjunction(node), and the right child receives rulecomp union Not(QuantifyAndConjunct(rule+)). This makes the right-branch body logically equivalent to 'not(previous tests and current test)', which holds when earlier tests on the path are false, so the resulting rule can fire in situations that cannot reach the right branch in the original decision tree. The correct right-branch body should be rulecomp plus the negation of the current test, with quantifiers handled accordingly. Please correct the pseudocode, or explain why the current formulation is intended, and confirm that the experiments use the corrected semantics. This point is load-bearing because the LTN fine-tuning signal is derived from these rules.","section":"Algorithm 1, line 13"},{"comment":"The similar/2 threshold tau is a dataset-specific free parameter (0.75 for hate speech, 0.70 for spam, 0.50 for drug response), and no sensitivity analysis is reported. Since the performance gains are attributed to the similarity predicate, the results should be shown to be robust to tau (e.g., a small grid around the chosen values), or the threshold should be learned as part of the pipeline. Without this, the reader cannot tell whether the reported improvements depend on hand-tuned thresholds that may not transfer to new data.","section":"Section 3.2 and Section 4.1"}],"minor_comments":[{"comment":"There are several typos: 'sytems' in Section 1, 'versality' in Section 1, 'Kontschieder el al.' in Section 2, 'simlilar' in Section 4.1, and 'three real-world domain' in the abstract should be 'three real-world domains'.","section":"Throughout"},{"comment":"The hand-crafted rules are only partially specified because the offensive words are redacted as '[racist word]', '[sexist word]', etc.; please include the actual terms or a complete rule set in an appendix so that the baseline is reproducible.","section":"Section 4.1"},{"comment":"The validation split is mentioned, but the paper does not state how it was used (e.g., for early stopping, hyperparameter selection, or threshold selection). Please clarify to rule out any selection based on the test set.","section":"Section 4.1"},{"comment":"The LTN implementation of similar/2 uses a 'shifted sigmoid' to model the similarity threshold, but the slope or temperature of the sigmoid and the way the threshold is encoded are not specified; adding these details would improve reproducibility.","section":"Section 3.3"},{"comment":"The figures would benefit from more detailed captions explaining the SOM color scale and how the clusters around constants were identified, since the qualitative claims in Section 4.2 rely on these visualizations.","section":"Figures 3-5"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the venue and the core idea is worth pursuing, but the gap between the strength of the abstract claim and the evidence in Table 1 is substantial. I would support a major revision that adds uncertainty quantification, external baselines, and a fix or clarification of the rule-extraction algorithm."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper describes a genuinely simple and useful way to give an ILP learner (TILDE) access to subsymbolic information—precompute a similar/2 predicate from pretrained embeddings, learn rules, then fine-tune the embeddings with an LTN using the extracted rules. The combination is new as far as I know, and the ablation shows the similarity predicate changes behavior and helps. That part is worth taking seriously.\n\nWhat's good: the method is clearly described, the idea of closing the loop between structure learning and embedding refinement is neat, and the three datasets are genuinely different (text, spam, genomics). The SOM visualizations are a nice touch. The authors also flag data provenance, which is honest.\n\nThe soft spots are real and mostly in the evaluation. Table 1 is a single split, no standard deviations, no significance tests, and the only baselines are TILDE variants and hand-crafted rules. On hate speech the margin between TILDE+LTN all and TILDE+LTN constants is 0.007 F1—noise territory. There's no comparison to standard classifiers (logistic regression, SVM, a small neural net) or to actual end-to-end neurosymbolic systems like DeepProbLog, which is a gap given the framing. The per-dataset thresholds for similar/2 are hand-set and not analyzed; that's a minor caveat but worth mentioning.\n\nNot all of this is fatal. The central claim as literally stated—'outperforms all other baselines'—is not supported by the protocol, but the trend is consistent across three datasets and the ablations make sense. I'd want to see repeated runs, error bars, and at least one or two strong external baselines before trusting the headline. The method itself is plausible and the paper is honest about its scope.\n\nWho is this for? People working on neurosymbolic learning, ILP, or interpretable ML. It's a useful practical contribution that deserves a serious referee, not a desk reject, but it needs a revised evaluation before I'd call the claim established.\n\nRecommendation: send it out for peer review, with the expectation that the authors add variance estimates and external baselines.","headline":"A sensible incremental neurosymbolic trick—similarity predicates over pretrained embeddings plus LTN fine-tuning—that shows a consistent F1 trend but is oversold relative to a single-split evaluation with no external baselines.","tokens_in":9298,"tokens_out":2396,"would_cite":true,"duration_ms":20545,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes that equipping the symbolic rule learner TILDE with a similarity predicate over pretrained neural embeddings, followed by fine-tuning those embeddings with a logic tensor network, yields the best F1 score among all…","keywords":["neuro-symbolic AI","inductive logic programming","TILDE","neural embeddings","similarity predicate","Logic Tensor Networks","embedding fine-tuning","F1 score"],"falsifier":"Replacing the pretrained word and gene vectors with random vectors of equal dimension, while keeping all thresholds and rules fixed, would falsify the semantic-similarity claim if the F1 gains persisted.","tokens_in":8286,"feed_emoji":"🧠","tokens_out":8399,"duration_ms":80164,"temperature":0.7,"pith_summary":"The paper's aim is to give a symbolic machine learning system access to neural embeddings without giving up its interpretability. It introduces a similarity predicate over the embeddings, lets the symbolic learner use it as background knowledge, and then refines the embeddings so the learned rules are better satisfied. In three classification tasks the complete pipeline beats plain symbolic learning and hand-crafted rules in F1.","feed_headline":"Similarity over word vectors gives symbolic AI top F1 in 3 tasks","feed_subtitle":"A cosine-similarity predicate plus embedding fine-tuning beats baselines on hate speech, spam, and drug response.","key_machinery":"The central object is the predicate similar/2, defined by cosine similarity: similar(X,Y) holds when the dot product of the embeddings of X and Y, divided by the product of their norms, is at least a threshold tau. TILDE is biased to use this predicate with a constant on one side, so it learns words or genes that act as semantic prototypes. The accompanying logic tensor network converts the induced tree into fuzzy rules and backpropagates through the cosine-similarity predicate, moving embeddings so that the rules they trigger match the labels; the updated embeddings then replace the originals for classification.","core_discovery":"The central discovery is that a purely relational learner can be lifted by a single cosine-similarity predicate. The paper reports that TILDE with the predicate raises F1 from 0.257 to 0.480 on hate speech, from 0.574 to 0.626 on spam, and from 0.167 to 0.242 on drug response. Fine-tuning all embeddings through a logic tensor network pushes these numbers to 0.576, 0.763, and 0.295, which the authors state is the best among all variants and baselines. This claim concerns F1 specifically: accuracy gains are mixed in the reported table.","pith_inferences":["One testable extension: sweep the similarity threshold tau on a single dataset; if the reported ordering of methods changes drastically, the fixed thresholds carry more weight than the paper's single-setting comparison shows.","The same wrapper could be applied to other symbolic learners or to similarities between whole instances, effectively implementing kernels inside a logical language; the paper lists this as future work, and it follows from the mechanism rather than from the experiments.","The authors' own data-provenance note leaves open a residual risk that public text embeddings overlapped with target texts during pretraining; a cleaner test would use embeddings trained without the evaluation corpus."],"forward_implications":["In all three domains, adding the similarity predicate alone improves F1 over plain TILDE, so the gain does not require the LTN refinement step.","Fine-tuning all embeddings, rather than only the constants appearing in the rules, gives the largest F1 gains, though it moves embeddings further from their original semantic positions.","Fine-tuning can repair poor symbolic choices: in spam, the embedding for 'urgent' shifts toward 'text', which TILDE independently considered the most predictive constant.","Because the final classifier is still a TILDE tree, the improved F1 comes with readable rules rather than a black box."],"supporting_citations":[{"why":"Supplies TILDE, the first-order decision-tree learner that forms the symbolic core being enhanced.","marker":"[3]"},{"why":"Supplies the logic tensor network used to convert TILDE trees into fuzzy rules and fine-tune the embeddings.","marker":"[2]"},{"why":"GenePT provides the gene embeddings used in the drug-response similarity predicate.","marker":"[4]"},{"why":"Provides the 200-dimensional GloVe vectors trained on tweets that carry the word similarities for hate speech and spam.","marker":"[6]"},{"why":"Defines the GloVe embedding model and the semantic-syntactic properties that the similarity predicate relies on.","marker":"[20]"},{"why":"Provides the hate speech dataset and its label construction for the first evaluation.","marker":"[11]"},{"why":"Provides the SMS spam corpus for the second evaluation.","marker":"[1]"},{"why":"Provides the multi-omics drug-response data used in the third evaluation.","marker":"[10]"},{"why":"Supplies the multi-omics preprocessing pipeline that the drug-response experiments follow.","marker":"[22]"}],"fun_headline_variants":["Cos-sim + embeddings land symbolic AI top F1 on 3 tasks","Fine-tuned embeddings lift symbolic TILDE to best F1 in 3 domains","One similarity predicate powers symbolic AI to beat baselines","Neural embeddings give symbolic learner edge in F1 across tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach succeeds only if cosine similarity over the pretrained embeddings is a meaningful relation for the target, with the per-dataset threshold tau chosen by hand and no sensitivity analysis reported.","fun_headline_variants_meta":{"raw":{"variants":["Cos-sim + embeddings land symbolic AI top F1 on 3 tasks","Fine-tuned embeddings lift symbolic TILDE to best F1 in 3 domains","One similarity predicate powers symbolic AI to beat baselines","Neural embeddings give symbolic learner edge in F1 across tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1426,"prompt_tokens":882,"completion_tokens":544,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":467}},"tokens_in":498,"tokens_out":544,"duration_ms":6055,"temperature":1.0,"reasoning_tokens":467,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:51:15.628549+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replacing the pretrained word and gene vectors with random vectors of equal dimension, while keeping all thresholds and rules fixed, would falsify the semantic-similarity claim if the F1 gains persisted.","supporting_citations":[{"cited_title":"https://huggingface.co/fse/glove-twitter-200 (2023), accessed: 2025-05-28","cited_arxiv_id":null,"evidence_quote":"Provides the 200-dimensional GloVe vectors trained on tweets that carry the word similarities for hate speech and spam."},{"cited_title":"Cell166(3), 740–754 (2016)","cited_arxiv_id":null,"evidence_quote":"Provides the multi-omics drug-response data used in the third evaluation."}],"review_version":1}