{"id":"08017847-d9fd-4edb-86da-5158eb3de688","arxiv_id":"2504.16312","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Pretrained language models score near random on a new Wikidata-derived symmetry and antisymmetry NLI task, while encoder retraining with a symmetry-aware distance reaches 100% on the task with as few as 48 training samples.","lead":"This paper builds a synthetic natural language inference test from Wikidata to check whether language models know that some relations are symmetric, such as 'A borders B' implying 'B borders A', and some are not. It finds pretrained models near chance, then shows that retraining the encoder with a RotatE-style distance fixes the task with very few examples and less forgetting.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark labels are a deterministic function of the fixed template/relation phrase, so the 100% retraining and near-chance pretraining results may reflect a template shortcut rather than general symmetry understanding.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the dataset's labels are assigned mechanically from Wikidata's declared relation property and the sentences come from fixed templates, so the benchmark may not measure general relational understanding. I agree, and I would go further in one respect: because each relation is realized by exactly one template, the label is a deterministic function of the relation phrase alone. This makes the task solvable by a trivial bag-of-words/relation-word classifier, which would also reach 100% on held-out entities. That is a sharper version of the syntactic-diversity objection: the issue is not only limited diversity but the presence of a direct shortcut that bypasses the entity swap entirely. If that shortcut is available, the pretrained models' near-chance scores and the retrained encoders' 100% scores do not establish that the proposed symmetry-aware objective teaches general symmetry or antisymmetry reasoning. The paper is transparent about the limitation, and the central empirical observation may still be interesting as a template-matched result, but the abstract and conclusion overstate the scope. The paraphrase test I propose would settle the concern directly: passing it would support the general claim, while failing it would restrict the claim to template-matched settings and weaken the practical implications. Because the reader already assigned a CONDITIONAL verdict, no adjustment is needed now; if the test fails, the verdict should move toward REJECT.","tokens_in":7640,"tokens_out":10052,"duration_ms":110458,"concrete_test":"Construct a held-out paraphrase set for the same 14 relations and Wikidata triples, using at least three novel syntactic realizations per relation (e.g., active/passive, 'X borders Y', 'X and Y share a border') with labels still derived from the property symmetry as in A.2. Run the best retrained encoder (Random Label Embeddings or k-NN) on this set without any further training. If accuracy drops appreciably below the reported 100%, the original results are attributable to template/relation-phrase memorization rather than general symmetry reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every headline result is measured on a dataset whose labels are assigned mechanically from the Wikidata property (Appendix A.2), and each relation is rendered by exactly one manually written template (Table 3). Consequently the NLI label is a deterministic function of the relation phrase alone: for a symmetric property all instances are Entailment, for an antisymmetric property all are Contradiction, regardless of the entities or the order of the swapped pair. A classifier that only recognizes the template/relation word (e.g., 'is a child of' vs 'is different from') can reach 100% on held-out entities without processing the premise-hypothesis direction or the entity swap. The paper's own Limitations section concedes the dataset 'lack[s] syntactic diversity,' but the abstract and conclusion still claim an enhanced 'understanding of symmetric and antisymmetric relations.' This is load-bearing because the near-chance LLM results, the few-shot 100% accuracies, and the forgetting comparisons are all computed on this same benchmark; if the task is solvable by template memorization, the central claim about relational understanding is not established for natural language variation. The forgetting comparison is additionally confounded by unequal training-sample counts (48 vs 336), but the benchmark shortcut is the more basic issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a Wikidata-derived NLI dataset for evaluating symmetric and antisymmetric relations, reports that pretrained LLMs perform near chance on it, and proposes retraining the encoder with a contrastive objective and an asymmetric, RotatE-inspired distance metric, optionally combined with k-NN. The authors report 100% accuracy on both lexicalized and delexicalized versions with tens of training examples, claim parity with fine-tuned classification heads, and report improved retention on MNLI. The central claim is that retraining encoders with symmetry-aware objectives yields a reusable encoder that understands relational symmetry and antisymmetry better than standard fine-tuning.","tokens_in":7786,"tokens_out":7686,"duration_ms":78594,"significance":"If the empirical results were valid, the paper would offer a practical, few-shot method for injecting relational symmetry and antisymmetry into sentence encoders, with a released dataset as a useful resource. The directional distance metric derived from RotatE is an interesting idea, and the paper is clearly written. However, the evaluation is undermined by a template shortcut: because every relation is rendered by exactly one template and labels are assigned mechanically from the Wikidata property type, the benchmark can be solved by recognizing the relation phrase alone. The near-chance pretrained results and the 100% retrained results therefore do not establish an understanding of symmetry and antisymmetry in any general sense, and the paper's own Limitations section partially concedes this. The current significance is accordingly low.","major_comments":[{"comment":"The dataset construction makes the NLI label a deterministic function of the relation phrase. For each of the 14 properties, exactly one natural-language template is used (Table 3), and Appendix A.2 assigns 'Entailment' to every symmetric property and 'Contradiction' to every antisymmetric property, regardless of the entities. A model that maps the template phrase (e.g., 'is a child of' vs. 'is a sibling of') to a label reaches 100% without comparing premise and hypothesis direction or inspecting the entity swap; the delexicalized version still contains the template word. The paper's own Limitations paragraph concedes that the dataset 'lack[s] syntactic diversity' and that the challenge is 'far from resolved,' yet the abstract and conclusion assert enhanced 'understanding of symmetric and antisymmetric relations.' This template shortcut is load-bearing because the near-chance pretrained results, the few-shot 100% accuracies, and the forgetting comparisons are all computed on this same benchmark.","section":"Appendix A.2, Table 3, Limitations"},{"comment":"The catastrophic-forgetting comparison is confounded by unequal training budgets. Random Label Embeddings uses 48 training samples, k-NN uses 64, k-NN with Learnt Distance Metric uses 400, and Fine-Tuning uses 336; the reported MNLI drops (5.8%, 7.7%, 21.5%, 11.2%) are therefore not comparable as measures of the training objective. Matching the number of training samples or training steps, or reporting forgetting as a function of update count, is needed before claiming improved knowledge retention.","section":"Table 2, Section 3.2"},{"comment":"The few-shot efficiency claim is not substantiated. The paper reports the number of training samples at which 100% accuracy is reached, but it does not describe how these counts were selected, whether they are minimal, or the variance across random seeds; no significance tests or error bars are given. Given the template shortcut, reaching 100% with 48 examples is consistent with memorizing the 14 relation templates rather than learning a reusable relational bias.","section":"Section 3.1, Table 2"}],"minor_comments":[{"comment":"There is a typo in the introduction: 'antisyemmetric' should be 'antisymmetric'.","section":"Section 1"},{"comment":"The two references to Lin and Ng (2022a) and (2022b) appear to be the same paper with the same title and venue; they should be merged or clearly distinguished.","section":"References"},{"comment":"The appendix says hyperparameters are detailed below, but the k-NN with Learnt Distance Metric setting is missing its hyperparameters (learning rate, batch size, margin, k).","section":"Appendix C"},{"comment":"All accuracies are point estimates with no confidence intervals or significance tests; given the repeated 100% values, the tables should report run-level variability or at least multiple seeds.","section":"Tables 2 and 4"}],"recommendation":"reject","confidential_remarks":"The template shortcut is fundamental: the benchmark label is a deterministic function of the relation phrase, so the central empirical claims are not supported by the reported experiments. This is a reject, not because the idea is without merit, but because the evaluation would need to be redesigned with multiple templates per relation, control conditions, and ideally external naturalistic data. If such a revision were carried out, I would be willing to reconsider the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper puts forward a clean idea and a clean writeup: use a RotatE-style asymmetric distance to make sentence encoders aware of relation direction. The dataset release on Hugging Face is a nice gesture. But the benchmark has a load-bearing flaw, and the Limitations section mentions the lack of syntactic diversity without facing what it does to the results.\n\nThe core problem: for each of the 14 relations, the NLI label is constant. A symmetric relation always gives Entailment after swapping entities; an antisymmetric one always gives Contradiction. Each relation appears in exactly one manually written template (Table 3), so the label is a deterministic function of the template phrase alone. A model that recognizes \"is a child of\" and maps it to Contradiction, or \"borders\" to Entailment, would score 100% on held-out entities without ever processing the direction of the swap. The delexicalized version replaces entities with Q-ids but keeps the same template, so it does not fix the shortcut. That makes the near-chance pretrained results and the perfect retraining results both consistent with template memorization. Fine-tuning also hitting 100% with 336 samples fits the same reading.\n\nTo be fair, the paper does contribute a concrete empirical observation: standard sentence encoders with 1-NN probing are near chance on this construction. And the asymmetric distance is plausibly useful. But the observation is about this specific template-bound task, not about general relational understanding. The catastrophic-forgetting comparison is further confounded by unequal training-sample counts (48 vs 336 vs 400), so the 5.4% advantage is not interpretable as reported.\n\nThis paper is for a reader interested in benchmark artifacts or in symmetry-aware metric designs. They might take the dataset and the distance idea as a starting point, not as evidence for the stated claims. A credible version needs multiple diverse templates per relation, held-out relations at test time, and a task where the relation word alone cannot determine the answer.\n\nMy recommendation: in the current form, I would not trust the headline claims, and I would probably desk-reject. But the underlying idea is not bad, and a serious editor could send it to reviewers with an invitation for major revision. The authors first need to confront the template shortcut head-on.","headline":"The template-bound benchmark makes the paper's central claims unsubstantiated, but the RotatE-inspired metric is a germ of a good idea.","tokens_in":8386,"tokens_out":3173,"would_cite":false,"duration_ms":34899,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retrained encoders learn swap-relation entailment that LLMs miss","keywords":["symmetric relations","antisymmetric relations","natural language inference","contrastive learning","k-nearest neighbors","encoder retraining","Wikidata","RotatE"],"falsifier":"Take the same 14 relations and build a test set from naturally occurring, paraphrased sentences with swapped arguments; if the retrained encoder, frozen and probed by k-NN, drops from 100 percent toward chance on that set, the claimed mastery of symmetric and antisymmetric semantics is an artifact of the template format.","tokens_in":7360,"feed_emoji":"🔄","tokens_out":6745,"duration_ms":57395,"temperature":0.7,"pith_summary":"The paper asks whether language models understand that some relations remain true when their arguments are swapped (symmetric relations) while others become false (antisymmetric relations). It constructs a natural-language-inference benchmark from Wikidata triples in which each premise asserts a relation and each hypothesis swaps the two entities. On that benchmark, pretrained encoders such as RoBERTa hover at the 50 percent random baseline, even after MNLI fine-tuning. The authors then retrain the encoder alone with a contrastive objective built on a symmetry-aware distance metric borrowed from the RotatE knowledge-graph embedding method, reaching 100 percent accuracy on both lexicalized and delexicalized versions with only dozens of training examples. They conclude that retraining the encoder, instead of a classification head, teaches swapped-entity entailment while forgetting less of the model's prior knowledge than standard fine-tuning.","feed_headline":"Retrained encoders learn swap-relation entailment that LLMs miss","feed_subtitle":"On a Wikidata-derived NLI benchmark, encoder-only retraining hits 100% with 48 samples and forgets less than fine-tuning.","key_machinery":"The load-bearing object is a symmetry-aware distance metric between a premise embedding $p$ and a hypothesis embedding $h$, $d_\\ell(p,h)=1-\\operatorname{sim}(h,\\,p\\circ \\ell)$, where $\\circ$ is the element-wise (Hadamard) product with a label embedding $\\ell$. This metric is asymmetric in its arguments: $d_\\ell(p,h)\\neq d_\\ell(h,p)$ in general, which makes 'p entails h' and 'h entails p' distinguishable. In the k-NN variant, each sample's label embedding is recovered as the element-wise ratio $h\\oslash p$ of the hypothesis and premise embeddings, and training pulls same-label ratios together while pushing different-label ratios apart; at inference, test labels come from a majority vote over the $k$ nearest training label embeddings. The fixed version of this metric is what lets the encoder learn the semantic distinction with only 48-64 training examples, because the rotational asymmetry is supplied by the metric rather than learned from data.","core_discovery":"The central discovery is that the missing relational competence is not in the classification head but in the encoder's geometry: pretrained sentence encoders place a sentence and its swapped-argument counterpart at symmetric distances, so entailment and contradiction cannot be told apart. The paper's fix is a distance metric defined on sentence embeddings, $d_\\ell(p,h)=1-\\operatorname{sim}(h,\\,p\\circ \\ell)$, that treats swapping the premise and hypothesis as a rotation in embedding space, exactly as RotatE rotates one entity into another in knowledge-graph embeddings. Training the encoder with this metric, using random label embeddings or a k-NN voting scheme that infers label embeddings as element-wise ratios $h\\oslash p$, brings accuracy to 100 percent on the new benchmark, matches the accuracy of a fine-tuned classification head, and does so with far fewer samples (48-64 vs. 336) while producing a smaller drop on MNLI after training (as low as -5.8 percent, versus -11.2 percent for fine-tuning). The authors' conclusion is that retraining the encoder through symmetry-aware objectives captures symmetric and antisymmetric relations that pretrained models miss, with better few-shot efficiency and knowledge retention than head fine-tuning.","pith_inferences":["Because the benchmark labels come mechanically from Wikidata's property declarations and the sentences from fixed templates (the paper's own stated limitation), the 100 percent scores should be read as a proof that the encoder can learn the declared relational axioms, not yet as evidence it would do so on the open-ended linguistic variety of natural text.","One untested corollary of the rotation-based distance is that the same objective could inject other knowledge-graph axiomatics—transitivity, irreflexivity, or hierarchy—into sentence encoders with similarly few examples; the paper only exercises symmetry and antisymmetry.","A natural stress test would apply the method to context-dependent relations such as 'x helps y' or 'x likes y', where symmetry is not a fixed property of the relation; success would require the metric to use the actual entity meanings rather than the relation label alone."],"forward_implications":["Retraining the encoder alone, without any classification head or adapter, is enough to reach 100 percent accuracy on both lexicalized and delexicalized versions of the new symmetry benchmark.","The fixed RotatE-derived distance metric reduces the data needed to roughly an eighth of what fine-tuning requires (48-64 samples vs. 336), an efficiency gain the paper attributes to the metric already encoding the directionality of relations.","The retrained encoder forgets less prior knowledge: the measured drop on MNLI is up to 5.4 percentage points smaller than with fine-tuning in the reported settings.","The approach transfers across encoder sizes and pretraining pipelines (RoBERTa-Large, MiniLM 6/12 layers, all-MiniLM 6/12 layers), which the paper reads as evidence the method is general rather than model-specific."],"supporting_citations":[{"why":"Supplies the RotatE rotation operator used to define the symmetry-aware distance metric, which is the paper's core machinery.","marker":"Sun et al., 2019"},{"why":"Source of the Wikidata relation declarations that determine the NLI labels for symmetric versus antisymmetric pairs.","marker":"Vrandečić and Krötzsch, 2014"},{"why":"Provides the Wikidata5m triple store from which the paper retrieves the concrete entity triples for the dataset.","marker":"Wang et al., 2021"},{"why":"Documents the reversal-curse failure (A is B vs B is A) that motivates the benchmark and the retraining goal.","marker":"Berglund et al., 2023"},{"why":"Prior evidence that BERT misses relational properties such as transitivity, framing the gap the paper targets.","marker":"Lin and Ng, 2022b"},{"why":"Defines catastrophic forgetting, the phenomenon used to compare knowledge retention between retraining and fine-tuning.","marker":"French, 1999"}],"fun_headline_variants":["Symmetry-aware retraining turns LLM blind spot into 100% accuracy","Retrained encoders nail swap-relations, forget less than fine-tuning","Encoder-only retraining: 100% on swap-relations with 48 samples","Swap-relation benchmark: retrained encoders hit 100%, match fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark labels are assigned from Wikidata's declared relation property and the sentences are produced by fixed templates, so if template-based pairs do not reflect genuine relational understanding then both the chance-level pretrained scores and the perfect retrained scores are test artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Symmetry-aware retraining turns LLM blind spot into 100% accuracy","Retrained encoders nail swap-relations, forget less than fine-tuning","Encoder-only retraining: 100% on swap-relations with 48 samples","Swap-relation benchmark: retrained encoders hit 100%, match fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001522,"raw_usage":{"total_tokens":6077,"prompt_tokens":909,"completion_tokens":5168,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":5084}},"tokens_in":525,"tokens_out":5168,"duration_ms":33464,"temperature":1.0,"reasoning_tokens":5084,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:05:52.136297+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 14 relations and build a test set from naturally occurring, paraphrased sentences with swapped arguments; if the retrained encoder, frozen and probed by k-NN, drops from 100 percent toward chance on that set, the claimed mastery of symmetric and antisymmetric semantics is an artifact of the template format.","supporting_citations":[],"review_version":1}