{"id":"447fb9e7-df38-4315-9b3a-75175ca6c9ef","arxiv_id":"2608.03512","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Five instruction-tuned LLMs show significant, dataset-dependent asymmetries in inverse relation classification, with relation descriptions rarely helping and performance shifting when entities are replaced or masked.","lead":"Large language models frequently get inverse relations backwards, confusing parent with child when asked which direction the sentence implies, and swapping the people in the sentence changes how often they err. The study builds a 5,457-example benchmark from FewRel and TekGen, tests five open models, and finds the errors are real but uneven across datasets and models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FewRel HT/TH asymmetry may reflect lexical overlap between answer labels and sentence words (e.g., 'father' in the sentence) rather than inverse-relation reasoning.","rationale":"I read the paper as trying to establish that LLMs systematically mishandle inverse relation directionality. The strongest evidence is the HT/TH asymmetry on FewRel. But the MCQ format makes the correct answer visible as a lexical string, and the sentences appear biased toward containing the TH label surface form. This confound is not addressed in Section 6 or the Limitations, where the authors only worry about entity anonymization. Since the asymmetry persists under masking (which removes entities but not relation words) and is absent on TekGen where labels are not surface-cued, the lexical-overlap explanation is credible. This is a load-bearing threat to the central claim. The reader's weakest assumption concerned the entity perturbation experiments, which affect a secondary finding; I identify a more central confound. The paper is otherwise careful (paired tests, bootstrap CIs, multiple models), and the benchmark is a useful resource. Therefore I keep the CONDITIONAL verdict but would require the lexical-cue control before fully accepting the asymmetry claim.","tokens_in":13941,"tokens_out":11129,"duration_ms":125733,"concrete_test":"Re-analyze the benchmark by tagging each instance for whether the sentence contains a token that is a morphological variant of the HT label, the TH label, both, or neither (lemmatize via WordNet or similar). Then recompute the paired HT-vs-TH Wilcoxon test separately in the 'neither' and 'both' subsets. If the asymmetry on FewRel disappears in the 'neither' subset, the effect is driven by lexical overlap. As a complementary check, replace the answer labels in the prompt with arbitrary codewords (e.g., R1/R2/R3) and provide a mapping; if the direction asymmetry vanishes, the original result is label-surface-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of 'systematic asymmetries' (Abstract; Table 4) is based on MCQ prompts where the answer options are the relation labels themselves (Section 4.2, Table 2). For each instance, the same sentence is shown in both HT and TH conditions; only the entity order in the question changes, so the model receives identical lexical cues in both conditions. On FewRel, several inverse pairs are kinship relations (child/father/mother) and sentences frequently contain words like 'son', 'father', 'mother'. In the Niels Bohr example, the sentence contains both 'son' and 'father'. For TH (Niels Bohr -> Aage), the correct label 'father' is literally present; for HT (Aage -> Niels Bohr), the correct label 'child' is only implied by 'son', while the distractor 'father' is explicit. This creates a spurious TH>HT advantage. The paper does not control for lexical overlap between labels and sentence tokens, and does not report per-relation asymmetries. The masked-entity condition preserves relation words, so persistence of the asymmetry under masking is consistent with a surface-cue explanation. On TekGen, the relation labels (e.g., 'characters', 'position held') are unlikely to appear verbatim, and indeed no significant asymmetry is observed for original/synthetic entities. Therefore the headline result may be an artifact of lexical cueing, not a property of inverse-relation directionality.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether instruction-tuned LLMs correctly recognize the directionality of inverse relations at the sentence level. The authors construct a benchmark from FewRel and TekGen by verifying head-to-tail and tail-to-head relation labels against Wikidata, and evaluate five open-source LLMs using zero-shot multiple-choice prompts. They compare two directions (HT vs. TH), two prompting variants (with/without relation descriptions), and three entity representations (original, synthetic, masked). The main reported findings are: a statistically significant HT/TH asymmetry on FewRel for all entity types; no significant asymmetry on TekGen for original and synthetic entities but a significant effect for masked entities; no consistent benefit from relation descriptions; and sensitivity to entity representations. The paper also introduces the benchmark design, including a Wikidata-based direction verification step, and provides the full data pipeline in the appendix.","tokens_in":14252,"tokens_out":7294,"duration_ms":84216,"significance":"If the headline claims are valid, this would be a useful systematic evaluation of inverse-relation directionality in LLMs, with a reusable benchmark construction methodology. Strengths include the use of Wikidata to verify relation directions, evaluation across five models, zero-shot prompting, and paired statistical tests. The null result on relation descriptions is a creditable negative finding. However, the central asymmetry claim is threatened by a lexical-overlap confound: answer labels are the relation names themselves, and several FewRel sentences contain those names verbatim, so the observed TH/HT difference may reflect surface-word matching rather than relational reasoning. The entity-familiarity interpretation is also explicitly acknowledged by the authors to be confounded by semantic changes from anonymization. Code and data are only promised upon acceptance, so reproducibility cannot be currently assessed. With additional controls, the benchmark could be valuable; in its present form, the main claim is not fully supported.","major_comments":[{"comment":"The FewRel directional asymmetry is vulnerable to a lexical-overlap confound. In the Bohr example (Table 2), the TH ground-truth 'father' appears verbatim in the sentence ('his father's assistant'), while the HT ground-truth 'child' is only present as 'son'. Because the answer choices are the relation labels themselves, TH receives a surface cue that HT lacks. Masked-entity prompts preserve all content words, so the persistence of the asymmetry under masking is exactly what a cue-based explanation predicts; the absence of the effect on TekGen, whose labels ('characters', 'position held') rarely occur as surface tokens, is also consistent. To support the claimed 'systematic asymmetries', the authors should report per-relation HT vs. TH scores and run a control with paraphrased answer labels or with relation-denoting words removed from the sentence.","section":"4.2 / Table 2 / Table 4"},{"comment":"Many TekGen inverse pairs have a single sentence (e.g., astronaut mission/crew member, reservoir created/dam in Figure 4), so per-pair macro-F1 takes values of 0 or 1 for those pairs. The paired Wilcoxon test in Table 4 pools 160 such observations, and this high-variance Bernoulli component can mask real effects or produce spurious ones, especially for the masked-entity comparison. Please report per-pair instance counts, exclude or down-weight pairs below a minimum support, and re-check the TekGen results with an instance-level bootstrap or a stratified permutation test.","section":"5.1 / Figure 4 / Table 4"},{"comment":"The third contribution, framed as analyzing the effect of entity familiarity, is not supported by the entity-replacement experiments. As the Limitations section states, Presidio may leave many entities unchanged, and replacing entities can break attribute-dependent relations (e.g., father/mother). The macro-F1 differences in Table 6 are therefore consistent with a change in sentence semantics rather than a loss of entity familiarity. The manuscript should either restrict the analysis to verified anonymizations or explicitly state that Table 6 provides evidence of representation sensitivity only, and adjust the abstract and conclusions accordingly.","section":"6 / Table 6 / Limitations"}],"minor_comments":[{"comment":"Reference [15] cites Thoppilan et al. (LaMDA) for Flan-T5 XL, which appears to be a citation error. Please cite the Flan-T5 source (e.g., Chung et al., 'Scaling Instruction-Finetuned Language Models').","section":"References"},{"comment":"The token list contains the artifact '/quotesingle.Vars' after 'father'; fix the tokenization display.","section":"Appendix B"},{"comment":"The counts in Figure 3 do not sum to the stated benchmark total; clarify whether these are direction-verified instances and how they relate to the overall 5,457 instances.","section":"Figures 3 and 4"},{"comment":"Please state whether the Wilcoxon tests are one- or two-sided, how ties are handled, and whether any multiple-comparison correction is applied. The 95% CI of the mean absolute difference does not indicate the direction of the effect.","section":"Tables 4-6"},{"comment":"Explain why TekGen negative relations are sampled from all 16 pairs, including those without any sentence instances; this choice may affect option difficulty and deserves justification.","section":"4.2"},{"comment":"Add error bars or confidence intervals for the macro-F1 values, since many reported differences are within a few percentage points and may be within noise.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central finding, the directional asymmetry on FewRel, is at risk from the lexical-overlap confound. I would ask the authors to add a per-relation analysis and a label-paraphrase control before this can be accepted. The paper's own Limitations are candid, but the abstract and contribution list go beyond what the experiments establish. Also, the code and data are not yet available, so the reproducibility of the benchmark construction cannot be independently assessed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper builds a new benchmark and runs a careful empirical study, but the load-bearing result on FewRel has a plausible surface-cue explanation that the authors never test. I'd send it to peer review, but the central claim needs more work.\n\nWhat's genuinely new: this is the first systematic sentence-level evaluation of inverse relation directionality in LLMs, and the benchmark construction is solid. They take FewRel and TekGen, extract entity pairs, verify directions against Wikidata, and only keep consistent pairs. The zero-shot MCQ design across five open-weight models is standard but appropriate, and the paired Wilcoxon tests with bootstrap CIs are a step beyond most papers in this area.\n\nThe paper is honest about its own limits: the limitations section explicitly acknowledges that synthetic anonymization may change sentence semantics and that they don't check this. That's real credit.\n\nThe soft spot is the stress-test concern, and it lands. In the MCQ, the answer options are the relation labels. On FewRel, many inverse pairs involve kinship words like 'father' and 'son' that appear verbatim in the sentences. In the Niels Bohr example, the TH question asks the relation from Niels Bohr to Aage, and the correct answer 'father' is literally in the sentence. The HT question asks from Aage to Niels Bohr, and 'child' is only implied by 'son'. So the model can pick the answer by lexical matching, not by reasoning about direction. Masking entities to XXX and YYY doesn't remove the relation words, so the persistence of the asymmetry under masking is consistent with this explanation. The paper doesn't report per-relation asymmetries or control for label-sentence overlap. That's a real gap.\n\nThe TekGen null result is actually informative: those relation labels (e.g., 'characters', 'position held') rarely appear verbatim in sentences, and no asymmetry appears for original or synthetic entities. That pattern is exactly what you'd predict if the FewRel effect is lexical.\n\nOther softer spots: code and data are only promised 'upon acceptance,' which weakens reproducibility; synthetic anonymization may leave some entities unchanged; and the abstract overstates the aggregate finding as 'systematic asymmetries across LLMs' when the evidence is dataset-dependent and possibly confounded.\n\nWho this is for: people working on LLM-based relation extraction or KG construction. It's a useful benchmark and a fair start, but the main conclusion should be treated as provisional.\n\nRecommendation: peer review, with a request that the authors address the lexical-overlap confound directly — per-relation analysis, and a prompt design that hides or abstracts the relation labels. I'd want to see that before believing the directional asymmetry is real.","headline":"A useful new benchmark for sentence-level inverse-relation directionality, but the headline FewRel asymmetry may be a lexical-overlap artifact rather than a genuine directional effect.","tokens_in":14714,"tokens_out":1838,"would_cite":false,"duration_ms":22590,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that instruction-tuned LLMs classify inverse relations such as mother/child in a direction-dependent way, with a significant head-to-tail vs tail-to-head gap on the FewRel-derived benchmark but not on the TekGen-derived one","keywords":["inverse relations","relation directionality","converse relations","large language models","relation classification","entity perturbation","FewRel","TekGen"],"falsifier":"Ask annotators to judge whether the target inverse relation still holds in a random sample (say, 100) of the synthetic- and masked-entity sentences; if a substantial share no longer support the relation, the familiarity interpretation is an artifact. As a direct check on the asymmetry claim, rerun the benchmark with the two inverse labels' positions swapped and see whether the head-to-tail vs tail-to-head gap persists.","tokens_in":13841,"feed_emoji":"🔄","tokens_out":5977,"duration_ms":62132,"temperature":0.7,"pith_summary":"The paper tries to establish that instruction-tuned language models do not reliably track the direction of inverse relations—pairs like mother/child where swapping the two entities flips the meaning. It builds a benchmark of 5,457 sentence-level instances spanning 27 verified inverse relation pairs, queries five open models with a three-way multiple-choice prompt, and perturbs entities to separate web familiarity from true relation reasoning. Its headline finding is a systematic asymmetry: on the FewRel-derived partition every model classified the same facts differently when asked head-to-tail versus tail-to-head, while on the TekGen partition that asymmetry mostly disappeared. A careful reader would care because knowledge-graph construction, question answering, and ontology learning all rely on the model keeping the arrow of a relation straight.","feed_headline":"Inverse relations expose lopsided reasoning in five LLMs","feed_subtitle":"On one benchmark every model showed head-to-tail vs tail-to-head gaps; on another the gap almost vanished.","key_machinery":"The central object is the inverse (converse) relation pair—two relations R and R⁻¹ with R⁻¹(x,y) ⇔ R(y,x)—evaluated as a three-way multiple-choice classification where the answer options are the target relation, its inverse, and an unrelated negative. The argument is carried by the benchmark-construction pipeline: entity pairs are extracted from two existing relation-extraction datasets, every direction is verified against a public knowledge base, and direction-specific labels (head-to-tail vs tail-to-head) are assigned only for consistent pairs. Statistical load is carried by paired Wilcoxon signed-rank tests comparing the same model on the same inverse pair in the two directions.","core_discovery":"On its own terms, the paper's central discovery is that inverse relation directionality is not a solved subskill of relation classification: statistical tests show that the direction you query the relation in systematically changes accuracy on the FewRel-derived benchmark. Specifically, paired Wilcoxon tests over 70 observations (7 inverse pairs × 2 description settings × 5 models) reject the null of equal head-to-tail and tail-to-head performance for original, synthetic, and masked entities, all with p<0.05. On the TekGen benchmark the same comparison over 160 observations shows no significant difference for original or synthetic entities but a significant one for masked entities. The autho","pith_inferences":["If the asymmetry reflects pretraining exposure to head-to-tail phrases, fine-tuning on direction-labeled triples or adding contrastive inverse examples should shrink the gap—a testable prediction the paper does not make.","The TekGen masked-only effect suggests a latent directional bias that entity names mask; a follow-up could correlate per-relation accuracy with the surface frequency of each relation phrase in the pretraining corpus.","Because the choice set always contains exactly the inverse pair plus one negative, part of the measured asymmetry may come from answer-position or label-format bias; varying the set size and options would separate that.","The same direction-verified construction can double as a probe for how relation direction is represented internally, for example by inspecting activation differences between correct and incorrect direction prompts."],"forward_implications":["Adding relation descriptions to the answer options does not consistently improve inverse-relation classification across the five models.","Directionality effects are dataset-dependent: every model showed head-to-tail vs tail-to-head asymmetry on the FewRel-derived partition, while the TekGen partition showed asymmetry only when entities were masked.","Substituting synthetic or masked entities changes performance materially, especially on FewRel, which the authors read as sensitivity to entity representations.","The 5,457-instance, 27-label benchmark gives the field a reusable testbed for direction-verified inverse relation classification."],"supporting_citations":[{"why":"Supplies the FewRel sentences and entity pairs used to build half the benchmark.","marker":"[5]"},{"why":"Supplies the TekGen partition sentences used for the other half of the benchmark.","marker":"[11]"},{"why":"Provides the public knowledge base queried via SPARQL to verify relation direction between head and tail.","marker":"[16]"},{"why":"The closest prior study of converse relations in LLMs, which the paper extends from triple-level to sentence-level classification.","marker":"[14]"},{"why":"Establishes direction-sensitive relation-label evaluation, which the paper distinguishes from true inverse pairs.","marker":"[10]"},{"why":"Reports that inverse relation pairs degrade few-shot relation classification, a prior result whose generalization the paper tests in the zero-shot, instruction-tuned setting.","marker":"[2]"}],"fun_headline_variants":["LLMs show systematic inverse relation bias","Direction flips accuracy in LLM relation tests","Inverse relations expose lopsided LLM reasoning","Same relation, reversed meaning: LLMs fail symmetry"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"Replacing a named entity with a synthetic name or a placeholder leaves the sentence's relation intact, so the drop in accuracy measures loss of entity familiarity rather than a sentence that no longer expresses the relation.","fun_headline_variants_meta":{"raw":{"variants":["LLMs show systematic inverse relation bias","Direction flips accuracy in LLM relation tests","Inverse relations expose lopsided LLM reasoning","Same relation, reversed meaning: LLMs fail symmetry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1188,"prompt_tokens":670,"completion_tokens":518,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":414,"completion_tokens_details":{"reasoning_tokens":459}},"tokens_in":414,"tokens_out":518,"duration_ms":6451,"temperature":1.0,"reasoning_tokens":459,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:32:32.138821+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask annotators to judge whether the target inverse relation still holds in a random sample (say, 100) of the synthetic- and masked-entity sentences; if a substantial share no longer support the relation, the familiarity interpretation is an artifact. As a direct check on the asymmetry claim, rerun the benchmark with the two inverse labels' positions swapped and see whether the head-to-tail vs tail-to-head gap persists.","supporting_citations":[{"cited_title":"In: Proceedings Reversing Arrows in Large Language Models 15 of the 2018 Conference on Empirical Methods in Natural Langu age Processing","cited_arxiv_id":null,"evidence_quote":"Supplies the FewRel sentences and entity pairs used to build half the benchmark."},{"cited_title":"In: Payne, T.R., Presutti, V ., Qi, G., Poveda-Villalón, M., Stoilos, G., Hollink, L., Kaoudi, Z., Cheng, G., Li, J","cited_arxiv_id":null,"evidence_quote":"Supplies the TekGen partition sentences used for the other half of the benchmark."},{"cited_title":"In: Bouamor, H., Pino , J., Bali, K","cited_arxiv_id":null,"evidence_quote":"The closest prior study of converse relations in LLMs, which the paper extends from triple-level to sentence-level classification."},{"cited_title":"IEEE Tra nsactions on Emerging Topics in Computational Intelligence 6(4), 883–892 (2022)","cited_arxiv_id":null,"evidence_quote":"Establishes direction-sensitive relation-label evaluation, which the paper distinguishes from true inverse pairs."},{"cited_title":"In: Raedt, L.D","cited_arxiv_id":null,"evidence_quote":"Reports that inverse relation pairs degrade few-shot relation classification, a prior result whose generalization the paper tests in the zero-shot, instruction-tuned setting."}],"review_version":1}