pith. sign in

arxiv: 2604.12176 · v1 · submitted 2026-04-14 · 💻 cs.AI

Evaluating Relational Reasoning in LLMs with REL

Lukas Fesser , Yasha Ektefaie , Ada Fang , Sham M. Kakade , Marinka Zitnik This is my paper

Pith reviewed 2026-05-10 16:21 UTC · model grok-4.3

classification 💻 cs.AI
keywords relational reasoninglarge language modelsrelational complexitybenchmarkalgebrachemistrybiologyarity
0
0 comments X

The pith

Frontier LLMs show steady performance drops on relational tasks as the number of entities that must bind together increases, even with fixed total entities and extra compute.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Relational reasoning means inferring how multiple entities, attributes, or variables connect under a single relation. The paper defines Relational Complexity as the smallest number of independent items that have to be bound at the same time to apply that relation. REL is a new generative benchmark that varies this complexity inside algebra, chemistry, and biology problems while holding input length, vocabulary, and entity count fixed. Across many large language models, accuracy falls in a regular, step-wise pattern as complexity rises, and the pattern survives both longer chain-of-thought reasoning and in-context examples. The result isolates a bottleneck in handling higher-arity bindings rather than in raw scale or step count.

Core claim

Across frontier LLMs, performance degrades consistently and monotonically as Relational Complexity increases, even when the total number of entities is held fixed. This failure mode persists with increased test-time compute and in-context learning, suggesting a limitation tied to the arity of the required relational binding rather than to insufficient inference steps or lack of exposure to examples.

What carries the argument

Relational Complexity (RC), defined as the minimum number of independent entities or operands that must be simultaneously bound to apply a relation; it is used to vary reasoning difficulty independently of input size or vocabulary.

If this is right

  • Models will underperform on any task whose logical structure requires simultaneous binding of four or more entities, regardless of scaling.
  • Standard benchmarks that only count total entities or inference steps will miss this specific failure mode.
  • Scientific reasoning applications involving multiple intertwined variables will remain limited until binding mechanisms improve.
  • Extra test-time compute and in-context examples will not close the gap for higher-arity problems.
  • New benchmarks should explicitly control for arity rather than aggregate entity count.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same arity limit may appear in other domains such as multi-object scene understanding or planning with many interdependent variables.
  • Architectures that maintain explicit bindings or use variable-binding mechanisms could be tested directly against the REL tasks.
  • If the degradation continues at still higher RC values, it would suggest a hard computational ceiling rather than a gradual scaling issue.

Load-bearing premise

The generative tasks in REL truly isolate the effect of relational arity without other uncontrolled differences in input structure or task framing that could cause the performance drop.

What would settle it

An experiment in which the same logical binding requirements are presented in a restructured format that lowers apparent arity while keeping entity count and content identical, yet models still show the same accuracy drop, would challenge the claim that the limit is specifically in arity.

Figures

Figures reproduced from arXiv: 2604.12176 by Ada Fang, Lukas Fesser, Marinka Zitnik, Sham M. Kakade, Yasha Ektefaie.

Figure 1
Figure 1. Figure 1: a Performance decreases as relational complexity increases, even when the number of entities varies across tasks. Entity count is therefore a noisy proxy for task difficulty. b Relational complexity increases with the number of entities that must be jointly bound to satisfy a shared constraint, i.e., when correctness depends on a higher-arity relation. c REL evaluates relational reasoning in LLMs across al… view at source ↗
Figure 2
Figure 2. Figure 2: REL evaluates relational reasoning across algebraic, biological, and chemical domains [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Three examples of Raven’s Progressive Matrices with increasing relational complexity. The answers are shown in bold. along the x-, y-, and z-axis. RC = 4. A6 (5-Moving Aver￾age). Same as previous, but with 5 predecessors. RC = 5. A7 (Neighborhood Sum). Each entry of the RPT is the sum of its neighbors modulo 7. RC = 6 in a 3 × 3 × 3 tensor. We use the same setup as in the original vision-based RPM task, th… view at source ↗
Figure 4
Figure 4. Figure 4: From provided parameters, we generate a phylogenetic tree, alignment, inject shared motifs, and ask the model to use this alignment and tree to identify the taxa that are in homoplasy. 4.3. Relational Reasoning in Chemistry (REL-C) Relational reasoning is key to understanding molecular func￾tion and the vast chemical space (Bemis & Murcko, 1996). This capability is central to chemical library design (Hajdu… view at source ↗
Figure 5
Figure 5. Figure 5: Model performance on REL-A tasks. RPMs at the top, with RPTs below. The models are given 8 answer choices, so trivial accuracy is 12.5%. All three model perform well on tasks with low RC (REL-A1 and REL-A2, top two rows), but struggle once RC increases: on REL-A3 and REL-A4, where RC increases with input size, performance drops by as much as 80%. RPTs (REL-A5, REL-A6, and REL-A7), which always have a highe… view at source ↗
Figure 6
Figure 6. Figure 6: Performance decreases with increasing RC controlled by increasing the number of homoplastic taxa increases in REL-B1. tributed by each variable. Across all three models, RC explains the largest share of explainable variance: 24% of explainable variance for Claude, 32% for Gemini, and 44% for GPT. In contrast, the next strongest factor, motif ratio for Claude, prompt length for Gemini, and distance between … view at source ↗
Figure 7
Figure 7. Figure 7: Top: Schema of variables in multivariate regression. Bottom: Explained variance of performance on REL-B1 across five measures of complexity. RC, which is number of homoplastic taxa, explains the most variance. SMILES strings and comparing the canonical forms for exact match, ensuring that chemically equivalent SMILES representations are treated as correct. Example prompts of the three tasks are provided in… view at source ↗
Figure 8
Figure 8. Figure 8: Task completion rate decreases as from C1 (RC=2 and OC easy) to C2 (RC=2 and OC medium) to C3 (RC=Nisomers, Nobserved and OC hard). This observation holds across different number of molecules in the task. across Nmolecules. We observe that REL-C1 increases in ac￾curacy from 56.0% at 5 molecules to 71.0% at 50 molecules. In REL-C2, the task completion rate remains relatively sta￾ble for averaging at 39.2% ±… view at source ↗
Figure 9
Figure 9. Figure 9: Task completion rate on REL-C2 evaluated with Is Substructure. For this task, RC is fixed at 2 in Left: increasing Nmolecules does not have an effect on performance until Nmolecules = 50, in Right: increasing the molecule size increases OC and leads to decreased IsSubstructure rate. Finally, for REL-C3, RC increases with both Nisomers and Nmissing = Nisomers − Nobserved. We observe that increasing both sou… view at source ↗
read the original abstract

Relational reasoning is the ability to infer relations that jointly bind multiple entities, attributes, or variables. This ability is central to scientific reasoning, but existing evaluations of relational reasoning in large language models often focus on structured inputs such as tables, graphs, or synthetic tasks, and do not isolate the difficulty introduced by higher-arity relational binding. We study this problem through the lens of Relational Complexity (RC), which we define as the minimum number of independent entities or operands that must be simultaneously bound to apply a relation. RC provides a principled way to vary reasoning difficulty while controlling for confounders such as input size, vocabulary, and representational choices. Building on RC, we introduce REL, a generative benchmark framework spanning algebra, chemistry, and biology that varies RC within each domain. Across frontier LLMs, performance degrades consistently and monotonically as RC increases, even when the total number of entities is held fixed. This failure mode persists with increased test-time compute and in-context learning, suggesting a limitation tied to the arity of the required relational binding rather than to insufficient inference steps or lack of exposure to examples. Our results identify a regime of higher-arity reasoning in which current models struggle, and motivate re-examining benchmarks through the lens of relational complexity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 3 minor

Summary. The paper defines Relational Complexity (RC) as the minimum number of independent entities or operands that must be simultaneously bound to apply a relation. It introduces the REL generative benchmark spanning algebra, chemistry, and biology, varying RC while claiming controls for total entity count, input size, vocabulary, and representational choices. Empirical evaluation of frontier LLMs shows consistent monotonic performance degradation as RC increases (even with fixed entity count), and this pattern persists under increased test-time compute and in-context learning, suggesting a limitation specific to higher-arity relational binding rather than inference steps or example exposure.

Significance. If the controls for input structure and other confounders prove effective, the work would usefully identify a regime of higher-arity reasoning where current LLMs fail in a manner not addressed by standard scaling of compute or few-shot examples. The multi-domain generative design and explicit focus on arity (distinct from entity count) are strengths that could inform more targeted benchmark development for scientific reasoning. The persistence results under ICL and extra compute add weight to the claim that the issue is tied to binding arity.

major comments (2)
  1. [§4] §4 (task generators): The manuscript states that RC is varied while holding total entities fixed and controlling for input size/vocabulary, but provides no quantitative metrics (e.g., average nesting depth, variables per clause, or syntactic complexity scores) comparing RC=2 vs. RC=3/4 conditions across domains. Without these, the monotonic degradation could reflect uncontrolled structural changes in the generated problems rather than arity of binding per se, directly undermining the central claim that the failure mode is 'tied to the arity of the required relational binding'.
  2. [§5.2] §5.2 (results on test-time compute and ICL): The persistence of the RC effect under extra compute and in-context examples is presented as evidence against insufficient inference or lack of exposure, but the analysis does not report whether prompt length, token count, or parsing demands also increase with RC; if they do, the controls claimed in the abstract are incomplete and the interpretation that the limitation is arity-specific remains unproven.
minor comments (3)
  1. [Introduction] The abstract and introduction cite prior relational reasoning benchmarks but omit direct comparison tables showing how REL differs in its control of arity vs. entity count from existing synthetic or graph-based evaluations.
  2. [Figures] Figure captions for performance plots should explicitly state the exact RC values tested and whether error bars represent standard error across models or runs.
  3. [§3] Notation for RC is introduced clearly but the operationalization in each domain (e.g., how a ternary binding is instantiated in the chemistry generator) could be illustrated with a short example in the main text rather than only appendix.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. We address each major point below and have incorporated revisions to strengthen the controls and reporting in the manuscript.

read point-by-point responses

  1. Referee: [§4] §4 (task generators): The manuscript states that RC is varied while holding total entities fixed and controlling for input size/vocabulary, but provides no quantitative metrics (e.g., average nesting depth, variables per clause, or syntactic complexity scores) comparing RC=2 vs. RC=3/4 conditions across domains. Without these, the monotonic degradation could reflect uncontrolled structural changes in the generated problems rather than arity of binding per se, directly undermining the central claim that the failure mode is 'tied to the arity of the required relational binding'.

    Authors: We agree that explicit quantitative verification of structural equivalence would strengthen the claim. The REL generators were explicitly designed to vary only the arity of individual relations (RC) while fixing the total number of entities, the number of clauses, vocabulary size, and overall input length across RC levels within each domain. However, we did not include comparative metrics such as average nesting depth or syntactic complexity scores in the original submission. In the revised manuscript, we have added a new table in §4.1 reporting these metrics (nesting depth, variables per clause, and a syntactic complexity score based on abstract syntax tree size) for RC=2, 3, and 4 conditions across algebra, chemistry, and biology. The values are statistically indistinguishable (p > 0.1), supporting that the observed performance degradation is attributable to relational arity rather than uncontrolled structural differences. revision: yes

  2. Referee: [§5.2] §5.2 (results on test-time compute and ICL): The persistence of the RC effect under extra compute and in-context examples is presented as evidence against insufficient inference or lack of exposure, but the analysis does not report whether prompt length, token count, or parsing demands also increase with RC; if they do, the controls claimed in the abstract are incomplete and the interpretation that the limitation is arity-specific remains unproven.

    Authors: We appreciate the referee highlighting the need for explicit verification of these controls in the experimental analysis. By construction, the generators maintain fixed total entity count and input size (token count) across RC levels, with higher-arity relations substituted in place of multiple lower-arity ones to preserve overall prompt length. Nevertheless, we did not report per-condition token counts or parsing demand estimates in §5.2. In the revision, we have added a supplementary table and brief analysis in §5.2 documenting average prompt token counts, maximum sequence lengths, and a proxy for parsing demand (average dependency parse depth) for each RC level under both standard and ICL settings. These remain comparable across conditions (within 5% variation), confirming that the persistent RC effect is not explained by increased input size or parsing load. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical benchmark with independent RC definition and falsifiable results

full rationale

The paper is an empirical benchmark study that defines Relational Complexity (RC) as the minimum number of independent entities that must be simultaneously bound, then generates tasks in algebra, chemistry, and biology while claiming to hold total entities, input size, and vocabulary fixed. Performance degradation with increasing RC is reported as an experimental observation across LLMs, not derived from any equation or parameter fit that reduces to the inputs by construction. No mathematical derivation chain exists, no fitted inputs are relabeled as predictions, and no load-bearing self-citation or uniqueness theorem is invoked to force the central claim. The results remain externally falsifiable via the released benchmark and do not rely on any of the enumerated circular patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that the generated tasks cleanly vary only relational complexity while holding other factors fixed. No free parameters are fitted to produce the main result; the work is benchmark construction and empirical measurement rather than derivation.

axioms (1)
  • domain assumption Relational complexity can be isolated as an independent variable in generative tasks without introducing correlated changes in input length, vocabulary distribution, or representational format.
    Invoked when constructing REL tasks across domains and claiming the performance drop is due to arity rather than confounders.

pith-pipeline@v0.9.0 · 5525 in / 1377 out tokens · 52714 ms · 2026-05-10T16:21:12.936530+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages

  1. [1]

    URL https://www.sciencedirect.com/science/article/pii/ S1359644623003616

    doi: https://doi.org/10.1016/j.drudis.2023.103845. URL https://www.sciencedirect.com/science/article/pii/ S1359644623003616. Alexander, P. A., Dumas, D., Grossnickle, E. M., List, A., and Firetto, C. M. Measuring relational reasoning. The Journal of Experimental Education, 84(1):119–151, 2016. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Do...

    work page doi:10.1016/j.drudis.2023.103845 2023

  2. [2]

    doi: 10.7554/eLife.71393

    ISSN 2050-084X. doi: 10.7554/eLife.71393. URL https://doi.org/10.7554/eLife.71393. Poelwijk, F. J., Krishna, V ., and Ranganathan, R. The context-dependence of mutations: A linkage of for- malisms.PLOS Computational Biology, 12(6):1–19, 06 2016. doi: 10.1371/journal.pcbi.1004771. URL https://doi.org/10.1371/journal.pcbi.1004771. Poelwijk, F. J., Socolich,...

    work page doi:10.7554/elife.71393 2050

  3. [3]

    Simulate a baseline alignment.Using Pyvolve, simulate a nucleotide alignment of length lseq under a standard substitution model

    work page

  4. [4]

    harmful together,

    Inject tree-aware convergent blocks.Inject a motif of length lmotif by enforcing a shared motif across taxa that are distant on the tree: • Select nht leaves whose pairwisetopological distance(the number of edges along the unique path between two leaves) is at least3. • For a randomly chosen contiguous block of lmotif columns, overwrite the nucleotides fo...

    work page 2000

  5. [5]

    ClCC=C(Cl)Cl Return exactly one of: <Yes> or <No> No explanation. REL-C2 Given the following list of SMILES, what is the largest *connected* common chemical motif (maximum common substructure) present in every molecule? Rules: - The motif must be a single connected fragment. - Do NOT tautomerize molecules. - Ignore stereochemistry unless it is explicitly ...

    work page

  6. [6]

    COc1ccc2c(c1)N(CC(C)CN(C)C)c1ccccc1S2

    work page

  7. [7]

    CC(CN(C)C)CN1c2ccccc2Sc2ccccc21

    work page

  8. [8]

    CCc1ccc2c(c1)N(CC(C)CN(C)C)c1ccccc1S2

    work page

  9. [9]

    CSc1ccc2c(c1)N(CC(C)CN(C)C)c1ccccc1S2

    work page

  10. [10]

    REL-C3 Given the following list of constitutional isomers, complete the set by identifying the missing constitutional isomers

    CC(CN(C)C)CN1c2ccccc2Sc2ccc(C#N)cc21 Return your final answer as a single SMILES wrapped exactly like: <smiles>YOUR SMILES HERE</smiles> No explanation. REL-C3 Given the following list of constitutional isomers, complete the set by identifying the missing constitutional isomers. Given SMILES:

    work page

  11. [11]

    C=CCCCF 30 Evaluating Relational Reasoning in LLMs with REL

    work page

  12. [12]

    REL-C4 Given the following 5 molecules, identify one continuous motif fromeachmolecule

    C=CCC(C)F Return the missing molecules as SMILES, one per line, each wrapped exactly like: <smiles>YOUR SMILES HERE</smiles> No explanation. REL-C4 Given the following 5 molecules, identify one continuous motif fromeachmolecule. Task

    work page

  13. [13]

    From each of the 5 molecules below, extract one continuous motif (substructure)

    work page

  14. [14]

    Constraints •Each motif must be a valid SMILES string (complete and parseable by RDKit)

    Ensure the total count of total carboxylic acids across all motifs equals 1. Constraints •Each motif must be a valid SMILES string (complete and parseable by RDKit). •Each motif must be a substructure that actually exists in its parent molecule. •Each motif must contain at least 6 heavy atoms (non-hydrogen). •The sum of total carboxylic acids across all s...

    work page

  15. [15]

    CCN(CC)C(C)=NN=Cc1c2c(O)c3c(O)c(C)c4c(c3c1O)C(=O)C(C(OC=CC(OC)C(C) C(OC(C)=O)C(C)C(O)C(C)C(O)C(C)C=CC=C(C)C(=O)N2)O4

    work page

  16. [16]

    CCC1OC(=O)C(C)C(=O)C(C)C(OC2OC(C)CC(N(C)C)C2O)C2(C)CC(C)C(=NC(C)= O)C(C)C(OCC(=NOCc3ccc(-n4cccn4)nc3)CO2)C1(C)O

    work page

  17. [17]

    CCC1OC(=O)C(C)C(=O)C(C)C(OC2OC(C)CC(N(C)C)C2O)C(C)(OC)CC(C)C(=O)C (C)C2C(C(N)=NOC(C)c3nnc(-c4ccccn4)s3)C(=O)OC12C

    work page

  18. [18]

    CCC12CN3CCc4c([nH]c5ccccc45)C(C(=O)OC)(c4cc5c(cc4OC)N(C=O)C4C(O) (C(=O)OC)C(OC(C)=O)C6(CC)C=CCN7CCC54C76)CC(C3)C1O2

    work page

  19. [19]

    CCOC(=O)CCC(=O)OC1C(OC2C(C)C(OC3CC(C)(OC)C(O)C(C)O3)C(C)C(=O)OC (CC)C(C)(O)C(O)C(C)C(=O)C(C)CC2(C)O)OC(C)CC1N(C)C Step-by-step approach

    work page

  20. [20]

    For each molecule, identify candidate motifs with at least 6 heavy atoms

    work page

  21. [21]

    Count total carboxylic acids in each candidate motif

    work page

  22. [22]

    Select one motif from each molecule such that the total sum equals 1

    work page

  23. [23]

    Some motifs may contain 0 total carboxylic acids; this is allowed

    work page

  24. [24]

    Extract the exact substructure from the parent molecule and copy it precisely

    work page

  25. [25]

    Ensure each SMILES is complete, with all rings properly closed (e.g., c1ccccc1)

    work page

  26. [26]

    Final check: each motif exists in its parent molecule and the total sum equals 1. Functional group examples (for reference) 31 Evaluating Relational Reasoning in LLMs with REL •Ketone: C(=O)C or CC(=O)CC •Carboxylic acid: C(=O)O or CC(=O)O •Ester: C(=O)OC or CC(=O)OC •Aldehyde: C(=O) at chain end •Primary amine: CNH2 or CCN •Alcohol: CO (hydroxyl on an sp...

    work page