Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structures

Colin White; Ellen Vitercik; Yingxi Li; Yu He

REVIEW 2 major objections 4 minor 9 references

Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structures

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces DSR-Bench, a benchmark of 4,140 data-structure reasoning tasks across 20 structures and 35 operations, and reports that even the best models score only 0.46 on the hardest instances.

desk verdict DSR-Bench is a genuinely useful diagnostic with solid reproducible engineering, but the headline 0.46 number depends on a ground-truth uniqueness guarantee that Appendix B does not actually deliver for at least four challenge tasks. read the letter →

arxiv 2505.24069 v4 pith:BYDQ3RH6 submitted 2025-05-29 cs.LG cs.AI

Yu He , Yingxi Li , Colin White , Ellen Vitercik This is my paper

classification cs.LGcs.AI

keywords datastructurereasoningstructuralLLMbenchmarkingalgorithmichierarchicaltaskorganizationautomatedevaluationlengthgeneralizationspatial

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large language models can reason structurally, meaning the ability to understand and manipulate relationships such as order, hierarchy, and connectivity. To answer it, the authors build DSR-Bench, which turns data structures like queues, trees, and graphs into a diagnostic test of structural reasoning, with fully automated generation and evaluation so that scores are not confounded by human or LLM judging. Across 13 state-of-the-art models, they find that the top model answers only 46 percent of the hardest instances correctly, and that performance degrades on longer inputs, multi-attribute structures, non-uniform spatial data, and realistic language. If the results hold, LLMs currently lack reliable structural reasoning, and DSR-Bench gives researchers a fine-grained tool to localize where that reasoning fails.

What carries the argument

The central object is DSR-Bench itself, a data-structure reasoning benchmark whose tasks are generated synthetically and scored automatically against a unique ground truth by forcing model outputs through a JSON schema. Each data structure is described in natural language with explicit operational rules, including tie-breaking conventions, hash functions, collision resolution, and traversal order, so that a correct answer is a single deterministic outcome rather than a judgment call. These rules are what allow the benchmark to attribute a wrong answer to a reasoning failure rather than to ambiguous specification, and the hierarchical ordering of tasks lets the authors localize which structural relationship, from simple arrays to composite graphs, first breaks a model's reasoning.

What would settle it

Find a single DSR-Bench instance where a model that correctly follows a standard textbook definition produces an output different from the paper's ground truth because of an arbitrary convention, such as a different but common hash function, children-visitation order, or tree-representation format; if such cases are common, the low accuracy scores would partly reflect evaluation ambiguity rather than a structural-reasoning deficit.

Watch

Extended reading notes

Core claim

The central claim is that contemporary LLMs cannot yet reason structurally, and the evidence is a new benchmark designed to isolate that capability. DSR-Bench spans 20 data structures, 35 operations, and 4,140 instances organized in six relationship categories: linear, temporal, associative, hierarchical, network, and hybrid. Evaluation is fully automatic and deterministic, with prompts that specify exact tie-breaking, hash functions, and output formats so that each instance has one unambiguous ground-truth answer. Across 13 models, the top scoring model averages 0.79 on the main suite but only 0.46 on the challenge subset of complex structures, with reasoning models still near zero on several spatial and hybrid structures. The authors also show that instruction-tuned models struggle with multi-attribute elements and multi-hop properties, that reasoning models can ignore user-defined constraints due to learned priors, that performance drops as input dimensionality rises and on non-uniform distributions, and that shifting from formal descriptions to realistic narratives hurts accuracy. Finally, they show that asking models to write code does not help them reason when they must simulate the code themselves, although an external interpreter can help on familiar textbook tasks.

Load-bearing premise

The benchmark's scores assume that the explicit tie-breaking, hash-function, and output-formatting rules in each prompt make every instance have exactly one correct answer, so that a wrong result must be a reasoning failure rather than a reasonable disagreement about a standard convention.

Editorial extensions

If this is right

If the results are correct, any deployment of current LLMs in domains that require step-by-step manipulation of order, hierarchy, or connectivity, such as scheduling, database indexing, or trip planning, cannot assume reliable structural reasoning and must supply external verification or fallback mechanisms.
The gap between formal and realistic performance implies that LLMs trained on textbook-style patterns may not transfer structural skills to context-rich language found in real user input, so safer deployment requires testing on these probes rather than on canonical task formulations.
The finding that models fail to reason over their own generated code, at least when they must simulate execution internally, suggests that relying on code generation to compensate for weak reasoning is not a cure; external interpreters help on familiar tasks but do not fix brittle pattern matching on non-standard variants.
The per-category scores give model developers a diagnostic map: multi-attribute and multi-hop structures, as well as high-dimensional spatial structures, are where current models collapse, so training or architecture improvements should target those specific relationship types.
The explicit specification of tie-breaks and hash functions makes the benchmark safe for repeated use and less vulnerable to contamination than coding benchmarks, so progress on structural reasoning can be measured over time with the same yardstick.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The replication protocol this benchmark invites is a strong test of the underlying claim: because every instance has a unique canonical output, one can directly measure whether a model's reasoning trace is producing the right internal representation at each step, not just the right final answer, which would localize the failure more precisely than an aggregate score.
The degradation on non-uniform spatial data is one of the more consequential results: if it reflects that models reason by matching training-distribution patterns rather than by applying algorithmic rules, then similar distribution-shift fragility is likely in other structured reasoning tasks, such as interpreting real map or sensor data.
An extension the paper leaves implicit is to use the same benchmark to test whether providing explicit procedural examples, rather than formal definitions, closes the realistic-formal gap; the prompting experiments already hint that carefully designed CoT can double accuracy on uncommon structures, so a systematic study of what kind of scaffolding converts formal competence into deployed competenc
The result that models cannot simulate their own code internally suggests a broader conjecture: structural reasoning in an LLM may not be grounded in the same symbolic state-transition semantics as an interpreter, which would predict that no amount of code-writing instruction will help without some external execution or a memory mechanism that tracks state externally.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: this is a well-built diagnostic benchmark that fills a real gap, but the headline 0.46 rests on a ground-truth uniqueness claim that doesn't hold for several challenge tasks. Get the authors to audit that before trusting the number.

What's actually new: DSR-Bench covers 20 data structures and 35 operations organized into six relationship categories, with five evaluation probes. That goes beyond NLGraph/GraphQA (graphs only) and CLRS-Text (algorithm simulation). Generation is synthetic and deterministic; scoring is exact match against programmatically computed outputs; schema violations are zero across 1,624 trials; results are averaged over three runs with standard deviations. They also include control analyses, like matching human difficulty for the spatial distribution shift, and they're transparent about prompting sensitivity. The code and data are public. That's real, reproducible work and I'd cite it as a benchmark if I were building on it.

The soft spot is exactly what the stress-test note says. Appendix B specifies rules for each structure, but four tasks don't actually pin down a unique ground truth. DSU only says lower-rank roots attach to higher-rank ones; equal-rank unions are left arbitrary. Skip List says promotion probabilities are specified, but probabilities don't determine the realized coin flips that produce the layers. LRU Cache's final state is a set, but string comparison needs a canonical order. Priority Queue gives tie-break rules for level-order output but doesn't fix Fibonacci heap consolidation/merge order. Any of these can make a correct computation fail exact-match scoring. The paper even says the central evaluation challenge is tie-breaking ambiguity, yet it reports no uniqueness audit across the released instances.

How much does this matter? The affected tasks are concentrated in the challenge subset that produces the 0.46 headline, so that number is not clean. The qualitative conclusion — LLMs are brittle on complex structural reasoning and degrade on long inputs, spatial data, and realistic language — is consistent across many unambiguous tasks (arrays, stacks, queues, BST, graph traversal), so I expect it to survive an audit. But the specific accuracy ranking, and the claim of unambiguous ground truth, are overstated as written.

Minor: three runs per model is acceptable at this scale, though it adds noise; the Levenshtein ablation is a useful sanity check but doesn't fix the uniqueness problem.

Who this is for: anyone building LLM evaluation suites, or studying algorithmic reasoning and model brittleness. It deserves a serious referee, but the review should require a reference-implementation audit of ground-truth uniqueness per task, plus re-estimation of any affected scores. If the audit comes back clean, this is a solid benchmark paper.

Referee Report

2 major / 4 minor

Summary. DSR-Bench is a benchmark for evaluating the structural reasoning of large language models through data structure tasks. It comprises 4,140 instances spanning 20 data structures, 35 operations, six relationship categories (linear, temporal, associative, hierarchical, network, hybrid), and three length-based difficulty levels, plus a challenge subset and three auxiliary probes (spatial, realistic, code). Instances are generated synthetically and scored automatically by binary exact match against programmatically computed ground truth, with model outputs constrained to a JSON schema. The authors evaluate 13 instruction-tuned and reasoning models from open and closed sources, three runs each, and report that the best model scores 0.79 on the main suite and only 0.46 on the challenge subset; performance degrades with input length, with higher dimensionality, on non-uniform data distributions, and under realistic paraphrased language, and internal code simulation does not help where external execution does. The paper concludes that LLMs cannot yet reason structurally and positions DSR-Bench as a fine-grained diagnostic for algorithmic reasoning.

Significance. If the measurements hold, DSR-Bench is a valuable and influential diagnostic resource. The paper's concrete strengths are substantial: fully synthetic instance generation for contamination resistance; programmatic ground-truth computation; deterministic, schema-enforced evaluation with zero JSON violations across the reported trials; three-run averaging with standard deviations in the appendix; a human-difficulty matching control for the non-uniform distribution study (Appendix F); an ablation justifying binary scoring over Levenshtein distance (Section I); and public releases of code and data. The six-category taxonomy and per-operation scoring support the paper's claim of fine-grained failure localization, and the auxiliary probes usefully extend the evaluation beyond formal textbook prompts. The headline result—that the best reasoning model remains below 0.5 on the hard subset—is falsifiable and, if correct, directly relevant to LLM deployment in scheduling, database, robotics, and healthcare domains.

major comments (2)

[Section 3.4 / Appendix B] Section 3.4 states that “any potential ambiguity (hash functions, tie-breaking rules) is explicitly specified in the prompts” and that every instance is scored against “a single well-defined solution.” This premise is load-bearing for all of the paper's headline accuracies, but the specifications in Appendix B do not establish uniqueness for at least four tasks. (1) DSU: “lower-rank roots are always attached to higher-rank ones” is silent for equal-rank unions, where standard union-by-rank may break ties arbitrarily. (2) Skip List: the prompt is said to specify only “promotion probabilities,” and probabilities do not determine the realized coin flips that fix the final layer structure, so the ground truth appears to be a random variable that the model cannot infer from the prompt. (3) LRU Cache: the final state is defined as a “set of elements,” but exact-match scoring against a serialized answer requires a canonical order that the prompt text does not define. (4) Priority Queue: the output is a level-order traversal of a Fibonacci heap forest, but consolidation order, cascading cuts, and merge order are not pinned down by the stated tie-break rules, so different valid implementations can yield different forests for the same operation sequence. Since Table 3's challenge score (0.46) includes Skip List and Priority Queue, these gaps could materially affect the reported reasoning deficit. The high observed scores on DSU and LRU (Table 2) and the non-negligible Skip List scores (Table 3) suggest some of these concerns may be resolved by the actual prompts, which is precisely why an audit is needed rather than a full re-evaluation. I recommend (i) quoting the actual prompt text for these tasks verbatim in the appendix; (ii) reporting a uniqueness audit, e.g., the fraction of instances on which two independent reference implementations or two reasonable readings of the stated rules produce identical graded outputs; and (iii) revising any prompt whose stated rules admit multiple answers, such as listing realized promotion outcomes for Skip List and specifying equal-rank DSU attachment.
[Section 3.2 / Table 3] The abstract's flagship quantitative claim—that “the top-performing model achieves only 0.46/1 on challenging instances”—cannot be reconstructed from the definitions given in the main text. Section 3.2 defines the challenge subset only as “particularly complex structures,” and Table 3 does not state which difficulty levels are included, how many instances per task, or that the Challenge Score row is the unweighted mean of the eleven per-task rows. I verified from Appendix D that each Challenge Score entry equals the corresponding “Long” (length 21–30) value of the starred structures in Tables 7–19 (for GPT-5, Hashmap 0.71, Bloom Filter 0.47, Geom Graph 0.19, and B+ Tree 0.98 match the long-level columns), so the intended construction is recoverable by cross-referencing, but it should be stated explicitly. A one-sentence definition of the challenge subset and its aggregation rule in Section 3.2 or the Table 3 caption would make the headline number directly verifiable.

minor comments (4)

[Section 3.4 / Table 30] The text reports zero schema violations across “1,624 trials with nine models and six structures,” but Table 30 contains 9 models × 6 structures × 30 trials = 1,620 trials; the trial count should be reconciled.
[Section 4] The paper states that “each problem [was] evaluated three times (166,230 total evaluations),” but 4,140 instances × 13 models × 3 runs equals 161,460; the composition of the reported total, presumably including the auxiliary probes, should be stated explicitly.
[Section 4.1.1 / Figure 3] The claim that the None prompt “performs worst” is an aggregate over tasks (Figure 3), and Table 20 shows that the pattern is task-dependent; for example, on Trie with Claude-3.5-Sonnet, None scores 0.89 while Stepwise scores 0.02, so the generalization should be qualified as an average effect.
[Appendix D] Section D's enumeration of per-model accuracy tables stops at Table 16 (Llama-3.3) and omits Tables 17–19 (Qwen3-8B, Phi-4-reasoning-14B, Mixtral-8x7B), and Table 6's caption says “instruction-tuned models” while including only five of the seven instruction-tuned models; the cross-references and captions should be completed.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper's central claims are empirical measurements of 13 LLMs against independent, programmatically generated ground truth, with no fitted parameters or self-citation chain; only a mild self-definitional framing of 'structural reasoning' via the paper's own task taxonomy.

full rationale

This is a benchmark paper, not a derivation paper, and its claim chain is predominantly self-contained and externally anchored. The headline result — "the top-performing model achieves only 0.46/1 on challenging instances" (Section 5) — is a measurement against ground truth computed by programmatic implementations of each data structure (Section 3.4: "Each data structure and its operations are programmatically implemented to produce ground-truth outputs"). There are no fitted parameters renamed as predictions, no predicted quantity that is equal to a task definition by construction, and no equation-level reduction of an output to an input. The evaluation protocol (binary 0/1 scoring against a single reference solution, Section 3.4) is exactly the standard external-check design; model outputs are not derived from the benchmark's own definitions. The three auxiliary probes (spatial, realistic, code) compare the same models on held-out, matched settings and include internal controls, such as the formal-versus-realistic comparison with "identical problem distribution" (Section 4.3) and the uniform-versus-non-uniform difficulty-matching control in Appendix F, which argues against the benchmark being tuned to manufacture failures. Citations to prior benchmarks (CLRS-Text, NLGraph, GraphQA) are comparative and not load-bearing; no uniqueness theorem or ansatz is imported from the authors' own prior work. If there is any circular flavor, it is the ordinary operationalization inherent to all benchmarks: "structural reasoning" is defined in Section 1 as handling "order, hierarchy, and connectivity" and in Section 3.1 as the six relationship categories (Linear, Temporal, Associative, Hierarchical, Network, Hybrid), so the verdict in Section 5 ("not yet") restates, in construct terms, the measured score on the taxonomy the paper itself chose. That is a framing choice, not a reduction: the scores themselves are not implied by the definitions. The weakest load-bearing assumption — that every instance has a unique ground truth with all tie-breaking specified (Section 3.4) — is a validity and correctness concern about possible underspecification in Appendix B tasks (e.g., DSU equal-rank unions, Skip List realized coin flips, LRU set ordering, Fibonacci-heap level-order format), not a circularity: ambiguity would bias the measured accuracies, but it would not make the conclusion equivalent to the inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The benchmark introduces no fitted parameters or invented entities beyond the two hand-chosen design constants listed above. Its conclusions rest on domain assumptions about the validity of data structure tasks as a probe of structural reasoning, the representativeness of synthetic uniform sampling, and the uniqueness of ground truth under explicit tie-breaking rules.

free parameters (2)

Prompt token budget = 8000 tokens
Fixed verbosity ceiling appended to all prompts; chosen by hand, not tuned against results, but it constrains every model response and could influence output quality.
Difficulty length thresholds = short 5-10, medium 11-20, long 21-30
Input-length cutoffs used to define difficulty levels; chosen by hand and used in aggregation and length-generalization analysis, though the overall low scores do not hinge on the exact boundaries.

assumptions (3)

domain assumption Data structure tasks are a valid operationalization of structural reasoning.
The paper justifies this lens in Section 1 and relies on it to draw conclusions about structural reasoning from benchmark scores.
domain assumption Synthetic uniform random inputs produce contamination-resistant instances representative of data structure tasks.
Data generation in Section 3.4 relies on uniform sampling of integers and lowercase letters; non-uniform distributions are separately probed later.
domain assumption Ground truth outputs are unambiguous given the specified tie-breaking and representation rules.
Section 3.4 and Appendix B specify hash functions, tie-breaking, and traversal orders to ensure a single correct answer; this is load-bearing for binary scoring.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structures." pith.science (2026). https://pith.science/paper/BYDQ3RH6

@misc{pith2026250524069,
  author       = {Pith},
  title        = {Pith review of: Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BYDQ3RH6}},
  note         = {Machine review of arXiv:2505.24069}
}

read the original abstract

Large language models (LLMs) are deployed on increasingly complex tasks that require multi-step decision-making. Understanding their algorithmic reasoning abilities is therefore crucial. However, we lack a diagnostic benchmark for evaluating these capabilities. We propose to use data structures as a principled lens: as fundamental building blocks of algorithms, they naturally probe structural reasoning - the ability to understand and manipulate relationships such as order, hierarchy, and connectivity that underpin algorithmic reasoning. We introduce DSR-Bench (Data Structure Reasoning Benchmark), spanning 20 data structures, 35 operations, and 4,140 problem instances. DSR-Bench features hierarchical task organization, fully automated generation and evaluation, and fine-grained diagnostics. Evaluating 13 state-of-the-art LLMs reveals critical limitations: the top-performing model achieves only 0.46/1 on challenging instances. Three auxiliary probes targeting more realistic usages expose further weaknesses: models perform poorly on spatial data and context-rich scenarios, and they struggle to reason over their own code.

Figures

Figures reproduced from arXiv: 2505.24069 by the authors.

**Figure 1.** High-level overview of DSR-Bench-main with six data structure categories capturing distinct relationships, plus the challenge subset. Three specialized probes that holistically evaluate structural reasoning under different settings: spatial (multidimensional data), realistic (realistic context-rich scenarios), and code (code generation). See Section B for full list of tasks. collections/vitercik-lab/dsr-bench for c… view at source ↗

**Figure 2.** Left: Scores of thirteen models on DSR-Bench-main, averaged across three runs. Right: Radar chart showing scores of top-performing models across six relationship categories. Note that the best model on DSR-Bench-challenge scores only 0.46. time-sensitive applications, including scheduling. • Associative (Key-value mapping): This category includes HASHMAP, TRIE, SUFFIX TREE, and SKIP LIST. We evaluate the ability to… view at source ↗

**Figure 3.** Average scores across all tasks for five instruction-tuned models under different prompting strategies. Instruction-tuned models struggle with multi-attribute reasoning. As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 5.** Figure 5: (a) Performance on the two metrics for K-D HEAP. (b) K-D Tree with varying input data distributions. Limited robustness to non-uniform data distributions. We assess LLM robustness to distribution shifts by comparing performance on uniformly sampled versus skewed or cl…

**Figure 4.** Figure 4: Scores for the three spatial data structures with input data of varying dimensionality (k = 1, 2, 3, 5). Performance degrades as dimensionality increases. In [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 6.** Figure 6: Model performance on formal and realistic usages. LLMs struggle when shifting from formal to realistic usage. Performance drops when tasks are using realistic, context-rich language compared to formal descriptors, despite identical problem distribution ( [PITH_FULL_I…

**Figure 8.** Figure 8: Example K-D Tree instances from three non-uniform distributions. the root node is determined by a median over the entire dataset, this ambiguity leads to more errors. Supporting this, we observed 13 out of 30 correct root nodes on uniform data, but only 5 out of 30 on …

**Figure 9.** Figure 9: The pipeline for generating natural language prompts [PITH_FULL_IMAGE:figures/full_fig_p045_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

9 extracted references · 2 canonical work pages

[3]

Jiang, A

URL https://openreview.net/forum? id=chfJJYC3iL. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T....

arXiv 2024
[5]

Malfa, E

Accessed: 2025-09-10. Malfa, E. L., Weinhuber, C., Torre, O., Lin, F., Marro, S., Cohn, A., Shadbolt, N., and Wooldridge, M. Code simu- lation challenges for large language models, 2024. URL https://arxiv.org/abs/2401.09074. Malfa, E. L., Weinhuber, C., Torre, O., Lin, F., Huang, X. A., Marro, S., Cohn, A., Shadbolt, N., and Wooldridge, M. Code simulation...

arXiv 2025
[560]

acl-long.560/

URL https://aclanthology.org/2024. acl-long.560/. Sui, Y ., Zhou, M., Zhou, M., Han, S., and Zhang, D. Table meets llm: Can large language models under- stand structured table data? a benchmark and empirical study. InProceedings of the 17th ACM International Conference on Web Search and Data Mining, WSDM ’24, pp. 645–654, New York, NY , USA, 2024. Associa...

arXiv 2024
[765]

$” to ensure a unique structure. The final state is a pre-order traversal collecting edge labels, with child edges visited in lexicographical order and “$

URL https://aclanthology.org/2024. findings-emnlp.765/. Ye, J., Wang, Y ., Huang, Y ., Chen, D., Zhang, Q., Moniz, N., Gao, T., Geyer, W., Huang, C., Chen, P.-Y ., Chawla, N. V ., and Zhang, X. Justice or prejudice? quantifying biases in LLM-as-a-judge. InThe Thirteenth International Confer- ence on Learning Representations, 2025. URL https: //openreview....

work page 2024
[2021]

Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I

URL https://openreview.net/forum? id=7Bywt2mQsCe. Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Live- codebench: Holistic and contamination free evaluation of large language models for code. InThe Thirteenth International Conference on Learning Representations,

work page
[2022]

Wang, B., Yue, X., Su, Y ., and Sun, H

URL https://proceedings.mlr.press/ v162/velickovic22a.html. Wang, B., Yue, X., Su, Y ., and Sun, H. Grokking of im- plicit reasoning in transformers: A mechanistic journey to the edge of generalization. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https://openreview.net/forum? id=D4QgSWxiOb. Wang, H., Feng, S....

arXiv 2023
[2023]

doi: 10.1145/3564240

ISSN 0360-0300. doi: 10.1145/3564240. URL https://doi.org/10.1145/3564240. Silpa-Anan, C. and Hartley, R. Optimised kd-trees for fast image descriptor matching. In2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8, 2008. doi: 10.1109/CVPR.2008.4587638. Singh, H., Gupta, N., Bharadwaj, S., Tewari, D., and Taluk- dar, P. IndicGenBench:...

arXiv
[2024]

Kanithi, P., Christophe, C., Pimentel, M

URL https://openreview.net/forum? id=VTF8yNQM66. Kanithi, P., Christophe, C., Pimentel, M. A., Raha, T., Munjal, P., Saadi, N., Javed, H. A., Maslenkova, S., Hayat, N., Rajan, R., and Khan, S. Medic: Compre- hensive evaluation of leading indicators for llm safety and utility in clinical applications, 2026. URL https: //arxiv.org/abs/2409.07314. Koo, R., L...

arXiv 2026

Show all 9 references

[2025]

Fatemi, B., Halcrow, J., and Perozzi, B

URL https://openreview.net/forum? id=eax2ixyeQL. Fatemi, B., Halcrow, J., and Perozzi, B. Talk like a graph: Encoding graphs for large language models. InThe Twelfth International Conference on Learning Represen- tations, 2024. URL https://openreview.net/ forum?id=IuXR1CCrSi. ...

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [3]

Jiang, A

URL https://openreview.net/forum? id=chfJJYC3iL. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T....

arXiv 2024

[2] [5]

Malfa, E

Accessed: 2025-09-10. Malfa, E. L., Weinhuber, C., Torre, O., Lin, F., Marro, S., Cohn, A., Shadbolt, N., and Wooldridge, M. Code simu- lation challenges for large language models, 2024. URL https://arxiv.org/abs/2401.09074. Malfa, E. L., Weinhuber, C., Torre, O., Lin, F., Huang, X. A., Marro, S., Cohn, A., Shadbolt, N., and Wooldridge, M. Code simulation...

arXiv 2025

[3] [560]

acl-long.560/

URL https://aclanthology.org/2024. acl-long.560/. Sui, Y ., Zhou, M., Zhou, M., Han, S., and Zhang, D. Table meets llm: Can large language models under- stand structured table data? a benchmark and empirical study. InProceedings of the 17th ACM International Conference on Web Search and Data Mining, WSDM ’24, pp. 645–654, New York, NY , USA, 2024. Associa...

arXiv 2024

[4] [765]

$” to ensure a unique structure. The final state is a pre-order traversal collecting edge labels, with child edges visited in lexicographical order and “$

URL https://aclanthology.org/2024. findings-emnlp.765/. Ye, J., Wang, Y ., Huang, Y ., Chen, D., Zhang, Q., Moniz, N., Gao, T., Geyer, W., Huang, C., Chen, P.-Y ., Chawla, N. V ., and Zhang, X. Justice or prejudice? quantifying biases in LLM-as-a-judge. InThe Thirteenth International Confer- ence on Learning Representations, 2025. URL https: //openreview....

work page 2024

[5] [2021]

Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I

URL https://openreview.net/forum? id=7Bywt2mQsCe. Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Live- codebench: Holistic and contamination free evaluation of large language models for code. InThe Thirteenth International Conference on Learning Representations,

work page

[6] [2022]

Wang, B., Yue, X., Su, Y ., and Sun, H

URL https://proceedings.mlr.press/ v162/velickovic22a.html. Wang, B., Yue, X., Su, Y ., and Sun, H. Grokking of im- plicit reasoning in transformers: A mechanistic journey to the edge of generalization. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https://openreview.net/forum? id=D4QgSWxiOb. Wang, H., Feng, S....

arXiv 2023

[7] [2023]

doi: 10.1145/3564240

ISSN 0360-0300. doi: 10.1145/3564240. URL https://doi.org/10.1145/3564240. Silpa-Anan, C. and Hartley, R. Optimised kd-trees for fast image descriptor matching. In2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8, 2008. doi: 10.1109/CVPR.2008.4587638. Singh, H., Gupta, N., Bharadwaj, S., Tewari, D., and Taluk- dar, P. IndicGenBench:...

arXiv

[8] [2024]

Kanithi, P., Christophe, C., Pimentel, M

URL https://openreview.net/forum? id=VTF8yNQM66. Kanithi, P., Christophe, C., Pimentel, M. A., Raha, T., Munjal, P., Saadi, N., Javed, H. A., Maslenkova, S., Hayat, N., Rajan, R., and Khan, S. Medic: Compre- hensive evaluation of leading indicators for llm safety and utility in clinical applications, 2026. URL https: //arxiv.org/abs/2409.07314. Koo, R., L...

arXiv 2026