Pith. sign in

REVIEW 1 cited by

Transformers Struggle to Learn to Search

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.04703 v2 pith:PBLVRACQ submitted 2024-12-06 cs.CL cs.AIcs.LG

Abulhair Saparov , Srushti Pawar , Shreyas Pimpalgaonkar , Nitish Joshi , Richard Yuanzhe Pang , Vishakh Padmakumar , Seyed Mehran Kazemi , Najoung Kim
show 1 more author
He He
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords searchfindgraphlearnmodeltransformertransformersperform
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Search is an ability foundational in many important tasks, and recent studies have shown that large language models (LLMs) struggle to perform search robustly. It is unknown whether this inability is due to a lack of data, insufficient model parameters, or fundamental limitations of the transformer architecture. In this work, we use the foundational graph connectivity problem as a testbed to generate effectively limitless high-coverage data to train small transformers and test whether they can learn to perform search. We find that, when given the right training distribution, the transformer is able to learn to search. We analyze the algorithm that the transformer has learned through a novel mechanistic interpretability technique that enables us to extract the computation graph from the trained model. We find that transformers perform search at every vertex in parallel: For each vertex in the input graph, transformers compute the set of vertices reachable from that vertex. Each layer then progressively expands these sets, allowing the model to search over a number of vertices exponential in $n_{\text{layers}}$. However, we find that as the input graph size increases, the transformer has greater difficulty in learning the task. This difficulty is not resolved even as the number of parameters is increased, suggesting that increasing model scale will not lead to robust search abilities. We also find that performing search in-context (i.e., chain-of-thought) does not resolve this inability to learn to search on larger graphs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning Model Successors

    cs.LG 2025-01 conditional novelty 6.0 of 10

    A meta-learner that transforms a model for difficulty level k into a model for level k+1 can generalize far beyond its training range, demonstrated on a bracket-matching task.

Reference graph

Works this paper leans on

55 extracted references · 19 linked inside Pith · cited by 1 Pith paper

  1. [1]

    The pitfalls of next-token prediction

    Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=76zq8Wkl6Z

    2024

  2. [2]

    Chayes, and Oliver Riordan

    B \' e la Bollob \' a s, Christian Borgs, Jennifer T. Chayes, and Oliver Riordan. Directed scale-free graphs. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms, January 12-14, 2003, Baltimore, Maryland, USA , pp.\ 132--139. ACM/SIAM , 2003. URL http://dl.acm.org/citation.cfm?id=644108.644133

    arXiv 2003

  3. [3]

    Navigating the labyrinth: Evaluating and enhancing llms' ability to reason about search problems

    Nasim Borazjanizadeh, Roei Herzig, Trevor Darrell, Rog \'e rio Feris, and Leonid Karlinsky. Navigating the labyrinth: Evaluating and enhancing llms' ability to reason about search problems. ArXiv, abs/2406.12172, 2024. URL https://api.semanticscholar.org/CorpusID:270562930

    arXiv 2024

  4. [4]

    A mechanistic analysis of a transformer trained on a symbolic multi-step reasoning task

    Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt. A mechanistic analysis of a transformer trained on a symbolic multi-step reasoning task. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, ...

    doi:10.18653/v1/2024.findings-acl.242 2024

  5. [5]

    Broken neural scaling laws

    Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=sckjveqlCZ

    2023

  6. [6]

    Selection-inference: Exploiting large language models for interpretable logical reasoning

    Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=3Pf3Wg6o-A4

    2023

  7. [7]

    MANGO : A benchmark for evaluating mapping and navigation abilities of large language models

    Peng Ding, Jiading Fang, Peng Li, Kangrui Wang, Xiaochen Zhou, Mo Yu, Jing Li, Hongyuan Mei, and Matthew Walter. MANGO : A benchmark for evaluating mapping and navigation abilities of large language models. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=6vEfyp0o68

    2024

  8. [8]

    Understanding emergent abilities of language models from the loss perspective

    Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. CoRR, abs/2403.15796, 2024. doi:10.48550/ARXIV.2403.15796. URL https://doi.org/10.48550/arXiv.2403.15796

    arXiv doi:10.48550/arxiv.2403.15796 2024

Show all 55 references
  1. [9]

    On random graphs i

    P Erd\"os and A R\'enyi. On random graphs i. Publicationes Mathematicae Debrecen, 6: 0 290--297, 1959

    1959

  2. [10]

    Nphardeval: Dynamic benchmark on reasoning ability of large language models via complexity classes

    Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling, and Yongfeng Zhang. Nphardeval: Dynamic benchmark on reasoning ability of large language models via complexity classes. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the As...

    2024 doi

  3. [11]

    Isobench: Benchmarking multimodal foundation models on isomorphic representations

    Deqing Fu, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. Isobench: Benchmarking multimodal foundation models on isomorphic representations. CoRR, abs/2404.01266, 2024. doi:10.48550/ARXIV.2404.01266. URL https://doi.org/10.485...

    2024 doi arXiv

  4. [12]

    Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D. Goodman. Stream of search (sos): Learning to search in language. CoRR, abs/2404.03683, 2024. doi:10.48550/ARXIV.2404.03683. URL https://doi.org/10.48550/arXiv.2404.03683

    2024 doi arXiv

  5. [13]

    Causal abstractions of neural networks

    Atticus Geiger, Hanson Lu, Thomas F Icard, and Christopher Potts. Causal abstractions of neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=RmuXDtjDhG

    2021

  6. [14]

    Reasoning with language model is planning with world model

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Langua...

    2023 doi

  7. [15]

    How to use and interpret activation patching

    Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching. ArXiv, abs/2404.15255, 2024. URL https://api.semanticscholar.org/CorpusID:269302704

    2024 arXiv

  8. [16]

    Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M

    Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish...

    2010 arXiv

  9. [17]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 doi arXiv

  10. [18]

    Towards a mechanistic interpretation of multi-step reasoning capabilities of language models

    Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, and Mrinmaya Sachan. Towards a mechanistic interpretation of multi-step reasoning capabilities of language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedin...

    2023 doi

  11. [19]

    Ivanitskiy, Alex F

    Michael I. Ivanitskiy, Alex F. Spies, Tilman R \" a uker, Guillaume Corlouer, Chris Mathwin, Lucia Quirke, Can Rager, Rusheb Shah, Dan Valentine, Cecilia G. Diniz Behn, Katsumi Inoue, and Samy Wu Fung. Linearly structured world representations in maze-solving transformers. In ...

    2023

  12. [20]

    Evidence of learned look-ahead in a chess-playing neural network

    Erik Jenner, Shreyas Kapur, Vasil Georgiev, Cameron Allen, Scott Emmons, and Stuart Russell. Evidence of learned look-ahead in a chess-playing neural network. CoRR, abs/2406.00877, 2024. doi:10.48550/ARXIV.2406.00877. URL https://doi.org/10.48550/arXiv.2406.00877

    2024 doi arXiv

  13. [21]

    Llms can't plan, but can help planning in llm-modulo frameworks

    Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Kaya Stechly, Mudit Verma, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. Llms can't plan, but can help planning in llm-modulo frameworks. CoRR, abs/2402.01817, 2024. doi:10.48550/ARXIV.2402.01817. URL https://doi.org/10.485...

    2024 doi arXiv

  14. [22]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020. URL https://arxiv.org/abs/2001.08361

    2001 arXiv

  15. [23]

    LAMBADA : Backward chaining for automated reasoning in natural language

    Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, and Deepak Ramachandran. LAMBADA : Backward chaining for automated reasoning in natural language. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Com...

    2023 doi

  16. [24]

    A mechanistic interpretation of syllogistic reasoning in auto-regressive language models

    Geonhee Kim, Marco Valentino, and Andr \'e Freitas. A mechanistic interpretation of syllogistic reasoning in auto-regressive language models. ArXiv, abs/2408.08590, 2024. URL https://api.semanticscholar.org/CorpusID:271892176

    2024 arXiv

  17. [25]

    COGS: A compositional generalization challenge based on semantic interpretation

    Najoung Kim and Tal Linzen. COGS: A compositional generalization challenge based on semantic interpretation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online...

    2020 doi

  18. [26]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual ...

    2022

  19. [27]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training

    Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . Open...

    2024

  20. [28]

    The expressive power of transformers with chain of thought

    William Merrill and Ashish Sabharwal. The expressive power of transformers with chain of thought. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=NjNGlPh8Wh

    2024

  21. [29]

    Maxwell I. Nye, Anders Johan Andreassen, Guy Gur - Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. C...

    2021 arXiv

  22. [30]

    Language models are unsupervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019

    2019

  23. [31]

    Lewis, Joel Veness, and Tim Genewein

    Anian Ruoss, Gr\'egoire Del\'etang, Sourabh Medapati, Jordi Grau-Moya, Li Kevin Wenliang, Elliot Catt, John Reid, Cannada A. Lewis, Joel Veness, and Tim Genewein. Amortized planning with large-scale transformers: A case study on chess. CoRR, abs/2402.04494, 2024. doi:10.48550/...

    2024 doi arXiv

  24. [32]

    Understanding transformer reasoning capabilities via graph algorithms

    Clayton Sanford, Bahare Fatemi, Ethan Hall, Anton Tsitsulin, Seyed Mehran Kazemi, Jonathan Halcrow, Bryan Perozzi, and Vahab Mirrokni. Understanding transformer reasoning capabilities via graph algorithms. CoRR, abs/2405.18512, 2024 a . doi:10.48550/ARXIV.2405.18512. URL https...

    2024 doi arXiv

  25. [33]

    Transformers, parallel computation, and logarithmic depth

    Clayton Sanford, Daniel Hsu, and Matus Telgarsky. Transformers, parallel computation, and logarithmic depth. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024 b . URL https://openreview.net/forum?id...

    2024

  26. [34]

    Language models are greedy reasoners: A systematic formal analysis of chain-of-thought

    Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. CoRR, abs/2210.01240, 2022. doi:10.48550/ARXIV.2210.01240. URL https://doi.org/10.48550/arXiv.2210.01240

    2022 doi arXiv

  27. [35]

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conferen...

    2023

  28. [36]

    Causal language modeling can elicit search and reasoning capabilities on logic puzzles

    Kulin Shah, Nishanth Dikkala, Xin Wang, and Rina Panigrahy. Causal language modeling can elicit search and reasoning capabilities on logic puzzles. CoRR, abs/2409.10502, 2024. doi:10.48550/ARXIV.2409.10502. URL https://doi.org/10.48550/arXiv.2409.10502

    2024 doi arXiv

  29. [37]

    Autoplanbench: Automatically generating benchmarks for llm planners from pddl

    Katharina Stein and Alexander Koller. Autoplanbench: Automatically generating benchmarks for llm planners from pddl. arXiv preprint arXiv:2311.09830, 2023

    2023 arXiv

  30. [38]

    A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis

    Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Na...

    2023 doi

  31. [39]

    Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024. doi:10.1016/J.NEUCOM.2023.127063. URL https://doi.org/10.1016/j.neucom.2023.127063

    2024

  32. [40]

    Large language models still can't plan (a benchmark for LLM s on planning and reasoning about change)

    Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for LLM s on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022. URL https://openreview.net...

    2022

  33. [41]

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. Investigating gender bias in language models using causal mediation analysis. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria - Florina Balcan, and Hsu...

    2020

  34. [42]

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. Causal mediation analysis for interpreting neural nlp: The case of gender bias. ArXiv, abs/2004.12265, 2020 b . URL https://api.semanticscholar.org/CorpusID:216553696

    2004 arXiv

  35. [43]

    Can language models solve graph problems in natural language? In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language? In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=UDqHhbqYJV

    2023

  36. [44]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. T...

    2022

  37. [45]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in N...

    2022

  38. [46]

    Do large language models latently perform multi-hop reasoning? In Annual Meeting of the Association for Computational Linguistics, 2024

    Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? In Annual Meeting of the Association for Computational Linguistics, 2024. URL https://api.semanticscholar.org/CorpusID:268032051

    2024

  39. [47]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  40. [48]

    Towards best practices of activation patching in language models: Metrics and methods

    Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/...

    2024

  41. [49]

    On the paradox of learning to reason from data

    Honghua Zhang, Liunian Harold Li, Tao Meng, Kai - Wei Chang, and Guy Van den Broeck. On the paradox of learning to reason from data. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, Ch...

    2023 doi

  42. [50]

    Can llm graph reasoning generalize beyond pattern memorization? ArXiv, abs/2406.15992, 2024

    Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xiaochuang Han, Tianxing He, and Yulia Tsvetkov. Can llm graph reasoning generalize beyond pattern memorization? ArXiv, abs/2406.15992, 2024. URL https://api.semanticscholar.org/CorpusID:270702523

    2024 arXiv

  43. [51]

    Transformers can achieve length generalization but not robustly

    Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, and Denny Zhou. Transformers can achieve length generalization but not robustly. CoRR, abs/2402.09371, 2024. doi:10.48550/ARXIV.2402.09371. URL https://doi.org/10.48550/arXiv.2402.09371

    2024 doi arXiv

  44. [52]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  45. [53]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  46. [54]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  47. [55]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools