Pith. sign in

REVIEW 4 major objections 5 minor 66 references

Procedural knowledge in LLMs is not low-rank, so LoRA fails to match full fine-tuning on multi-step procedures.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 13:11 UTC pith:2A3SG3QR

load-bearing objection A careful empirical demonstration that uniform-rank LoRA underperforms full FT on procedural tasks, with an overreaching title and a mechanistic story the evidence doesn't quite support. the 4 major comments →

arxiv 2607.21612 v1 pith:2A3SG3QR submitted 2026-05-23 cs.AI cs.LG

Procedural Knowledge Is Not Low-Rank: Why LoRA Fails to Internalize Multi-Step Procedures

Simon Dennis , Kevin Shabahang , Hao Guo , Rivaan Patil This is my paper
classification cs.AI cs.LG
keywords LoRAlow-rank adaptationprocedural knowledgemulti-step proceduresagentic taskseffective ranksingular value decompositionparameter-efficient fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that procedural knowledge—the ability to follow multi-step procedures with conditional branching and terminal states—is distributed across the full parameter space of a language model in a way that low-rank approximations cannot capture at practical ranks. Across three procedural domains (travel booking, Zoom support, insurance claims) and two model sizes (3B and 8B), LoRA at ranks 16–128 scores far below full fine-tuning on task success and other behavioral criteria, even though conversation completion rates stay above 95%. The paper explains the failure by analyzing the weight updates produced by full fine-tuning: they have mean effective rank 761–1,026, and a rank-128 approximation captures only 43–51% of the update energy—an upper bound set by the Eckart–Young theorem. If true, this means parameter-efficient fine-tuning cannot substitute for full fine-tuning in agentic applications where reliable procedure-following matters.

Core claim

LoRA, the standard parameter-efficient fine-tuning method, fails to internalize procedural knowledge across three domains and two model sizes. Even at rank 128, which modifies 3.4% of all parameters, LoRA reaches only about half of full fine-tuning's task-success score on a 14-node travel procedure, and in the most complex 55-node insurance procedure its task success is 2.10 versus 4.47 for full fine-tuning. The paper's central structural claim is that procedural knowledge is high-rank: singular value decomposition of the full-fine-tuning weight updates shows mean effective rank 761–1,026, with rank 128 capturing only 43–51% of the squared Frobenius norm. Because Eckart–Young guarantees this

What carries the argument

The central analytical tool is the effective rank of weight updates, combined with the Eckart–Young theorem. The paper computes ΔW = W_full − W_base for every weight matrix, decomposes it via singular value decomposition, and measures the fraction of squared Frobenius norm captured by the top r singular vectors. This fraction is an upper bound on what any rank-r adapter could express of that specific update. Layer-wise analysis identifies MLP projections as the bottleneck: their effective ranks exceed 1,300, so rank 128 covers only about 10% of the needed dimensions.

Load-bearing premise

The representational argument assumes that the weight change produced by full fine-tuning is the target that a low-rank method must approximate; if a different low-rank update with the same behavior exists, the SVD energy bound loses force, leaving only the empirical behavioral results.

What would settle it

Train a rank-128 or lower adapter, including a heterogeneous layer-aware variant, on the 55-node insurance-claims procedure with the same 8B base model and training data. If any such adapter achieves task success near full fine-tuning's ~4.5 (rather than LoRA's ~2.1), the structural claim that procedural knowledge cannot be captured at rank 128 would be refuted; a rank-32 adapter matching full fine-tuning on travel would similarly falsify the claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Full fine-tuning, not LoRA, is the appropriate method for adapting models to agentic tasks that require reliable multi-step procedure following.
  • Increasing LoRA rank from 16 to 128 provides no benefit and can degrade performance, so the efficiency advantage of LoRA does not extend to procedural knowledge.
  • The LoRA deficit cannot be explained by underfitting: LoRA achieves lower held-out token loss than full fine-tuning while behaving worse, showing it fits surface language without acquiring the state-to-action mapping.
  • Changes to learning rate, epochs, or initialization will not close the gap at these ranks, because the binding constraint is representational capacity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same SVD diagnostic could be applied to other task families (mathematical reasoning, long-horizon planning, code generation) to test whether high-rank update profiles are a general signature of procedural or multi-step competencies beyond customer-service workflows.
  • A testable extension of the paper's proposal is a per-layer rank allocation proportional to the effective rank of the corresponding full-fine-tuning update; such a heterogeneous LoRA could close part of the gap at a constant parameter count.
  • Because the paper uses one model family, replication on other base architectures (e.g., Llama or Mistral) would determine whether the high-rank profile is universal or specific to models with grouped-query attention and similar layer shapes.
  • The clean dissociation between conversation completion and procedural success suggests that form-based completion metrics are insufficient for evaluating agentic fine-tuning; task-success scoring should accompany perplexity in future PEFT comparisons.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper claims that procedural knowledge—learning multi-step, conditional-branching task procedures—cannot be captured by low-rank LoRA updates at practical ranks. Evidence includes a LoRA rank ablation (r = 16–128) on a travel-booking task at 3B, cross-domain replication on Zoom support and insurance claims at 8B, SVD analysis of the full fine-tuning weight differences ΔW, and training-dynamics/loss analysis. The behavioral results show a consistent, large LoRA deficit relative to full fine-tuning across domains, ranks, and two independent judges; the SVD results show that rank-128 captures only 43–51% of the squared Frobenius norm of ΔW. The paper concludes that procedural knowledge is high-rank and that LoRA's rank budget is structurally insufficient.

Significance. If the behavioral finding holds, it is practically valuable: it identifies a class of agentic tasks where standard LoRA's efficiency-accuracy tradeoff breaks down, with a clear rank ablation and cross-domain replication. The two-judge evaluation and the training-dynamics control are methodological strengths, and the SVD analysis provides a useful descriptive characterization of full-FT weight updates. However, the paper's structural conclusion—that procedural knowledge is 'not low-rank' and 'cannot be captured at practical ranks'—is not established by the evidence. LoRA is not required to reproduce the specific matrix ΔW; it is required to find a low-rank update that yields the same behavior. The limitations section itself concedes that only uniform-rank LoRA was tested and that heterogeneous-rank variants remain untested. The contribution should therefore be reframed as a robust empirical demonstration of uniform-rank LoRA's failure rather than a representational impossibility proof.

major comments (4)
  1. [§3.3, Table 3; Discussion, Limitations] The Eckart–Young argument bounds approximation of the particular full-FT weight difference ΔW, but the paper's central claim requires showing that no low-rank update can reproduce the procedural behavior. A different rank-128 update, not lying in the top-128 singular subspace of ΔW, could in principle implement the same state-to-action mapping. The abstract's 'cannot capture at practical ranks' and the title's 'not low-rank' therefore outrun the evidence. The limitations paragraph correctly says: 'The “LoRA fails” result should therefore be read as “uniform-rank LoRA at standard ranks fails.”' I ask the authors to either (a) reframe the title/abstract and the §3.3/§4 conclusions to the supported empirical claim, or (b) add an experiment that tests representational possibility, e.g., heterogeneous-rank LoRA/AdaLoRA or a deliberate search for a low-rank update that achieves high task succe
  2. [Table 3] The table reports 'effective rank' to three significant figures but never defines it. Without a definition (e.g., rank at 90% cumulative energy, participation ratio, or a stated singular-value threshold), the 'Eff. rank' and 'r=128/Eff.' columns are not reproducible. This matters because the SVD analysis is one of the two main pillars of the 'high-rank' claim. Please define the measure, state the threshold if any, and ideally report the full singular-value spectrum or a percentile band across the 252 matrices.
  3. [Tables 1–2; §3.1] All behavioral comparisons include full fine-tuning but no zero-shot base-model condition. The paper claims LoRA 'fails to internalize' the procedure and 'fails to add the procedural following,' which requires knowing what the base model already does. If the base model starts at a similar low task-success score, LoRA simply fails to improve it; if the base model already achieves partial success, the interpretation shifts. This is a cheap and important baseline to report. Without it, the practical conclusion ('LoRA cannot be used for agentic applications') is still supported, but the mechanistic interpretation is not.
  4. [Limitations; §5] The generalization claim is further limited by two acknowledged scope boundaries: all experiments use the Qwen model family, and heterogeneous-rank LoRA is untested. The Discussion nevertheless concludes that 'procedural knowledge is not low-rank' and that LoRA's failure is 'fundamental.' Given the paper's own Limitations list, the title and abstract should be toned down to the domains/base models tested. At minimum, the revision should make the conclusion's scope match the Limitations; ideally, a non-Qwen replication should be added.
minor comments (5)
  1. [Abstract; Table 1] The abstract says 'with scores decreasing at higher ranks,' but Table 1 shows Task Success 2.50 for r=16 and 2.54 for r=32—an increase, not a decrease. The text later correctly says 'decrease monotonically beyond r=32.' Adjust the abstract and Table 1 caption for accuracy.
  2. [Appendix B] The text says the average LoRA-vs-full-FT gap at r=32 is 1.7 for travel under Claude, but the values in Table 1 imply 1.94. Please check the arithmetic and state how the average is computed.
  3. [§2.3] The claim that 'the LoRA parameter count exceeds full fine-tuning's once r approaches min(m,n)/2' is imprecise for rectangular matrices. The exact threshold for one layer is d_in·d_out/(d_in + d_out), not min(d_in,d_out)/2. Recommend correcting this sentence.
  4. [Overall] No code, data, or evaluation-harness availability statement is included. Since the procedures and user-simulator conversations are synthetic, releasing the flowchart specifications and evaluation scripts would substantially aid reproducibility.
  5. [Appendix A] 'Zoom' is lowercased as 'zoom' in the sentence 'the smallest is on zoom'—trivial typo.

Circularity Check

0 steps flagged

No circular derivation: behavioral LoRA-vs-full-FT comparison is self-contained; SVD analysis is descriptive and uses an external theorem; companion-work self-citation is motivational only.

full rationale

The paper's core behavioral result is a direct comparison: the same base model and training data are used for LoRA and full fine-tuning (§2.3, Tables 1–2), with outcomes scored by an independent judge and replicated by a second judge (§3.2, Appendix B). No parameter is fitted to the conclusion and then re-predicted: there is no fitted-input-called-prediction step. The SVD analysis (§3.3) computes ΔW = W_FT − W_base and applies Eckart–Young, an external theorem, to bound the Frobenius energy captured by rank-r truncations; it is descriptive of the measured full-FT update, not constructed from LoRA's outputs. The paper itself concedes that LoRA optimizes a token-prediction loss rather than directly approximating ΔW, which is a validity caveat about the explanatory step, not a circularity: the behavioral failure is established independently. The only same-author citation (companion work [Dennis et al., 2026]) is motivational; the paper's own full-FT baseline supplies the load-bearing evidence. Limitations explicitly narrow the claim to uniform-rank LoRA and state that a heterogeneous-rank variant is untested, so the title's 'not low-rank' phrasing goes beyond the demonstrated scope, but that overreach is not a definitional equivalence or self-citation loop.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper's central empirical comparison depends on standard hyperparameters but not on fitted constants. The main hidden parameter is the effective-rank threshold used to compute Table 3. The SVD mechanism additionally assumes that the full-FT update is the right target for LoRA, which is not logically forced. No new theoretical entities are introduced.

free parameters (4)
  • LoRA rank r = 16, 32, 64, 128
    Experimental variable, not fitted; but the central comparison depends on choosing ranks that define the efficiency regime. No per-rank LR tuning was performed.
  • LoRA scaling α = 2r
    Chosen by standard practice; affects update magnitude and may explain rank degradation, not controlled.
  • Learning rates = 2e-4 LoRA, 2e-5 full FT
    Hand-picked; LoRA gets 10x LR. A different LoRA LR might change results; authors argue not from val loss.
  • Effective rank threshold = not stated
    The paper never defines how effective rank is computed (e.g., energy threshold); this is a hidden methodological parameter.
axioms (4)
  • standard math Eckart–Young theorem
    Used to upper-bound any rank-r adapter's captured energy; valid but not sufficient to prove behavioral equivalence.
  • ad hoc to paper The full-FT weight diff ΔW is the target LoRA must approximate
    LoRA can in principle find a different low-rank ΔW with same function; energy capture of full-FT ΔW is not a proven necessity. This is load-bearing for the SVD mechanism (§3.3).
  • domain assumption Three customer-service flowcharts represent procedural knowledge broadly
    Authors limit, but the title claims procedural knowledge in general; generalization to other procedural families is open.
  • domain assumption LLM judges (Claude/GPT) score procedural correctness validly
    Cross-checked with two judges, but both are LLMs; no human validation.

pith-pipeline@v1.3.0-alltime-deepseek · 11678 in / 10149 out tokens · 102425 ms · 2026-08-02T13:11:54.191818+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Procedural Knowledge Is Not Low-Rank: Why LoRA Fails to Internalize Multi-Step Procedures." pith.science (2026). https://pith.science/paper/2A3SG3QR

@misc{pith2026260721612,
  author       = {Pith},
  title        = {Pith review of: Procedural Knowledge Is Not Low-Rank: Why LoRA Fails to Internalize Multi-Step Procedures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2A3SG3QR}},
  note         = {Machine review of arXiv:2607.21612}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Parameter-efficient fine-tuning methods like LoRA have become the default for adapting large language models, succeeding across instruction following, style transfer, and factual adaptation. We show that for procedural knowledge--the ability to follow multi-step procedures with conditional branching through to terminal states--LoRA fails to match full fine-tuning at the ranks where it retains its efficiency advantage. In a systematic ablation (r = 16--128) on a procedural travel booking task (14 nodes), all LoRA configurations fail uniformly (task success <= 2.54 vs. 4.11 for full fine-tuning, all p < 0.001), with scores decreasing at higher ranks--despite maintaining 95--99% conversation completion rates. Cross-domain replication on Zoom support (14 nodes) and insurance claims (55 nodes) at 8B confirms the failure generalizes: LoRA underperforms full fine-tuning by 0.8--2.2 points on average at both r = 32 and r = 128, with the largest gap on the most complex procedure. Quadrupling rank from 32 to 128 provides marginal improvement but does not close the gap. SVD analysis of the weight changes produced by full fine-tuning explains why: across three domains at both 3B and 8B, the mean effective rank of the update ranges from 761 to 1,026, and rank 128 captures only 43--51% of the squared Frobenius norm. Together, these findings establish that for procedural tasks LoRA falls well short of full fine-tuning--a fundamental limitation for agentic applications.

Figures

Figures reproduced from arXiv: 2607.21612 by Hao Guo, Kevin Shabahang, Rivaan Patil, Simon Dennis.

Figure 1
Figure 1. Figure 1: Travel booking procedure (14 nodes, 3 decision hubs, 3 terminal states). The agent opens, [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Zoom support procedure (14 nodes, 3 decision hubs, 3 terminal states). The agent triages [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Insurance claims procedure (55 nodes, 6 decision hubs, 4 terminal states). Nearly [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 16 linked inside Pith

  1. [1]

    Compiling Agentic Workflows into

    Dennis, Simon and Patil, Rivaan and Shabahang, Kevin and Guo, Hao , journal=. Compiling Agentic Workflows into

  2. [2]

    arXiv preprint , year=

    In-Context Prompting Obsoletes Agent Orchestration for Procedural Tasks , author=. arXiv preprint , year=

  3. [3]

    Procedural Knowledge Is Not Low-Rank: Why

    Dennis, Simon and Shabahang, Kevin and Guo, Hao and Patil, Rivaan , journal=. Procedural Knowledge Is Not Low-Rank: Why

  4. [4]

    Efficient Memory Management for Large Language Model Serving with

    Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph and Zhang, Hao and Stoica, Ion , booktitle=. Efficient Memory Management for Large Language Model Serving with

  5. [5]

    arXiv preprint arXiv:2602.13692 , year=

    ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System , author=. arXiv preprint arXiv:2602.13692 , year=

    Pith/arXiv arXiv

  6. [6]

    Advances in Neural Information Processing Systems , volume=

    A Simple Language Model for Task-Oriented Dialogue , author=. Advances in Neural Information Processing Systems , volume=

  7. [9]

    arXiv preprint arXiv:2411.05451 , year=

    WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models , author=. arXiv preprint arXiv:2411.05451 , year=

    Pith/arXiv arXiv

  8. [10]

    Findings of ACL , year=

    AgentTuning: Enabling Generalized Agent Abilities for LLMs , author=. Findings of ACL , year=

  9. [11]

    ACL , year=

    Agent Lumos: Unified and Modular Training for Open-Source Language Agents , author=. ACL , year=

  10. [12]

    ACL , year=

    Rethinking Task-Oriented Dialogue Systems: From Complex Modularity to Zero-Shot Autonomous Agent , author=. ACL , year=

  11. [14]

    ICLR , year=

    AgentBench: Evaluating LLMs as Agents , author=. ICLR , year=

  12. [15]

    Zhu, Kunlun and Liu, Zijia and Li, Bingxuan and Tian, Muxin and Yang, Yingxuan and Zhang, Jiaxun and Han, Pengrui and Xie, Qipeng and Cui, Fuyang and Zhang, Weijia and Ma, Xiaoteng and Yu, Xiaodong and Ramesh, Gowtham and Wu, Jialian and Liu, Zicheng and Lu, Pan and Zou, James and You, Jiaxuan , journal=. Where

  13. [16]

    Gupta, Aayush , journal=

  14. [17]

    and Nadgir, Nitya and Narayanan, Arvind , journal=

    Kapoor, Sayash and Stroebl, Benedikt and Siegel, Zachary S. and Nadgir, Nitya and Narayanan, Arvind , journal=

  15. [18]

    An Empirical Study of Agent Developer Practices in

    Wang, Yanlin and Xu, Xinyi and Chen, Jiachi and Bi, Tingting and Gu, Wenchao and Zheng, Zibin , journal=. An Empirical Study of Agent Developer Practices in

  16. [19]

    Findings of ACL , year=

    Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes , author=. Findings of ACL , year=

  17. [20]

    Findings of ACL , year=

    Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models , author=. Findings of ACL , year=

  18. [21]

    EMNLP , year=

    Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs , author=. EMNLP , year=

  19. [22]

    arXiv preprint arXiv:2210.03629 , year=

    ReAct: Synergizing Reasoning and Acting in Language Models , author=. arXiv preprint arXiv:2210.03629 , year=

    Pith/arXiv arXiv

  20. [23]

    Advances in Neural Information Processing Systems , volume=

    Reflexion: Language Agents with Verbal Reinforcement Learning , author=. Advances in Neural Information Processing Systems , volume=

  21. [24]

    1987 , publisher=

    Intention, Plans, and Practical Reason , author=. 1987 , publisher=

    1987

  22. [25]

    Advances in Neural Information Processing Systems , volume=

    Training Language Models to Follow Instructions with Human Feedback , author=. Advances in Neural Information Processing Systems , volume=

  23. [26]

    arXiv preprint arXiv:2109.01652 , year=

    Finetuned Language Models Are Zero-Shot Learners , author=. arXiv preprint arXiv:2109.01652 , year=

    Pith/arXiv arXiv

  24. [28]

    Advances in Neural Information Processing Systems , volume=

    QLoRA: Efficient Finetuning of Quantized LLMs , author=. Advances in Neural Information Processing Systems , volume=

  25. [29]

    Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages=

    MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling , author=. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages=

    2018

  26. [30]

    arXiv preprint arXiv:2212.10560 , year=

    Self-Instruct: Aligning Language Models with Self-Generated Instructions , author=. arXiv preprint arXiv:2212.10560 , year=

    Pith/arXiv arXiv

  27. [31]

    2023 , note=

    Stanford Alpaca: An Instruction-following LLaMA Model , author=. 2023 , note=

    2023

  28. [32]

    arXiv preprint arXiv:2304.12244 , year=

    WizardLM: Empowering Large Language Models to Follow Complex Instructions , author=. arXiv preprint arXiv:2304.12244 , year=

    Pith/arXiv arXiv

  29. [33]

    Proceedings of the First International Conference on Multi-Agent Systems , pages=

    BDI Agents: From Theory to Practice , author=. Proceedings of the First International Conference on Multi-Agent Systems , pages=

  30. [34]

    arXiv preprint arXiv:2212.01681 , year=

    Language Models as Agent Models , author=. arXiv preprint arXiv:2212.01681 , year=

    Pith/arXiv arXiv

  31. [35]

    2024 , howpublished=

    LangGraph: Build Resilient Language Agents as Graphs , author=. 2024 , howpublished=

    2024

  32. [36]

    2024 , howpublished=

    CrewAI: Framework for Orchestrating Role-Playing AI Agents , author=. 2024 , howpublished=

    2024

  33. [37]

    2026 , howpublished=

    Agent Development Kit , author=. 2026 , howpublished=

    2026

  34. [38]

    2026 , howpublished=

    OpenAI Agents SDK , author=. 2026 , howpublished=

    2026

  35. [39]

    2026 , howpublished=

    Semantic Kernel: Multi-Agent Orchestration , author=. 2026 , howpublished=

    2026

  36. [40]

    2026 , howpublished=

    Strands Agents SDK , author=. 2026 , howpublished=

    2026

  37. [41]

    2026 , howpublished=

    LlamaIndex Workflows , author=. 2026 , howpublished=

    2026

  38. [42]

    Why Do Multi-Agent

    Cemri, Muhammed and Shi, Yue and Jeyakumar, Jayaganesh and Kislal, Oznur and Karypis, George and Srivastava, Akash , journal=. Why Do Multi-Agent

  39. [43]

    arXiv preprint arXiv:1503.02531 , year=

    Distilling the Knowledge in a Neural Network , author=. arXiv preprint arXiv:1503.02531 , year=

    Pith/arXiv arXiv

  40. [44]

    Advances in Neural Information Processing Systems , volume=

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , author=. Advances in Neural Information Processing Systems , volume=

  41. [45]

    arXiv preprint arXiv:2412.15115 , year=

    Qwen2.5 Technical Report , author=. arXiv preprint arXiv:2412.15115 , year=

    Pith/arXiv arXiv

  42. [46]

    arXiv preprint arXiv:2404.13076 , year=

    LLM Evaluators Recognize and Favor Their Own Generations , author=. arXiv preprint arXiv:2404.13076 , year=

    Pith/arXiv arXiv

  43. [47]

    Transactions of the Association for Computational Linguistics , year=

    Lost in the Middle: How Language Models Use Long Contexts , author=. Transactions of the Association for Computational Linguistics , year=

  44. [48]

    arXiv preprint arXiv:2512.08296 , year=

    Towards a Science of Scaling Agent Systems , author=. arXiv preprint arXiv:2512.08296 , year=

    Pith/arXiv arXiv

  45. [49]

    arXiv preprint arXiv:2601.04748 , year=

    When Single-Agent with Skills Replace Multi-Agent Systems and When They Fail , author=. arXiv preprint arXiv:2601.04748 , year=

    arXiv

  46. [50]

    arXiv preprint arXiv:2601.12307 , year=

    Rethinking the Value of Multi-Agent Workflow: A Strong Single Agent Baseline , author=. arXiv preprint arXiv:2601.12307 , year=

    arXiv

  47. [51]

    arXiv preprint arXiv:2307.09923 , year=

    Large Language Models can accomplish Business Process Management Tasks , author=. arXiv preprint arXiv:2307.09923 , year=

    Pith/arXiv arXiv

  48. [52]

    , journal=

    Schneider, Walter and Shiffrin, Richard M. , journal=. Controlled and Automatic Human Information Processing:. 1977 , publisher=

    1977

  49. [53]

    Patel, Krishna and Patel, Tirth and Vij, Mihir and Zhu, Yueqing and Jain, Siddharth and Franusich, Matthew and Liang, Yin and Liu, Xiao and Liu, Zhengyu and Athiwaratkun, Ben and Zou, Yanqi and Vishwanath, Shreyas and Basu, Arindam and Guan, Hui , journal=

  50. [54]

    arXiv preprint arXiv:2001.08361 , year=

    Scaling Laws for Neural Language Models , author=. arXiv preprint arXiv:2001.08361 , year=

    Pith/arXiv arXiv 2001

  51. [55]

    Advances in Neural Information Processing Systems , volume=

    Training Compute-Optimal Large Language Models , author=. Advances in Neural Information Processing Systems , volume=

  52. [56]

    Psychometrika , volume=

    The Approximation of One Matrix by Another of Lower Rank , author=. Psychometrika , volume=

  53. [57]

    Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling

    Pawe Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Ga s i \'c . Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016--5026, 2018

    2018

  54. [58]

    Fireact: Toward language agent fine-tuning

    Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915, 2023

    Pith/arXiv arXiv 2023

  55. [59]

    Agent-flan: Designing data and methods of effective agent tuning for large language models

    Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. Findings of ACL, 2024

    2024

  56. [60]

    Compiling agentic workflows into LLM weights: Near-frontier quality at two orders of magnitude less cost

    Simon Dennis, Rivaan Patil, Kevin Shabahang, and Hao Guo. Compiling agentic workflows into LLM weights: Near-frontier quality at two orders of magnitude less cost. arXiv preprint, 2026

    2026

  57. [61]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Arturo Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2023

    2023

  58. [62]

    The approximation of one matrix by another of lower rank

    Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1 0 (3): 0 211--218, 1936

    1936

  59. [63]

    A simple language model for task-oriented dialogue

    Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. A simple language model for task-oriented dialogue. Advances in Neural Information Processing Systems, 33, 2020

    2020

  60. [64]

    Procedural knowledge improves agentic llm workflows

    Victor Hsiao. Procedural knowledge improves agentic llm workflows. arXiv preprint arXiv:2511.07568, 2026

    arXiv 2026

  61. [65]

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. Findings of ACL, 2023

    2023

  62. [66]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    Pith/arXiv arXiv 2021

  63. [67]

    Simulating task-oriented dialogues with state transition graphs and large language models

    Chris Samarinas, Pattaramanee Promthaw, Atharva Nijasure, Hansi Zeng, Julian Killingback, and Hamed Zamani. Simulating task-oriented dialogues with state transition graphs and large language models. arXiv preprint arXiv:2404.14772, 2024

    Pith/arXiv arXiv 2024

  64. [68]

    Agent lumos: Unified and modular training for open-source language agents

    Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Agent lumos: Unified and modular training for open-source language agents. ACL, 2024

    2024

  65. [69]

    Agenttuning: Enabling generalized agent abilities for llms

    Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms. Findings of ACL, 2024

    2024

  66. [70]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2023

    2023