Can a Large Language Model Learn Matrix Functions In Context?

Evangelos E. Papalexakis; Paimon Goulart

REVIEW 4 major objections 5 minor 15 references

Can a Large Language Model Learn Matrix Functions In Context?

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that large language models can learn nonlinear matrix functions in context, matching classical baselines on easy norms and beating them on nuclear-norm and top-k singular-value tasks.

desk verdict A plausible proof-of-concept that LLMs can approximate SVD-derived quantities in context, but missing controls and single-run stats make the 'learning' claim under-supported. read the letter →

arxiv 2411.15675 v1 pith:5TZW6CUK submitted 2024-11-24 cs.LG

Paimon Goulart , Evangelos E. Papalexakis This is my paper · ORCID

classification cs.LG

keywords in-contextlearninglargelanguagemodelsmatrixfunctionssingularvaluedecompositionnuclearnormtop-kvaluesnonlinearregressionnumericalcomputation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a large language model can learn nonlinear matrix functions purely from input-output examples shown in its prompt, without any fine-tuning. The authors test vector $p$-norms, the nuclear norm, and the top-3 singular values of random matrices, comparing the Gemini 1.5 Flash model against SGD linear regression, a two-layer neural network, and a CNN. Their central claim is that on the harder SVD-based tasks Gemini produces lower root mean square error (RMSE) than all classical baselines, needs fewer prior examples to converge, and avoids the overfitting that afflicts SGD at larger example counts. They also report that the advantage persists for 10x10 and 25x25 matrices and that open-weight models show the same in-context ability. If true, this would make in-context learning a plausible tool for approximate high-dimensional matrix computations where training data or compute is scarce.

What carries the argument

The load-bearing mechanism is a meta-in-context-learning prompt that frames each numeric task as a machine producing an output for a given input, with no instruction to compute a norm or a singular value. Because the prompt never discloses the task, any task-specific behavior must come from the demonstrated input-output pairs, and the output is scored by RMSE against ground-truth values. The target quantities share a common mathematical backbone through the singular value decomposition, which is exactly the nonlinear structure the paper claims in-context learning can approximate.

What would settle it

Rerun the same prompts on several independent random sequences of 5x5, 10x10, and 25x25 matrices, computing the distribution of final-average RMSE for each model; if Gemini's interval overlaps the CNN's or SGD's intervals in the top-3 singular-value task, the paper's main claim would fail.

Watch

Extended reading notes

Core claim

The discovery the authors argue for is that in-context learning can carry out nonlinear matrix functions, not just the linear regression previously associated with the technique. Using a fixed prompt that never names the task, the model is shown pairs where the input is a vector or matrix and the output is the target quantity, then asked to predict the output for a new input; the same prompt handles scalar and vector outputs. Across repeated incremental evaluations, Gemini's computed RMSE is lower than the SGD, two-layer NN, and CNN baselines for the nuclear norm and for the top-3 singular values, with average RMSE over the last 25 trials of 45.52 versus 60.64, 51.44, and 55.34 for the nuclear norm, and 21.79 versus 28.87, 23.58, and 24.37 for top-3 singular values. The authors interpret this as evidence that task complexity favors the LLM: simpler p-norms are roughly tied or slightly in favor of the neural network, while SVD-based tasks are won by Gemini consistently and without overfitting as the number of prior examples grows.

Load-bearing premise

The claim rests on the assumption that one randomly generated sequence of matrices, evaluated once per model, is enough to establish the reported performance ordering; if another equally likely sequence reshuffles the RMSE rankings, the central outperformance claim would not survive.

Editorial extensions

If this is right

Nonlinear numerical operations such as nuclear norms and top-k singular values can be approximated by LLMs at inference time, without training a model or running a dedicated SVD routine.
ICL reaches low error with fewer demonstrations, suggesting it may be useful in small-data regimes where classical regressors overfit.
Gemini's accuracy remains stable as matrix size grows to 25x25 while the neural-network and SGD baselines degrade, pointing to scalability of the prompting approach.
Open-weight models reproduce the effect, implying the capability is not exclusive to large proprietary systems and is open to further study.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The authors do not test this, but one extension is to repeat the protocol over several independent random matrix sequences and report error bars; the current single-run RMSE curves leave open the possibility that the Gemini-versus-baseline ordering is sequence-specific.
The apparent ability may not be general in-context regression; an alternative explanation is that pretraining exposed the model to linear-algebra patterns, so probing with matrices from other distributions, such as ill-conditioned or non-real matrices, would separate learned arithmetic from true ICL.
If the trend holds, prompting could become a cheap fallback for quick spectral estimates during prototyping, with deterministic SVD reserved for the final answer.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague, here's the short version: this is a legitimate proof-of-concept that LLMs can produce reasonable estimates of matrix functions (nuclear norm, top-k singular values) from a handful of demonstrations, and the prompt design is clean. But the paper's central claim—that the LLM is *learning* these functions in context—is under-supported because there is no zero-shot or shuffled-label control, and the comparative evaluation is based on a single random sequence with no error bars.

What's actually new: the extension of the meta-in-context-learning prompt from Coda-Forno et al. to SVD-based quantities is straightforward but previously untested. The authors also check open models (Qwen, Hermes) and compare against CNN/SGD/NN baselines, and they release code. Those are real contributions. Citation pattern is fair: the only self-citation is the SV-learn baseline, which is appropriate.

The soft spots are non-trivial. First, the lack of controls: Gemini has almost certainly seen matrices, norms, and SVD during pretraining. A model that ignores the demonstrations and emits a distribution-appropriate guess would produce a low, flat RMSE that plateaus quickly. Without a condition where the demonstrations carry information the model doesn't already have, or a shuffled-label control, you cannot attribute the performance to in-context learning. The prompt tells the model to 'try to find a pattern', but that's not evidence it uses the examples. So the stress-test note is on target.

Second, the evaluation is statistically thin. The text says N=50 for all experiments, but Figures 3 and 5 plot up to 100 prior examples; Figure 7 shows 50. The authors regenerate queries until the output format matches, which can introduce selection bias. There are no repeated runs, seeds, confidence intervals, or archived data. The 'Gemini consistently outperforms' claims are essentially single-trajectory observations. That doesn't make the result false; it makes it unverified.

The paper is honest about being a proof-of-concept, and I believe that framing is appropriate. I'd send it to peer review because the question is worth asking and a good referee could push the authors to add the needed controls. But in its current form, the evidence supports 'LLMs can guess these functions well', not 'LLMs learn these functions from the provided examples.' A serious revision should fix the statistics and add at least one control.

Recommendation: give it a chance at a workshop or a tolerant venue, but only after major revision. If it comes back with controls and repeated trials, the finding could be solid.

Referee Report

4 major / 5 minor

Summary. The paper presents an empirical study of whether large language models can perform nonlinear numerical regression in context, using p-norms of vectors, the nuclear norm of matrices, and the top-3 singular values. The experimental protocol builds prompts that show input-output examples without naming the target operation, and compares Gemini 1.5 Flash, Qwen2.5-72B, and Hermes-3-Llama-3.1-8B against SGD linear regression, a 2-layer neural network, and a CNN. The paper reports that Gemini is comparable on p-norms, outperforms the classical baselines on nuclear norm and top-3 singular values, needs fewer examples, and scales better to 10x10 and 25x25 matrices. The conclusions are framed as preliminary proof-of-concept results.

Significance. If the reported comparisons were statistically solid, the paper would be a useful demonstration that a generic ICL prompt can approximate nonlinear matrix functionals without task-specific training. The strengths include the public code repository, the use of external non-LLM baselines, and the inclusion of open-weight models. However, the central comparative claim currently rests on single-run RMSE curves, inconsistent trial counts, and no controls for pretraining knowledge; as a result the significance is conditional on additional experimentation rather than established by the present evidence.

major comments (4)

[Section III; Figures 3, 5, 7] The experimental section states that 'we used 50 examples in total (N = 50)', yet Figure 5 plots RMSE over 100 prior examples and Figure 7 plots 50 trials, with tables reporting 'last 25 trials'. This internal inconsistency affects the reproducibility of every quantitative claim. More importantly, all RMSE curves are single trajectories from one randomly generated sequence; there are no repeated seeds, resampled sequences, or confidence intervals. Consequently, the frequent statements that Gemini 'consistently outperforms' the baselines (Sections IV-B and IV-C) or that classical models 'deteriorate' (Section IV-D) cannot be distinguished from noise in a single sequence. The central comparative claim needs repeated trials and an explicit definition of the reported error bars or intervals.
[Section III, Figure 1 prompt] The prompt explicitly instructs the model to 'try to find a pattern', but the design never tests whether the demonstrations are actually driving performance. There is no zero-shot control (same prompt with no examples), no shuffled-label control, and no task where the target mapping is deliberately unlike anything in pretraining. For random matrices with entries in [-100,100], the singular-value spectrum is fairly concentrated and predictable, so a model emitting a sensible prior guess could produce exactly the reported pattern: moderate RMSE, quick plateau after roughly 20 examples, and no overfitting. Without these controls, the conclusion that LLMs 'learn' matrix functions in context rather than rely on prior statistics is not established.
[Section III, response regeneration] The protocol says that if the LLM output did not adhere to the expected format, 'the query was regenerated until the output format was correct'. Since only one accepted response is used per data point, this is a selection procedure, and its effect on RMSE is unreported. The manuscript also does not state the sampling temperature or how many regenerations occurred. To make the comparison fair and reproducible, the authors should sample multiple completions per prompt, report the dispersion, and describe or bound the effect of regeneration. This is load-bearing because the claimed superiority margins in Figure 5 are small (e.g., average RMSE of 45.5 vs 60.6 for the nuclear norm, and 21.8 vs 23.6 for top-3 singular values).
[Section IV-C; Figure 5(b)] The paper does not define the RMSE used for the vector-valued top-3 singular value predictions; it is unclear whether RMSE is averaged over the three components, summed, or computed as the Euclidean norm of the vector difference. This matters because the numerical comparison between models depends on the metric definition. The definition should be stated once, in Section III, and applied consistently in the tables.

minor comments (5)

[Figure captions] Subfigures in Figures 2 and 6 are referred to as 'Figures 1a-1c' and '1a-1c'; these labels do not match the parent figure numbers.
[Figure 6, middle panel] The legend contains the entry '2 Layer Gemini', which appears to be a typo for '2 Layer NN'.
[Figures 2, 4, and 6] The phrase '3 randomly selected experiments' is vague; the selection method should be specified, and since only one run exists, the selection is not random across seeds.
[Figure 4 caption] The caption says 'Average predictions compared to average actual values', but it is not clear over what the averaging is performed; the text should clarify.
[Reference [9]] The baseline section cites SV-learn as reference [9], but the relation between the 2-layer NN baseline and the SV-learn architecture is not described; if the baseline is not SV-learn, the citation is unnecessary or should be explained.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the LLM-vs-baseline comparison is empirically self-contained, with only a non-load-bearing self-citation to the authors' prior SV-learn work.

full rationale

This is an empirical comparison study, not a derivation. The LLM predictions are scored against NumPy-computed ground-truth values, and the classical baselines (SGD linear regression, 2-layer NN, CNN) are trained on the same demonstrations and tested on the same held-out matrices. No fitted parameter is relabeled as a prediction, and no target quantity is defined in terms of another predicted quantity, so the reported RMSE comparisons do not reduce to the experimental inputs by construction. The only apparent self-citation is reference [9] (SV-learn, co-authored by Papalexakis), which is used to name the 2-layer neural network baseline; that citation is not load-bearing because the NN baseline is a standard architecture and the paper's central LLM-versus-baseline comparison does not depend on any claim imported from [9]. The skeptical concern that LLM performance may reflect pretrained knowledge of singular-value statistics rather than in-context learning is a control-condition or validity issue, not a circularity, because it does not exhibit any equation, fitted parameter, or self-citation chain that makes the result equivalent to its inputs. Similarly, the use of a single randomly generated sequence without repeated seeds is a statistical robustness limitation, not a circular step. Therefore, no concrete circular step is identified in the paper's own derivation chain.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No new entities or fitted parameters are introduced; the paper is an empirical comparison. The listed axioms are the key unverified assumptions that the central claims depend on.

assumptions (3)

domain assumption The random generation of matrices from [-100, 100] is representative of the problem distribution.
Section III uses this distribution for all inputs. If the distribution is narrow or atypical, the reported RMSE and performance ordering may not generalize to other matrix distributions.
domain assumption The LLM solves the task by learning from the provided demonstrations rather than by pattern-matching or memorization.
The paper's interpretation rests on the model actually performing in-context regression. No control experiments, such as shuffled labels or random baselines, are reported to rule out alternative explanations.
ad hoc to paper A single run of the random sequence is sufficient to compare models.
Section III evaluates each model once on one random sequence. The paper implicitly assumes the resulting RMSE curves are stable, which is load-bearing for the 'outperform' claims.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Can a Large Language Model Learn Matrix Functions In Context?." pith.science (2026). https://pith.science/paper/5TZW6CUK

@misc{pith2026241115675,
  author       = {Pith},
  title        = {Pith review of: Can a Large Language Model Learn Matrix Functions In Context?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5TZW6CUK}},
  note         = {Machine review of arXiv:2411.15675}
}

read the original abstract

Large Language Models (LLMs) have demonstrated the ability to solve complex tasks through In-Context Learning (ICL), where models learn from a few input-output pairs without explicit fine-tuning. In this paper, we explore the capacity of LLMs to solve non-linear numerical computations, with specific emphasis on functions of the Singular Value Decomposition. Our experiments show that while LLMs perform comparably to traditional models such as Stochastic Gradient Descent (SGD) based Linear Regression and Neural Networks (NN) for simpler tasks, they outperform these models on more complex tasks, particularly in the case of top-k Singular Values. Furthermore, LLMs demonstrate strong scalability, maintaining high accuracy even as the matrix size increases. Additionally, we found that LLMs can achieve high accuracy with minimal prior examples, converging quickly and avoiding the overfitting seen in classical models. These results suggest that LLMs could provide an efficient alternative to classical methods for solving high-dimensional problems. Future work will focus on extending these findings to larger matrices and more complex matrix operations while exploring the effect of using different numerical representations in ICL.

Figures

Figures reproduced from arXiv: 2411.15675 by the authors.

**Figure 2.** Average predictions compared to average actual values for the vector norm learning task. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** RMSE of the vector norm predictions for every model in the vector norm learning task. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Average predictions compared to average actual values for the nuclear norm learning task at randomly selected experiments for 5x5 matrix inputs. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: RMSE for the nuclear norm and SVD learning tasks when given 5x5 matrices as inputs. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: Average predictions compared to average actual values for the SVD learning task. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 7.** Figure 7: RMSE for SVD learning tasks when given 10x10 and 25x25 matrices as inputs. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

**Figure 8.** Figure 8: RMSE of the top-3 singular values for Open Source Models in SVD learning task on 5x5 matrices over all 50 trials. [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 7 canonical work pages

[1]

Rethinking the role of demonstrations: What makes in-context learning work?,

S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?,” arXiv preprint arXiv:2202.12837 , 2022

arXiv 2022
[2]

Chain-of-thought prompting elicits reasoning in large language models,

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou, et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24824–24837, 2022

2022
[3]

Meta-in-context learning in large language models,

J. Coda-Forno, M. Binz, Z. Akata, M. Botvinick, J. Wang, and E. Schulz, “Meta-in-context learning in large language models,” Advances in Neural Information Processing Systems , vol. 36, pp. 65189–65201, 2023

work page 2023
[4]

Talk like a graph: Encoding graphs for large language models,

B. Fatemi, J. Halcrow, and B. Perozzi, “Talk like a graph: Encoding graphs for large language models,” arXiv preprint arXiv:2310.04560 , 2023

arXiv 2023
[5]

Tempo- ral knowledge graph forecasting without knowledge using in-context learning,

D.-H. Lee, K. Ahrabian, W. Jin, F. Morstatter, and J. Pujara, “Tempo- ral knowledge graph forecasting without knowledge using in-context learning,” arXiv preprint arXiv:2305.10613 , 2023

arXiv 2023
[6]

G. H. Golub and C. F. Van Loan, Matrix Computations . Baltimore, MD: Johns Hopkins University Press, 4th ed., 2013

work page 2013
[7]

R. A. Horn and C. R. Johnson, Matrix Analysis . New York: Cambridge University Press, 1985

work page 1985
[8]

Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization,

B. Recht, M. Fazel, and P. A. Parrilo, “Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization,” SIAM Review, vol. 52, no. 3, pp. 471–501, 2010

work page 2010

Show all 15 references

[9]

Sv-learn: Learning matrix singular values with neural networks,

D. Xu, W. Shiao, J. Chen, and E. E. Papalexakis, “Sv-learn: Learning matrix singular values with neural networks,” in 2022 IEEE International Conference on Data Mining Workshops (ICDMW) , pp. 232–237, IEEE, 2022

2022
[10]

Array programming with NumPy,

C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T....

2020
[11]

Random search for hyper-parameter optimiza- tion,

J. Bergstra and Y . Bengio, “Random search for hyper-parameter optimiza- tion,” Journal of Machine Learning Research , vol. 13, no. 2, pp. 281–305, 2012

2012
[12]

Gemini: a family of highly capable multimodal models,

Gemini, R. Anil, S. Borgeaud, Y . Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. , “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805 , 2023

2023 arXiv
[13]

Qwen2 technical report,

A. Yang et al. , “Qwen2 technical report,” arXiv preprint arXiv:2407.10671, 2024

2024 arXiv
[14]

Qwen2.5: A party of foundation models,

Q. Team, “Qwen2.5: A party of foundation models,” September 2024

2024
[15]

Hermes 3 technical report,

R. Teknium, J. Quesnelle, and C. Guang, “Hermes 3 technical report,” 2024

2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Rethinking the role of demonstrations: What makes in-context learning work?,

S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?,” arXiv preprint arXiv:2202.12837 , 2022

arXiv 2022

[2] [2]

Chain-of-thought prompting elicits reasoning in large language models,

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou, et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24824–24837, 2022

2022

[3] [3]

Meta-in-context learning in large language models,

J. Coda-Forno, M. Binz, Z. Akata, M. Botvinick, J. Wang, and E. Schulz, “Meta-in-context learning in large language models,” Advances in Neural Information Processing Systems , vol. 36, pp. 65189–65201, 2023

work page 2023

[4] [4]

Talk like a graph: Encoding graphs for large language models,

B. Fatemi, J. Halcrow, and B. Perozzi, “Talk like a graph: Encoding graphs for large language models,” arXiv preprint arXiv:2310.04560 , 2023

arXiv 2023

[5] [5]

Tempo- ral knowledge graph forecasting without knowledge using in-context learning,

D.-H. Lee, K. Ahrabian, W. Jin, F. Morstatter, and J. Pujara, “Tempo- ral knowledge graph forecasting without knowledge using in-context learning,” arXiv preprint arXiv:2305.10613 , 2023

arXiv 2023

[6] [6]

G. H. Golub and C. F. Van Loan, Matrix Computations . Baltimore, MD: Johns Hopkins University Press, 4th ed., 2013

work page 2013

[7] [7]

R. A. Horn and C. R. Johnson, Matrix Analysis . New York: Cambridge University Press, 1985

work page 1985

[8] [8]

Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization,

B. Recht, M. Fazel, and P. A. Parrilo, “Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization,” SIAM Review, vol. 52, no. 3, pp. 471–501, 2010

work page 2010