{"id":"01056737-1769-4c94-a87d-0c14dbf7476b","arxiv_id":"2411.15675","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In-context learning lets LLMs approximate p-norms, nuclear norms, and top singular values of matrices, outperforming SGD, 2-layer nets, and CNNs on the harder SVD tasks in the authors' small-scale tests.","lead":"Large language models can approximate matrix norms and top singular values from a handful of examples, sometimes beating small classical regressors in the authors' tests. The result is a proof-of-concept that in-context learning might handle nonlinear numerical tasks, but the experiments lack statistical rigor.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'outperform' result may reflect pretrained knowledge of singular-value statistics rather than in-context learning; no control condition distinguishes the two.","rationale":"The reader's conditional verdict identifies the single-run protocol as the weakest assumption. That is valid and important, but I think the more load-bearing weakness is interpretive: the experiments do not establish that the LLM is learning the target input-output mapping from the demonstrations. The models have substantial pretraining exposure to matrix norms and SVD. For the tested distributions (iid uniform entries in [-100,100]), the marginal distribution of singular values is concentrated and predictable, so a model that ignores the demonstrations and emits a sensible prior estimate would produce exactly the reported pattern: moderate RMSE, a quick plateau near 20 examples, no overfitting, and apparent robustness to matrix size. Section IV-C explicitly says performance 'plateaus after around 20 prior demonstrations'—for a genuine in-context regressor learning a nonlinear map from 50 examples, one would expect continued improvement rather than a flat prior-like floor. Without a zero-shot baseline or a shuffled-label control, the 'learn in context' claim is not adequately supported. The paper does provide a public code repository, which is helpful, but the missing control conditions are not covered by that code. In addition, the inconsistency between the stated N=50 and the 100-trial axes/captions in Figures 3 and 5, and the absence of variance estimates, undermine the 'outperform' comparisons even if the confound is resolved. These are addressable with additional experiments, so rejection is not warranted; the conditional stance is appropriate, with the pretraining/ICL control added as a primary requirement. I therefore leave the reader's verdict unchanged but partially disagree with the stated weakest assumption, since the stability issue is secondary to the confound between pretraining and in-context learning.","tokens_in":8412,"tokens_out":4814,"duration_ms":49014,"concrete_test":"Run the same 5x5 top-3 singular-value prompt under three conditions: (a) zero demonstrations, (b) demonstrations with ground-truth singular values randomly permuted across examples, and (c) demonstrations pairing matrices with outputs from an unrelated function (e.g., trace or random values), while asking for the top-3 singular values. Measure RMSE for each condition over at least 10 independently sampled matrix sequences and compare with the reported real-demonstration RMSE. If shuffled or unrelated labels produce the same RMSE as true demonstrations, the performance is not caused by in-context learning of the matrix function; if zero-shot equals few-shot, prior knowledge is the explanation. Separately, rerun the full experiment with 10 different random seeds/sequences and report mean ± standard deviation to test the stability of the claimed ordering.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLMs 'learn' nonlinear matrix functions from demonstrations and beat classical baselines on top-k singular values. But the design cannot separate in-context learning from knowledge already in the model. Gemini, Qwen, and Hermes have been pretrained on text involving matrices, norms, and SVD; for random 5x5/10x10/25x25 matrices with entries drawn uniformly from [-100,100], the distribution of singular values is fairly concentrated and predictable. A model that ignores the demonstrations and emits a reasonable prior guess would produce exactly the reported pattern: moderate RMSE, rapid plateau after ~20 examples, no overfitting, and apparent robustness to matrix size. The paper includes no zero-shot control, no shuffled-label control, and no control task where the correct mapping is a nonlinear function unlikely to be in pretraining. The prompt itself tells the model to 'try to find a pattern', so stable performance with few examples is also consistent with reliance on priors rather than on the provided input-output pairs. The conclusion that LLMs 'can learn matrix functions in context' requires evidence that predictions actually improve because of the demonstrations; that evidence is missing. Compounding this, the evaluation uses a single random sequence (Section III says N=50, while figures and tables show trials up to 100), with no repeated seeds or error bars, so the comparison against baselines is also statistically unanchored.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of whether large language models can perform nonlinear numerical regression in context, using p-norms of vectors, the nuclear norm of matrices, and the top-3 singular values. The experimental protocol builds prompts that show input-output examples without naming the target operation, and compares Gemini 1.5 Flash, Qwen2.5-72B, and Hermes-3-Llama-3.1-8B against SGD linear regression, a 2-layer neural network, and a CNN. The paper reports that Gemini is comparable on p-norms, outperforms the classical baselines on nuclear norm and top-3 singular values, needs fewer examples, and scales better to 10x10 and 25x25 matrices. The conclusions are framed as preliminary proof-of-concept results.","tokens_in":8813,"tokens_out":4696,"duration_ms":42462,"significance":"If the reported comparisons were statistically solid, the paper would be a useful demonstration that a generic ICL prompt can approximate nonlinear matrix functionals without task-specific training. The strengths include the public code repository, the use of external non-LLM baselines, and the inclusion of open-weight models. However, the central comparative claim currently rests on single-run RMSE curves, inconsistent trial counts, and no controls for pretraining knowledge; as a result the significance is conditional on additional experimentation rather than established by the present evidence.","major_comments":[{"comment":"The experimental section states that 'we used 50 examples in total (N = 50)', yet Figure 5 plots RMSE over 100 prior examples and Figure 7 plots 50 trials, with tables reporting 'last 25 trials'. This internal inconsistency affects the reproducibility of every quantitative claim. More importantly, all RMSE curves are single trajectories from one randomly generated sequence; there are no repeated seeds, resampled sequences, or confidence intervals. Consequently, the frequent statements that Gemini 'consistently outperforms' the baselines (Sections IV-B and IV-C) or that classical models 'deteriorate' (Section IV-D) cannot be distinguished from noise in a single sequence. The central comparative claim needs repeated trials and an explicit definition of the reported error bars or intervals.","section":"Section III; Figures 3, 5, 7"},{"comment":"The prompt explicitly instructs the model to 'try to find a pattern', but the design never tests whether the demonstrations are actually driving performance. There is no zero-shot control (same prompt with no examples), no shuffled-label control, and no task where the target mapping is deliberately unlike anything in pretraining. For random matrices with entries in [-100,100], the singular-value spectrum is fairly concentrated and predictable, so a model emitting a sensible prior guess could produce exactly the reported pattern: moderate RMSE, quick plateau after roughly 20 examples, and no overfitting. Without these controls, the conclusion that LLMs 'learn' matrix functions in context rather than rely on prior statistics is not established.","section":"Section III, Figure 1 prompt"},{"comment":"The protocol says that if the LLM output did not adhere to the expected format, 'the query was regenerated until the output format was correct'. Since only one accepted response is used per data point, this is a selection procedure, and its effect on RMSE is unreported. The manuscript also does not state the sampling temperature or how many regenerations occurred. To make the comparison fair and reproducible, the authors should sample multiple completions per prompt, report the dispersion, and describe or bound the effect of regeneration. This is load-bearing because the claimed superiority margins in Figure 5 are small (e.g., average RMSE of 45.5 vs 60.6 for the nuclear norm, and 21.8 vs 23.6 for top-3 singular values).","section":"Section III, response regeneration"},{"comment":"The paper does not define the RMSE used for the vector-valued top-3 singular value predictions; it is unclear whether RMSE is averaged over the three components, summed, or computed as the Euclidean norm of the vector difference. This matters because the numerical comparison between models depends on the metric definition. The definition should be stated once, in Section III, and applied consistently in the tables.","section":"Section IV-C; Figure 5(b)"}],"minor_comments":[{"comment":"Subfigures in Figures 2 and 6 are referred to as 'Figures 1a-1c' and '1a-1c'; these labels do not match the parent figure numbers.","section":"Figure captions"},{"comment":"The legend contains the entry '2 Layer Gemini', which appears to be a typo for '2 Layer NN'.","section":"Figure 6, middle panel"},{"comment":"The phrase '3 randomly selected experiments' is vague; the selection method should be specified, and since only one run exists, the selection is not random across seeds.","section":"Figures 2, 4, and 6"},{"comment":"The caption says 'Average predictions compared to average actual values', but it is not clear over what the averaging is performed; the text should clarify.","section":"Figure 4 caption"},{"comment":"The baseline section cites SV-learn as reference [9], but the relation between the 2-layer NN baseline and the SV-learn architecture is not described; if the baseline is not SV-learn, the citation is unnecessary or should be explained.","section":"Reference [9]"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as a preliminary workshop-level study rather than a fully supported journal submission. The central claims may survive additional experimentation, but the single-run protocol, missing controls, and inconsistent trial counts are load-bearing. I would advise asking for a revision that adds repeated trials, control conditions, and a precise definition of the evaluation metric before considering publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's the short version: this is a legitimate proof-of-concept that LLMs can produce reasonable estimates of matrix functions (nuclear norm, top-k singular values) from a handful of demonstrations, and the prompt design is clean. But the paper's central claim—that the LLM is *learning* these functions in context—is under-supported because there is no zero-shot or shuffled-label control, and the comparative evaluation is based on a single random sequence with no error bars.\n\nWhat's actually new: the extension of the meta-in-context-learning prompt from Coda-Forno et al. to SVD-based quantities is straightforward but previously untested. The authors also check open models (Qwen, Hermes) and compare against CNN/SGD/NN baselines, and they release code. Those are real contributions. Citation pattern is fair: the only self-citation is the SV-learn baseline, which is appropriate.\n\nThe soft spots are non-trivial. First, the lack of controls: Gemini has almost certainly seen matrices, norms, and SVD during pretraining. A model that ignores the demonstrations and emits a distribution-appropriate guess would produce a low, flat RMSE that plateaus quickly. Without a condition where the demonstrations carry information the model doesn't already have, or a shuffled-label control, you cannot attribute the performance to in-context learning. The prompt tells the model to 'try to find a pattern', but that's not evidence it uses the examples. So the stress-test note is on target.\n\nSecond, the evaluation is statistically thin. The text says N=50 for all experiments, but Figures 3 and 5 plot up to 100 prior examples; Figure 7 shows 50. The authors regenerate queries until the output format matches, which can introduce selection bias. There are no repeated runs, seeds, confidence intervals, or archived data. The 'Gemini consistently outperforms' claims are essentially single-trajectory observations. That doesn't make the result false; it makes it unverified.\n\nThe paper is honest about being a proof-of-concept, and I believe that framing is appropriate. I'd send it to peer review because the question is worth asking and a good referee could push the authors to add the needed controls. But in its current form, the evidence supports 'LLMs can guess these functions well', not 'LLMs learn these functions from the provided examples.' A serious revision should fix the statistics and add at least one control.\n\nRecommendation: give it a chance at a workshop or a tolerant venue, but only after major revision. If it comes back with controls and repeated trials, the finding could be solid.","headline":"A plausible proof-of-concept that LLMs can approximate SVD-derived quantities in context, but missing controls and single-run stats make the 'learning' claim under-supported.","tokens_in":9186,"tokens_out":4264,"would_cite":false,"duration_ms":37615,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that large language models can learn nonlinear matrix functions in context, matching classical baselines on easy norms and beating them on nuclear-norm and top-k singular-value tasks.","keywords":["in-context learning","large language models","matrix functions","singular value decomposition","nuclear norm","top-k singular values","nonlinear regression","numerical computation"],"falsifier":"Rerun the same prompts on several independent random sequences of 5x5, 10x10, and 25x25 matrices, computing the distribution of final-average RMSE for each model; if Gemini's interval overlaps the CNN's or SGD's intervals in the top-3 singular-value task, the paper's main claim would fail.","tokens_in":8229,"feed_emoji":"🧮","tokens_out":7534,"duration_ms":64311,"temperature":0.7,"pith_summary":"The paper asks whether a large language model can learn nonlinear matrix functions purely from input-output examples shown in its prompt, without any fine-tuning. The authors test vector $p$-norms, the nuclear norm, and the top-3 singular values of random matrices, comparing the Gemini 1.5 Flash model against SGD linear regression, a two-layer neural network, and a CNN. Their central claim is that on the harder SVD-based tasks Gemini produces lower root mean square error (RMSE) than all classical baselines, needs fewer prior examples to converge, and avoids the overfitting that afflicts SGD at larger example counts. They also report that the advantage persists for 10x10 and 25x25 matrices and that open-weight models show the same in-context ability. If true, this would make in-context learning a plausible tool for approximate high-dimensional matrix computations where training data or compute is scarce.","feed_headline":"LLMs learn top-k singular values in context, beating classical models","feed_subtitle":"With a few examples and no fine-tuning, Gemini beats SGD, NN, and CNN on nuclear norm and SVD tasks.","key_machinery":"The load-bearing mechanism is a meta-in-context-learning prompt that frames each numeric task as a machine producing an output for a given input, with no instruction to compute a norm or a singular value. Because the prompt never discloses the task, any task-specific behavior must come from the demonstrated input-output pairs, and the output is scored by RMSE against ground-truth values. The target quantities share a common mathematical backbone through the singular value decomposition, which is exactly the nonlinear structure the paper claims in-context learning can approximate.","core_discovery":"The discovery the authors argue for is that in-context learning can carry out nonlinear matrix functions, not just the linear regression previously associated with the technique. Using a fixed prompt that never names the task, the model is shown pairs where the input is a vector or matrix and the output is the target quantity, then asked to predict the output for a new input; the same prompt handles scalar and vector outputs. Across repeated incremental evaluations, Gemini's computed RMSE is lower than the SGD, two-layer NN, and CNN baselines for the nuclear norm and for the top-3 singular values, with average RMSE over the last 25 trials of 45.52 versus 60.64, 51.44, and 55.34 for the nuclear norm, and 21.79 versus 28.87, 23.58, and 24.37 for top-3 singular values. The authors interpret this as evidence that task complexity favors the LLM: simpler p-norms are roughly tied or slightly in favor of the neural network, while SVD-based tasks are won by Gemini consistently and without overfitting as the number of prior examples grows.","pith_inferences":["The authors do not test this, but one extension is to repeat the protocol over several independent random matrix sequences and report error bars; the current single-run RMSE curves leave open the possibility that the Gemini-versus-baseline ordering is sequence-specific.","The apparent ability may not be general in-context regression; an alternative explanation is that pretraining exposed the model to linear-algebra patterns, so probing with matrices from other distributions, such as ill-conditioned or non-real matrices, would separate learned arithmetic from true ICL.","If the trend holds, prompting could become a cheap fallback for quick spectral estimates during prototyping, with deterministic SVD reserved for the final answer."],"forward_implications":["Nonlinear numerical operations such as nuclear norms and top-k singular values can be approximated by LLMs at inference time, without training a model or running a dedicated SVD routine.","ICL reaches low error with fewer demonstrations, suggesting it may be useful in small-data regimes where classical regressors overfit.","Gemini's accuracy remains stable as matrix size grows to 25x25 while the neural-network and SGD baselines degrade, pointing to scalability of the prompting approach.","Open-weight models reproduce the effect, implying the capability is not exclusive to large proprietary systems and is open to further study."],"supporting_citations":[{"why":"Establishes that demonstrations alone can drive in-context learning, the premise the paper builds on.","marker":"[1]"},{"why":"Supplies the meta-in-context prompt design and the prior linear-regression result this paper extends to nonlinear matrix functions.","marker":"[3]"},{"why":"Defines the SVD and singular values, the target quantities whose deterministic cost motivates the search for faster approximations.","marker":"[6]"},{"why":"Provides a neural-network method for learning matrix singular values that informs the two-layer NN baseline.","marker":"[9]"},{"why":"Supplies the numerical routines used to compute ground-truth norms and singular values for RMSE evaluation.","marker":"[10]"},{"why":"Grid search selects baseline hyperparameters, supporting the claim that classical models were reasonably tuned.","marker":"[11]"},{"why":"Identifies the main LLM, Gemini 1.5 Flash, whose in-context predictions are the subject of the experiments.","marker":"[12]"},{"why":"Provides the Qwen2.5 open-weight model used in the open-source comparison.","marker":"[13]"},{"why":"Provides the Hermes-3-Llama open-weight model used in the open-source comparison.","marker":"[15]"}],"fun_headline_variants":["In-context LLMs beat classical models on top-k singular values","LLM in-context learning handles nonlinear matrix functions, edges out SGD","Without fine-tuning, LLMs match and beat neural nets on SVD tasks","Gemini outperforms SGD and NNs on SVD via few-shot in-context learning","LLMs learn matrix functions in context: strong on SVD, scales with size"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the assumption that one randomly generated sequence of matrices, evaluated once per model, is enough to establish the reported performance ordering; if another equally likely sequence reshuffles the RMSE rankings, the central outperformance claim would not survive.","fun_headline_variants_meta":{"raw":{"variants":["In-context LLMs beat classical models on top-k singular values","LLM in-context learning handles nonlinear matrix functions, edges out SGD","Without fine-tuning, LLMs match and beat neural nets on SVD tasks","Gemini outperforms SGD and NNs on SVD via few-shot in-context learning","LLMs learn matrix functions in context: strong on SVD, scales with size"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000853,"raw_usage":{"total_tokens":3718,"prompt_tokens":964,"completion_tokens":2754,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":2652}},"tokens_in":580,"tokens_out":2754,"duration_ms":18666,"temperature":1.0,"reasoning_tokens":2652,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:00:35.213257+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the same prompts on several independent random sequences of 5x5, 10x10, and 25x25 matrices, computing the distribution of final-average RMSE for each model; if Gemini's interval overlaps the CNN's or SGD's intervals in the top-3 singular-value task, the paper's main claim would fail.","supporting_citations":[{"cited_title":"Meta-in-context learning in large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the meta-in-context prompt design and the prior linear-regression result this paper extends to nonlinear matrix functions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the SVD and singular values, the target quantities whose deterministic cost motivates the search for faster approximations."},{"cited_title":"Sv-learn: Learning matrix singular values with neural networks,","cited_arxiv_id":null,"evidence_quote":"Provides a neural-network method for learning matrix singular values that informs the two-layer NN baseline."},{"cited_title":"Array programming with NumPy,","cited_arxiv_id":null,"evidence_quote":"Supplies the numerical routines used to compute ground-truth norms and singular values for RMSE evaluation."},{"cited_title":"Random search for hyper-parameter optimiza- tion,","cited_arxiv_id":null,"evidence_quote":"Grid search selects baseline hyperparameters, supporting the claim that classical models were reasonably tuned."}],"review_version":1}