pith. sign in

arxiv: 2607.00329 · v1 · pith:POABO2DMnew · submitted 2026-07-01 · 💻 cs.LG · cs.AI

K-Inverse-RFM: A Modified RFM that Bridges the Gap to Neural Networks for Data-Corrupted Mathematical Tasks

Gil Pasternak This is my paper

Pith reviewed 2026-07-02 16:04 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords Recursive Feature MachinesK-Inverse transformationlabel transformationneural networksdata corruptionfeature learningmachine learning
0
0 comments X

The pith

A K-Inverse label transformation enables Recursive Feature Machines to match or exceed neural network performance on corrupted mathematical data.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Recursive Feature Machines replicate neural network feature learning through the Average Gradient Outer Product but underperform in noisy or imbalanced settings. The paper proposes applying a K-Inverse transformation to the training labels as a fix. This adjustment allows RFMs to close the gap with feedforward neural networks and sometimes surpass them in data-corrupted mathematical tasks. Sympathetic readers would see this as evidence that a simple label change can make kernel methods competitive without extra complexity.

Core claim

Recursive Feature Machines (RFMs) that use the Average Gradient Outer Product (AGOP) for feature learning replicate the dynamics of Feedforward Neural Networks (FNNs) but show significantly lower performance in data-corrupted scenarios. Introducing the K-Inverse transformation on training labels promotes learning in noisy, complexly represented, and class-imbalanced data, enabling RFMs to close the performance gap with FNNs and in some cases even surpass them.

What carries the argument

The K-Inverse transformation, a modification applied to the training labels in Recursive Feature Machines to handle data corruption.

If this is right

  • RFMs with K-Inverse can achieve performance levels comparable to FNNs on mathematical tasks with noise or imbalance.
  • The transformation works without requiring additional mechanisms beyond the label change.
  • Modified RFMs can sometimes outperform standard neural networks in these settings.
  • Feature learning similarities between RFMs and FNNs become performance-equivalent after the adjustment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Label preprocessing may be a more general lever for improving kernel methods on real-world data.
  • Testing the K-Inverse approach on image or text datasets could reveal if the benefit extends beyond mathematical problems.
  • The performance gap between RFMs and FNNs might stem from how labels are represented rather than differences in feature extraction capacity.

Load-bearing premise

The K-Inverse transformation on labels will promote effective learning in noisy, complex, and imbalanced data without any other specified conditions or mechanisms.

What would settle it

Running RFMs with and without K-Inverse on a benchmark mathematical classification task with added noise and measuring if the transformed version fails to improve accuracy over the baseline RFM.

Figures

Figures reproduced from arXiv: 2607.00329 by Gil Pasternak.

Figure 3.1
Figure 3.1. Figure 3.1: RFM vs. neural network test set performances by training step for modular addition (top, p=61) and multiplication (bottom, p=61). . . . . . . . . . . . . . . . 10 [PITH_FULL_IMAGE:figures/full_fig_p006_3_1.png] view at source ↗
Figure 3.1
Figure 3.1. Figure 3.1: RFM vs. neural network test set performances by training step for modular addition (top, p=61) and multiplication (bottom, p=61). In our experiments, we attempt a plethora of variations upon these and other tasks in comparison of neural networks and RFMs. In this light, we uncover three interesting phenomena: 1. Neural networks perform far better than RFMs as label noise is introduced, scaling at a 10 [… view at source ↗
Figure 3.2
Figure 3.2. Figure 3.2: RFM vs. neural network test set Performances by training step for modular addition (left, p=61) and multiplication (right, p=61). What we observe in [PITH_FULL_IMAGE:figures/full_fig_p022_3_2.png] view at source ↗
Figure 3.3
Figure 3.3. Figure 3.3: Comparison of various methods for modular addition with noisy training data. Left: Plot Comparing the test set performance of RFMs with neural networks and Laplace kernel regression trained on neural network features at 16% / 32% Label Noise. The kernel regression trained on the neural features outperforms other methods, indicating RFMs primary issue in noisy settings is feature learning. Right: kernel r… view at source ↗
Figure 3.4
Figure 3.4. Figure 3.4: RFM vs. neural network performance on various reweightings of divisible-by-3 input pairs and labels. All “unweighted” elements are assigned a weight of 1. Top row: RFM vs. NN on modular addition and multiplication (p=61) for various input weights of pairs divisible by 3. As can be observed, any reweighting greatly damages RFM performance. Bottom Row: RFM vs. NN on modular addition and multiplication (p=6… view at source ↗
Figure 3.5
Figure 3.5. Figure 3.5: Ratio of RFM modular addition performance on divisible-by-3 inputs vs. not￾divisible-by-3 inputs as the weight of divisible-by-3 inputs in the training set increases. As the weight becomes much larger, the vast majority of what the RFM learns becomes predicated on inputs divisible by 3. Similarly to our previous section, we also conduct a test as to whether these phenomena can be accounted for by the fea… view at source ↗
Figure 3.6
Figure 3.6. Figure 3.6: 3.2.2 Systematic Exclusion While the imbalancing of inputs and outputs divisible by 3 sheds signficiant light on the disparity between Neural Networds and RFMs, it is a somewhat random form of imbalance. To see whether different forms of imbalances would behave differently, we opted to experiment with seven forms of systematic exclusions: Exclusion of pairs where operands had the same value (e.g. 15 [PI… view at source ↗
Figure 3.6
Figure 3.6. Figure 3.6: A comparison of methods on a modular multiplication task with p=61 and various weights for divisibile-by-3 inputs (input sampling with replacement). Simply incorporating random circulant features bridges a significant portion fo the gap, and training on first layer post-nonlinearity features exceeds neural network performance altogether. (2,2)), exclusion of pairs where operands had different values, exc… view at source ↗
Figure 3.7
Figure 3.7. Figure 3.7: A comparison of methods on a modular multiplication task with p=61 on various forms of input exclusion. The neural network handily beats the RFM in 6/7 tasks, and struggles only when input pairs with different operands are excluded. by the tuple (x mod m1, x mod m2,..., x mod mn). Using a prime of 61, we leverage this property to encode our inputs as the tuple (x mod 3, x mod 5,..., x mod 7), since all n… view at source ↗
Figure 3.8
Figure 3.8. Figure 3.8: RFM vs. neural network performance on various CRT encodings. Top: A thorough performance comparison of neural networks and RFMs across all tasks. Bottom: A comparison of neural networks and RFMs across different CRT encodings. The results show that modular addition becomes an easier task for both models as the size of the CRT representation increases, whereas modular multiplication becomes more difficult… view at source ↗
Figure 3.9
Figure 3.9. Figure 3.9: Laplace kernel with NN first Layer (post nonlinearity) features vs. neural network performance on various CRT encodings. Both models are tasked with solving CRT-encoded modular multiplication with p=61. This time, the Laplace kernel using the first layer features underperforms the network itself, showing the gap isn’t entirely one of first layer feature learning. 19 [PITH_FULL_IMAGE:figures/full_fig_p03… view at source ↗
Figure 4.1
Figure 4.1. Figure 4.1: RFM, K-Inverse-RFM, and neural network performance comparison across modular addition and multiplication. While it does not reach the performance of neural networks, the K-Inverse-RFM showcases much improved scaling relative to the RFM and on average bridges 64% of the gap to the performance of neural networks. 22 [PITH_FULL_IMAGE:figures/full_fig_p033_4_1.png] view at source ↗
Figure 4.2
Figure 4.2. Figure 4.2: A comparison of RFM, K-Inverse-RFM and neural network performance for modular addition and multiplication. The top plot shows test set accuracy for different relative weightings of inputs divisible by 3, whereas the bottom plot shows test accuracy for different relative weightings of labels divisible by 3. clusion types introduced in 3.7. K-Inverse-RFMs once again beat RFMs across the board, with particu… view at source ↗
Figure 4.3
Figure 4.3. Figure 4.3: A reframing of [PITH_FULL_IMAGE:figures/full_fig_p036_4_3.png] view at source ↗
Figure 4.4
Figure 4.4. Figure 4.4: A comparison of K-Inverse, RFMs, and neural networks across different excluded inputs for modular addition. K-Inverse-RFMs outperform RFMs across the board, in particular on tasks with a larger number of unique input points. two is just that K-Inverse-RFMs have a superior scaling rate, likely as a result of the multiclass feature learning capabilities encouraged by their feature-space-to-feature-space ma… view at source ↗
Figure 4.5
Figure 4.5. Figure 4.5: A comparison of RFMs, K-Inverse-RFMs, and neural networks across the different CRT encoding sizes given in 3.8. In 5/6 encoding-task pairs, the K-Inverse-RFM outperforms both the RFM and neural network. 26 [PITH_FULL_IMAGE:figures/full_fig_p037_4_5.png] view at source ↗
read the original abstract

Recursive Feature Machines (RFMs) are a class of kernel machines that utilize the Average Gradient Outer Product (AGOP) as a mechanism for feature learning. They have been shown to effectively replicate the learning dynamics and feature representations of Feedforward Neural Networks (FNNs) across various settings. However, despite comparable capacity for feature learning and the similarities in the features they acquire, RFMs exhibit significantly lower performance than neural networks in certain data-corrupted scenarios. In this work, we investigate these limitations in mathematical problems. As a solution, we introduce a remarkably effective transformation applied to the training labels which promotes learning in noisy, complexly represented, and class-imbalanced data. This simple yet powerful adjustment enables RFMs to close the performance gap with FNNs and, in some cases, even surpass them.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The manuscript introduces K-Inverse-RFM, a variant of Recursive Feature Machines (RFMs) that applies a K-Inverse transformation to the training labels. RFMs are kernel machines that use the Average Gradient Outer Product (AGOP) for feature learning and have been shown to replicate aspects of Feedforward Neural Network (FNN) dynamics, yet they underperform FNNs on data-corrupted mathematical tasks involving noise, complex representations, and class imbalance. The central claim is that the proposed label transformation is a simple, effective fix that closes this performance gap and can even allow RFMs to surpass FNNs.

Significance. If the empirical results hold under proper controls, the work would demonstrate that a minimal label-space adjustment can make RFMs competitive with neural networks on corrupted mathematical data without altering the core AGOP-based feature-learning mechanism. This could be useful for settings where kernel methods are preferred for interpretability or theoretical tractability, provided the transformation is shown to be robust across multiple corruption types and task formulations.

major comments (1)
  1. Abstract: The central claim that the K-Inverse transformation 'promotes learning in noisy, complexly represented, and class-imbalanced data' and 'enables RFMs to close the performance gap with FNNs' is asserted without any experimental details, dataset descriptions, baseline comparisons, or derivation. This prevents evaluation of whether the reported improvement is load-bearing for the claim or an artifact of unspecified conditions.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their review. We address the single major comment below.

read point-by-point responses

  1. Referee: Abstract: The central claim that the K-Inverse transformation 'promotes learning in noisy, complexly represented, and class-imbalanced data' and 'enables RFMs to close the performance gap with FNNs' is asserted without any experimental details, dataset descriptions, baseline comparisons, or derivation. This prevents evaluation of whether the reported improvement is load-bearing for the claim or an artifact of unspecified conditions.

    Authors: We agree that the abstract, as currently written, is a high-level summary that does not include the requested specifics. While the full experimental details, dataset descriptions, baseline comparisons, and derivation of the K-Inverse transformation appear in Sections 2–4 of the manuscript, we acknowledge that a brief indication of these elements in the abstract would improve readability and allow readers to assess the claims more readily. In the revised version we will expand the abstract with a concise statement of the tasks, corruption types, and main baselines used. revision: yes

Circularity Check

0 steps flagged

No derivation chain or equations present; purely empirical claim

full rationale

The abstract and description introduce an empirical label transformation (K-Inverse) to improve RFM performance on corrupted data, with no equations, derivations, first-principles results, or load-bearing self-citations provided. The central assertion is an experimental performance improvement rather than any claimed mathematical reduction that could be circular by construction. No steps match the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no equations, methods, or results from which to extract free parameters, axioms, or invented entities; the transformation is presented at a high level without supporting structure.

pith-pipeline@v0.9.1-grok · 5668 in / 1000 out tokens · 27977 ms · 2026-07-02T16:04:41.422732+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

9 extracted references · 4 canonical work pages

  1. [1]

    Mechanism for feature learning in neural networks and backpropagation-free machine learning models.Science, 383(6690):1461–1467, 2024

    Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Mechanism for feature learning in neural networks and backpropagation-free machine learning models.Science, 383(6690):1461–1467, 2024. doi: 10.1126/science.adi5639. URL https: //www.science.org/doi/abs/10.1126/science.adi5639

    work page doi:10.1126/science.adi5639 2024

  2. [2]

    Kernel methods

    Percy Liang. Kernel methods. Stanford CS229T/STAT231: Statistical Learning Theory (Winter 2016), 2016. URL https://web.stanford.edu/class/cs229t/2017/Lectures/percy-notes. pdf. Accessed: 2025-03-18

    2016

  3. [3]

    Wahba.Spline Models for Observational Data, volume 59 ofCBMS-NSF Regional Conference Series in Applied Mathematics

    G. Wahba.Spline Models for Observational Data, volume 59 ofCBMS-NSF Regional Conference Series in Applied Mathematics. SIAM, Philadelphia, 1990

    1990

  4. [4]

    Emergence in non-neural models: grokking modular arithmetic via average gradient outer product, 2024

    Neil Mallinar, Daniel Beaglehole, Libin Zhu, Adityanarayanan Radhakrishnan, Parthe Pandit, and Mikhail Belkin. Emergence in non-neural models: grokking modular arithmetic via average gradient outer product, 2024. URL https://arxiv.org/abs/2407.20199

    work page arXiv 2024

  5. [5]

    Aggregate and conquer: detecting and steering llm concepts by combining nonlinear predictors over multiple layers, 2025

    Daniel Beaglehole, Adityanarayanan Radhakrishnan, Enric Boix-Adser` a, and Mikhail Belkin. Aggregate and conquer: detecting and steering llm concepts by combining nonlinear predictors over multiple layers, 2025. URL https://arxiv.org/abs/2502.03708

    work page arXiv 2025

  6. [6]

    Linear recursive feature machines provably recover low-rank matrices.Proceedings of the National Academy of Sciences, 122(13):e2411325122, 2025

    Adityanarayanan Radhakrishnan, Mikhail Belkin, and Dmitriy Drusvyatskiy. Linear recursive feature machines provably recover low-rank matrices.Proceedings of the National Academy of Sciences, 122(13):e2411325122, 2025. doi: 10.1073/pnas.2411325122. URL https://www. pnas.org/doi/abs/10.1073/pnas.2411325122

    work page doi:10.1073/pnas.2411325122 2025

  7. [7]

    Average gradient outer product as a mechanism for deep neural collapse

    Daniel Beaglehole, Peter S´ uken´ ık, Marco Mondelli, and Mikhail Belkin. Average gradient outer product as a mechanism for deep neural collapse. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, volume 37, pages 130764–130796. Curran As- sociates, Inc., 2024. U...

    2024

  8. [8]

    The k-inverse rfm framework

    Neil Mallinar and Mikhail Belkin. The k-inverse rfm framework. Unpublished manuscript, 2025

    2025

  9. [9]

    Deep learning through the lens of example difficulty

    Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman 34 Vaughan, editors,Advances in Neural Information Processing Systems, volume 34, pages 10876–10889. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper files...

    2021