Pith.

Machine Learning · arXiv:2610.37553 v1 · submitted 27 Oct 2026 · 14 references

Collective Dynamics of Self-Reinforcement: How Cross-Task Coupling Drives Policy Collapse in Unsupervised RLVR

Peize Ding

The paper says

“Label-free reinforcement learning with verifiable rewards (RLVR) rewards a language model for agreeing with its own majority vote.”

more of the abstract

It improves reasoning at first but eventually collapses: the self-assessed reward goes to one while accuracy falls, often until the model gives the same answer to every problem. The collapse appears only when training uses many problems. We explain this dataset-size dependence as a collective instability. In a solvable many-problem model in which problems share parameters, a population whose problems prefer different answers amplifies any shared perturbation for as long as the population-averaged slope ⟨F′⟩ of the single-problem self-reinforcement drift F is positive. This holds while problems are still uncertain about their answers. Per-problem sharpening, slowed by 1/N for N training problems, races against the shared mode, which grows at a rate ∝ λ2 independent of N, where λ is the strength of the shared coupling. This yields a critical number of problems Nc ≃ ln(1/δ)/(λ2G), where δ seeds the shared mode and G decreases with the initial confidence. Simulations of the model confirm the predictions: a sharp transition, Nc ∝ λ−2 up to a logarithmic correction, a single critical amplification λ2NG across couplings, confidence profiles and training variants, a logarithmic dependence on batch size, and no collapse for confident initial models. A kernel version of the theory identifies a protective coupling (generalization of sharpening) and a marginal-entropy bonus that stabilizes the uniform collective mode. In label-free RL of a small Transformer, removing the shared direction prevents collapse entirely, restoring it causes collapse, and collapse grows with training-set size and base-model entropy.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 28 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A genuinely new toy-model mechanism for the dataset-size threshold in unsupervised RLVR, with clean math and an honest but quantitatively undetermined network transfer that the paper itself discloses. This line is wrong

2 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 14 references say.

The paper says, at Section 6.2, Table 3, Appendices D.3-D.4

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.60):

“In short, the initial kernel identifies which architectures are protected and orders conditions by risk, but it does not locate a reliable quantitative stability boundary for a given network.”

Pith says: Objection 1 of 2 · major

The quantitative theory is contradicted by two conditions in Table 3, and the paper's own kernel-drift analysis does not resolve the contradiction. First, plain base-120 at τ=1 has initial G=0.000 to three decimals; Eq. (7) and prediction P4 then give a decaying shared mode and no collapse, yet 17 of 42 runs collapse.

more of objection 1

Second, the antisymmetric base-20 condition at τ=1, ρ_b=0.3 has ku/k'_d = 0.436 < ks/k'_d = 0.446 with G = 0.177, so Eq. (8) with Λ_c = 2.64 gives Λ_c k'_d/(G ku − Λ_c ks) < 0: this is the feature-protected regime in which no training-set size should cause collapse, yet 22 of 42 runs collapse. Appendix D.4 shows that kernel drift is small in the plain networks (k_d grows at most 2.2× over eight epochs and ku/k'_d erodes mildly to 0.254), so drift cannot rescue the prediction there; the authors concede exactly this in D.4. Because the disagreement is one-sided (43 of 44 cells, D.3), the issue is not calibration noise but a systematic gap between the sign of the predicted stability and the observed outcome. The authors state the limitation explicitly (Sections 6.2 and 7), but the consequence for the headline claim — that the dataset-size dependence is explained by this specific mechanism — is not drawn: a substantial fraction of the network collapses reported (the τ=1 confident plain conditions and the ρ_b=0.3 antisymmetric cells) remain unexplained by the mechanism as formulated.

The paper says, at Sections 4.2-4.3 and 6.2, Appendix D.4

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.64):

“Measuring ⟨F ′⟩ and the kernel ratios on a small pretrained language model, and testing whether they predict the dataset-size threshold of He et al. (2026), is the most important next step.”

Pith says: Objection 2 of 2 · major

The decisive trajectory-level check of the mechanism is missing. The paper never measures the instability index ⟨F'(z_i(t)/τ)⟩, or the projection of the logits onto the shared direction, along actual training trajectories in the Transformer; it measures only the initial G and the evolving kernel ratios (Table 4).

more of objection 2

Consequently it cannot answer whether collapsing runs satisfy the window condition ⟨F'>0 while the shared mode is amplified, or whether they collapse through a different route. The paper itself lists other collective kernel modes and finite-sample answer imbalance as plausible contributors that it has not separated (D.4). This measurement is cheap in the authors' setup — it is the same quantity used to compute G on the initial model — and it is decisive for the scope of the central claim. I ask that the authors either (i) add it and show, for the conditions of Table 3, that collapse coincides with a positive integrated window along the trajectory and that stable runs are separated by this index, or (ii) narrow the abstract's claim so that the explanation refers to the solvable model plus the qualitative necessity/sufficiency result for networks (antisymmetric never collapses; added coupling restores collapse), with the quantitative law N_c ≈ ln(1/δ)/(λ²G) explicitly scoped to the fixed-coupling model. Without one of these, the boundary of the explanation remains undetermined.

5 minor objections

The paper says, at Section 3, Eq. (3)

Pith says: minor objection 1 of 5

The displayed formula F'(z) = p(1−p)(12 − 6q²) and the sentence 'Setting 6q² = 12 locates the zero' appear to garble the fraction 1/2; the subsequent values z* = ln(2+√3) and p ∈ (0.21, 0.79) are computed with 6q² = 1/2. Please fix the typesetting.

The paper says, at Section 5.1 and Appendix B.1

Pith says: minor objection 2 of 5

The fitted critical amplifications spread over [2.1, 3.6] in the main sweeps and over [2.8, 4.0] across group sizes (Appendix C). Since Λ_c is the theory's only free constant (it absorbs the unspecified noise seed), the claim of a single critical value would be strengthened by reporting this spread relative to the transition width in Λ and by stating explicitly which part of the spread is attributed to seed-size variation.

show 3 more

The paper says, at Table 3 and Appendix D.2

Pith says: minor objection 3 of 5

The collapsed column aggregates 42 runs across seven values of N. Because D.2 attributes part of the small-N collapse to initial-answer imbalance acting as a large seed, please give the N-wise breakdown at least for the τ=1 plain conditions and the ρ_b=0.3 antisymmetric cells, so the reader can separate the small-N seed effect from the large-N collapse that the theory cannot explain.

The paper says, at Section 6.2 and Appendix D.3

Pith says: minor objection 4 of 5

The sign-agreement comparison with the always-collapse baseline (73%, with overlapping bootstrap intervals) appears only in Appendix D.3; since it is the correct reading of the headline 71% figure, it should move to the main text next to the 71% claim.

The paper says, at Appendix B.2

Pith says: minor objection 5 of 5

The measured batch-size slope is about 1.5× the crude seed estimate of Appendix A.3; a sentence identifying the likely source of the discrepancy (for example, time-dependence of F' over the growth interval) would prevent readers from treating the slope as a sharp predicted value.

What would settle it

Run the plain base-120, τ=1 condition (G=0.000) at N=1024, where initial answer imbalance is negligible, for roughly six seeds, and record per-problem logits z_i(t) every few optimizer steps during label-free RL. From these trajectories compute the instability index ⟨F′(z_i(t)/τ)⟩, the fraction of problems inside the window |z| < z⋆, and the order parameter Φ. The paper's mechanism requires ⟨F′⟩ > 0 before the shared mode can amplify, so the outcome discriminates cleanly: (i) if collapsing runs show ⟨F′⟩ > 0, with logits shrinking into the window, before Φ rises, the mechanism operates and the required fix is a time-varying-kernel version of Equation (8) using the tracked kernel; (ii) if runs collapse while ⟨F′⟩ ≤ 0 throughout, the undecided-window condition is not necessary for network collapse and the central explanation must be scoped down or extended.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 28 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic partly agrees 2 min 10 s · 30,210 tokens read, 19,857 written
  3. Editor wrote the verdict above 15 s · 31,265 tokens read, 1,793 written
  4. Referee major revision: 2 major objections, 5 minor 3 min 5 s · 30,554 tokens read, 27,729 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The useful part of this paper is the toy model, and it is good. The single-problem drift F(z) = p(1-p)(2p-1), the exact proportionality between REINFORCE, mean-baseline and GRPO drifts at K=3, the linearization of the shared mode, and the deterministic mean-field check with unit slope in ln(1/δ) are all clean and correct as far as I can tell. The mechanism—local sharpening slowed by 1/N racing an N-independent shared-mode growth—is genuinely new relative to the single-problem sharpening analyses you mentioned, and the closed-form Nc ≈ ln(1/δ)/(λ²G) with the single constant Λc ≈ 2.6 is a real, falsifiable statement. The simulations confirming the sharp transition, the λ−2 scaling, the logarithmic batch-size dependence and the stability of confident models are convincing. The protective prediction-aligned coupling and the marginal-entropy bonus are sensible extensions, and the honest asymmetric transfer results deserve credit.

The network section is where the scope stays undetermined, and the stress-test note has it right. The antisymmetric readout experiment is a clean causal test: no shared direction, no collapse; add the bias back dose-dependently, collapse returns. That supports the qualitative claim. But the quantitative claim does not transfer. Initial G=0.000 yet 17 of 42 runs collapse. Kernel drift is measured and is too small to explain it in those plain networks, and the paper says so itself—Appendix D.4 calls the remaining candidates plausible but unseparated. The sign-agreement statistic (71%) being indistinguishable from always-collapse (73%) is a real weakness, not a nitpick, and the paper reports it openly. The decisive question the stress-test poses is fair: do the collapsing network trajectories ever satisfy the window condition ⟨F′⟩ > 0, or do some collapse with the instability index non-positive throughout? The paper does not answer this, and without it the mechanism's scope over actual networks is not established.

None of this is hidden. The paper states its limitations plainly, including that Λc is toy-calibrated, that no pretrained language model was tested, and that the initial kernel is a screening tool rather than a stability guarantee. That candor is to its credit and should not be penalized. What remains is a genuine open boundary: the toy-model theory is solid, the causal role of the shared direction in the network is demonstrated, and the quantitative Nc claim has no demonstrated network support.

Worth a serious referee, without question. The toy-model result alone justifies citation, and the transformation of a vague dataset-size observation into a closed-form prediction with an honest failure report is more than most papers in this area deliver. I would bring it to reading group and cite it.

The strongest objection

The paper's central claim is that dataset-size collapse is a shared-direction collective instability that grows only while the population-averaged drift slope ⟨F′⟩ is positive, i.e., while problems remain undecided (|z| < z⋆ ≈ 1.32, Equation (3)), producing the closed-form threshold Nc ≈ ln(1/δ)/(λ²G). That mechanism is exact and verified in the toy model, and the paper candidly discloses that its quantitative predictions fail in the Transformer: the initial-kernel surrogate is indistinguishable from always-collapse by sign agreement (71% vs 73%) and underpredicts collapse in 43 of 44 disagreement cells (Sections 6.2, D.3). What remains unresolved is the boundary of the explanation. Table 3 lists plain base-120 at τ=1 with initial G=0.000 to three decimals; P4 and Equation (7) predict no amplification and no collapse, yet 17 of 42 runs collapse. Table 4 shows this is not a kernel-drift rescue: for that network k_d grows only 1.6× over 4 epochs and 2.2× over 8, k_s/k′d falls mildly from 0.059 to 0.037, and Section D.4 states kernel drift therefore 'does not explain why the confident plain networks at τ=1 collapse more often than predicted.' The remaining candidates (other collective kernel modes, finite-sample answer imbalance) are 'plausible contributors there, which we have not separated' (Appendix D.4). The decisive question is whether the mechanism operates at all in these runs: do collapsing trajectories ever satisfy the window condition ⟨F′(z_i(t)/τ)⟩ > 0, or do they collapse while the instability index stays non-positive throughout? If the latter, the collective-race explanation does not cover a measured collapse regime and P4-type stability cannot be claimed for networks from initial G; if the former, the mechanism survives but initial-kernel G is not a valid input and the quantitative claim has no demonstrated network support. Either way, the central claim's scope is undetermined and the paper's stated candidate explanations do not resolve it.

Reading in good faith: the paper does what it says. The toy-model theory is internally consistent; exact enumeration gives F(z) = p(1−p)(2p−1), Proposition 1 is correct, the deterministic mean-field check in Appendix A.4 confirms X_c G(a) = ln(1/δ) − c with unit slope, stochastic simulations verify P1–P4 including long-training stability of confident models, and the Λc ≈ 2.6 universality across couplings, confidences and batch sizes is a real, parameter-light result. The Transformer evidence is also genuinely informative: 0 of 168 antisymmetric runs collapse, 335 of 336 collapse once the added shared coupling reaches ρ_b ≥ 1, and collapse grows with N and temperature. The paper is unusually candid about the one-sided quantitative failure, stating in the abstract itself that initial-kernel predictions are poorly calibrated and no better than always-collapse in sign agreement, and listing the absent pretrained-LM test as the most important next step. The load-bearing residual concern is not the acknowledged underprediction, and not the missing LM experiment (a stated scope limit). It is that the mechanism's defining condition, amplification requiring ⟨F′⟩ > 0 with problems undecided, has not been checked along the trajectories of the networks that collapse, and the starkest case (G=0.000, 17/42 collapsed) sits exactly where the condition is absent at initialization while kernel drift is measured and found insufficient by the authors themselves (Table 4, Section D.4). Because the remaining candidate causes are left 'not separated,' the central claim has an undetermined boundary: the confident-plain collapses are either inside the theory (logits later enter the window) or outside it (collapse with no undecided-window amplification). The trajectory-level measurement of ⟨F′⟩ settles this cheaply. This sharpens rather than replaces the reader's weakest assumption: the reader attributes the transfer failure to kernel movement and other collective modes; the G=0.000 case shows predictions fail even where the kernel barely moves, so the failure is not fully attributable to kernel drift and the scope of the mechanism itself is unproven (partial agreement). The verdict remains CONDITIONAL: the concern is concrete and testable, the toy model and qualitative causal evidence stand, and nothing in my reading justifies rejection. The condition to add is the trajectory-level window check on the confident-plain conditions.

The referee's report

If the result holds, the paper supplies a clean, falsifiable mechanism for a puzzling empirical phenomenon: per-problem sharpening is slowed by 1/N while the shared mode grows at an N-independent rate, so enlarging the training set gives a shared bias more time to be amplified. The within-model verification is strong: exact drift enumerations (Appendix A.1), a correct-looking linearization (Proposition 1), and a deterministic mean-field check that reproduces the closed form with unit slope in ln(1/δ) (Table 1). The causal architecture experiment is the most convincing part of the network study: removing the shared direction eliminates collapse entirely and adding it back restores collapse monotonically, a rare direct necessity/sufficiency test. The paper is also exemplary in disclosing its failures: the abstract, Section 6.2 and Section 7 state that the initial-kernel predictions are poorly calibrated, and Appendices D.3-D.4 document the one-sided underprediction and the kernel drift. The reproducibility package (code and raw results at https://github.com/pd2714/RLVR_instability) supports the empirical claims.

Pith's reading of the paper (a machine's summary, not the authors')

This paper tries to explain a striking empirical pattern in label-free RLVR, in which a model is rewarded for agreeing with its own majority vote: the method sharpens reasoning at first, then collapses — the self-assessed reward goes to one while true accuracy falls to chance, often until the model gives the same answer to every problem — and the collapse appears only when training uses many problems. The proposed explanation is a collective instability: because problems share network parameters, any parameter direction that moves all answers the same way collects the majority-vote signal from every problem at once, while a problem-specific direction receives only a $1/N$ share. While enough problems are still undecided (answer logits inside the window $|z| < z_\star \approx 1.32$), such a shared perturbation is amplified, and a larger training set gives it more time to grow, producing a critical dataset size $N_c \simeq \ln(1/\delta)/(\lambda^2 G)$ set by the coupling strength $\lambda$, the initial confidence, and the noise seed $\delta$. A sympathetic reader should care because the paper turns a vague 'reward hacking' failure into a quantitative threshold with testable scaling laws, and it identifies both a protective coupling and a structural remedy: removing the shared direction eliminated collapse entirely in the paper's own small Transformer experiments.

What carries the argument. The load-bearing object is the single-problem self-reinforcement drift $F(z)$ and its slope $F'(z)$, together with the collective race they set up once problems share a parameter. For binary answers and $K=3$ samples, $F(z) = p(1-p)(2p-1)$ with $p = \sigma(z)$, and $F'(z) > 0$ inside the unstable window $|z| < z_\star = \ln(2+\sqrt{3}) \approx 1.317$, whose edge supplies the drift value $F_\star$ entering the amplification. Coupling $N$ problems through one shared direction, $z_i = u_i + \lambda v$, gives the mean-field equations $\dot z_i = (\eta/N)F(z_i) + \eta\lambda^2\langle F\rangle$, whose two terms scale differently with $N$: local sharpening slows as $1/N$ while the shared mode grows at the $N$-independent rate $\propto \lambda^2$. The ratio of the two rates integrates exactly to $\Lambda = \lambda^2 N \ln[F_\star/F(a)]$, the total log-amplification available before sharpening carries the population out of the window. For networks the same race is expressed through the empirical neural tangent kernel, summarized by four numbers — the self-kernel $k_d$, the shared coupling $k_u$ (the collapse driver, playing the role of $\lambda^2$), the prediction-aligned coupling $k_s$ (protective), and the remainder $k'_d$ — and the kernel formula exhibits a protected regime $G k_u < \Lambda_c k_s$ in which no dataset size causes collapse.

Core claim. On the paper's own terms, the central discovery is a race between two time scales in a mean-field model of $N$ problems coupled through a shared parameter: each problem has logit $z_i = u_i + \lambda v$, and the single-problem self-reinforcement drift $F$, exact for binary answers with $K=3$ samples as $F(z) = p(1-p)(2p-1)$ with $p = \sigma(z)$, pushes each problem toward its current majority answer. Per-problem sharpening proceeds at rate $1/N$, while a common logit shift grows as $\dot w = \eta\lambda^2\langle F'\rangle w$ at a rate independent of $N$, for as long as the population-averaged slope $\langle F'\rangle$ stays positive — that is, while problems are uncertain, $|z| < \ln(2+\sqrt{3}) \approx 1.32$. Integrating this transient growth along the sharpening trajectory yields a single critical amplification $\Lambda = \lambda^2 N \ln[F_\star/F(a)]$: collapse occurs when $\Lambda$ exceeds $\Lambda_c \approx 2.6$, equivalently a critical number of problems $N_c \simeq \ln(1/\delta)/(\lambda^2 G)$. The paper verifies the resulting scaling laws in stochastic simulations (a sharp transition, $N_c \propto \lambda^{-2}$, only logarithmic batch-size dependence, and stability of confident initial models) and carries the mechanism to a small Transformer, where an antisymmetric readout that removes the shared direction prevents collapse entirely, adding a controlled shared bias restores it, and a kernel version of the theory uncovers a protective prediction-aligned coupling that can make collapse vanish at every dataset size.

The premise it rests on. The quantitative threshold transfers to a real network only if the coupling between problems is fixed by the kernel measured at initialization and stays roughly constant during training — an assumption the authors acknowledge their own runs violate, since the kernel drifts sharply in some networks and a plain network with predicted negligible growth still collapsed in 17 of 42 runs.

If the paper is right (Pith's inferences, not the authors')

The critical dataset size is computable before training from measurable quantities — the coupling strength and the initial confidence profile — so the formula turns a post-hoc diagnosis into an a-priori threshold for when an unlabeled training set will destabilize self-rewarded RL.

Dataset size is a dial into the instability rather than a cure: because sharpening slows as $1/N$ while the shared mode grows independently of $N$, adding undecided problems crosses the transition, and above $N_c$ every problem converges to the same answer while the self-assessed reward rises to one.

Batch size matters almost not at all — its effect on the threshold is only logarithmic and vanishes under linear step scaling — so tuning the optimizer cannot fix the collapse.

Confident initial models are stable, not merely slow: if every problem starts outside the window ($a > z_\star$), a shared perturbation decays at every instant and no amount of extra training flips an answer, which directly predicts that higher-entropy base models collapse at smaller $N$.

The failure is not inevitable: a prediction-aligned coupling that makes sharpening one problem sharpen others raises the threshold and can eliminate collapse at every dataset size, and removing the shared direction with an antisymmetric readout prevented collapse in all of the paper's small-Transformer runs.

Editorial extension: the same $1/N$-versus-$\lambda^2$ race should apply to any training signal that depends on the policy's own collective output — self-certainty, entropy minimization, semantic entropy — so $N_c$-style thresholds may organize a whole family of self-referential collapse phenomena, not just majority-vote RLVR.

Editorial extension: the paper's one-sided calibration failure suggests the practical version for real models will be a kernel-tracking formula, with $k_u/k'_d$ and $k_s/k'_d$ measured during the first epochs instead of at initialization; the drift data reported in the paper (self-kernel growth up to roughly $245\times$, protective coupling eroding from $0.42$ to $0.07$) show where such a correcti

Editorial extension: because the marginal-entropy bonus damps only the uniform mode and the instability then migrates to the next collective kernel mode, a testable remedy is to penalize the leading task-unrelated eigenvectors of the kernel directly — a route the paper identifies but does not implement.

Editorial extension: if the threshold law survives transfer, architectures that cancel answer-independent shared directions (format normalization, frozen output biases, odd or antisymmetric readouts) should be more collapse-resistant at scale than any reward or entropy shaping, because they remove $\lambda$ at its source rather than compensating for its effects.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Label-free reinforcement learning with verifiable rewards (RLVR) rewards a language model for agreeing with its own majority vote. It improves reasoning at first but eventually collapses: the self-assessed reward goes to one while accuracy falls, often until the model gives the same answer to every problem. The collapse appears only when training uses many problems. We explain this dataset-size dependence as a collective instability. In a solvable many-problem model in which problems share parameters, a population whose problems prefer different answers amplifies any shared perturbation for as long as the population-averaged slope ⟨F′⟩ of the single-problem self-reinforcement drift F is positive. This holds while problems are still uncertain about their answers. Per-problem sharpening, slowed by 1/N for N training problems, races against the shared mode, which grows at a rate ∝ λ2 independent of N, where λ is the strength of the shared coupling. This yields a critical number of problems Nc ≃ ln(1/δ)/(λ2G), where δ seeds the shared mode and G decreases with the initial confidence. Simulations of the model confirm the predictions: a sharp transition, Nc ∝ λ−2 up to a logarithmic correction, a single critical amplification λ2NG across couplings, confidence profiles and training variants, a logarithmic dependence on batch size, and no collapse for confident initial models. A kernel version of the theory identifies a protective coupling (generalization of sharpening) and a marginal-entropy bonus that stabilizes the uniform collective mode. In label-free RL of a small Transformer, removing the shared direction prevents collapse entirely, restoring it causes collapse, and collapse grows with training-set size and base-model entropy.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-28

The authors' main claim, as the classifier picked it out 0.72

“We explain this dataset-size dependence as a collective instability.”

How the main claim is stated: asserted 0.98

What the abstract itself states, as probabilities: claims first 0.03 · states limitation 0.33 · human participants 0.02 · controlled comparison 0.67 · randomized assignment 0.02 · states quantitative result 0.88 · compares with prior results 0.38 · mentions code or data availability 0.02

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Label-free reinforcement learning with verifiable rewards (RLVR) rewards a language model for agreeing with its own majority vote. background 0.99 0.08 0.03 asserted 1.00
It improves reasoning at first but eventually collapses: the self-assessed reward goes to one while accuracy falls, often until the model gives the same answer to every problem. result 0.56 0.50 0.16 asserted 0.99
The collapse appears only when training uses many problems. result 0.92 0.68 0.07 asserted 0.99
We explain this dataset-size dependence as a collective instability. interpretation 0.72 0.73 0.27 asserted 0.98
In a solvable many-problem model in which problems share parameters, a population whose problems prefer different answers amplifies any shared perturbation for as long as the population-averaged slope $\langle F'\rangle$ of the single-problem self-reinforcement drift $F$ is positive. interpretation 0.33 0.79 0.48 asserted 1.00
This holds while problems are still uncertain about their answers. result 0.43 0.57 0.05 asserted 0.98
Per-problem sharpening, slowed by $1/N$ for $N$ training problems, races against the shared mode, which grows at a rate $\propto\lambda^2$ independent of $N$, where $\lambda$ is the strength of the shared coupling. result 0.41 0.80 0.13 asserted 1.00
This yields a critical number of problems $N_c\simeq\ln(1/\delta)/(\lambda^2G)$, where $\delta$ seeds the shared mode and $G$ decreases with the initial confidence. result 0.91 0.86 0.18 asserted 0.99
Simulations of the model confirm the predictions: a sharp transition, $N_c\propto\lambda^{-2}$ up to a logarithmic correction, a single critical amplification $\lambda^2NG$ across couplings, confidence profiles and training variants, a logarithmic dependence on batch size, and no collapse for confident initial models. result 0.99 0.94 0.24 asserted 1.00
A kernel version of the theory identifies a protective coupling (generalization of sharpening) and a marginal-entropy bonus that stabilizes the uniform collective mode. contribution 0.51 0.87 0.09 asserted 1.00
In label-free RL of a small Transformer, removing the shared direction prevents collapse entirely, restoring it causes collapse, and collapse grows with training-set size and base-model entropy. result 0.96 0.91 0.28 asserted 1.00

What kind of paper the classifier reports this is: theoretical result 1.00 · evidence from simulation or computation 0.76

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-28: 1 request, 8,466 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (7)
Figure 1 of the paper
Figure 1. Mechanism. (a) Single-problem drift F and its slope F ′ . The shaded window |z| < z⋆ is where a shared push is amplified. (b) Majority-vote reward (solid) and accuracy against the hidden labels (dashed) at λ=0.1, a=0.5, averaged over 24 seeds. The two runs differ only in the number of training problems, which lies below (blue) or above (orange) the fitted Nc ≈ 536. Below Nc, local sharpening wins: the reward rises and accuracy stays near its initial level. Above Nc, the reward rises faster and accuracy falls to chance, because every problem converges to the same answer. (c) Order…
Figure 2 of the paper
Figure 2. Critical number of problems. (a) Probability of collapse against N for seven coupling strengths (a=0.5). (b) The same data, together with a sweep over initial confidence a (orange, λ=0.1), plotted against the predicted amplification Λ = λ 2N ln[F ⋆/F(a)]. Curves spanning a 64× range in λ −2 and critical sizes from 60 to 8740 collapse onto a single transition at Λc ≈ 2.6 (dotted). (c) Nc against λ. The fitted slope is −2.22, compared with the predicted −2 plus a logarithmic correction. (d) Nc against initial confidence. The dashed line is the theory with the single constant Λc.…
Figure 3 of the paper
Figure 3. Generalizations. (a) Heterogeneous initial confidence. P(collapse) against the amplification λ 2NG[ρ0] predicted from the initial distribution alone; the dotted line is Λc from
Figure 4 of the paper
Figure 4. Label-free RL on a small Transformer. (a) Majority-vote reward (solid) and held-out accuracy (dashed) of one run at N=512, for base 90 at τ=2 (orange) and base 120 at τ=1 (blue). (b) Plain Transformer, base 90: probability of collapse against N at three temperatures, as the observed fraction of 6 seeds. (c) Antisymmetric Transformer: probability of collapse, averaged over N, against the added shared coupling ρb, for each base and τ . Solid: observed. Dotted: kernel￾calibrated surrogate in the same color, from 24 simulated seeds per point. ρb=0 is drawn at the left edge. (d) Observed…
Figure 5 of the paper
Figure 5. Batch size and long training. (a) Nc against batch size, with a fixed step (blue) and with linear step scaling (orange). The dashed line is the logarithmic prediction anchored at B=32. (b) Final order parameter after 10× longer training for confident initial models (a > z⋆ ) at values of λ 2N up to 1474. Uncertain models collapse for λ 2N ≳ 5. (c) Order-parameter trajectories: confident models stay balanced, and sub-critical uncertain models remain balanced after sharpening. Kernel-calibrated surrogate. The main predictor of Section 6 keeps the same projected kernel but replaces the…
Figure 6 of the paper
Figure 6. Robustness. (a) GRPO and REINFORCE for K ∈ {3, 5, 8, 16} against Λ = λ 2NGK, where GK (Section 4.3) is computed from the exact K-sample drift FK (Section 3). (b) M-answer softmax policies against Λ = λ 2NGM. (c) Adam: probability of collapse against N for several couplings. the noise per sample fixed, Nc is flat (521–572 for B = 8–128, orange in Figure 5a), again as predicted by Appendix A.3. At B = 512 with linear scaling the step on the shared parameter (ηλ2 = 0.64) is too large for the small-step approximation, and all runs collapse. For comparison, a 16× change in λ −2 moves Nc…
Figure 7 of the paper
Figure 7. Marginal-entropy bonus on the plain base-90 Transformer at τ = 2. Lines: mean over 4 seeds; dots: individual seeds. The dashed line marks β = 1 2 , above which the theory predicts stability (Equation (9)). (a) Final order parameter Φ on the training problems; runs above the dotted line count as collapsed. The bonus removes collapse for β ≥ 1 2 . (b) Held-out accuracy stays near chance at every β, far below the base model’s initial accuracy (dotted, top). 5.1×103 , 2.5×103 , 2.3×103 , . . . , all far above the self-kernel kd = 94. Proposition 1 applies to any such mode: when the…

Figures are the authors', reproduced from the paper.

References (14)
  1. [3] Mingyuan Fan, Weiguang Han, Daixin Wang, Cen Chen, Zhiqiang Zhang, and Jun Zhou. When sharpening becomes collapse: Sampling bias and semantic cou- pling in RL with verifiable rewards.arXiv preprint arXiv:2601.15609, on PitharXiv
  2. [6] Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388, on PithPith's reviewarXiv
  3. [7] Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. Can large reasoning models self-train?arXiv preprint arXiv:2505.21444, on PitharXiv
  4. [8] Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer. Spuri- ous rewards: Rethinking training signals in RLVR.arXiv prepr... on PithPith's reviewarXiv
  5. [9] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, on PithPith's reviewarXiv
  6. [11] k′dk′d +N ks . The second term is the self-term. It is at most G and independent of N, and it is the growth that remains with the coupling switched off, when each problem follows its own sign-preserving flow and cannot flip. It is therefore absorbed into the O(1) of the thresh... on Pith
  7. [12] D.1. Models and conditions Ties x1 +x 2 +x 3 =x 4 +x 5 +x 6 are excluded, so the labels are balanced. The plain model has ku/k′d = 0.40–0.70 and ks/k′d ≈0.05 . We use base 90 and base 120, with 82% and 89% held-out accuracy. The antisymmetric model has ku/k′d <0.002 and ks/k′d... on Pith
  8. [14] The kernel ratios and the amplification G are measured on the initial network before any RL step. The base column gives the number of supervised warm-up steps and the resulting held-out accuracy. The last column counts collapsed runs out of 42 (7 values ofNand 6 seeds). model ... on Pith
  9. [128] on fresh labeled samples. RL uses the first N problems of a fixed pool of 8192 unlabeled problems, plain SGD with step 0.125B/kd (where kd is the mean self-kernel at the given temperature), B= 16 , K= 3 , REINFORCE with majority-vote rewards, and 100 epochs. Accuracy is measur... on Pith
  10. [1991] Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reason- ing language ... on PithPith's reviewarXiv
  11. [2001] Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximiz- ing confidence alone improves reasoning.arXiv preprint arXiv:2505.22660, on PithPith's reviewarXiv
  12. [2015] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Lu... on PithPith's reviewarXiv
  13. [2024] doi: 10.1038/s41586-024-07566-y. Takashi Takahashi. The role of pseudo-labels in self-training linear classifiers on high-dimensional Gaussian mixture data.Journal of Machine Learning Research, 27(138): 1–68, on Pithdoi:10.1038/s41586-024-07566-y
  14. [2025] DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capa- bility in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, on PithPith's reviewarXiv
Cited by 2 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment