{"id":"4ef8a9fb-6987-49a2-907b-b2807091bb48","arxiv_id":"2611.17019","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"TeReL turns VICReg-style self-supervision into per-neuron, temporally local Hebbian and anti-Hebbian updates that learn useful MNIST features with batch size one.","lead":"TeReL is a self-supervised learning rule that updates each neuron using only the current input, a short memory of nearby past activity, and same-layer signals, with no backward pass. On a label-sorted MNIST stream it beats random features and classical slow feature analysis, and it can update during inference without losing accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The class-chunked stream supplies a label-aligned boundary oracle through the I_t flag in Eq. (2); no ablation separates boundary gating from temporal adjacency, so the rule's gain is not yet isolated from experimenter-provided segmentation.","rationale":"The reader's weakest assumption is that the temporal signal is injected by the label-chunked ordering. I agree, and I sharpen it: the stream also provides exact boundary times, and these times enter the loss directly through I_t in Eq. (2) and through the p_t reset in Section 2.4. The authors are transparent about the constructed order and explicitly list natural-stream evaluation as the most important extension (Section 6), so this is not a hidden flaw. It is nonetheless load-bearing for the broad 'self-supervised temporal representation learning' framing, because the current experiments cannot tell whether TeReL learns from temporal coherence or from a label-derived segmentation signal. The math in Proposition 1 is sound, the protocol is careful about validation/test separation, and the ablations support H2 within the constructed setting, so the paper's scoped empirical claims stand. But the condition that the reader already placed on the verdict remains necessary. The proposed boundary manipulations are cheap and would settle whether the boundary oracle is the active ingredient. I therefore recommend no change to the CONDITIONAL verdict; the condition should be stated in terms of the boundary-gating flag as well as the label ordering.","tokens_in":12766,"tokens_out":11273,"duration_ms":81403,"concrete_test":"Run the main TeReL protocol on the same class-chunked MNIST stream under two boundary manipulations: (i) no boundaries, I_t = 1 for every observation so the temporal term always pulls toward the previous sample; (ii) the same number of boundaries but placed at random positions, uncorrelated with label changes. Compare fixed-encoder linear-probe accuracy against the reported TeReL 96.48%, shuffled-order 95.40%, and no-temporal-term 94.98% values. If either condition drops to the shuffled or no-temporal level, the central gain depends on the label-aligned boundary oracle rather than on temporal coherence alone.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 constructs the stream as randomly permuted class-homogeneous chunks of 16, and Section 2.4 says that at a sequence boundary only the validity of p_t is cleared. Because the chunk boundaries are exactly the label changes, the I_t term in Eq. (2) gives the network a precise oracle for when the 'slow feature' changes: within a chunk it pulls z_t toward the previous activation, and at the first sample of a new class it suppresses that pull. The reported ablations do not separate these two signals. Shuffling the stream destroys same-label adjacency but keeps random sequence boundaries; removing the temporal term removes the whole temporal objective. Neither condition tests whether TeReL would work when the stream is continuous or when boundary times are unknown, as in natural video or sensor data. This is a sharper version of the acknowledged limitation in Section 6 (sentences b93, b133): the unresolved issue is not only that positives are label-sorted, but that the algorithm is told exactly when the latent class changes. If the gain over random features is largely due to this boundary oracle, the evidence for a self-supervised temporal learning rule is considerably weaker than the abstract suggests.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper introduces Temporal Regularized Learning (TeReL), a layer-local, online learning rule that instantiates a VICReg-style objective with temporally adjacent positives, detached running statistics, and a lagged lateral decorrelation matrix. The authors derive the per-neuron gradient in Proposition 1, rewrite it as a Hebbian feedforward update with an anti-Hebbian lateral update plus a one-step lateral correction, and evaluate the rule on a class-chunked MNIST stream with batch size one. TeReL reaches 96.48% linear-probe accuracy, versus 95.09% for a random encoder and about 90% for batch and incremental SFA; a relaxed offline variant matches a fully supervised network, and continued unlabeled updates along the evaluation stream preserve accuracy. Ablations show that shuffling the stream or removing the temporal term degrades performance, that removing the variance term causes collapse, and that removing the covariance term causes divergence.","tokens_in":13136,"tokens_out":10863,"duration_ms":73736,"significance":"If the empirical claims hold, TeReL is a useful demonstration that a spatially and temporally local, batch-free self-supervised rule can learn a non-trivial representation in a small network, and it connects modern regularized self-supervised learning to SFA, trace learning, and Hebbian/anti-Hebbian synaptic rules. The paper has clear strengths: the gradient computation in Proposition 1 is clean; the official test split is held out; model selection is on a separate validation split; results are reported with paired confidence intervals; and code and a DOI are provided. The ablations also behave as expected for a VICReg-style objective, with collapse and divergence in the intended control conditions. However, the central evidence for the temporal signal is confounded by the boundary flag I_t, and the lateral-correction component of the algorithm is not derived from the stated objective. The significance for natural, unsegmented streams is therefore not yet established, although the authors do acknowledge the label-ordered stream as a limitation in Section 6.","major_comments":[{"comment":"The empirical case for the temporal signal is confounded by the boundary flag I_t. In the described stream, chunks are class-homogeneous and sequence boundaries coincide with class changes, so I_t is a per-sample oracle that tells the layer when the latent class has changed: within a chunk it enforces z_t close to p_t, and at the first sample of a new class it suppresses that pull. The two ablations in Figure 2b do not separate these effects: shuffling changes both the same-label adjacency and the alignment of boundaries with label changes, while removing the temporal term removes both the within-chunk pull and the boundary gating. The limitation is acknowledged in Section 6 (sentences b93 and b133), but the specific boundary-oracle problem is not. Please add an ablation that fixes boundary positions while destroying class homogeneity, an ablation that keeps class chunks but removes the boundary gating (I_t = 1 across boundaries), and ideally a truly continuous ordering, so that the contribution to H2 can be attributed to slow-feature learning rather than to the experimenter-provided segmentation.","section":"Section 4, Eq. (2), Section 2.4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The update used in the experiments is not exactly the gradient derived in Proposition 1: with the selected kappa = 0.575, s_t = b_t - eta kappa M_t b_t is a heuristic modification, and Eq. (6) is only the exact feedforward gradient when kappa = 0. No statement or proof is given that this one-step correction preserves the descent direction of the VICReg-style loss. Moreover, the 'no covariance term' ablation sets lambda_C = 0 but leaves M_t active in s_t and in Eq. (7), so it does not isolate the covariance loss from the lateral-correction mechanism. Please report the kappa = 0 and lambda_C = 0 conditions separately, and either prove the correction's descent property or mark the correction explicitly as an empirical design choice outside the derived objective.","section":"Section 2.3, Eqs. (5)-(6)","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The compute budgets differ across methods: TeReL-Offline consumes 1,000,000 samples while supervised offline consumes 500,000 samples. The paper states that the comparison is about representation quality rather than equal compute, but a sentence reporting a matched-sample or matched-epoch condition would make the 'comparable to supervised backpropagation' statement less open to misinterpretation.","section":"Appendix A, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The phrase 'per-neuron online loss' is slightly overstated because the covariance term and the lateral correction couple neurons through M_t; 'neuron-local loss with same-layer lateral coupling' would be more precise.","section":"Section 2.1 and Section 2.3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The selected hyperparameters are listed, but the search ranges and the validation criterion used for model selection are not. Reporting them would improve reproducibility.","section":"Appendix A","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Reference [27] is hosted on the author's own page rather than in an archival venue; if it is used as a prior-method citation, a versioned DOI or an archival source would be helpful.","section":"References","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Worth reading if you work on local or neuromorphic learning rules. TeReL is a VICReg-style objective made per-neuron and temporally local: adjacent positives, detached running statistics, and a lagged lateral matrix yield a Hebbian feedforward / anti-Hebbian lateral update. That combination is genuinely new, the derivation in Proposition 1 is clean, and the paper is explicit about what is a derived gradient versus a design choice. What is solid: the protocol. Held-out test split, validation for model selection, five seeds with paired 95% intervals, raw values in the appendix, code and a Zenodo DOI. The ablations behave the way they should — the variance term collapses, the covariance term diverges, and both shuffling and dropping the temporal term degrade accuracy. That is a real sanity check and it passes. The paper also scopes itself honestly: proof of concept in a small network, not a backprop competitor. I agree with the reader's soundness-7 and low-circularity scores; the linear probe is external to the objective, and nothing fitted is dressed up as a prediction. The stress-test note holds up, and it is the sharpest version of the real problem. The I_t flag in Eq. (2) tells the network exactly when a class chunk ends — pull toward the previous activation or suppress that pull. Because chunk boundaries are precisely the label changes, this is an experimenter-provided boundary oracle, not something the rule discovers. Neither ablation — shuffling, or removing the temporal term — separates boundary gating from temporal adjacency, so you cannot tell how much of the 1.39-point gain over random features relies on being told when the class changes. Natural streams do not come with boundary flags. The authors acknowledge the constructed stream in Section 6, but the oracle point is sharper than the acknowledgment and deserves its own experiment. The gains themselves are modest: 1.39 points over a random encoder, and Layer-local SupCon — which uses labels — beats TeReL by half a point. The win over classical SFA is the cleaner result. The lateral correction strength and mixing weight are hand-set rather than derived from a single objective, and the online continuation result comes after validating the step scale on the validation stream; it preserves rather than improves. All of this is stated in the paper, nothing hidden. Send it to review. It is a clean, honest, reproducible small contribution with one real open question — whether the rule works without the boundary oracle and on natural streams — and a referee should push on exactly that.","headline":"A clean, honest, reproducible local learning rule with real but small gains — the sharpest unresolved question is the boundary oracle the class-chunked stream hands the network.","tokens_in":14332,"tokens_out":851,"duration_ms":23806},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"TeReL turns a VICReg-style objective into a per-neuron, batch-free update: a two-layer network trained on a class-ordered MNIST stream hits 96.48% linear-probe accuracy, above random features and classical Slow Feature Analysis, and…","keywords":["temporal locality","self-supervised learning","slow feature analysis","Hebbian learning","anti-Hebbian learning","VICReg","online learning","streaming representation learning"],"falsifier":"A decisive check: train TeReL with the same protocol on a natural stream, for example video frames where pose or object identity rather than a class label is the slow variable, and compare linear-probe accuracy and the same-object nearest-neighbor fraction against a random encoder of the same shape; if the gain over random features disappears whenever the slow variable is not engineered to equal the class label, the claim that temporal coherence alone yields useful representations fails outside the constructed ordering. A cheaper in-domain check follows from the paper's own ablations: as the chunk structure of the MNIST stream is progressively weakened (shuffling already costs 1.08 of the 1.39-point paired gain over random features), the gain should shrink continuously and vanish with the temporal signal.","tokens_in":14171,"feed_emoji":"🧠","tokens_out":21170,"duration_ms":145151,"temperature":0.7,"pith_summary":"The paper introduces Temporal Regularized Learning (TeReL), a self-supervised objective for sequential input whose update is local in both space and time: every neuron uses only its own pre- and postsynaptic activity plus a small amount of detached state, with no backward pass between layers and no gradient flowing through the past. The central claim is that a VICReg-style triple of invariance, variance, and covariance regularizers can be instantiated per neuron, using the previous observation as the positive pair and running statistics as fixed state, and that this is enough to learn a useful representation in a small network. Trained with batch size one on a stream of class-homogeneous MNIST chunks, a two-layer fully connected encoder reaches $96.48\\%$ linear-probe accuracy against $95.09\\%$ for a random encoder and about $90\\%$ for classical SFA; a variant that relaxes both locality constraints reaches $98.41\\%$, on par with supervised backpropagation at $98.44\\%$. Because training and inference use the same per-sample computation, the rule can keep learning from unlabeled data after training without losing accuracy. The practical point is that temporally local, batch-free rules with bounded per-neuron memory fit streaming and energy-constrained hardware.","feed_headline":"96.48%: a fully local learning rule beats random features on MNIST","feed_subtitle":"A VICReg-style objective becomes per-neuron Hebbian updates that keep learning online with no accuracy loss.","key_machinery":"The load-bearing object is the per-neuron regularized target. For observation $t$ the residual\n$$r_t = I_t(z_t - p_t) - \\frac{\\lambda_V}{\\lambda_T} g_t \\odot (z_t - \\mu_t) + \\frac{\\lambda_C}{2\\lambda_T} A_t (z_t - \\mu_t), \\qquad A_t = \\mathrm{offdiag}(M_t),$$\ncombines temporal invariance (pull the activation $z_t$ toward the previous activation $p_t$, gated by sequence validity $I_t$), variance expansion ($g_t = [\\tau - v_t]_+$), and a lagged covariance penalty. The residual defines a detached activation target $\\hat{z}_t = \\mathrm{sg}(z_t - r_t)$; through a ReLU it becomes the preactivation residual $b_t = r_t \\odot \\mathbb{1}[a_t > 0]$, and one matrix–vector pass through the lateral matrix gives the neuron state $s_t = b_t - \\eta\\kappa M_t b_t$. Feedforward weights update as $W_{t+1} = (1 - \\alpha\\lambda_W)W_t - \\alpha c\\, s_t x_t^\\top$, so each synapse sees one presynaptic scalar and one postsynaptic scalar, the Hebbian form; the lateral matrix updates as $M_{t+1} = \\rho M_t + (1-\\rho)\\left[(1-\\omega)\\,\\mathrm{offdiag}(\\bar{z}_t \\bar{z}_t^\\top) + \\omega s_t s_t^\\top\\right]$, whose negative is anti-Hebbian: same-signed states make the lateral connection more inhibitory. Persistent state is bounded: three scalars per neuron (previous activation, running mean, running variance) plus one $D \\times D$ lateral matrix per layer, and plain SGD adds no optimizer state. This machinery converts a batch-level self-supervised objective into a causal, per-neuron, online loss with no backward pass and no gradient through the history that produced the state.","core_discovery":"TeReL is a soft, causal form of Slow Feature Analysis expressed as a per-neuron loss. The temporal term penalizes the squared difference between consecutive activations, the discrete-time derivative that SFA minimizes; the variance term expands coordinates whose running variance lies below a target $\\tau$; and the covariance term applies a lagged decorrelation signal through a lateral matrix $M_t$. Running statistics and the previous activation are non-differentiated state, so the gradient at a preactivation is a signed neuron state $s_t$ that drives a Hebbian feedforward update (postsynaptic state times presynaptic input) and an anti-Hebbian lateral update. At batch size one on a stream of randomly permuted class-homogeneous chunks of 16 MNIST samples, the two-layer encoder reaches $96.48 \\pm 0.15\\%$ held-out linear-probe accuracy, a paired gain of 1.39 points over a random encoder and about 6.4 points over batch and incremental SFA. Every ingredient is shown to matter: shuffling the stream costs 1.08 paired points, removing the temporal term costs 1.50 points, removing the variance term collapses the representation (median feature variance near $1.4 \\times 10^{-5}$, accuracy $15.36\\%$), and removing the covariance term makes training diverge. When the same objective is optimized with backpropagation through subsequences of 128 observations, accuracy reaches $98.41 \\pm 0.04\\%$, on par with the fully supervised network at $98.44 \\pm 0.09\\%$, so the gap to backpropagation is attributed to the locality constraints rather than to the objective. Continued unlabeled updates on the evaluation stream, with the step scaled by $10^{-3}$, hold accuracy at $96.50\\%$ versus $96.48\\%$ for the fixed encoder while the feedforward weights keep changing ($\\|\\Delta W\\|_2 = 0.047$).","pith_inferences":["My inference: if temporal coherence is what carries the gain, TeReL should learn whichever latent variable is genuinely slow on a natural stream, such as pose or identity in video, and a linear probe on that slow variable rather than on the class label is the right test; the paper itself names video and sensor streams as the key extension.","My inference: the chunked-MNIST setting aligns the slow variable exactly with the class label, so a fixed coefficient balance $(\\lambda_T, \\lambda_V, \\lambda_C)$ may need to be scale-adaptive on natural streams, since the ablations show the outcome is sensitive to the ordering and to each term.","My inference: the $D^2$ lateral matrix dominates memory at width, so low-rank, sparse, or structured lateral matrices, which the paper flags as future work, are the natural test of whether decorrelation can be preserved at lower cost.","My inference: because training and inference share one computation with bounded state, TeReL is a candidate for neuromorphic and analog implementation; a testable prediction is that the signed neuron state $s_t$ alone is the information a synapse must read to update its weight."],"forward_implications":["Self-supervised learning can run on a stream with batch size one: large batches, stored negatives, and cross-sample statistics are replaced by temporally adjacent positives plus per-neuron running statistics, cutting the per-layer memory to $3D$ scalars and one $D \\times D$ lateral matrix.","Training and inference use the same per-sample computation, so a deployed encoder can keep learning from unlabeled observations: at a $10^{-3}$-scaled step, continued updates hold accuracy at $96.50\\%$ against $96.48\\%$ for the fixed encoder while the feedforward weights keep changing.","Each loss term and the temporal order carry weight: removing the temporal term costs 1.50 paired points, shuffling the stream costs 1.08 points, removing the variance term collapses the representation, and removing the covariance term makes training diverge, so the rule is not reducible to a single mechanism.","The objective, not the locality, sets the ceiling in these experiments: TeReL-Offline, which backpropagates the same three-term loss through subsequences, reaches $98.41\\%$, on par with the fully supervised network at $98.44\\%$.","TeReL positions modern regularized self-supervision as a descendant of Slow Feature Analysis and trace learning, and the paper argues on ablation evidence that temporal local rules gain from explicit variance control and decorrelation."],"supporting_citations":[{"why":"Supplies the invariance-variance-covariance decomposition (VICReg) that TeReL instantiates per neuron with causally available statistics.","marker":"[3]"},{"why":"Defines Slow Feature Analysis and trace learning, the temporal-slowness ideas that TeReL softens into a regularized per-neuron loss.","marker":"[10, 11]"},{"why":"Derives Hebbian feedforward and anti-Hebbian lateral updates from similarity matching, the exact synaptic form TeReL's gradient takes.","marker":"[12, 13]"},{"why":"Motivates the no-weight-sharing experimental design and the use of class-ordered chunks so unsupervised features support discrimination.","marker":"[20]"},{"why":"Incremental SFA, the streaming SFA variant used as a baseline for representation quality.","marker":"[28]"},{"why":"Bio-SFA, the biologically plausible SFA with hard whitening, contrasted with TeReL's soft covariance penalties.","marker":"[29]"},{"why":"MNIST, the dataset on which all experiments, ablations, and the online-learning demonstration are run.","marker":"[32]"}],"fun_headline_variants":["Local-in-space-and-time learning beats random features on MNIST at 96.48%","Per-neuron Hebbian updates learn online from an MNIST stream at 96.48%","TeReL: a fully local self-supervised rule that keeps learning during inference","Temporal local rule nears supervised backprop when locality constraints are relaxed","Soft slow-feature analysis with Hebbian and anti-Hebbian updates keeps learning online"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The load-bearing premise is that temporally adjacent observations share the slow feature the representation should capture: in the experiments this is injected by sorting MNIST into class-homogeneous chunks of 16, so adjacency means same label, and Section 6 acknowledges that natural streams contain several time scales, so the same gain on video or sensor streams is not established.","fun_headline_variants_meta":{"raw":{"variants":["Local-in-space-and-time learning beats random features on MNIST at 96.48%","Per-neuron Hebbian updates learn online from an MNIST stream at 96.48%","TeReL: a fully local self-supervised rule that keeps learning during inference","Temporal local rule nears supervised backprop when locality constraints are relaxed","Soft slow-feature analysis with Hebbian and anti-Hebbian updates keeps learning online"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000107,"raw_usage":{"cost":0.0001065428,"is_byok":false,"cost_details":{"upstream_inference_cost":0.0001065428,"upstream_inference_prompt_cost":3.22868e-05,"upstream_inference_completions_cost":7.4256e-05},"total_tokens":1674,"prompt_tokens":1266,"completion_tokens":408,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":1139,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":309}},"tokens_in":1266,"tokens_out":408,"cost_source":"openrouter_usage","duration_ms":5682,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":309,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T07:21:42.387700+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"A decisive check: train TeReL with the same protocol on a natural stream, for example video frames where pose or object identity rather than a class label is the slow variable, and compare linear-probe accuracy and the same-object nearest-neighbor fraction against a random encoder of the same shape; if the gain over random features disappears whenever the slow variable is not engineered to equal the class label, the claim that temporal coherence alone yields useful representations fails outside the constructed ordering. A cheaper in-domain check follows from the paper's own ablations: as the chunk structure of the MNIST stream is progressively weakened (shuffling already costs 1.08 of the 1.39-point paired gain over random features), the gain should shrink continuously and vanish with the temporal signal.","supporting_citations":[{"cited_title":"VICReg: Variance-Invariance-Covariance Regular- ization for Self-Supervised Learning","cited_arxiv_id":null,"evidence_quote":"Supplies the invariance-variance-covariance decomposition (VICReg) that TeReL instantiates per neuron with causally available statistics."},{"cited_title":"The Forward-Forward Algorithm: Some Preliminary Investigations","cited_arxiv_id":"2212.13345","evidence_quote":"Motivates the no-weight-sharing experimental design and the use of class-ordered chunks so unsupervised features support discrimination."},{"cited_title":"Incremental Slow Feature Analy- sis: Adaptive and Episodic Learning from High-Dimensional Input Streams.Neural Computation, 24(11):2994–3024, 2012","cited_arxiv_id":null,"evidence_quote":"Incremental SFA, the streaming SFA variant used as a baseline for representation quality."},{"cited_title":"A Biologically Plausible Neural Network for Slow Feature Analysis","cited_arxiv_id":null,"evidence_quote":"Bio-SFA, the biologically plausible SFA with hard whitening, contrasted with TeReL's soft covariance penalties."},{"cited_title":"Gradient-based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 1998","cited_arxiv_id":null,"evidence_quote":"MNIST, the dataset on which all experiments, ablations, and the online-learning demonstration are run."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}