{"id":"234a7605-1184-4ef9-bf88-e82a8cd732c3","arxiv_id":"2604.06169","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"In-Place TTT adapts LLM MLP projection matrices at test time with a next-token-aligned objective and chunk-wise updates, enabling better long-context performance as a drop-in enhancement.","lead":"The paper presents In-Place Test-Time Training, a method that lets large language models update a subset of their parameters (the final projection in MLP blocks) during inference using a next-token prediction objective. This drop-in approach aims to improve adaptation on long contexts without retraining the entire model from scratch.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Adapting only the final MLP projection may not suffice for stable, effective test-time updates on long contexts","rationale":"The reader's weakest assumption matches the load-bearing point exactly. The abstract's experimental claims cannot be verified from the provided text alone, but the sufficiency of the narrow adaptation scope remains the key untested condition for the drop-in and scalability assertions.","tokens_in":1793,"tokens_out":290,"duration_ms":43274,"concrete_test":"Re-run the 4B-model 128k-context experiments while also adapting the first linear projection inside the same MLP blocks (keeping all other hyperparameters identical); if the original final-projection-only version underperforms the expanded variant by more than the reported margin over baselines, the restriction is insufficient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that restricting fast-weight updates to the final projection matrix inside each MLP block (plus the new next-token objective) produces stable adaptation that improves performance on 128k contexts without degrading the rest of the model or needing architectural changes. This restriction is what enables the 'drop-in' property and the reported gains for the 4B model. If the limited capacity cannot capture the necessary parameter shifts for new information, the updates would either be ineffective or would leak into other weights, undermining both the in-place enhancement and the outperformance claims versus other TTT methods.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces In-Place Test-Time Training (In-Place TTT) as a drop-in framework for LLMs that adapts only the final projection matrix within each MLP block as fast weights during inference. It replaces generic TTT reconstruction objectives with a new next-token-prediction-aligned objective and uses chunk-wise updates for scalability with context parallelism. Experiments claim that this enables a 4B model to outperform baselines on tasks with up to 128k contexts as an in-place enhancement, and that pretraining from scratch with In-Place TTT consistently beats competitive TTT methods, supported by ablations on design choices.","tokens_in":1930,"tokens_out":557,"duration_ms":28957,"significance":"If the empirical results and stability claims hold under the restricted adaptation, this could meaningfully advance practical test-time adaptation for existing LLMs by avoiding architectural changes or full retraining. The emphasis on a theoretically aligned objective and compatibility with long contexts addresses real barriers in the TTT literature for language modeling. The drop-in property and reported outperformance on 128k contexts would be notable strengths if the limited fast-weight capacity proves sufficient without side effects.","major_comments":[{"comment":"§3 (Method) and Eq. for the new objective: the claim that the objective is 'theoretically-grounded' and independent of experimental outcomes is not demonstrated in the provided description; the derivation must be shown explicitly to confirm it does not reduce to a fitted quantity or introduce circularity with the reported gains.","section":"§3"},{"comment":"Experiments section (4B model results on 128k contexts): the central claim that restricting updates to only the final MLP projection matrix produces stable, effective adaptation without degrading the rest of the model or requiring changes rests on unverified assumptions about capacity; additional controls or analysis are needed to show why this restriction suffices rather than leaking or underfitting on long contexts.","section":"Experiments"}],"minor_comments":[{"comment":"Abstract: notation for 'fast weights' and 'chunk size' should be defined on first use for clarity.","section":"Abstract"},{"comment":"The description of 'context parallelism' compatibility would benefit from a brief diagram or pseudocode in the methods to illustrate the chunk-wise mechanism.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":"The citation pattern appears light on prior TTT work in vision; ensure the related-work section positions the contribution against the full literature rather than only LLM-specific papers. The low-confidence empirical grounding noted in the reader's report aligns with the need for the major revisions above."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed feedback. We address each major comment below and will revise the manuscript to incorporate the requested clarifications and analyses.","responses":[{"response":"We appreciate this observation. The objective is obtained by replacing the generic reconstruction loss of prior TTT methods with the standard autoregressive cross-entropy loss applied to the next token, where the loss is evaluated after the in-place update of the fast weights. This construction follows directly from the next-token-prediction objective that defines language-model training and does not depend on any post-hoc fitting to the reported results. To make the grounding fully explicit and to rule out any appearance of circularity, we will insert the complete derivation (including the precise loss expression and the justification for its independence from experimental outcomes) into the revised Section 3.","revision_made":"yes","referee_comment":"[§3] §3 (Method) and Eq. for the new objective: the claim that the objective is 'theoretically-grounded' and independent of experimental outcomes is not demonstrated in the provided description; the derivation must be shown explicitly to confirm it does not reduce to a fitted quantity or introduce circularity with the reported gains."},{"response":"We agree that stronger evidence for the sufficiency of the restricted adaptation is warranted. The final projection matrix is chosen because it is the linear transformation that produces the MLP block output after the non-linearity, thereby providing a compact yet expressive site for fast-weight updates while preserving the rest of the model unchanged. The 4B-model experiments already demonstrate stable gains up to 128k contexts without degradation on shorter contexts or unrelated tasks, which is consistent with adequate capacity. Nevertheless, we will add in the revised experiments section (i) an ablation comparing adaptation of the final projection versus other matrices inside the MLP block and (ii) a capacity analysis that tracks the effective rank and gradient norms of the updated weights across long contexts, thereby directly addressing concerns about leakage or underfitting.","revision_made":"yes","referee_comment":"[Experiments] Experiments section (4B model results on 128k contexts): the central claim that restricting updates to only the final MLP projection matrix produces stable, effective adaptation without degrading the rest of the model or requiring changes rests on unverified assumptions about capacity; additional controls or analysis are needed to show why this restriction suffices rather than leaking or underfitting on long contexts."}],"tokens_in":1471,"tokens_out":518,"duration_ms":68310,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that this paper turns test-time training into something you can bolt onto existing LLMs. They designate the final projection matrix inside each MLP block as the fast weights and update them at inference with a loss built directly around next-token prediction rather than a generic reconstruction target. Chunk-wise processing keeps it compatible with context parallelism, so the whole thing scales without major rewrites to the model or training pipeline.","headline":"In-Place TTT makes test-time adaptation a drop-in by updating only the final MLP projection per block with a next-token objective, delivering reported gains on 128k contexts for a 4B model while raising capacity questions for the restricted updates.","tokens_in":2413,"tokens_out":180,"would_cite":false,"duration_ms":36370,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[{"relation":"unclear","rs_module":"IndisputableMonolith/Cost/FunctionalEquation.lean","rs_theorem":"washburn_uniqueness_aczel","paper_passage":"treats the final projection matrix of the ubiquitous MLP blocks as its adaptable fast weights... replace TTT's generic reconstruction objective with a tailored... Next-Token-Prediction task"},{"relation":"unclear","rs_module":"IndisputableMonolith/Foundation/RealityFromDistinction.lean","rs_theorem":"reality_from_one_distinction","paper_passage":"chunk-wise update rule... context parallelism... 8-tick period absent"}],"headline":"In-Place TTT operates in LLM adaptation with fast-weight updates and NTP objectives; no overlap with RS forcing chain, J-cost, φ-ladder or 8-tick structures.","alignment":"orthogonal","rationale":"The paper's machinery (MLP projection as fast weights, chunk-wise gradient updates, Conv1D-derived next-token targets) addresses architectural compatibility and causal language modeling. RS derives J(x)=½(x+x⁻¹)−1, φ, 8-tick periodicity and constants from a single distinction (reality_from_one_distinction, AbsoluteFloorClosure, Cost.FunctionalEquation). No shared primitives, cost functions or periodicity appear; domain is machine-learning engineering, not recognition-logic forcing.","tokens_in":56894,"confidence":"high","tokens_out":322,"duration_ms":15747,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"In-Place Test-Time Training endows large language models with the ability to adapt weights at inference time by updating the final projection matrices of their MLP blocks.","keywords":["test time training","large language models","MLP","fast weights","next token prediction","continual adaptation","inference time","context length"],"falsifier":"A direct comparison where a model with In-Place TTT fails to improve or worsens on long-context benchmarks relative to its non-adapting counterpart would falsify the central effectiveness claim.","tokens_in":2700,"feed_emoji":"🔄","tokens_out":511,"duration_ms":66212,"temperature":0.7,"pith_summary":"Large language models are currently limited by a fixed set of weights after training, which prevents them from responding to new data streams during use. In-Place TTT overcomes this by selecting the final projection matrix in every MLP block as the fast weights that get updated at test time. The method introduces a next-token prediction objective that matches the core task of language modeling, along with chunk-wise updates that work with parallel processing of long contexts. This results in better performance for a 4 billion parameter model on inputs as long as 128 thousand tokens, and stronger results than other test-time training techniques when the model is trained from the start. A reader would care if they want models that keep learning after deployment without full retraining.","feed_headline":"MLP final projections become fast weights for test-time LLM adaptation","feed_subtitle":"The method replaces reconstruction with next-token objectives and uses chunked updates to handle contexts up to 128k tokens effectively.","key_machinery":"The final projection matrix of MLP blocks as fast weights, updated with a next-token-prediction objective through chunk-wise mechanisms.","core_discovery":"In-Place TTT treats the final projection matrix of the ubiquitous MLP blocks as its adaptable fast weights, enabling a drop-in enhancement for LLMs without costly retraining from scratch. It replaces TTT's generic reconstruction objective with a tailored objective aligned with next-token prediction. Combined with an efficient chunk-wise update mechanism, this produces a scalable algorithm. Experiments show superior performance on long-context tasks and outperformance of competitive approaches when pretrained from scratch.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["In-Place TTT makes MLP final projections fast weights","In-Place TTT replaces reconstruction with next-token objective","Chunked updates enable In-Place TTT up to 128k contexts","Drop-in fast weight adaptation for LLMs via In-Place TTT"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"That adapting only the final projection matrices inside the MLP blocks using the new next-token objective produces stable updates that improve performance without degrading the model or needing other changes.","fun_headline_variants_meta":{"raw":{"variants":["In-Place TTT makes MLP final projections fast weights","In-Place TTT replaces reconstruction with next-token objective","Chunked updates enable In-Place TTT up to 128k contexts","Drop-in fast weight adaptation for LLMs via In-Place TTT"]},"model":"grok-4.3","cost_usd":0.010727,"raw_usage":{"total_tokens":4768,"prompt_tokens":739,"num_sources_used":0,"completion_tokens":72,"cost_in_usd_ticks":107274500,"prompt_tokens_details":{"text_tokens":739,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3957,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":739,"tokens_out":72,"duration_ms":138071,"temperature":1.0,"reasoning_tokens":3957,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-10T19:00:51.067928+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A direct comparison where a model with In-Place TTT fails to improve or worsens on long-context benchmarks relative to its non-adapting counterpart would falsify the central effectiveness claim.","supporting_citations":[],"review_version":1}