{"id":"bb57aa70-6916-4b93-82d6-5b4cc17ef446","arxiv_id":"2606.05257","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Across 600 runs from 10^15 to 10^19 FLOPs, behavioral models show a 2% embedder is compute-optimal at all scales, training is data-heavy at low compute, and optimal negatives increase with budget until memory-limited.","lead":"The paper measures how performance of behavioral foundation models on user event sequences scales with compute across hundreds of training runs on real interaction data. A smart generalist might read it to calibrate training budgets and negative sampling for recommendation, payments, or fraud systems.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"Claim that embedder repetition exposure drives s*≈2% optimality rests on untested assumption about item repetition rates in the data.","rationale":"Reader's weakest assumption already flags generalization beyond the specific real datasets. The repetition mechanism is the precise causal link offered for the 2% figure, so the synthetic-repetition check directly tests whether that link holds. This moves the verdict from UNVERDICTED to CONDITIONAL rather than full acceptance until the isolation experiment is performed.","tokens_in":1811,"tokens_out":359,"duration_ms":22577,"concrete_test":"Construct synthetic event sequences with the same sequence length distribution but three different item-frequency exponents (α=0.8, 1.2, 1.6). At a fixed budget of ~10^17 FLOPs, sweep embedder fraction s from 1% to 10% for each α; if the minimizing s shifts by >1.5 percentage points across α values, the repetition-based explanation does not generalize.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim states that s*≈2% is optimal 'because embedder parameters are both more expensive per step and exposed to far more repeated items than contextualizer parameters.' This explanation is load-bearing for the 'why' part of the result. All 600 runs use real interaction datasets, which typically exhibit high item repetition (popular items appear across many user sequences). The paper does not report controlled experiments that vary repetition rate (e.g., via synthetic data with adjustable Zipf exponents) while holding other factors fixed. Without that isolation, it remains possible that the observed optimum is an artifact of the repetition statistics rather than a general architectural property of the two-part model.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that for two-part behavioral foundation models (feature-based event embedder + decoder-only transformer) trained on user event sequences, a small embedder fraction s* ≈ 2% of total parameters is compute-optimal across all tested budgets from 10^15 to 10^19 FLOPs. This is attributed to embedder parameters being more expensive per step and exposed to more repeated items. The work also reports that compute-optimal D/N ratios start data-heavy relative to Chinchilla but approach it at higher compute, that critical batch size and optimal negative count after freezing shift with scale, and that sampled loss and ranking metrics disagree in ways that themselves scale with compute and metric choice. These conclusions rest on ~600 runs on real interaction datasets jointly varying parameter split, batch size, model/data allocation, and negative count.","tokens_in":1968,"tokens_out":543,"duration_ms":21048,"significance":"If the central empirical trends hold, the paper supplies the first large-scale compute-optimal calibration for behavioral models on event sequences, directly relevant to recommendation, payments, and fraud domains. The scale of the experimental campaign (600 runs spanning five orders of magnitude in FLOPs) is a clear strength and provides substantial empirical support for the observed trends in embedder fraction and negative-sampling preferences.","major_comments":[{"comment":"Abstract: the explanatory claim that s* ≈ 2% optimality arises 'because embedder parameters are both more expensive per step and exposed to far more repeated items than contextualizer parameters' is load-bearing for the interpretation. All 600 runs use real interaction datasets that exhibit high item repetition; the manuscript reports no controlled experiments that vary repetition rate (e.g., synthetic data with adjustable Zipf exponents) while holding other factors fixed, leaving open the possibility that the observed optimum is an artifact of the repetition statistics rather than a general architectural property.","section":"Abstract"},{"comment":"Abstract and experimental description: the 2% optimality claim and the scaling trends for critical batch size and negative count lack reported error bars, statistical significance tests for the 2% figure, explicit data-exclusion rules, or analysis of whether post-hoc metric choices affect the central trends. These omissions make it difficult to assess robustness of the reported optima.","section":"Abstract"}],"minor_comments":[{"comment":"Abstract: the symbol s* is used before any definition or parenthetical explanation; a brief inline clarification would improve readability for readers encountering the abstract first.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive review and detailed comments. We address each major point below, indicating planned revisions where appropriate.","responses":[{"response":"We agree that our experiments are confined to real interaction datasets exhibiting high item repetition and that we did not perform controlled synthetic experiments varying repetition rates (e.g., via adjustable Zipf exponents). The 2% optimum and its proposed mechanism are therefore tied to the statistical properties of the real data used. While the trend is consistent across multiple distinct real-world datasets, this does not fully rule out dataset-specific artifacts. We will revise the abstract and discussion sections to present the explanation as a hypothesis grounded in architectural differences and observed repetition patterns in behavioral data, rather than a proven general causal factor, and will explicitly note the lack of synthetic controls as a limitation.","revision_made":"partial","referee_comment":"[Abstract] Abstract: the explanatory claim that s* ≈ 2% optimality arises 'because embedder parameters are both more expensive per step and exposed to far more repeated items than contextualizer parameters' is load-bearing for the interpretation. All 600 runs use real interaction datasets that exhibit high item repetition; the manuscript reports no controlled experiments that vary repetition rate (e.g., synthetic data with adjustable Zipf exponents) while holding other factors fixed, leaving open the possibility that the observed optimum is an artifact of the repetition statistics rather than a general architectural property."},{"response":"We acknowledge these gaps in the current version. In revision we will add error bars (computed from replicate runs where available) to all figures reporting the 2% optimum and scaling trends for batch size and negative count. We will include statistical significance tests for the identified optima. Data-exclusion criteria (based on convergence thresholds and outlier detection) will be stated explicitly in the experimental section. We will also add an analysis of how the central trends vary with different post-hoc metric choices and report the sensitivity of the scaling conclusions to metric selection.","revision_made":"yes","referee_comment":"[Abstract] Abstract and experimental description: the 2% optimality claim and the scaling trends for critical batch size and negative count lack reported error bars, statistical significance tests for the 2% figure, explicit data-exclusion rules, or analysis of whether post-hoc metric choices affect the central trends. These omissions make it difficult to assess robustness of the reported optima."}],"tokens_in":1572,"tokens_out":512,"duration_ms":28967,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core result is that across 600 runs on interaction sequences, a two-part model with roughly 2% of parameters in the embedder is compute-optimal at every scale tested, training is data-heavy at low compute before approaching Chinchilla ratios, and loss versus ranking metrics diverge more as compute grows. Negative count preferences also shift with budget until memory bounds them.\n\nThe experiment jointly sweeps the parameter split, batch size, data allocation, and post-freeze negatives over five orders of magnitude in FLOPs. That joint variation on real recommendation-style data is the main addition relative to language-model scaling work. The observation that the choice of evaluation metric changes the optimal recipe is a useful practical note.\n\nThe repetition-based account for why the embedder should be small is reasonable given how item frequencies work in these datasets, but the runs do not include controlled variation of repetition rate, so the causal link stays observational rather than isolated. Details on error bars, data exclusion, and whether metric choices were pre-specified would help judge how stable the reported optima are.\n\nThis is aimed at teams training behavioral models for commerce or recommendations who need allocation heuristics beyond text scaling laws. The empirical volume is large enough that a serious referee should see it, even if the explanatory part would benefit from tighter controls.","headline":"The paper maps practical compute splits for embedder-transformer behavioral models on real event data, with a consistent 2% embedder optimum and metric-dependent trends, but the repetition explanation for that optimum is not isolated from the data statistics.","tokens_in":2447,"tokens_out":353,"would_cite":true,"duration_ms":17209,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A small embedder of about 2% of parameters is compute-optimal at every budget for behavioral foundation models on user event sequences.","keywords":["scaling laws","behavioral foundation models","user event sequences","embedder parameters","compute optimal allocation","negative sampling","ranking metrics","recommendation models"],"falsifier":"A new set of scaling runs on different user event data or with a different architecture showing that the optimal embedder fraction changes substantially with compute budget or is not around 2%.","tokens_in":2704,"feed_emoji":"","tokens_out":708,"duration_ms":38185,"temperature":0.7,"pith_summary":"The paper investigates scaling laws for training foundation models on sequences of user actions in areas like recommendation and commerce. It uses a two-part model with an event embedder and a transformer contextualizer, varying the parameter allocation, batch size, data allocation, and negative samples across 600 runs spanning many orders of compute. The key finding is that a small embedder fraction is always optimal because those parameters are costlier per step and see more repeats. It also reveals that the best data-to-parameter ratio shifts with scale and that loss and ranking metrics diverge in a scale-dependent way, making the metric choice part of the scaling law.","feed_headline":"Tiny embedder is compute-optimal for user sequence models","feed_subtitle":"Only 2 percent of parameters in the embedder minimizes compute across 10^15 to 10^19 FLOPs on real interaction data.","key_machinery":"The parameter split between the feature-based event embedder and the decoder-only transformer, which determines the optimal allocation because of differing per-step costs and repetition rates.","core_discovery":"Across 600 runs on real interaction data from 10^15 to 10^19 FLOPs, the optimal embedder size s* is approximately 2% of total parameters at every compute budget. Embedder parameters are more expensive per training step and are exposed to far more repeated items than the contextualizer parameters. As a result, compute-optimal training starts data-heavy relative to language models but the D/N ratio approaches the Chinchilla heuristic at higher compute. The sampled training objective and ranking metrics disagree in ways that scale with compute and metric choice, with larger budgets preferring more negatives until memory limits are hit.","pith_inferences":["Production systems could use this to allocate fewer parameters to embeddings and reduce overall training cost.","Similar scaling might hold for other sequence models with high item repetition.","Future work could test if unifying the embedder and contextualizer changes the optimal split.","Disagreement scaling suggests loss functions may need to be adjusted or combined with ranking objectives at large scales."],"forward_implications":["Embedder size should stay small even as total model size grows.","Data allocation should be larger relative to parameters at smaller compute budgets.","Negative sample count should increase with scale until candidate memory becomes the bottleneck.","The choice of ranking metric affects the optimal training configuration.","Scaling laws for these models must incorporate the evaluation metric as a variable."],"fun_headline_variants":["Embedder at 2% parameters optimal for all tested budgets","Compute optimal split allocates 2% to user event embedder","Scaling laws indicate 2% embedder optimal on real data","For user sequences 2% embedder minimizes training compute","2% embedder parameters optimal across interaction data scales"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The trends in optimal embedder fraction and scaling behaviors will continue to hold outside the specific real interaction datasets and two-part architecture used in the experiments.","fun_headline_variants_meta":{"raw":{"variants":["Embedder at 2% parameters optimal for all tested budgets","Compute optimal split allocates 2% to user event embedder","Scaling laws indicate 2% embedder optimal on real data","For user sequences 2% embedder minimizes training compute","2% embedder parameters optimal across interaction data scales"]},"model":"grok-4.3","cost_usd":0.007476,"raw_usage":{"total_tokens":3493,"prompt_tokens":790,"num_sources_used":0,"completion_tokens":81,"cost_in_usd_ticks":74762000,"prompt_tokens_details":{"text_tokens":790,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2622,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":790,"tokens_out":81,"duration_ms":18969,"temperature":1.0,"reasoning_tokens":2622,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T06:46:47.207338+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A new set of scaling runs on different user event data or with a different architecture showing that the optimal embedder fraction changes substantially with compute budget or is not around 2%.","supporting_citations":[],"review_version":1}