{"id":"bfec0530-5316-4cc5-b95b-c1178de8aaa9","arxiv_id":"2508.05640","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A request-level training data format eliminates duplicate user features, increasing storage efficiency and training throughput while enabling larger recommendation architectures.","lead":"This paper describes a way to store and train recommendation models that treats a user's request as one data sample instead of creating one sample per item shown. This removes duplicated user data, cuts storage and training cost, and makes it practical to train much larger models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The drop-in quality claim rests on unverified distributional parity of request-level joins; aggregate label mismatch rates (Table 3) could hide systematic bias in rare item types or long sessions.","rationale":"The paper's efficiency arithmetic is credible: deduplicating user-side features at request level genuinely removes redundant embedding lookups and communication, and the throughput numbers are directionally consistent with the B_NRO-to-B_RO reduction. The production deployment across multiple products is meaningful supporting evidence, as are the clearly described architecture changes. The central soft spot is causal attribution: Tables 6 and 7 change the data format, sample volume, and model architecture simultaneously, so the observed quality lifts do not establish that the request-level join itself is quality-neutral. The reader's weakest-assumption analysis correctly identifies the join-window distribution-preservation question. I would sharpen it by noting that the join algorithm can split long sessions and that Table 3's aggregate mismatch rates cannot rule out concentrated bias in rare but important label classes. The proposed controlled comparison using the ROO expansion adapter is feasible from material the paper already describes, and it would settle whether the efficiency gains are purchased with a training-distribution shift. This does not overturn the paper; it reinforces that conditional acceptance should require direct distributional-parity evidence in addition to the aggregate mismatch table.","tokens_in":16671,"tokens_out":7833,"duration_ms":89956,"concrete_test":"Use the ROO expansion adapter in Appendix C to expand ROO training samples back into impression-level samples. Train the same LSR or ESR architecture on (i) impression-level logged data, (ii) ROO data as-is, and (iii) ROO-expanded impression-level data, holding the number of samples, optimizer steps, and compute budget fixed. Evaluate all three on the same held-out impression-level test set. If (iii) differs from (i) beyond the product's stated significant NE threshold, the request-level join changes the training distribution and the efficiency gains are not quality-neutral; follow up by stratifying Table 3 mismatches by item category and conversion latency to check concentration.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that ROO yields efficiency gains with no model-quality cost—requires that request-level join windows preserve the impression-level training distribution. Section 2.1.2 reports only an average 16-minute gap between the first event in a request and the last item; the join pseudocode (Algorithm 1) closes windows on fixed time or engagement thresholds, so long sessions can be split into multiple ROO records and feedback arriving after close is lost or deferred. Section 2.1.3 defends parity solely with aggregate label mismatch rates (Table 3: 0.01% to 1.07%). Small averages can coexist with systematic bias in a rare but important slice—e.g., high-value conversions with long decision latency or long-dwell requests—in which case the 43–150% sample-volume increase and throughput gains are bought partly by changing what the model learns, not merely by removing duplication. The paper never reports a same-architecture, same-compute comparison of impression-level data versus ROO-derived data, so the quality lifts in Tables 6 and 7 cannot be attributed to the data format alone. Appendix B's assertion that ROO adds no data-landing latency also sits in tension with Section 2.1.2's admitted latency consideration, reinforcing the need for direct measurement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Request-Only Optimization (ROO), a training-data and modeling paradigm for industry-scale DLRMs. Instead of logging one training sample per impression, ROO logs one sample per user request, storing user-side (RO) features once and item-side (NRO) features as an array. The authors argue this removes feature duplication at the source, increases the number of training samples storable under fixed storage, reduces user-side compute and communication from O(B_NRO) to O(B_RO), and enables scaled-up user-side architectures such as UserArch and HSTU. They report production results across several Meta products: 43% to 150% more training samples, 32% to 570% training-throughput gains, and offline/online quality improvements for retrieval, early-stage, and late-stage ranking models. The paper includes a request-level join algorithm, data-quality mismatch tables, and implementation appendices.","tokens_in":16889,"tokens_out":3719,"duration_ms":40927,"significance":"If the reported results hold, ROO is a significant industrial contribution: it identifies a pervasive duplication inefficiency in impression-level training data, provides a clean data-format fix, and demonstrates large efficiency gains across multiple billion-user products. The core deduplication argument is simple and correct arithmetic: with 4-7 impressions per request, user-side computation is reduced from B_NRO to B_RO examples, and the amortization calculation for sequential encoders in Section 3.3 is explicit and parameter-free. The paper also gives a concrete pseudocode join algorithm and deployment details, which strengthens reproducibility of the method itself, even though the production code and data are proprietary. The main risk is that the central 'drop-in quality parity' claim rests on aggregate statistics that may not capture systematic biases in the request-level join, and the quality experiments do not cleanly isolate the data-format effect from the architectural changes it enables.","major_comments":[{"comment":"The claim that ROO data preserves the impression-level training distribution is not sufficiently supported. Section 2.1.3 and Table 3 report only aggregate label mismatch rates (0.01% to 1.07%) for conversion and video-view-duration labels. These aggregates can hide systematic bias for rare item types, long sessions, or delayed feedback. Algorithm 1 closes the join window on fixed time or engagement thresholds, so a long user session is split into multiple ROO records and any feedback arriving after close is either lost or deferred to a different sample. No per-slice analysis (e.g., by session length, item type, or feedback latency) is provided. This matters because the paper's efficiency gains are only valuable if the resulting training data does not change what the model learns; the current evidence is too coarse to establish that.","section":"Section 2.1.3 and Algorithm 1"},{"comment":"There is a direct contradiction about data-landing latency. Section 2.1.2 states that request-level joining 'could potentially increase activity-to-training latency' and that ROO samples' data landing latency is about half the fixed-time join window. Appendix B, however, asserts that 'ROO data does not require additional data infra feature, nor does it add any data landing latency increase.' These statements cannot both be true without clarification. If the fixed-time join window can be as long as tens of minutes, then ROO adds latency relative to impression-level logging, which is material for freshness-sensitive models. The paper should reconcile these statements and report the actual fixed-time window values and the resulting activity-to-training latency increase, not just an average 16-minute gap between the first and last event in a request.","section":"Section 2.1.2 vs. Appendix B"},{"comment":"The quality experiments do not isolate the effect of the ROO data format from the effect of the new architectures or the larger training sample volume. Table 6 compares HSTU (Impression) at 6.8x FLOPs per example with HSTU (ROO) at 0.99x, but there is no matched-compute comparison of the same architecture trained on impression-level data versus ROO data. Table 7 reports lifts from UserArch and HSTU on top of production models, but these architectures are enabled by ROO's efficiency, so the improvements could come from the architecture, the data format, the increased sample volume (Table 4), or any combination. To support the paper's stronger claim that ROO data is quality-neutral or quality-improving relative to impression data, the authors should include a same-architecture, matched-compute comparison (e.g., impression-level data expanded from ROO samples, or ROO data expanded with the Appendix C adapter) versus native impression-level data.","section":"Section 4.3 and Tables 6-7"}],"minor_comments":[{"comment":"The procedure name 'ReqestLevelJoin' is misspelled; it should be 'RequestLevelJoin'.","section":"Algorithm 1"},{"comment":"The custom tensor name is spelled 'InverseKeyedJaggeredTensor' in Appendix B but 'InverseKeyedJaggedTensors' in Section 5; please unify the spelling.","section":"Appendix B"},{"comment":"The sentence 'where 0.24% and 0.03% are significant, respectively' is ambiguous because the preceding sentence lists three metrics (consumption, topline) and two percentages; please state which significance threshold applies to which metric.","section":"Section 4.3.2"},{"comment":"The phrase 'data landing latency is about half the fixed-time join window' needs a definition of the fixed-time join window and typical values; otherwise the reader cannot assess the freshness impact.","section":"Section 2.1.2"},{"comment":"The reference-format block still contains the placeholder dates '2018' and 'Conference acronym ’XX'; this should be updated for the actual submission venue.","section":"ACM Reference Format"}],"recommendation":"major_revision","confidential_remarks":"This is a strong industrial-systems paper with credible arithmetic and large production-scale results, but the empirical claims are entirely from proprietary systems with no public code or data, which is normal for this venue. The main concern is the unverified distributional-parity assumption; the authors could address it with additional offline analyses of the request-level join (per-slice mismatch rates) and a cleaner same-compute comparison. The heavy reliance on the same group's HSTU architecture for quality gains is not improper, but it means the quality results are not an independent validation of the data format. I recommend major revision rather than rejection because the central efficiency contribution is sound and the missing evidence appears obtainable within the paper's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe central idea here is straightforward and genuinely useful: make the user request the unit of training data, with request-level (RO) features stored and processed once per request instead of once per impression. With 4-7 impressions per request, that cuts user-side embedding lookups, all-to-all communication, and storage roughly proportionally. The arithmetic is correct, the throughput numbers (32-570% depending on stage) are consistent with the mechanism, and the paper is honest about where the gains come from. The comparison in Table 6—HSTU on impression data at 6.8x FLOPs vs HSTU on ROO data at 0.99x, with slightly better recall—is the right kind of control and suggests the data format itself is not hurting quality.\n\nThe soft spots are real but not fatal. The biggest is the distribution-parity claim: request-level join preserves the impression-level training distribution, supported only by aggregate label mismatch rates (Table 3, 0.01-1.07%). Those averages could mask systematic problems for long sessions or slow-converting item types, especially since the join window closes on fixed time or engagement thresholds and late feedback is either lost or deferred. They should provide mismatch rates broken down by session length and label type. Table 6 is not a same-compute comparison, and the Table 7 quality lifts all come from new architectures (UserArch/HSTU) on top of ROO, so the data format's independent contribution to quality is not cleanly isolated. Online A/B metrics have no confidence intervals; 'significant' is asserted without variance. The abstract's 32-100% late-stage throughput range doesn't match Table 5 (max LSR is 50%). And Appendix B says ROO adds no data-landing latency, while Section 2.1.2 admits the join window costs about half the fixed-time window—a small inconsistency that should be reconciled. No code or data is released, which is normal for this kind of industry paper but does limit independent verification.\n\nOverall, the core claim holds up: request-level dedup is a real efficiency win, and the quality evidence is encouraging, not conclusive. This deserves a serious referee. I'd ask the authors to tighten the distributional-parity evidence, add variance estimates, and fix the minor inconsistencies. I'd bring it to the reading group and would cite it for the ROO data schema and the throughput analysis.","headline":"Credible industrial contribution: request-level training data is a genuine efficiency win, but the distribution-parity evidence is thinner than the headline claims.","tokens_in":17532,"tokens_out":4063,"would_cite":true,"duration_ms":42287,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By logging training data per user request instead of per impression, the paper claims recommender systems can cut user-side computation from one pass per impression to one pass per request.","keywords":["request-only optimization","request-level training data","feature deduplication","recommender systems","DLRM training","HSTU","UserArch","training throughput"],"falsifier":"Compare two models trained on the same production stream, one on impression-level data and one on ROO data with a shortened join window, and measure per-item-type label recall and offline normalized entropy for item categories whose feedback typically arrives after the window closes; if ROO systematically degrades recall for those categories or its normalized-entropy advantage reverses, the drop-in equivalence claim fails. The paper's Table 3 reports only conversion and video-view-duration mismatches, so a mismatch measurement on rarer, slower labels such as long-term engagement or purchase would be a direct test.","tokens_in":16473,"feed_emoji":"⚡","tokens_out":6317,"duration_ms":62845,"temperature":0.7,"pith_summary":"The paper tries to establish that deep learning recommendation models train better and cheaper when the unit of training data is a user request rather than a single impression. In current pipelines, one request that produces four to seven items yields four to seven nearly identical training samples, each carrying a full copy of the user's features. Collapsing these into one request-level sample removes that duplication at the source, increases usable training volume by 43% to 150% under fixed storage, and raises training throughput by 32% to 570% across production models. The authors further argue that the saved compute is what makes heavy sequential user-history architectures such as HSTU and UserArch practical to train. If right, the paper shifts the default unit of DLRM data engineering from impression-level to request-level logging, storage, and model design.","feed_headline":"Request-level training data cuts recommendation compute up to 6x","feed_subtitle":"Same storage yields 43–150% more training samples and enables bigger user-history models.","key_machinery":"The load-bearing object is the request-level training sample produced by the request-level joiner. It separates request-only (RO) features, one copy of user-side dense, id-list, and sequence features, from non-request-only (NRO) features, an array of item-side features with an impressions-per-sample count tensor. This schema does three jobs at once: it eliminates feature duplication at logging time, it lets tensor preprocessing and embedding lookup run the RO part once per request instead of once per impression, and it gives model architects a place to put expensive user-side computation, such as UserArch's Linear Compress Embedding and HSTU self-attention over the user sequence, whose cost is amortized across all impressions in the request. The reduction from $B_{NRO}$ to $B_{RO}$ examples is the mechanism that carries both the storage and the throughput claims.","core_discovery":"On its own terms, this paper discovers that the whole DLRM pipeline can be restructured around a request-level sample schema in which user-side (RO) features appear once per request and item-side (NRO) features appear as an array, one entry per impression in that request. A request-level joiner buffers user-item events by request id until the join window closes, then emits a single compact example. Because user-side features dominate the feature space and are identical across impressions of one request, user-side training computation and all-to-all communication drop from $O(B_{NRO})$ to $O(B_{RO})$, with $B_{NRO}/B_{RO}$ typically 4 to 7; in production this appears as multi-fold throughput gains and as the ability to run generative-recommender-style sequential models at 7x the FLOPs under the same training compute. The paper also reports offline and online quality gains from ROO-based architectures, including up to 4.77% lift in a retrieval consumption metric and 0.04% to 0.25% topline lifts in late-stage ranking.","pith_inferences":["If the join-window distributional parity claim generalizes beyond the measured labels, conversion and video view duration, ROO could also change how online-training freshness is budgeted, since the request-level join window of about 16 minutes becomes a tunable freshness and latency knob rather than a fixed logging artifact.","The same amortization argument should transfer to any multi-candidate training or inference setting where one context is scored against many items, such as ads, search, and feed surfaces, so the 4-7 impressions-per-request ratios reported here may understate gains on surfaces with longer candidate lists.","A testable extension is to combine ROO with target-aware sequence selection methods: request-level amortization removes one of the main cost objections to per-candidate attention, so the comparison between amortized pairwise attention and HSTU-style self-attention becomes an empirical question the paper does not settle."],"forward_implications":["Retrieval and early-stage ranking models, whose two-tower design puts most cost in the user tower, can see 220% to 570% training-throughput gains from ROO with minimal code changes.","Late-stage ranking models gain 32% to 100% throughput, and the freed compute can be spent on user-side compression (UserArch) and sequential user modeling (HSTU) that improve offline normalized entropy and online metrics.","The same storage budget holds 43% to 150% more training samples, reducing the need for downsampling in data logging.","ROO unifies training and inference feature preprocessing, removing client-side user-feature broadcast and server-side deduplication in serving.","ROO expansion adapters provide backward compatibility, letting existing impression-level models consume ROO data while a fleet migrates."],"supporting_citations":[{"why":"Establishes that user-side sparse features dominate DLRM training samples, the premise for request-level deduplication.","marker":"[29]"},{"why":"Supplies HSTU and the generative-recommender sequential modeling formulation whose cost ROO amortizes.","marker":"[42]"},{"why":"Provides RecD, the existing deduplication pipeline that ROO compares against on storage, latency, and preprocessing overhead.","marker":"[46]"},{"why":"Supplies the feature-flattened columnar storage format that the ROO schema leverages for compression.","marker":"[45]"},{"why":"Provides TorchRec's KeyedJaggedTensor and variable-length batch sharding used to represent RO and NRO tensors.","marker":"[19]"},{"why":"Documents single-user-multiple-candidates inference deduplication that ROO extends to training.","marker":"[40]"},{"why":"Defines normalized entropy, the offline metric used for the ranking quality comparisons.","marker":"[16]"},{"why":"Supplies the database-normalization framing, second normal form, that motivates the schema redesign.","marker":"[5]"}],"fun_headline_variants":["Request-level batching cuts recommender compute 4-7x","Request-only storage yields up to 150% more samples","Request-only data enables 7x larger user-history models","Deduplicate user features by grouping impressions per request","Request-only paradigm cuts storage and compute in recommender training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that joining at the request level, waiting roughly 16 minutes for a request's items to arrive, does not systematically change which labels and user signals appear in the training data; if late-arriving feedback is concentrated on particular item types, the efficiency gains could come with a quality loss that the aggregate mismatch rates of 0.01% to 1.07% would miss.","fun_headline_variants_meta":{"raw":{"variants":["Request-level batching cuts recommender compute 4-7x","Request-only storage yields up to 150% more samples","Request-only data enables 7x larger user-history models","Deduplicate user features by grouping impressions per request","Request-only paradigm cuts storage and compute in recommender training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000588,"raw_usage":{"total_tokens":2800,"prompt_tokens":1025,"completion_tokens":1775,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":1693}},"tokens_in":641,"tokens_out":1775,"duration_ms":17326,"temperature":1.0,"reasoning_tokens":1693,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:38:59.331870+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare two models trained on the same production stream, one on impression-level data and one on ROO data with a shortened join window, and measure per-item-type label recall and offline normalized entropy for item categories whose feedback typically arrives after the window closes; if ROO systematically degrades recall for those categories or its normalized-entropy advantage reverses, the drop-in equivalence claim fails. The paper's Table 3 reports only conversion and video-view-duration mismatches, so a mismatch measurement on rarer, slower labels such as long-term engagement or purchase would be a direct test.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides RecD, the existing deduplication pipeline that ROO compares against on storage, latency, and preprocessing overhead."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the feature-flattened columnar storage format that the ROO schema leverages for compression."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides TorchRec's KeyedJaggedTensor and variable-length batch sharding used to represent RO and NRO tensors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the database-normalization framing, second normal form, that motivates the schema redesign."}],"review_version":1}