{"id":"7957ae7a-ddff-4783-b3d9-f08cec62378d","arxiv_id":"2501.09706","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Continued pretraining Llama 3.1 on 1 trillion e-commerce tokens produces e-Llama 8B/70B models that improve in-house e-commerce benchmarks by roughly 25-30% while retaining most general-domain accuracy.","lead":"eBay trained two large language models, e-Llama 8B and 70B, by continuing to pretrain Llama 3.1 on 1 trillion tokens of e-commerce data. The models score about 25 to 30 percent better on e-commerce-specific tests with only modest losses on general benchmarks, and merging with the base model allows fine control over that trade-off.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"E-commerce gains may reflect memorization of training listings rather than domain adaptation; the author-built benchmarks have no reported contamination control.","rationale":"The reader's weakest assumption—that the five novel e-commerce benchmarks are valid measures of e-commerce competence rather than tests of in-distribution memorization—matches my own reading. The paper's central positive result is measured exclusively on these author-constructed tasks, and the training data is drawn from the same eBay listing distribution. The absence of any overlap analysis or external benchmark makes the improvements in Table 4 vulnerable to the objection that the model has memorized exact or near-duplicate listings. This is the single most load-bearing concern because if it lands, the main contribution ('adapted towards the e-commerce domain') is unsupported; if it is resolved by contamination-controlled evaluation, the paper's central claim stands on much firmer ground. I also note a secondary issue: the abstract's 'without sacrificing significant performance' is overstated for the 8B model, which drops 4.6 points on the Open LLM Leaderboard English aggregate (17.2 to 12.6 in Table 4). That overstatement weakens the general-performance claim but does not invalidate it, whereas the benchmark contamination issue affects the core positive result. The reader's CONDITIONAL verdict already accounts for this uncertainty, so no verdict change is needed.","tokens_in":12364,"tokens_out":4765,"duration_ms":51068,"concrete_test":"Build a contamination-controlled evaluation. For every evaluation example in the five e-commerce tasks, compute exact and fuzzy overlap (e.g., 13-gram Jaccard similarity or embedding proximity) against the continued-pretraining corpus, including eBay listings and the Fineweb-derived subset. Remove any evaluation item with a near-duplicate in training, then recompute the Table 4 e-commerce accuracies for Llama-3.1 and e-Llama 8B/70B. If the 25%/30% gains shrink to near zero on the deduplicated subset, memorization explains the result. Alternatively or additionally, evaluate on listings timestamped after the training-data cutoff or on a public non-eBay e-commerce dataset; if the gains persist there, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two premises: (1) e-Llama improves e-commerce benchmarks by about 25% (English) and 30% (non-English) over Llama-3.1, and (2) general-domain performance is roughly preserved. Premise (1) rests entirely on five novel tasks (AP, APMC, PPMC, MCA, MCAMC) introduced in Section 3.3, which are constructed from eBay listing data. Section 3.2 states that continued pretraining uses 'listings and product reviews from the eBay website'—the same data source. The paper reports no heldout protocol, no deduplication, no temporal split, and no train/evaluation overlap analysis for these benchmarks. The task formats amplify the risk: AP and MCA ask the model to generate the most probable continuation of aspect key-value pairs for a listing, while APMC, PPMC, and MCAMC ask it to identify the correct value among corrupted alternatives. If the exact title, price, or aspect values appeared in the 1-trillion-token training corpus, these tasks reward retrieval rather than domain competence. The paper's limitation (iii) concedes that the evaluation suite could be improved in diversity and quantity, but the deeper concern is not quantity—it is that the benchmarks may be in-distribution probes of the training data. Without external validation or contamination analysis, the abstract's assertion that the Llama 3.1 models are 'adapted towards the e-commerce domain' is supported only by author-constructed, same-distribution benchmarks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents e-Llama 8B and 70B, obtained by continued pretraining of Llama 3.1 base models on 1 trillion tokens of a mixture containing 50% e-commerce data (eBay listings, product reviews, and classifier-selected Fineweb content) and 50% general-domain data. The authors describe hyperparameter studies (learning rate, e-commerce data ratio, context size) and define five new multilingual e-commerce evaluation tasks (AP, APMC, PPMC, MCA, MCAMC). They report that e-Llama improves e-commerce benchmark accuracy by about 25% on English and 30% on non-English tasks relative to Llama 3.1, while general NLU aggregates remain roughly flat, though the 8B model drops notably on the Open LLM Leaderboard English aggregate. They also propose linear interpolation (model merging) between the base and adapted checkpoints as a way to control the general/domain performance trade-off.","tokens_in":12610,"tokens_out":3585,"duration_ms":39479,"significance":"If the central empirical claim is valid, the paper provides a practically useful recipe for large-scale domain adaptation of 8B and 70B foundation models, with unusually large training scale (1T tokens) and a useful set of ablations. The learning-rate and data-mixing experiments are informative, and the model-merging result is a simple but potentially valuable practical tool. The main significance, however, is conditional: all e-commerce gains are measured on author-constructed tasks built from the same data source as the continued-pretraining corpus, and the paper reports no contamination analysis, no external benchmark, and no heldout description for these tasks. The general-domain preservation claim is also overstated for the 8B model in light of a 4.6-point drop on the Open LLM Leaderboard English aggregate. The paper is therefore a useful engineering report whose central conclusion requires additional validation before it can be fully accepted.","major_comments":[{"comment":"The five e-commerce benchmarks introduced in Section 3.3 are constructed from eBay listing data, while Section 3.2 states that continued pretraining uses 'listings and product reviews from the eBay website.' The paper gives no heldout split, no deduplication procedure, no temporal split, and no analysis of overlap between the evaluation instances and the 1-trillion-token training corpus. For tasks like AP and MCA, which ask the model to generate the most probable continuation of aspect key-value pairs, and APMC/PP/MCAMC, which ask it to select the correct value among corrupted alternatives, high scores may reflect retrieval of memorized training strings rather than generalizable e-commerce competence. This is load-bearing for the abstract's claim that the models are 'adapted towards the e-commerce domain.' The limitation statement (iii) acknowledges that evaluation diversity could be improved, but the issue is not only diversity: it is the lack of any contamination control or external validation. I ask the authors to add a contamination analysis (e.g., n-gram overlap statistics, deduplication, or a temporal split) or to evaluate on an independent e-commerce benchmark suite to demonstrate that the measured gains are not in-distribution fitting.","section":"§3.2, §3.3, §7"},{"comment":"The abstract states that the models can be adapted 'without sacrificing significant performance on general domain tasks,' but Table 4 shows that the 8B model drops from 17.2 to 12.6 on the Open LLM Leaderboard English aggregate, a 4.6-point (approximately 27% relative) decrease. The NLU English aggregate is stable (71.8 to 71.6), but the leaderboard drop is substantial and should be reported prominently. The sentence should be revised to acknowledge the model-size-dependent degradation, or the authors should provide statistical significance or confidence intervals to support the word 'significant.' As written, the general-domain preservation claim is too strong for the 8B model.","section":"§5.2, Table 4, Abstract"},{"comment":"The data-weighting ablation is run for only about 30 billion tokens and reports only a single averaged 'general benchmark' score (e.g., 43.2 to 43.3 across ratios). The paper concludes that varying the e-commerce ratio 'does not effect the model scores on the general domain benchmarks,' but this conclusion is not supported by the short training horizon or by the aggregate-only reporting. Since the final 1-trillion-token model does show a measurable leaderboard drop, the ablation should at least report per-benchmark numbers and, ideally, longer training runs, so that the choice of the 50% e-commerce ratio can be properly evaluated against the final-model trade-off.","section":"§4.2, Table 3"}],"minor_comments":[{"comment":"The task abbreviations are inconsistent: the text uses 'PP MC' and 'PP MC' in the task list, while Table 4 and Appendix A.1 use 'PPMC' and 'PP MC'; please standardize to a single abbreviation.","section":"§3.3 and Appendix A.2"},{"comment":"The learning-rate comparison reports only the averaged general benchmark score. Since the final claim concerns the preservation of general capabilities, including the separate NLU and LLM Leaderboard components in Table 2 would make the learning-rate choice more transparent and reduce reliance on a potentially misleading average.","section":"§4.1, Table 2"},{"comment":"Figure 1 plots perplexity against input sequence length, but the figure caption does not state whether the heldout test set is general-domain or e-commerce data, nor how many sequences are used. Please add this information, as the interpretation of the long-context behavior depends on it.","section":"§4.3, Figure 1"},{"comment":"The hardware description gives node and GPU counts but no total training time or estimated FLOPs. Reporting wall-clock time or throughput would help readers assess the practical cost of the 1-trillion-token continued pretraining and would better contextualize the paper's scale claims.","section":"§3.1"},{"comment":"The model-merging figure shows a nearly linear trade-off, but the caption does not report the evaluation tasks behind the 'general' and 'e-commerce' aggregates, nor whether this linearity holds for the 70B model. Please clarify which benchmarks are used in the figure and whether the result is consistent across model sizes.","section":"§5.3, Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The central concern is benchmark validity: the e-commerce evaluation tasks are built from eBay data, the same source as the training corpus, and the paper provides no contamination analysis or external validation. This is a fixable issue within the manuscript's scope if the authors can add overlap statistics or evaluate on an independent task suite. If they cannot, the abstract's main claim about successful domain adaptation should be substantially weakened. The general-domain preservation claim is also overstated for the 8B model. I recommend major revision rather than rejection, because the ablation studies and the merging analysis are useful contributions that can stand even if the e-commerce evaluation claims are made more modest."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The practical core of this paper is solid: continued pretraining Llama-3.1 8B/70B on 1T tokens of e-commerce data, with careful ablations of learning rate, data mix, and context size. The model-merging result—near-linear interpolation between base and adapted checkpoints for fine-grained trade-off—is genuinely useful and costs almost nothing. The five multilingual e-commerce tasks fill a real gap, since prior e-commerce work focused on instruction tuning. Credit also for reporting the 8B model's 4.6-point drop on the English LLM Leaderboard aggregate; that is honest and matches the abstract's claim far better than a selective table would.\n\nThe soft spot is exactly where the stress-test lands. The e-commerce benchmarks are built from eBay listing data, the same broad distribution as the continued-pretraining corpus. The paper gives no deduplication, no temporal split, and no train/evaluation overlap analysis for these tasks. AP and MCA are open-ended continuations of aspect-key–value strings; APMC, PPMC, and MCAMC are corruption-detection tasks. If the exact titles, prices, or aspect values appear in the training tokens, these tasks reward retrieval rather than domain competence. The authors' limitation (iii) mentions improving evaluation diversity, but the deeper issue is validity, not quantity. This does not sink the paper—the tasks are not trivial, and the per-language scores in the appendix look like real adaptation—but it should push the central claim from \"adapted to e-commerce\" to \"adapted to eBay-like listings,\" at least until external benchmarks or contamination controls appear.\n\nMinor issues: no error bars or significance tests across the multiple evaluation runs, and the abstract's \"without sacrificing significant performance\" is a stretch given the 8B drop. The saturated 8B learning curve after 20% of training is noted but not explored, which is a shame.\n\nRecommendation: this deserves a serious referee. The recipe and benchmarks will be cited by practitioners even if the evaluation validity is unresolved. A referee should ask for heldout protocols or an external validation step before the strong version of the claim is accepted.","headline":"A useful industrial-scale continued-pretraining recipe with new benchmarks, but the author-built e-commerce tasks are too close to the training data to fully support the abstract's claim.","tokens_in":13197,"tokens_out":1540,"would_cite":false,"duration_ms":17799,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Continued pretraining of Llama 3.1 on one trillion e-commerce tokens yields 8B and 70B models that gain roughly 25–30% on new e-commerce benchmarks while general-language scores stay nearly flat.","keywords":["continued pretraining","domain adaptation","e-commerce","foundation models","Llama 3.1","catastrophic forgetting","model merging","multilingual evaluation"],"falsifier":"Rebuild the five e-commerce tasks on a held-out sample of marketplace listings that were created after the training data was collected, or from a different marketplace, and compare the adapted models against the base models on that sample; if the accuracy gains shrink to a few points, the reported adaptation gains are largely memorization of the training distribution rather than transferable e-commerce knowledge.","tokens_in":12148,"feed_emoji":"🛒","tokens_out":7777,"duration_ms":70427,"temperature":0.7,"pith_summary":"Large language models trained on general web text know little about the specifics of online marketplaces — product categories, aspect attributes, prices, and how listings are phrased. The paper asks whether a general-purpose model can be turned into an e-commerce foundation model by continued pretraining, and whether that can be done without destroying its general abilities. It answers yes: after training 8B and 70B Llama 3.1 models on one trillion tokens of mixed e-commerce and general data, the adapted models improve average accuracy on five new multilingual e-commerce tasks by about 25% in English and 30% in other languages, while standard NLU aggregates move by about one point. The recipe that makes this work is a 50/50 domain/general data mix, a learning rate about one tenth of the original, and a short context window. A final trick — averaging the weights of the base and adapted models — gives a near-linear dial for trading domain gain against general competence.","feed_headline":"Adapting Llama 3.1 to e-commerce lifts domain scores up to 30%","feed_subtitle":"8B and 70B models gain 25-30% on new e-commerce tasks while standard NLU scores move about one point.","key_machinery":"Three components carry the argument. Continued pretraining with a carefully chosen setup — a 50/50 mix of e-commerce and general-domain data at a maximum learning rate of about 10% of the original pretraining rate — injects domain knowledge while replaying general data to limit forgetting. The evaluation is a set of five new multilingual benchmarks built from marketplace listings: aspect prediction, aspect prediction multiple choice, price prediction multiple choice, most common aspects, and most common aspects multiple choice. Finally, model merging — averaging the parameters of the base and adapted checkpoints — provides a compute-free control knob, since the resulting performance varies almost linearly with the interpolation weight.","core_discovery":"The central discovery is that domain adaptation of a modern foundation model can be done at trillion-token scale without catastrophic forgetting, provided the training setup is tuned explicitly for retention. The e-Llama models are the evidence: continued pretraining of Llama 3.1 8B and 70B on one trillion tokens (50% e-commerce, 50% general data, learning rate capped at about one tenth of the base pretraining value) raises the average accuracy across the paper's five new e-commerce benchmarks from 35.8 to 46.8 for the 8B model and from 40.4 to 52.8 for the 70B model, corresponding to roughly 25–30% relative gains. On general-language aggregates the models stay within about one point, with one caveat: the 8B variant drops 4.6 points on the English leaderboard aggregate, a loss the authors attribute to smaller model size and distribution differences in their general data. The paper's additional claim is that merging the adapted and base checkpoints by simple weight averaging produces a near-linear performance curve, so practitioners can dial the domain/general balance without retraining.","pith_inferences":["The headline gain is only as trustworthy as the five new benchmarks; since they are constructed from the same marketplace data used for training and no overlap analysis is given, some of the improvement could reflect memorization of the training distribution rather than transferable e-commerce knowledge.","The near-linear merging curve suggests the base and adapted models occupy a shared loss basin; if that holds for other domain pairs, weight averaging could become a general control mechanism for continued pretraining beyond e-commerce.","The paper evaluates only base models, so the natural next test is whether the domain gains survive instruction tuning — the authors list this as future work, but it is also the setup that would matter for most real deployments.","Because the paper's general-domain data mix differs from the original pretraining distribution, the 'no significant sacrifice' claim is benchmark-dependent; the 8B leaderboard drop of 4.6 points shows the trade-off is real even when aggregate NLU scores look flat."],"forward_implications":["A general-purpose base model can be repurposed for a vertical domain with one trillion tokens of mixed data, which is far cheaper than pretraining from scratch.","The 70B model forgets less than the 8B model while learning the domain at least as well, so larger models are the safer choice when both domain gain and general retention matter.","Weight averaging gives an exact, inference-time-free way to pick a point on the domain/general trade-off curve, so a single training run can serve multiple product requirements.","The five e-commerce benchmarks give the field a reusable multilingual evaluation for foundation-model e-commerce knowledge, filling a gap left by instruction-tuning-only evaluations.","Because the non-English benchmarks improve more than the English ones, the adaptation also transfers knowledge across languages within the marketplace domain."],"supporting_citations":[{"why":"Provides the Llama 3.1 base models and the pretraining learning-rate schedule from which the adaptation starts.","marker":"Dubey et al., 2024"},{"why":"Supplies the e-commerce and general-domain data mixture and the NLU evaluation aggregates that the paper follows.","marker":"Herold et al., 2024"},{"why":"Motivates the learning-rate and data-ratio experiments that determine the final training setup.","marker":"Ibrahim et al., 2024"},{"why":"Provides the web corpus from which e-commerce examples are extracted for part of the domain data.","marker":"Penedo et al., 2024"},{"why":"Provides the classifier-based data-filtering idea used to extract e-commerce text from the web corpus.","marker":"Lozhkov et al., 2024"},{"why":"Defines the English general-domain leaderboard benchmark on which the adapted models show a small degradation.","marker":"Fourrier et al., 2024"},{"why":"Provides the evaluation harness used to compute the NLU benchmark scores.","marker":"Gao et al., 2023"},{"why":"Introduces the weight-averaging technique used for model merging.","marker":"Wortsman et al., 2022"}],"fun_headline_variants":["Trillion-token tune-up: Llama 3.1 masters e-commerce, keeps general smarts","e-Llama: 25-30% domain boost with near-flat general scores","Adapt Llama 3.1 to e-commerce without forgetting: 1T tokens, 30% gain","Merging checkpoints dials e-commerce vs. general trade-off","8B and 70B Llama adapt to e-commerce, gain up to 30%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole 'adapted to e-commerce' conclusion rests on the five new e-commerce benchmarks being fair, unbiased tests of e-commerce knowledge rather than easy to game by memorizing the training data.","fun_headline_variants_meta":{"raw":{"variants":["Trillion-token tune-up: Llama 3.1 masters e-commerce, keeps general smarts","e-Llama: 25-30% domain boost with near-flat general scores","Adapt Llama 3.1 to e-commerce without forgetting: 1T tokens, 30% gain","Merging checkpoints dials e-commerce vs. general trade-off","8B and 70B Llama adapt to e-commerce, gain up to 30%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000529,"raw_usage":{"total_tokens":2555,"prompt_tokens":955,"completion_tokens":1600,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":1480}},"tokens_in":571,"tokens_out":1600,"duration_ms":11547,"temperature":1.0,"reasoning_tokens":1480,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:44:00.489598+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the five e-commerce tasks on a held-out sample of marketplace listings that were created after the training data was collected, or from a different marketplace, and compare the adapted models against the base models on that sample; if the accuracy gains shrink to a few points, the reported adaptation gains are largely memorization of the training distribution rather than transferable e-commerce knowledge.","supporting_citations":[{"cited_title":"Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timoth \\' e e Lesort, and Irina Rish","cited_arxiv_id":null,"evidence_quote":"Motivates the learning-rate and data-ratio experiments that determine the final training setup."}],"review_version":1}