{"id":"7f16bba3-0f03-4066-bc06-973178a4d3e6","arxiv_id":"2506.11999","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A single recommendation model with task-aware Mixture of Low-rank Experts and convergence-based sample scheduling beats baselines on a new 13-task benchmark.","lead":"This paper introduces RecFound, a single recommendation model that handles both embedding tasks, such as user-to-item retrieval, and generation tasks, such as answering questions or writing product text. The authors build a 13-task dataset and add three training modules, reporting that RecFound beats existing models on that benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset split may leak entities across the 13 tasks: no user/item-disjoint partition is stated, so test items/users seen in other tasks' training could inflate Tables 2/3 SOTA claims; needs an entity-disjoint re-evaluation.","rationale":"The paper's central empirical claim is that RecFound achieves state-of-the-art performance across 13 tasks. That claim depends entirely on a clean evaluation on the new RecFound dataset. The most critical unverified condition is the train/test split: since all tasks are carved from overlapping real-world interaction data, a test sample's user or item can easily appear in training samples of a different task (or the same task, if the split is at sample level). An LLM finetuned on item titles and user sequences can then retrieve or generate the correct answer by direct memorization, especially in tasks like U2I, I2I, and SR where the target is a specific item. The paper neither states nor empirically checks for entity overlap across splits, so the reported SOTA numbers could be inflated. This is precisely the weakest assumption the reader identified; no other issue (e.g., small test sets, lack of error bars, baseline tuning details) is as decisive because the split is upstream of all evaluation. If a strict entity-disjoint re-evaluation still shows RecFound ahead of baselines, the central claim would be substantially supported; if not, the claim must be weakened. Therefore the verdict should remain CONDITIONAL pending this check.","tokens_in":20450,"tokens_out":5391,"duration_ms":63421,"concrete_test":"Using the released code and dataset, reconstruct the split. 1) Compute the fraction of test users and test items that occur in any training sample of any of the 13 tasks. 2) Build a strict entity-disjoint split: for every user/item appearing in any test sample, remove all training and validation samples containing that user or item from all tasks; retrain or re-evaluate the best RecFound checkpoint (and the strongest baseline) on this clean split. If the Table 2/3 average ranks or headline metrics change by more than a small tolerance (e.g., 0.01 in F1/MRR), the SOTA claim is not supported by the current evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 splits the RecFound dataset into train/validation/test with no statement that the split is user-disjoint or item-disjoint, and every task is built from the same underlying sources (Amazon Reviews, MovieLens, Shopping Queries). Because the same user/item can therefore appear in training samples of one task and test samples of another, the decoder-only backbone (which sees all item titles and user sequences) can 'solve' test cases by memorizing entities seen during training, rather than by generalizing. This is most acute for retrieval/generation tasks like U2I, SR, and I2I, where the target item's text can be matched to a training example from another task. The reported SOTA gains in Tables 2 and 3—and the ablations that use the same split—would then be inflated. The paper provides no overlap statistics or entity-level split, so the central claim rests on an unverified assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RecFound proposes a unified generative-representational framework for recommendation foundation models. The paper contributes a new benchmark dataset covering 13 tasks (3 embedding and 10 generative) assembled from Amazon Reviews, MovieLens, Shopping Queries, and related sources; a Task-wise Mixture of Low-rank Experts (TMoLE) module to separate and share knowledge across tasks; a Step-wise Convergence-Oriented Sample Scheduler (S2Sched) that adjusts per-task sampling ratios from validation loss slopes; and a Model Merge module based on TIES-Merging to balance checkpoint specialization. Experiments compare RecFound against general-purpose LLMs, GritLM, eCeLLM, and an embedding-only baseline, reporting improved average ranks on both generative and embedding tasks, with ablations supporting the three proposed components. The authors state they will release code, data, and checkpoints.","tokens_in":20825,"tokens_out":5512,"duration_ms":186055,"significance":"If the claims hold, RecFound is a useful step toward recommendation foundation models that handle both embedding and generative tasks in one backbone, and the released dataset could become a reference benchmark for the community. The paper's strengths are its broad task coverage, the explicit attempt to handle knowledge conflict and convergence imbalance in multi-task training, and the promised public release of code, data, and checkpoints. The significance is currently tempered, however, because the SOTA claims rest entirely on the authors' own new benchmark, the train/test split is not shown to be entity-disjoint across tasks, and the main tables contain an unresolved numerical inconsistency. The central ideas are plausible, but the empirical evidence needs to be made internally consistent and leakage-free before the claims can be accepted.","major_comments":[{"comment":"The RecFound dataset is split into training, validation, and testing sets without any stated user- or item-disjoint partitioning, even though all 13 tasks are derived from overlapping sources (Amazon Reviews, MovieLens, Shopping Queries). Because the same user or item can appear in the training data of one task and the test data of another task, the shared backbone can memorize entities (item titles, user sequences) seen in other tasks' training splits, which is especially acute for retrieval-oriented tasks such as U2I, I2I, and SR. The paper should report cross-task entity-overlap statistics and re-run the main comparison on splits where no test entity appears in any task's training set, or otherwise justify that such leakage cannot affect the reported SOTA gains.","section":"§3, Tables 2 and 3"},{"comment":"The same configuration 'RecFound' reports inconsistent numbers in Table 3 and Table 6. Table 3 gives Item Emb 0.824, Query Emb 0.988, User Emb 0.570, while Table 6 gives 0.823, 0.993, 0.510 for the same model; the embedding average 0.775 in Table 4 matches Table 6, not Table 3. This discrepancy affects the main results and the ablation comparison, so it must be resolved before the SOTA claim can be assessed.","section":"§5.2 vs Appendix C, Tables 3 and 6"},{"comment":"The sign convention in the non-convergence estimator is unclear and appears inconsistent with the stated intent. If α_i(t) is the validation loss slope, then a task that is still improving has negative α, while a task whose loss is rising (for example due to overfitting) has positive α. Eq. (3) maps positive α to negative γ, so the statement that 'higher γ indicates less convergence' does not hold under the usual sign convention, and Eq. (6)'s -β γ_inter term would tend to up-weight tasks with negative slopes. The authors should define the normalized slope precisely, state what positive and negative α mean, and show explicitly that the scheduler up-weights genuinely non-converged tasks. In addition, the Softmax_t in Eq. (5) is not defined: the index set over which the softmax is computed is missing.","section":"§4.3, Eqs. (3)-(6)"},{"comment":"All reported metrics are single point estimates without error bars or significance tests. Several headline differences are small (e.g., AVE 0.344 vs 0.341, AG 0.853 vs 0.843, SA 0.533 vs 0.532 in Table 2), so without variance estimates or significance tests it is not possible to determine whether RecFound's SOTA claim is statistically meaningful. Please add multiple seeds, confidence intervals, or significance tests for the main comparisons and the ablations.","section":"§5.2, Tables 2, 3, 5, 6"},{"comment":"The Model Merge module is not specified at the implementation level. The text says TIES-Merging is applied 'over the learned low-rank experts across different checkpoints,' but it does not state how many checkpoints are merged, at which training steps they are selected, whether the base backbone weights are also merged, or how the merged experts interact with the trained router. Without these details, the 'w/o M.M.' ablation and the main RecFound results cannot be fully reproduced.","section":"§4.4"}],"minor_comments":[{"comment":"There are several typos: 'perfromance' in §4.4, 'S2Shed' in §5.2, and 'noramlized' in §4.3.2.","section":"§4.4, §5.2, §4.3.2"},{"comment":"In the QR column, eCeLLM and RecFound both report F1 = 0.973, but only RecFound is bolded; if ties are allowed, both should be marked, and the text should not describe RecFound as best on that task without qualification.","section":"Table 2"},{"comment":"The statement that Mask(·) suppresses 'irrelevant task dimensions' by assigning them −∞ is loose; it is the logits before the softmax that are set to −∞, not dimensions of the routing distribution itself.","section":"Eq. (1) and surrounding text"},{"comment":"The description of S2Sched says 'we uniformly sample 128 validation data instances' but does not clarify whether this sample is drawn per task or per task type; please specify.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The authors' commitment to release code, data, and checkpoints is commendable and will help the community verify the benchmark. Before further review, the editor should ask the authors to reconcile Tables 3 and 6, provide the dataset split construction and overlap statistics, and clarify the S2Sched equations. The paper's fit to cs.IR is appropriate, but the central empirical claims need to be placed on a leakage-free and statistically supported footing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRecFound is a serious systems effort: the first recommendation foundation model I know of that trains one backbone for both embedding and generative tasks, on a new benchmark of 3 embedding and 10 generative tasks. The design choices are sensible: TMoLE with task-masked low-rank experts gives a clean way to share parameters while allowing task-specific specialization; S2Sched's validation-slope-based sampling is a plausible fix for the embedding/generative convergence mismatch; TIES merge is a standard belt-and-suspenders addition. The paper is clearly written, the ablations are reasonably controlled, and the authors release code, checkpoints, and the dataset. That last part is real value—this dataset is a reusable resource even if the model results were wrong.\n\nThe empirical support for the headline SOTA claim has two soft spots. The first is statistical: nearly every metric in Tables 2, 3, 5, and 6 is a single point estimate. With no error bars or significance tests, gains like +0.01 F1 or +0.02 MRR are hard to distinguish from noise. The second, more serious, is the possibility of cross-task entity leakage. The benchmark is built entirely from Amazon Reviews, MovieLens, and Shopping Queries, and Section 3 does not state that the train/validation/test split is user-disjoint or item-disjoint. The same user or item can plausibly appear in the training set of one task and the test set of another. For retrieval/generation tasks where the target item's text appears in the input, the decoder could memorize entities rather than generalize. The paper should at minimum report entity-overlap statistics, and ideally re-run the comparison on an entity-disjoint split. Because the finetuned baselines see the same split, the relative comparison may survive, but the absolute numbers and the 'state-of-the-art' phrasing are vulnerable until this is addressed.\n\nWhere does that leave the paper? The mechanics are sound, the contribution is real, and the resource is valuable. It deserves peer review—an editor should send it out. But the reviewers should ask for the leakage analysis and the error bars, and the authors should soften the SOTA claim until those are in hand.","headline":"A real systems contribution with a new benchmark, but the SOTA claim needs entity-leakage analysis and error bars before I'd trust the numbers.","tokens_in":21176,"tokens_out":2714,"would_cite":true,"duration_ms":60895,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RecFound claims to be the first recommendation foundation model to train three embedding tasks and ten generative tasks in one shared backbone, reporting the best average rank on both task branches—ahead of larger LLMs and finetuned…","keywords":["recommendation foundation models","generative representational learning","multi-task training","mixture of low-rank experts","convergence-aware sample scheduling","model merging","embedding and generative tasks"],"falsifier":"Measure entity overlap across task boundaries: for each pair of tasks, count how many user IDs, item IDs, or review texts occur in one task's training samples and the other task's test samples. If, for instance, an item that appears in a U2I or I2I training sample also appears as the ground-truth next item in an SR test sample (or in its candidate set), re-run the comparison on a strictly entity-disjoint re-split; if RecFound's margins shrink or invert, the state-of-the-art claim rests on leakage rather than multi-task transfer.","tokens_in":20282,"feed_emoji":"🎯","tokens_out":11169,"duration_ms":117625,"temperature":0.7,"pith_summary":"The paper tries to establish that one foundation model can serve the two halves of recommendation usually handled by separate models: embedding tasks that power retrieval (representing users, queries, and items as vectors) and generative tasks that produce text (next-item prediction, user profiles, sentiment, product descriptions, and answer generation). Previous recommendation foundation models trained mostly the generative side; RecFound is claimed to be the first to train 3 embedding and 10 generative tasks together in one backbone. To make that joint training work, the authors contribute a dataset covering all 13 tasks, a task-routed mixture of low-rank experts, a convergence-aware sample scheduler, and a checkpoint-merging step. If the claim holds, a single model could replace the two-model setup typical of deployed recommendation systems, and the reported state-of-the-art average ranks (1.30 on generative tasks, 1.33 on embedding tasks) would be the first evidence that retrieval and text capabilities reinforce rather than damage each other in a recommendation foundation model.","feed_headline":"One model tops 13 recommendation tasks at once","feed_subtitle":"RecFound trains one backbone for retrieval embeddings and text generation, beating larger LLMs and tuned baselines.","key_machinery":"Three modules carry the argument. (1) Task-wise Mixture of Low-Rank Experts (TMoLE): each attention projection is augmented with $N=6$ parallel low-rank adapters of rank $r=16$, partitioned into two embedding experts, two generative experts, and two shared experts; a shared MLP router maps a learned task embedding to a gating vector $v_i = \\mathrm{Softmax}(\\mathrm{Mask}(\\mathrm{MLP}(e_i)))$ that zeroes out experts irrelevant to the sample's task, and the projection output is $o = W_0 x + \\sum_{j=1}^{N} v_i A_j B_j x$. The task mask is what lets related tasks share knowledge while unrelated tasks keep separate pathways. (2) Step-wise Convergence-oriented Sample Scheduler (S2Sched): at each step it fits a linear regression to the last $L=64$ validation losses per task to get normalized slopes, computes inter-task and intra-task non-convergence rates and a balancing weight $\\beta_b(t)$ shared within embedding or generative task groups, and reallocates the batch budget $n^{(b)}_i(t) = \\lfloor B^{(b)} \\omega_{i,b}(t) \\rfloor$ toward the slower-converging tasks, without touching loss weights. (3) Model Merge: TIES-Merging over low-rank experts across checkpoints to resolve conflicting parameter updates. The embedding branch is trained with InfoNCE on mean-pooled last-layer representations and the generative branch with token-level cross-entropy, both on the same backbone.","core_discovery":"The paper's central claim is that RecFound achieves the best average performance on both branches of its own benchmark: average rank 1.30 across the ten generative tasks and 1.33 across the three embedding tasks, ahead of finetuned unified baselines (GritLM, eCeLLM) and of far larger untuned models (Qwen2.5-32B-Instruct, Mixtral-8x7B). Ablations show that each of the three modules contributes: removing S2Sched drops the average generative score from 0.703 to 0.679 and the average embedding score from 0.775 to 0.762, and removing TMoLE drops them to 0.678 and 0.759; the model-merge step adds a small further gain. The authors also observe that the router learns task structure by itself, assigning similar routing distributions to related tasks and divergent ones to unrelated tasks, and that the scheduler's per-task sample ratios rise and fall alternately, which they interpret as protection against forgetting.","pith_inferences":["A direct test the authors did not run: re-split the dataset so each user and item appears in exactly one of train/validation/test across all 13 tasks, since all tasks are carved from the same five source corpora and the paper never shows that overlapping entities are partitioned consistently. If RecFound's margins shrink on that split, part of the reported gains would be entity memorization rather","The TMoLE-plus-S2Sched recipe is not recommendation-specific: any multi-task LLM setup with incommensurable losses and uneven convergence (search-plus-QA, retrieval-augmented generation) could reuse the same task-masked expert grouping and slope-based sample reallocation.","The router keys on a fixed task-ID embedding table, so the framework cannot absorb new tasks without retraining; the authors' own limitation note (sensitivity to task data imbalance, no continual task expansion) points at where the design would need to grow—for example, deriving the mask from instruction text or adding experts dynamically."],"forward_implications":["A single trained model can serve both the retrieval (embedding) and user-facing text (generative) stages of a recommendation pipeline, removing the need to maintain separate embedding and generation models.","Task-grouped routing over low-rank experts is sufficient to reconcile knowledge sharing and conflict across 13 recommendation tasks in a 7B backbone, at parameter-efficient cost.","Adaptively reallocating sample budgets by validation-loss slope outperforms both uniform sampling and direct loss reweighting when tasks use incomparable loss functions (InfoNCE versus cross-entropy).","The three modules are each load-bearing: ablations that remove S2Sched and TMoLE independently degrade both branches, so the state-of-the-art result is attributed to the combination rather than to any single component."],"supporting_citations":[{"why":"GritLM, the unified generative-representational instruction-tuning method used as the main finetuned baseline it must beat on both branches.","marker":"[43]"},{"why":"eCeLLM, the generative e-commerce LLM used as a finetuned baseline on text tasks and as a data source (ECInstruct) for the Query2Item task.","marker":"[44]"},{"why":"LoRA, the low-rank adaptation whose adapters and PEFT-MoE extensions TMoLE builds on.","marker":"[16]"},{"why":"TIES-Merging, the model-merging algorithm the Model Merge module applies to resolve conflicting expert updates across checkpoints.","marker":"[65]"},{"why":"Mistral-7B-Instruct, the backbone LLM that every RecFound variant finetunes.","marker":"[19]"},{"why":"The citation group the paper attaches to the Amazon Reviews dataset, which supplies most embedding and generative task samples (U2I, I2I, QR, AVE, AG, SR, SA, UP, AP, PRP, IP).","marker":"[64, 12, 40, 14]"},{"why":"MovieLens, the source of the user-profile, item-profile, and sequential-recommendation samples.","marker":"[11]"},{"why":"Shopping Queries, the source of the Query2Item embedding task samples.","marker":"[50]"}],"fun_headline_variants":["RecFound: one model tops 13 recommendation tasks","Single model beats tuned baselines on 13 tasks","RecFound unifies generative and embedding tasks","Task-wise experts help one model rule recommendations","13 tasks, one backbone: RecFound wins them all"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the RecFound dataset's train/validation/test split is leakage-free across tasks: the same user or item, drawn from the shared source corpora, must never appear in one task's training data and another task's test data; the paper states only that it split the dataset into training, validation, and testing sets and never shows that overlapping entities are partitioned consistently.","fun_headline_variants_meta":{"raw":{"variants":["RecFound: one model tops 13 recommendation tasks","Single model beats tuned baselines on 13 tasks","RecFound unifies generative and embedding tasks","Task-wise experts help one model rule recommendations","13 tasks, one backbone: RecFound wins them all"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00057,"raw_usage":{"total_tokens":2695,"prompt_tokens":942,"completion_tokens":1753,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":1680}},"tokens_in":558,"tokens_out":1753,"duration_ms":14276,"temperature":1.0,"reasoning_tokens":1680,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:57:57.785419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure entity overlap across task boundaries: for each pair of tasks, count how many user IDs, item IDs, or review texts occur in one task's training samples and the other task's test samples. If, for instance, an item that appears in a U2I or I2I training sample also appears as the ground-truth next item in an SR test sample (or in its candidate set), re-run the comparison on a strictly entity-disjoint re-split; if RecFound's margins shrink or invert, the state-of-the-art claim rests on leakage rather than multi-task transfer.","supporting_citations":[{"cited_title":"Generative representational instruction tuning, 2025","cited_arxiv_id":null,"evidence_quote":"GritLM, the unified generative-representational instruction-tuning method used as the main finetuned baseline it must beat on both branches."},{"cited_title":"ecellm: Generalizing large language models for e-commerce from large-scale, high-quality instruction data","cited_arxiv_id":null,"evidence_quote":"eCeLLM, the generative e-commerce LLM used as a finetuned baseline on text tasks and as a data source (ECInstruct) for the Query2Item task."},{"cited_title":"Ties-merging: Resolving interference when merging models","cited_arxiv_id":null,"evidence_quote":"TIES-Merging, the model-merging algorithm the Model Merge module applies to resolve conflicting expert updates across checkpoints."},{"cited_title":"Maxwell Harper and Joseph A","cited_arxiv_id":null,"evidence_quote":"MovieLens, the source of the user-profile, item-profile, and sequential-recommendation samples."}],"review_version":1}