{"id":"dff9aabc-6ebf-4378-a203-f53b494493ca","arxiv_id":"2607.15730","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"RECAP trains a streaming LLM profile updater with GRPO rewards from a dual-tower evaluator, gaining +0.0084 uAUC (cleaned eval) and +0.139% online usage time.","lead":"RECAP is a system that keeps a short structured text profile of each user's interests and updates it over time using feedback from watched and skipped videos, trained offline. In Kuaishou tests it improved profile-video matching and raised average app usage time by 0.139% in an online A/B test.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Online A/B compares RECAP against no-profile control, not against the open-loop Base generator; the only evidence for the headline closed-loop-vs-open-loop claim is the dual-tower offline eval, which shares its reward family and cleaned-label filter with training.","rationale":"The reader's conditional verdict is reasonable, and my read does not move it: the paper is a plausible engineering contribution with a positive but under-controlled online signal. The most load-bearing gap is not just that the dual-tower evaluator serves as both reward and offline metric; it is that the one source of evidence not using that evaluator—the online A/B—does not compare the closed-loop RECAP against the open-loop Base generator. The control group is 'original model without this feature,' so the +0.139% result cannot discriminate between 'semantic profiles help' and 'feedback-driven optimization helps.' The offline Table 2 does make the RECAP-vs-Base comparison, but it is weakened by the shared reward/evaluation family and by the much smaller raw-eval gains. A three-arm A/B (no profile, Base profile, RECAP profile) would directly test the claimed advantage. If RECAP beats Base online, the central claim is independently supported and the paper could move toward accept; if not, the claim currently rests only on a circular offline metric. I therefore keep the CONDITIONAL verdict.","tokens_in":13979,"tokens_out":9231,"duration_ms":78869,"concrete_test":"Run a three-arm online A/B on Kuaishou overseas traffic for the same 7-day window: (1) control with no profile feature, (2) Base open-loop profile rendered, encoded, and added to U2I retrieval, (3) RECAP profile added identically. Pre-register average app usage time per user as the primary metric, allocate 10% traffic per arm, and report bootstrap CIs and p-values for arm 3 vs arm 2. If RECAP-vs-Base is not statistically significant, the closed-loop-over-open-loop claim is unsupported by independent evidence; if it is significant, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that feedback-driven (closed-loop) profile optimization beats open-loop summarization. The independent online evidence does not test this. Section 5.5 describes a seven-day A/B in which treatment users receive the RECAP-generated profile as an additional retrieval feature and control users receive the original model with no profile feature. There is no arm serving a Base or SFT profile. Thus +0.139% usage time supports 'adding a semantic profile feature helps,' not 'optimizing that profile with GRPO helps.' The only direct RECAP-vs-Base comparison is Table 2, but that comparison is not independent of the training signal: the reward in Eq. (5) and the cleaned offline evaluator are both dual-tower models trained on behavior pairs filtered by the same LLM-judge consistency criterion. Raw-eval uAUC improves only 0.5871 to 0.5886 and Recall from 0.0088 to 0.0089, much smaller than the cleaned-eval gains. So the headline generalization claim rests on a metric family that is entangled with the optimization objective. This is an experimental-design gap, not a claim of gaming; it can be closed by adding the missing control arm.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RECAP, an offline closed-loop framework for optimizing streaming structured semantic user profiles in short-video recommendation. Profiles are maintained as bounded slot-based interest entries, with an LLM producing semantic diffs and a deterministic state machine handling lifecycle and capacity control. To obtain a recommendation-aligned reward, the paper filters implicit behavior pairs with an LLM judge, trains a dual-tower evaluator over rendered profiles and video captions, and uses the evaluator's matching score as a GRPO reward for policy optimization. Offline experiments on internal Kuaishou data report cleaned-eval uAUC gains from 0.7519 to 0.7603 and Recall@2000 from 0.0122 to 0.0128 for RECAP over Base, with smaller raw-eval gains, and a seven-day online A/B test reports a statistically significant +0.139% relative improvement in average application usage time per user.","tokens_in":14296,"tokens_out":4246,"duration_ms":37694,"significance":"If the closed-loop optimization result is substantiated, the paper would make a useful contribution to LLM-based user profiling: it treats the profile as a stateful bounded memory, separates semantic updates from deterministic bookkeeping, and demonstrates that feedback construction quality matters for GRPO-based profile optimization. The paper has real strengths: a concrete industrial setting, a held-out user set for offline evaluation, a disjoint evaluator-instance protocol, a controlled comparison of raw vs. cleaned feedback rewards, and rich profile-level diagnostics that connect reward signal quality to update behavior. These go beyond many purely offline LLM-profile papers. The main weakness is that the central claim — closed-loop optimization over an open-loop generator — is supported primarily by an offline evaluator that shares its dual-tower architecture and label-cleaning procedure with the training reward, while the online A/B test does not include an open-loop profile arm. With an additional control arm or a reframed claim, the contribution could be solid, but the current evidence is insufficient for the headline generalization claim.","major_comments":[{"comment":"The online A/B test compares RECAP against a no-profile control, not against Base or SFT. The reported +0.139% usage-time gain therefore supports 'adding a semantic profile feature to retrieval helps,' not 'GRPO closed-loop optimization beats open-loop summarization.' The only direct RECAP-vs-Base comparison is Table 2, and that comparison is entangled with the training reward (see next comment). To support the central claim, the paper should either add an A/B arm serving the Base or SFT profile, or explicitly restrict the online claim to 'profile feature helps' and present RECAP-vs-Base as an offline-only result.","section":"§5.5, Table 2"},{"comment":"The GRPO reward is the negative BCE loss plus cosine margin of a dual-tower evaluator over rendered profile text and video captions. The final offline evaluation uses the same dual-tower architecture and the same LLM-judge label-consistency cleaning to construct training and evaluation pairs. Because the policy is optimized against the same scoring family used for evaluation, the cleaned-eval gains (0.7519→0.7603 uAUC) may partly reflect better alignment to that scoring family rather than better recommendation utility. The held-out user split and separately trained evaluators are good, but they do not break the architectural and label-construction correlation. An independent evaluation signal — e.g., a production ranker score, a different architecture, or a downstream online outcome measured per profile variant — is needed for the RECAP-vs-Base claim.","section":"§4.3.3 Eq. (5), §5.1"},{"comment":"No error bars, confidence intervals, or significance tests are reported for the offline metrics. The raw-evaluation improvements are very small: uAUC 0.5871→0.5886 and Recall 0.0088→0.0089, while GRPO-raw decreases Recall relative to Base. Without variance estimates across seeds or user subsets, it is impossible to tell whether the raw-eval differences are noise. Since the abstract emphasizes the 0.0084 cleaned-uAUC improvement, the paper should also make explicit that this is on cleaned evaluation and should report whether the raw-eval differences are statistically distinguishable.","section":"§5.2, Table 2"},{"comment":"The parallel chunk-level rollout trains the policy only on diffs conditional on SFT-computed prefix profiles, while during inference the optimized policy's own outputs are fed forward sequentially through the state machine. This train/serve distribution shift is never analyzed. The approximation is justified by cost, but the paper should quantify its effect, e.g., by comparing against sequential GRPO on a subset of users or by measuring prefix-profile drift. Without this, the streaming objective in Eq. (1) is only loosely connected to the actual training objective.","section":"§4.4, Eq. (7)–(10)"},{"comment":"The label-consistency threshold τ is set to 0.45, which is barely above chance for a binary A/B judge. No analysis is provided of judge agreement, threshold sensitivity, or the resulting cleaning rate. Because the cleaned-feedback reward is a core contribution, the paper should report how performance varies with τ and ideally evaluate the LLM judge's accuracy against a small labeled sample. Relatedly, the penalty weights λ_fmt and λ_div in Eq. (8) are never reported; they should be listed in the implementation details.","section":"§4.3.2, §5.1"}],"minor_comments":[{"comment":"The values of λ_fmt, λ_div, and ε_clip are missing from the implementation details. The paper states w_rec=1 and w_bce=1, w_nce=0.1, but not the Eq. (8) penalty weights.","section":"§5.1"},{"comment":"Bar plots and line plots in Figure 2 do not include error bars or any variance information. For a paper whose headline offline differences are small, this is important for interpretation.","section":"§5.2, Figure 2"},{"comment":"The A/B test is described only as 'statistically significant' with +0.139%. Report the confidence interval, p-value, or the test used, and state the number of users or traffic volume in each arm beyond the 10%/20% allocation.","section":"§5.5"},{"comment":"The bar labels 'Raw', 'Clean', and 'S+G' are used in the caption but not defined there; the caption should state that these abbreviate GRPO-raw, GRPO-clean, and SFT+GRPO.","section":"Figure 4"},{"comment":"The description of the LLM judge says 'the first valid A/B token is parsed as the pairwise preference,' but no prompt template or parsing details are given. Since the cleaning pipeline is central, an appendix with the prompt and a few judged examples would help reproducibility.","section":"§4.3.2"},{"comment":"The diagnostics use 'gpt-oss-120b' as an LLM judge for coverage and semantic annotation, but no calibration or agreement with human annotation is reported. A small human-annotated validation set would strengthen these analyses.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The missing A/B arm for Base/SFT is the key experimental-design gap. The offline evaluation is internally consistent and well-controlled, but the reward–evaluation entanglement makes the headline closed-loop-vs-open-loop claim under-supported. I believe this is fixable within the scope of a major revision: either add the missing control arm or carefully reframe the claims. The related work and citations appear appropriate; no novelty disclosure concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Marco, quick take on arXiv:2607.15730. The paper is worth reading, but the splashy claim—feedback-driven closed-loop profile optimization beats open-loop summarization—is not actually tested by the online A/B. The real contribution is a concrete recipe for bounded streaming semantic profiles: an LLM that emits only semantic diffs, a deterministic state machine for lifecycle and capacity, LLM-judge filtering of implicit feedback, and GRPO against a dual-tower evaluator. That combination is new in the cited literature. The authors also deserve credit for reporting raw and cleaned evals separately, and for the careful decoupling of semantic updates from state bookkeeping.\n\nWhat I like: the state-machine split is a genuinely practical idea. LLMs are unreliable at maintaining counts and spans; keeping that deterministic and letting the LLM focus on topics and descriptions is the kind of engineering insight that survives contact with production. The qualitative diagnostics in Section 5.4 give a plausible account of why cleaned rewards help—grounded refinement instead of surface-feature matching. The seven-day A/B is a real production test, and +0.139% usage time is positive, even if small.\n\nThe soft spots are real, though. The stress-test note is correct: the A/B serves the RECAP profile as an extra retrieval feature and the control has no profile at all, so it supports “adding a semantic profile helps,” not “GRPO-optimized profiles beat Base or SFT-generated profiles.” The only direct RECAP-vs-Base comparison is Table 2, where the evaluation uses a dual-tower model of the same family as the GRPO reward, trained on the same cleaned-label filter. Raw-eval gains are tiny (uAUC 0.5871→0.5886, Recall 0.0088→0.0089). So the headline generalization claim rests on a metric family that shares its training signal with the optimization, and the independent online evidence does not cover the crucial comparison. That’s an experimental-design gap, not fraud. It can be closed with a Base/SFT profile arm.\n\nMinor issues: no error bars or significance tests for Table 2, no code/data release, and λ_fmt/λ_div are never reported, so reproducibility is limited. τ=0.45 is barely above chance, which makes me wonder how much cleaning actually happened. The parallel chunk rollouts (SFT-generated prefix profiles) approximate the streaming objective; the paper states this but doesn’t discuss how much bias it introduces.\n\nIf I were handling this, I’d send it to review. The recipe is useful, the problem is well-motivated, and the missing control arm is fixable.","headline":"Solid engineering contribution with a useful design recipe, but the headline closed-loop-vs-open-loop claim is not tested by the online A/B, and the offline evidence is entangled with the reward evaluator's own family.","tokens_in":14879,"tokens_out":2164,"would_cite":true,"duration_ms":18044,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Feedback-driven profile updates beat open-loop summarization in short-video recommendation.","keywords":["user profiling","user modeling","large language models","short video recommendation","streaming profile update","closed-loop optimization","implicit feedback","GRPO"],"falsifier":"Run a deployment A/B test in which the profile updater is trained on a reward from a freshly trained evaluator using post-deployment interaction logs, and check whether the offline cleaned-evaluation gains persist. If they shrink or invert, the original evaluator reward was overfit. Alternatively, compare RECAP against the base generator on ranking metrics produced by the production recommender itself, rather than by the dual-tower evaluator.","tokens_in":13843,"feed_emoji":"📺","tokens_out":5433,"duration_ms":40830,"temperature":0.7,"pith_summary":"This paper tries to establish that a user profile for short-video recommendation should be a bounded, structured state updated incrementally and optimized with downstream recommendation feedback, rather than a free-text summary rewritten from scratch. The proposed framework, RECAP, maintains a fixed-size set of interest entries with deterministic lifecycle rules, and trains an LLM-based updater using a reward derived from a dual-tower evaluator trained on label-consistency-filtered behavior pairs. Offline, this closed-loop optimization raises uAUC from 0.7519 to 0.7603 and Recall@2000 from 0.0122 to 0.0128 on cleaned evaluation, and a seven-day online A/B test shows a statistically significant 0.139% gain in average app usage time per user. The paper argues that reward-driven semantic updates generalize better than open-loop summarization because they produce more grounded refinements and user-level abstraction.","feed_headline":"Feedback-driven profiles beat open-loop summaries in short-video feeds","feed_subtitle":"Optimizing the profile updater with recommendation reward raises recall and user time in production.","key_machinery":"The central mechanism is the decoupling of semantic updates from lifecycle management: the LLM produces only minimal structured semantic diffs, while a deterministic state machine applies update/add/remove operations with strength decay, recency tracking, and capacity control. This bounded structured memory keeps the profile consistent over long streams, and the dual-tower evaluator's matching score, trained on cleaned feedback, serves as the GRPO reward that closes the loop.","core_discovery":"RECAP's central discovery is that a streaming profile generator can be optimized offline with a recommendation-aligned reward even when the recommender itself is unavailable. Each profile is a bounded structured memory of up to eight interest entries with lifecycle metadata; the LLM emits only a semantic diff (confirmed and new interests), while a deterministic state machine handles strength decay, eviction, and capacity. The reward is built by filtering logged implicit watch/skip pairs with an LLM judge to keep label-consistent examples, training a dual-tower semantic evaluator over rendered profile text and video captions, and using its matching score as a GRPO reward. Applying this closed","pith_inferences":["If the dual-tower evaluator is a faithful reward proxy, the same closed-loop recipe could transfer to other recommendation surfaces where items have semantic captions and implicit feedback is logged.","A testable extension: replacing the LLM-judge pairwise filtering with a learned consistency model could reduce dependence on a large judge at scale and enlarge the usable feedback set.","The bounded structured profile is a reusable user state for tasks beyond retrieval, such as ranking, explanation, or cross-domain transfer, since it is an interpretable text artifact.","Because the reward is a proxy trained on logged behavior, gains may plateau once profile text stops changing the evaluator's score; periodically retraining the evaluator on new interactions could sustain improvements."],"forward_implications":["Cleaned label-consistent feedback provides a more stable reward than raw implicit labels, leading to more grounded profile edits and fewer empty or unsupported interest entries.","SFT initialization and GRPO optimization work complementarily: SFT improves behavioral coverage and interest separation, while GRPO shapes how the profile is updated and abstracted.","The decoupled design (LLM semantics plus deterministic state machine) avoids lifecycle-field errors that a coupled full-state updater introduces.","Larger profile capacity improves offline profile utility but at higher generation and encoding cost; eight interest entries balance diversity and compactness.","Feeding RECAP-rendered profiles as text embeddings to a production retrieval model yields a statistically significant 0.139% increase in average usage time per user."],"fun_headline_variants":["Closed-loop profiles lift short-video recall and watch time","Feedback-tuned user profiles improve short-video recommendations","Optimizing user profiles with feedback boosts video recall","Training profiles on recommendation feedback lifts engagement","RECAP: feedback-driven profiles beat open-loop baselines"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the dual-tower evaluator's matching score, used both as the training reward and as the offline evaluation metric, faithfully reflects how useful a profile is for real recommendation; if optimizing it mainly aligns profile embeddings with captions rather than improving user satisfaction, the offline gains could be artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Closed-loop profiles lift short-video recall and watch time","Feedback-tuned user profiles improve short-video recommendations","Optimizing user profiles with feedback boosts video recall","Training profiles on recommendation feedback lifts engagement","RECAP: feedback-driven profiles beat open-loop baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1198,"prompt_tokens":756,"completion_tokens":442,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":368}},"tokens_in":500,"tokens_out":442,"duration_ms":4323,"temperature":1.0,"reasoning_tokens":368,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T22:27:44.440392+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a deployment A/B test in which the profile updater is trained on a reward from a freshly trained evaluator using post-deployment interaction logs, and check whether the offline cleaned-evaluation gains persist. If they shrink or invert, the original evaluator reward was overfit. Alternatively, compare RECAP against the base generator on ranking metrics produced by the production recommender itself, rather than by the dual-tower evaluator.","supporting_citations":[],"review_version":1}