{"id":"b3f6391c-8cbf-4fc7-b1bf-fc58748c45f7","arxiv_id":"2606.05336","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"BUMP trains profile generators via bidirectional ranking on in-batch interactions using GRPO and an LLM judge, matching supervised methods on LaMP without task labels.","lead":"The paper introduces BUMP, a self-supervised framework that trains an LLM to generate user profiles from raw interaction histories using bidirectional in-batch ranking with an LLM judge and GRPO. Smart generalists might read it because label-free personalization could reduce costs for deploying customized LLMs in recommendations, search, and dialogue.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Small LLM judge's NDCG scores may not provide reliable, unbiased supervision for profile quality","rationale":"The reader's weakest_assumption directly identifies the same load-bearing point. The abstract-only review already flags insufficient detail to verify the judge's reliability; the concrete_test above would resolve whether that assumption holds without requiring the full manuscript.","tokens_in":1739,"tokens_out":318,"duration_ms":18914,"concrete_test":"On a random 200-example subset of the training logs, compute the small judge's NDCG rankings for generated profiles vs. held-out interactions; independently re-score the identical ranking pairs with a larger model (e.g., the LaMP evaluator LLM) or human raters; if Spearman rank correlation between the two sets of NDCG scores falls below 0.65, the supervision signal is likely misaligned with downstream utility.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the bidirectional in-batch ranking objective (scored via multi-positive NDCG by a small LLM judge) supplying sufficient supervision that generalizes to LaMP downstream tasks. The abstract provides no evidence that the small judge's rankings correlate with actual personalization utility; small LLMs frequently exhibit position bias, lexical bias, and limited semantic depth when scoring free-form profiles against held-out interactions. If the judge rewards superficial matches rather than task-relevant user traits, the GRPO-trained generator could succeed on the proxy objective while failing to improve real personalization, even if it matches supervised baselines on LaMP.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces BUMP, a self-supervised framework for training an LLM-based profile generator from raw user interaction logs. It optimizes via GRPO under a bidirectional in-batch ranking objective: a small LLM judge scores multi-positive NDCG both when the generated profile ranks the user's held-out interactions above in-batch negatives and when a held-out interaction ranks the user's profile above other profiles. No downstream task labels are used at training time. On the LaMP benchmark the method is reported to match or exceed closed-source APIs and prior supervised baselines.","tokens_in":1879,"tokens_out":434,"duration_ms":20661,"significance":"If the bidirectional objective with the small LLM judge supplies supervision that generalizes beyond the proxy ranking task, the result would remove a major practical barrier (need for per-task labeled rewards) and enable scalable personalization across recommendation, dialogue, and generation settings.","major_comments":[{"comment":"The central performance claim on LaMP rests on the small LLM judge supplying unbiased, task-relevant supervision. The manuscript provides no correlation analysis, human validation, or ablation showing that the judge's NDCG rankings align with downstream personalization utility rather than superficial lexical or positional cues (Evaluation section).","section":"Evaluation"},{"comment":"The bidirectional objective uses the same small LLM judge for both training reward and (implicitly) the ranking signal; without an independent held-out metric or cross-task transfer experiment, it is unclear whether reported LaMP gains reflect genuine profile quality or optimization toward the judge's biases (Method / Objective definition).","section":"Method"}],"minor_comments":[{"comment":"Clarify the exact model size and prompting details of the 'small LLM judge' and whether it is frozen or updated during GRPO.","section":null},{"comment":"The abstract states 'matches or outperforms' without reporting per-task numbers or statistical significance; add these in the main results table.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments highlighting the need for stronger validation of the LLM judge. We respond to each major comment below.","responses":[{"response":"We agree that explicit validation of the judge would strengthen the paper. The LaMP results demonstrate that BUMP matches supervised baselines across multiple personalization tasks without task labels, providing indirect support that the judge captures relevant signals rather than superficial cues. In the revision we will add an ablation comparing judge NDCG against random rankings and report correlation between judge scores and downstream task metrics.","revision_made":"partial","referee_comment":"[Evaluation] The central performance claim on LaMP rests on the small LLM judge supplying unbiased, task-relevant supervision. The manuscript provides no correlation analysis, human validation, or ablation showing that the judge's NDCG rankings align with downstream personalization utility rather than superficial lexical or positional cues (Evaluation section)."},{"response":"The bidirectional design requires consistency across two distinct ranking directions (profile-to-interaction and interaction-to-profile) with held-out interactions and in-batch negatives, which reduces the risk of exploiting judge-specific biases. Performance on the diverse, held-out LaMP tasks further indicates generalization beyond judge artifacts. We will add a clarifying paragraph in the Method section explaining this design choice but maintain that no independent metric is required for the self-supervised claim.","revision_made":"partial","referee_comment":"[Method] The bidirectional objective uses the same small LLM judge for both training reward and (implicitly) the ranking signal; without an independent held-out metric or cross-task transfer experiment, it is unclear whether reported LaMP gains reflect genuine profile quality or optimization toward the judge's biases (Method / Objective definition)."}],"tokens_in":1382,"tokens_out":377,"duration_ms":22297,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that this paper trains a profile generator from raw logs alone by turning in-batch negatives into a bidirectional ranking signal scored with multi-positive NDCG. One direction treats the generated profile as query and ranks the user's held-out interactions; the other treats a held-out interaction as query and ranks the profile. GRPO turns those scores into a dense reward. On LaMP it reportedly matches or beats both closed APIs and prior methods that needed task labels.\n\nThe bidirectional objective is the clearest novelty. Most prior profile work optimizes directly against downstream rewards; here the supervision comes only from the ranking game inside the batch. That framing is clean and removes the need for annotated task data.\n\nThe soft spot is exactly where the stress-test note points: the judge is a small LLM. Nothing in the abstract or framing shows that its NDCG scores track actual personalization utility rather than lexical overlap or position artifacts. If the judge is biased, the generator can optimize the proxy while the profiles remain shallow for real tasks. The paper would be stronger with a direct check that judge rankings correlate with held-out task performance or with human ratings of profile quality.\n\nThe rest of the setup looks standard: GRPO is off-the-shelf, in-batch negatives are free, LaMP is a reasonable benchmark. No obvious circularity in the metrics themselves, but the judge is still an external model whose reliability is assumed rather than demonstrated.\n\nThis is for groups working on practical LLM personalization who want to avoid per-task labeling. A reader already running LaMP experiments would get immediate value from the method description and the reported numbers. It deserves a serious referee because the core claim is testable and the engineering is straightforward; the review would mainly pressure-test the judge validation and any ablations on batch size or judge model scale.","headline":"BUMP gives a workable label-free route to user profiles via bidirectional GRPO ranking, but the small-LLM judge supplying the NDCG scores is the load-bearing and unvalidated piece.","tokens_in":2347,"tokens_out":447,"would_cite":false,"duration_ms":18706,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"BUMP trains user profile generators from raw interaction logs alone via bidirectional ranking, matching labeled methods on LaMP without task labels.","keywords":["self-supervised learning","user profile generation","LLM personalization","bidirectional ranking","LaMP benchmark","in-batch negatives","NDCG scoring"],"falsifier":"Direct evaluation on the LaMP benchmark where BUMP's personalization performance falls below that of methods trained with labeled downstream rewards.","tokens_in":2656,"feed_emoji":"🤖","tokens_out":674,"duration_ms":24804,"temperature":0.7,"pith_summary":"The paper establishes that a profile generator for LLM personalization can be trained entirely self-supervised, without any labeled rewards from downstream tasks. It optimizes an LLM to emit free-form textual profiles from user histories under a bidirectional in-batch ranking loss: the profile must rank the user's held-out interactions above those of others in the batch, and a held-out interaction must rank the correct profile above others. Both directions are scored with multi-positive NDCG from a small LLM judge, turning other users into free negatives and yielding dense supervision from logs alone. On the LaMP benchmark this approach matches or exceeds closed-source APIs and prior supervised methods.","feed_headline":"Self-supervised profiles match labeled methods on LaMP","feed_subtitle":"BUMP trains profile generators from raw logs alone via bidirectional ranking, eliminating need for per-task labels.","key_machinery":"Bidirectional in-batch ranking objective scored with multi-positive NDCG by a small LLM judge","core_discovery":"BUMP is a self-supervised framework that trains an LLM profile generator under a bidirectional in-batch ranking objective. Given raw user interaction histories, the model emits a textual profile; a small LLM judge then scores how well that profile, treated as a query, retrieves the user's own held-out interactions over batch negatives, and how well a held-out interaction retrieves the correct profile over other profiles. Both directions use multi-positive NDCG, the scores are combined into a dense reward, and GRPO optimizes the generator. This produces profiles that personalize LLMs on LaMP tasks at the level of methods trained with explicit task labels.","pith_inferences":["The method could lower the barrier to deploying personalized LLMs in new domains where collecting task labels is impractical.","Profiles optimized for retrieval across users might transfer more readily between tasks than those tuned to specific reward signals.","Increasing the capacity of the judge model could tighten the supervision signal and improve profile quality further."],"forward_implications":["Profile generators can be trained on raw interaction logs without any task-specific annotations or rewards.","Personalization performance reaches or exceeds that of closed-source APIs and prior supervised methods on LaMP tasks.","Supervision becomes available for every training example because other users in the batch serve as free negatives.","The same profile generator can be applied across multiple downstream tasks without retraining for each labeled reward."],"fun_headline_variants":["BUMP trains LaMP profiles via bidirectional ranking","Self-supervised BUMP rivals labeled personalization methods","Raw logs yield competitive profiles on LaMP benchmark","Bidirectional NDCG enables label-free profile optimization"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The bidirectional in-batch ranking objective scored with multi-positive NDCG by a small LLM judge supplies sufficient and unbiased supervision to produce profiles that generalize to downstream personalization tasks.","fun_headline_variants_meta":{"raw":{"variants":["BUMP trains LaMP profiles via bidirectional ranking","Self-supervised BUMP rivals labeled personalization methods","Raw logs yield competitive profiles on LaMP benchmark","Bidirectional NDCG enables label-free profile optimization"]},"model":"grok-4.3","cost_usd":0.003865,"raw_usage":{"total_tokens":2033,"prompt_tokens":759,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":38649500,"prompt_tokens_details":{"text_tokens":759,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1217,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":759,"tokens_out":57,"duration_ms":11302,"temperature":1.0,"reasoning_tokens":1217,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T06:10:54.202836+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Direct evaluation on the LaMP benchmark where BUMP's personalization performance falls below that of methods trained with labeled downstream rewards.","supporting_citations":[],"review_version":1}