Self-supervised User Profile Generation for Personalization

Clark Mingxuan Ju; Neil Shah; Tong Zhao; Yuwei Qiu

REVIEW 2 major objections 2 minor 42 references

BUMP trains user profile generators from raw interaction logs alone via bidirectional ranking, matching labeled methods on LaMP without task labels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-28 06:10 UTC pith:KFYP5M6A

load-bearing objection BUMP gives a workable label-free route to user profiles via bidirectional GRPO ranking, but the small-LLM judge supplying the NDCG scores is the load-bearing and unvalidated piece. the 2 major comments →

arxiv 2606.05336 v1 pith:KFYP5M6A submitted 2026-06-03 cs.CL

Self-supervised User Profile Generation for Personalization

Clark Mingxuan Ju , Yuwei Qiu , Tong Zhao , Neil Shah This is my paper

classification cs.CL

keywords self-supervised learninguser profile generationLLM personalizationbidirectional rankingLaMP benchmarkin-batch negativesNDCG scoring

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that a profile generator for LLM personalization can be trained entirely self-supervised, without any labeled rewards from downstream tasks. It optimizes an LLM to emit free-form textual profiles from user histories under a bidirectional in-batch ranking loss: the profile must rank the user's held-out interactions above those of others in the batch, and a held-out interaction must rank the correct profile above others. Both directions are scored with multi-positive NDCG from a small LLM judge, turning other users into free negatives and yielding dense supervision from logs alone. On the LaMP benchmark this approach matches or exceeds closed-source APIs and prior supervised methods.

Core claim

BUMP is a self-supervised framework that trains an LLM profile generator under a bidirectional in-batch ranking objective. Given raw user interaction histories, the model emits a textual profile; a small LLM judge then scores how well that profile, treated as a query, retrieves the user's own held-out interactions over batch negatives, and how well a held-out interaction retrieves the correct profile over other profiles. Both directions use multi-positive NDCG, the scores are combined into a dense reward, and GRPO optimizes the generator. This produces profiles that personalize LLMs on LaMP tasks at the level of methods trained with explicit task labels.

What carries the argument

Bidirectional in-batch ranking objective scored with multi-positive NDCG by a small LLM judge

Load-bearing premise

The bidirectional in-batch ranking objective scored with multi-positive NDCG by a small LLM judge supplies sufficient and unbiased supervision to produce profiles that generalize to downstream personalization tasks.

What would settle it

Direct evaluation on the LaMP benchmark where BUMP's personalization performance falls below that of methods trained with labeled downstream rewards.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Profile generators can be trained on raw interaction logs without any task-specific annotations or rewards.
Personalization performance reaches or exceeds that of closed-source APIs and prior supervised methods on LaMP tasks.
Supervision becomes available for every training example because other users in the batch serve as free negatives.
The same profile generator can be applied across multiple downstream tasks without retraining for each labeled reward.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The method could lower the barrier to deploying personalized LLMs in new domains where collecting task labels is impractical.
Profiles optimized for retrieval across users might transfer more readily between tasks than those tuned to specific reward signals.
Increasing the capacity of the judge model could tighten the supervision signal and improve profile quality further.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

BUMP gives a workable label-free route to user profiles via bidirectional GRPO ranking, but the small-LLM judge supplying the NDCG scores is the load-bearing and unvalidated piece.

read the letter

The main takeaway is that this paper trains a profile generator from raw logs alone by turning in-batch negatives into a bidirectional ranking signal scored with multi-positive NDCG. One direction treats the generated profile as query and ranks the user's held-out interactions; the other treats a held-out interaction as query and ranks the profile. GRPO turns those scores into a dense reward. On LaMP it reportedly matches or beats both closed APIs and prior methods that needed task labels.

The bidirectional objective is the clearest novelty. Most prior profile work optimizes directly against downstream rewards; here the supervision comes only from the ranking game inside the batch. That framing is clean and removes the need for annotated task data.

The soft spot is exactly where the stress-test note points: the judge is a small LLM. Nothing in the abstract or framing shows that its NDCG scores track actual personalization utility rather than lexical overlap or position artifacts. If the judge is biased, the generator can optimize the proxy while the profiles remain shallow for real tasks. The paper would be stronger with a direct check that judge rankings correlate with held-out task performance or with human ratings of profile quality.

The rest of the setup looks standard: GRPO is off-the-shelf, in-batch negatives are free, LaMP is a reasonable benchmark. No obvious circularity in the metrics themselves, but the judge is still an external model whose reliability is assumed rather than demonstrated.

This is for groups working on practical LLM personalization who want to avoid per-task labeling. A reader already running LaMP experiments would get immediate value from the method description and the reported numbers. It deserves a serious referee because the core claim is testable and the engineering is straightforward; the review would mainly pressure-test the judge validation and any ablations on batch size or judge model scale.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces BUMP, a self-supervised framework for training an LLM-based profile generator from raw user interaction logs. It optimizes via GRPO under a bidirectional in-batch ranking objective: a small LLM judge scores multi-positive NDCG both when the generated profile ranks the user's held-out interactions above in-batch negatives and when a held-out interaction ranks the user's profile above other profiles. No downstream task labels are used at training time. On the LaMP benchmark the method is reported to match or exceed closed-source APIs and prior supervised baselines.

Significance. If the bidirectional objective with the small LLM judge supplies supervision that generalizes beyond the proxy ranking task, the result would remove a major practical barrier (need for per-task labeled rewards) and enable scalable personalization across recommendation, dialogue, and generation settings.

major comments (2)

[Evaluation] The central performance claim on LaMP rests on the small LLM judge supplying unbiased, task-relevant supervision. The manuscript provides no correlation analysis, human validation, or ablation showing that the judge's NDCG rankings align with downstream personalization utility rather than superficial lexical or positional cues (Evaluation section).
[Method] The bidirectional objective uses the same small LLM judge for both training reward and (implicitly) the ranking signal; without an independent held-out metric or cross-task transfer experiment, it is unclear whether reported LaMP gains reflect genuine profile quality or optimization toward the judge's biases (Method / Objective definition).

minor comments (2)

Clarify the exact model size and prompting details of the 'small LLM judge' and whether it is frozen or updated during GRPO.
The abstract states 'matches or outperforms' without reporting per-task numbers or statistical significance; add these in the main results table.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments highlighting the need for stronger validation of the LLM judge. We respond to each major comment below.

read point-by-point responses

Referee: [Evaluation] The central performance claim on LaMP rests on the small LLM judge supplying unbiased, task-relevant supervision. The manuscript provides no correlation analysis, human validation, or ablation showing that the judge's NDCG rankings align with downstream personalization utility rather than superficial lexical or positional cues (Evaluation section).

Authors: We agree that explicit validation of the judge would strengthen the paper. The LaMP results demonstrate that BUMP matches supervised baselines across multiple personalization tasks without task labels, providing indirect support that the judge captures relevant signals rather than superficial cues. In the revision we will add an ablation comparing judge NDCG against random rankings and report correlation between judge scores and downstream task metrics. revision: partial
Referee: [Method] The bidirectional objective uses the same small LLM judge for both training reward and (implicitly) the ranking signal; without an independent held-out metric or cross-task transfer experiment, it is unclear whether reported LaMP gains reflect genuine profile quality or optimization toward the judge's biases (Method / Objective definition).

Authors: The bidirectional design requires consistency across two distinct ranking directions (profile-to-interaction and interaction-to-profile) with held-out interactions and in-batch negatives, which reduces the risk of exploiting judge-specific biases. Performance on the diverse, held-out LaMP tasks further indicates generalization beyond judge artifacts. We will add a clarifying paragraph in the Method section explaining this design choice but maintain that no independent metric is required for the self-supervised claim. revision: partial

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper trains a profile generator via GRPO under a bidirectional in-batch ranking objective whose reward is computed by an external small LLM judge on multi-positive NDCG using held-out interactions and in-batch negatives drawn from raw logs; this objective is independent of any downstream LaMP task labels. Evaluation on LaMP is a separate held-out benchmark. No equation, reward definition, or claim reduces by construction to a fitted parameter, self-citation chain, or renamed input; the central claim is an empirical statement about generalization from the proxy objective rather than a definitional equivalence.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The method rests on standard ranking metrics and the assumption that an LLM judge can reliably score profile-interaction alignment; no new entities or fitted parameters are introduced in the abstract.

axioms (1)

domain assumption Multi-positive NDCG is a suitable dense reward for bidirectional ranking of profiles and interactions.
Invoked as the scoring function for both directions of the objective.

pith-pipeline@v0.9.1-grok · 5799 in / 1233 out tokens · 38229 ms · 2026-06-28T06:10:54.202836+00:00 · methodology

0 comments

read the original abstract

Personalizing large language models (LLMs) has become a central challenge as LLMs are deployed across recommendation, search, dialogue, and content generation -- settings where the same query should yield different answers given different users. A promising route is to summarize each user's interaction history into a natural-language memory or profile and prepend it to the prompt to facilitate personalization. Existing methods learn such profile generators with explicit rewards derived from labeled downstream tasks, which are expensive and sparse as they require annotated supervision for every target task. In light of this challenge, we introduce Bidirectional User Modeling via Profiles (BUMP), a self-supervised framework that trains a profile generator without any downstream labels. Specifically, given a user's interaction history, we use GRPO to train an LLM to emit a free-form textual profile under a bidirectional in-batch ranking objective: a small LLM judge measures (i) how well the generated profile, used as a query, ranks the user's own held-out interactions above interactions from other users in the batch, and (ii) how well a held-out interaction, used as a query, ranks the user's own profile above profiles of other users. Both directions are scored with multi-positive NDCG and combined into a dense reward per rollout; other users in the batch supply free negatives, so every training example yields supervision from raw interaction logs alone. Evaluated on the LaMP benchmark, BUMP matches or outperforms closed-source APIs and prior methods relying on labeled rewards, while requiring no task label at training.

Figures

Figures reproduced from arXiv: 2606.05336 by Clark Mingxuan Ju, Neil Shah, Tong Zhao, Yuwei Qiu.

**Figure 3.** Figure 3: LaMP-2 forward and backward judge prompts. Forward / Backward judge prompts — LaMP-3 (product rating). Forward instruction: You are an expert judge. Given a user summary, rank the following product reviews from most likely to least likely to have been written by this user. Input — User summary su: “A pragmatic reviewer of kitchen and small-appliance products; rates strictly on durability and value; tends t… view at source ↗

**Figure 5.** Figure 5: LaMP-4 forward and backward judge prompts. Forward / Backward judge prompts — LaMP-5 (scholarly title generation). Forward instruction: You are an expert judge. Given a user summary, rank the following academic works (abstract + title) from most likely to least likely to have been titled by this user. Input — User summary su: “A graph machinelearning researcher whose titles favor “X for Y”-style framings … view at source ↗

**Figure 7.** Figure 7: LaMP-7 forward and backward judge prompts. Summary-generation prompt for BUMP — instruction given to πθ to elicit the profile su from a user’s visible history Hvis u . Instruction (LaMP-1, citation identification): Below is a list of academic works authored by a user: {profile} Based on these works, please write a concise summary about the user. Instruction (LaMP-2, movie tagging): Below is a list of movi… view at source ↗

**Figure 9.** Figure 9: Zero-shot Gemini profile-generation prompts. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗

**Figure 10.** Figure 10: LoRA-SFT input template. {summary} is the cached profile and {input} is the LaMP question (unchanged from the official LaMP release). Shared across every profile-based method in [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗

**Figure 11.** Figure 11: Training dynamics of the bidirectional reward across the six LaMP tasks. Each column shows one task; [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

42 extracted references · 3 canonical work pages · 2 internal anchors

[1]

InPro- ceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 1–14, Kyoto, Japan

Llara: Large language-recommendation assis- tant. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1785–1795. Jiacheng Lin, Tian Wang, and Kun Qian. 2025. Rec-r1: Bridging generative large language models and user- centric recommendation systems via reinforcement learning.arXiv prepri...

work page arXiv 2025
[2]

BPR: Bayesian Personalized Ranking from Implicit Feedback

Representation learning with large language models for recommendation. InProceedings of the ACM web conference 2024, pages 3464–3475. Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2012. Bpr: Bayesian personalized ranking from implicit feedback.arXiv preprint arXiv:1205.2618. Chris Richardson, Yao Zhang, Kellen Gillespie, ...

work page internal anchor Pith review Pith/arXiv arXiv 2024
[3]

Qwen3 Technical Report

Qwen3 technical report.arXiv preprint arXiv:2505.09388. An Zhang, Yuxin Chen, Leheng Sheng, Xiang Wang, and Tat-Seng Chua. 2024. On generative agents in recommendation. InProceedings of the 47th inter- national ACM SIGIR conference on research and development in Information Retrieval, pages 1807– 1817. Jinghao Zhang, Yuting Liu, Wenjie Wang, Qiang Liu, Sh...

work page internal anchor Pith review Pith/arXiv arXiv 2024
[4]

Tag:thriller; Title/Desc:Se7en– two detectives hunt a serial killer
[5]

Tag:action; Title/Desc:Fast & Furious 9– street racers reunite
[6]

[4]Tag:family; Title/Desc:Paw Patrol Movie

Tag:indie; Title/Desc:Manchester by the Sea– a janitor returns home. . . [4]Tag:family; Title/Desc:Paw Patrol Movie. . . [5]Tag:drama; Title/Desc:The Florida Project. . . Output (judge):[3, 5, 1, 2, 4] Expected Groundtruth:[1, 3, 5,· · ·] Backward instruction:You are an expert judge. Given a movie (with its tag), rank the following user summaries from mos...
[7]

Action / blockbuster fan tagging mostly franchise tentpoles

“Action / blockbuster fan tagging mostly franchise tentpoles. . . ”
[8]

Viewer who tags character-driven indie dramas and 90s thrillers

“Viewer who tags character-driven indie dramas and 90s thrillers. . . ”
[9]

Animation enthusiast focused on family con- tent

“Animation enthusiast focused on family con- tent. . . ”
[10]

Horror completionist with a taste for 80s slash- ers

“Horror completionist with a taste for 80s slash- ers. . . ” Output (judge):[2, 1, 4, 3] Expected Groundtruth:[2,· · ·] Figure 3: LaMP-2 forward and backward judge prompts. Forward / Backward judge prompts — LaMP-3 (product rating). Forward instruction:You are an expert judge. Given a user summary, rank the following product reviews from most likely to le...
[11]

Solid blender, gasket feels cheap but it works

Rating 4/5: “Solid blender, gasket feels cheap but it works. . . ”
[12]

OMG best mascara ever!!! holy grail

Rating 5/5: “OMG best mascara ever!!! holy grail. . . ”
[13]

Toaster died after 6 weeks, re- turned

Rating 2/5: “Toaster died after 6 weeks, re- turned. . . ”
[14]

Loved this thriller, couldn’t put it down

Rating 5/5: “Loved this thriller, couldn’t put it down. . . ”
[15]

Kettle boils fast, lid hinge already loose

Rating 4/5: “Kettle boils fast, lid hinge already loose. . . ” Output (judge):[1, 5, 3, 2, 4] Expected Groundtruth:[1, 3, 5,· · ·] Backward instruction:You are an expert judge. Given a product review (with its rating), rank the following user summaries from most likely to least likely to describe its author. Input — Review h(p) u :Rating 2/5: “Toaster die...
[16]

Pragmatic small-appliance reviewer rating on dura- bility

“Pragmatic small-appliance reviewer rating on dura- bility. . . ”
[17]

Sci-fi book reader leaving long narrative re- views

“Sci-fi book reader leaving long narrative re- views. . . ”
[18]

Brand-loyal tech reviewer; rarely below 4 stars

“Brand-loyal tech reviewer; rarely below 4 stars. . . ” Output (judge):[2, 4, 3, 1] Expected Groundtruth:[2,· · ·] Figure 4: LaMP-3 forward and backward judge prompts. Forward / Backward judge prompts — LaMP-4 (news headline generation). Forward instruction:You are an expert judge. Given a user summary, rank the following news articles from most likely to...
[19]

Why You Keep Waking Up at 3 AM

Headline: “Why You Keep Waking Up at 3 AM”; Article: sleep researcher Q&A on cortisol cycles
[20]

Senate Passes Defense Authorization Bill

Headline: “Senate Passes Defense Authorization Bill”
[21]

5 Tiny Habits That Saved My Mar- riage

Headline: “5 Tiny Habits That Saved My Mar- riage”. . . [4]Headline: “Quarterly Earnings: TSMC Beats Esti- mates”
[22]

Your Morning Coffee Might Be Lying to You

Headline: “Your Morning Coffee Might Be Lying to You”. . . Output (judge):[1, 3, 5, 4, 2] Expected Groundtruth:[1, 3, 5,· · ·] Backward instruction:You are an expert judge. Given a news article, rank the following user summaries from most likely to least likely to describe its headline writer. Input — Article h(p) u :Headline: “5 Tiny Habits That Saved My...
[23]

Hard-news political reporter; declarative, neutral headlines

“Hard-news political reporter; declarative, neutral headlines. . . ”
[24]

Lifestyle and wellness writer with punchy second- person headlines

“Lifestyle and wellness writer with punchy second- person headlines. . . ”
[25]

Financial markets writer; numeric, ticker-heavy headlines

“Financial markets writer; numeric, ticker-heavy headlines. . . ”
[26]

Sports columnist; rhetorical-question headlines about teams

“Sports columnist; rhetorical-question headlines about teams. . . ” Output (judge):[2, 4, 1, 3] Expected Groundtruth:[2,· · ·] Figure 5: LaMP-4 forward and backward judge prompts. Forward / Backward judge prompts — LaMP-5 (scholarly title generation). Forward instruction:You are an expert judge. Given a user summary, rank the following academic works (abs...
[27]

Scalable Heterogeneous GNNs for Web- Scale Recommendation

Title: “Scalable Heterogeneous GNNs for Web- Scale Recommendation”; Abstract:
[28]

A Randomized Controlled Trial of

Title: “A Randomized Controlled Trial of. . . ”; Ab- stract: clinical trial methodology
[29]

Subgraph Sampling for Trillion-Edge Rep- resentation Learning

Title: “Subgraph Sampling for Trillion-Edge Rep- resentation Learning”; Abstract:
[30]

On the Stability of Tokamak Plasma

Title: “On the Stability of Tokamak Plasma. . . ”; Abstract:
[31]

Graph Attention for Cross-Domain Cold Start

Title: “Graph Attention for Cross-Domain Cold Start”; Abstract: . . . Output (judge):[1, 3, 5, 4, 2] Expected Groundtruth:[1, 3, 5,· · ·] Backward instruction:You are an expert judge. Given an academic work (abstract + title), rank the follow- ing user summaries from most likely to least likely to describe its author. Input — Work h(p) u :Title: “Subgraph...
[32]

Clinical-trial methodologist; titles emphasize study design

“Clinical-trial methodologist; titles emphasize study design. . . ”
[33]

Graph ML researcher favoring ‘X for Y’ titles on scalable representation learning

“Graph ML researcher favoring ‘X for Y’ titles on scalable representation learning. . . ”
[34]

Plasma physicist; titles foreground stability and confinement

“Plasma physicist; titles foreground stability and confinement. . . ”
[35]

NLP researcher focused on instruction tuning and alignment

“NLP researcher focused on instruction tuning and alignment. . . ” Output (judge):[2, 4, 1, 3] Expected Groundtruth:[2,· · ·] Figure 6: LaMP-5 forward and backward judge prompts. Forward / Backward judge prompts — LaMP-7 (tweet paraphrasing). Forward instruction:You are an expert judge. Given a user summary, rank the following tweets from most likely to l...
[36]

Excited to announce my new role @MegaCorp! #blessed #newchapter

“Excited to announce my new role @MegaCorp! #blessed #newchapter”
[37]

ate cold pizza standing up at 1am, peak fine din- ing

“ate cold pizza standing up at 1am, peak fine din- ing”
[38]

URGENT: Sign the petition NOW #ActOnCli- mate

“URGENT: Sign the petition NOW #ActOnCli- mate” [5]“halftime score is rigged i cant do this anymore” Output (judge):[1, 3, 5, 2, 4] Expected Groundtruth:[1, 3, 5,· · ·] Backward instruction:You are an expert judge. Given a tweet, rank the following user summaries from most likely to least likely to describe its author. Input — Tweet h(p) u :“ate cold pizz...
[39]

Corporate-comms-style tweeter; capitalized, hash- tagged announcements

“Corporate-comms-style tweeter; capitalized, hash- tagged announcements. . . ”
[40]

Casual lowercase NBA + late-night-food tweeter with dry self-deprecating humor

“Casual lowercase NBA + late-night-food tweeter with dry self-deprecating humor. . . ”
[41]

Activist account; urgent calls-to-action with peti- tions and hashtags

“Activist account; urgent calls-to-action with peti- tions and hashtags. . . ”
[42]

Tech-industry thread writer; numbered lists and product takes

“Tech-industry thread writer; numbered lists and product takes. . . ” Output (judge):[2, 4, 1, 3] Expected Groundtruth:[2,· · ·] Figure 7: LaMP-7 forward and backward judge prompts. Summary-generation prompt for BUMP — in- struction given to πθ to elicit the profile su from a user’s visible historyH vis u . Instruction (LaMP-1, citation identification):Be...

1990

[1] [1]

InPro- ceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 1–14, Kyoto, Japan

Llara: Large language-recommendation assis- tant. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1785–1795. Jiacheng Lin, Tian Wang, and Kun Qian. 2025. Rec-r1: Bridging generative large language models and user- centric recommendation systems via reinforcement learning.arXiv prepri...

work page arXiv 2025

[2] [2]

BPR: Bayesian Personalized Ranking from Implicit Feedback

Representation learning with large language models for recommendation. InProceedings of the ACM web conference 2024, pages 3464–3475. Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2012. Bpr: Bayesian personalized ranking from implicit feedback.arXiv preprint arXiv:1205.2618. Chris Richardson, Yao Zhang, Kellen Gillespie, ...

work page internal anchor Pith review Pith/arXiv arXiv 2024

[3] [3]

Qwen3 Technical Report

Qwen3 technical report.arXiv preprint arXiv:2505.09388. An Zhang, Yuxin Chen, Leheng Sheng, Xiang Wang, and Tat-Seng Chua. 2024. On generative agents in recommendation. InProceedings of the 47th inter- national ACM SIGIR conference on research and development in Information Retrieval, pages 1807– 1817. Jinghao Zhang, Yuting Liu, Wenjie Wang, Qiang Liu, Sh...

work page internal anchor Pith review Pith/arXiv arXiv 2024

[4] [4]

Tag:thriller; Title/Desc:Se7en– two detectives hunt a serial killer

[5] [5]

Tag:action; Title/Desc:Fast & Furious 9– street racers reunite

[6] [6]

[4]Tag:family; Title/Desc:Paw Patrol Movie

Tag:indie; Title/Desc:Manchester by the Sea– a janitor returns home. . . [4]Tag:family; Title/Desc:Paw Patrol Movie. . . [5]Tag:drama; Title/Desc:The Florida Project. . . Output (judge):[3, 5, 1, 2, 4] Expected Groundtruth:[1, 3, 5,· · ·] Backward instruction:You are an expert judge. Given a movie (with its tag), rank the following user summaries from mos...

[7] [7]

Action / blockbuster fan tagging mostly franchise tentpoles

“Action / blockbuster fan tagging mostly franchise tentpoles. . . ”

[8] [8]

Viewer who tags character-driven indie dramas and 90s thrillers

“Viewer who tags character-driven indie dramas and 90s thrillers. . . ”

[9] [9]

Animation enthusiast focused on family con- tent

“Animation enthusiast focused on family con- tent. . . ”

[10] [10]

Horror completionist with a taste for 80s slash- ers

“Horror completionist with a taste for 80s slash- ers. . . ” Output (judge):[2, 1, 4, 3] Expected Groundtruth:[2,· · ·] Figure 3: LaMP-2 forward and backward judge prompts. Forward / Backward judge prompts — LaMP-3 (product rating). Forward instruction:You are an expert judge. Given a user summary, rank the following product reviews from most likely to le...

[11] [11]

Solid blender, gasket feels cheap but it works

Rating 4/5: “Solid blender, gasket feels cheap but it works. . . ”

[12] [12]

OMG best mascara ever!!! holy grail

Rating 5/5: “OMG best mascara ever!!! holy grail. . . ”

[13] [13]

Toaster died after 6 weeks, re- turned

Rating 2/5: “Toaster died after 6 weeks, re- turned. . . ”

[14] [14]

Loved this thriller, couldn’t put it down

Rating 5/5: “Loved this thriller, couldn’t put it down. . . ”

[15] [15]

Kettle boils fast, lid hinge already loose

Rating 4/5: “Kettle boils fast, lid hinge already loose. . . ” Output (judge):[1, 5, 3, 2, 4] Expected Groundtruth:[1, 3, 5,· · ·] Backward instruction:You are an expert judge. Given a product review (with its rating), rank the following user summaries from most likely to least likely to describe its author. Input — Review h(p) u :Rating 2/5: “Toaster die...

[16] [16]

Pragmatic small-appliance reviewer rating on dura- bility

“Pragmatic small-appliance reviewer rating on dura- bility. . . ”

[17] [17]

Sci-fi book reader leaving long narrative re- views

“Sci-fi book reader leaving long narrative re- views. . . ”

[18] [18]

Brand-loyal tech reviewer; rarely below 4 stars

“Brand-loyal tech reviewer; rarely below 4 stars. . . ” Output (judge):[2, 4, 3, 1] Expected Groundtruth:[2,· · ·] Figure 4: LaMP-3 forward and backward judge prompts. Forward / Backward judge prompts — LaMP-4 (news headline generation). Forward instruction:You are an expert judge. Given a user summary, rank the following news articles from most likely to...

[19] [19]

Why You Keep Waking Up at 3 AM

Headline: “Why You Keep Waking Up at 3 AM”; Article: sleep researcher Q&A on cortisol cycles

[20] [20]

Senate Passes Defense Authorization Bill

Headline: “Senate Passes Defense Authorization Bill”

[21] [21]

5 Tiny Habits That Saved My Mar- riage

Headline: “5 Tiny Habits That Saved My Mar- riage”. . . [4]Headline: “Quarterly Earnings: TSMC Beats Esti- mates”

[22] [22]

Your Morning Coffee Might Be Lying to You

Headline: “Your Morning Coffee Might Be Lying to You”. . . Output (judge):[1, 3, 5, 4, 2] Expected Groundtruth:[1, 3, 5,· · ·] Backward instruction:You are an expert judge. Given a news article, rank the following user summaries from most likely to least likely to describe its headline writer. Input — Article h(p) u :Headline: “5 Tiny Habits That Saved My...

[23] [23]

Hard-news political reporter; declarative, neutral headlines

“Hard-news political reporter; declarative, neutral headlines. . . ”

[24] [24]

Lifestyle and wellness writer with punchy second- person headlines

“Lifestyle and wellness writer with punchy second- person headlines. . . ”

[25] [25]

Financial markets writer; numeric, ticker-heavy headlines

“Financial markets writer; numeric, ticker-heavy headlines. . . ”

[26] [26]

Sports columnist; rhetorical-question headlines about teams

“Sports columnist; rhetorical-question headlines about teams. . . ” Output (judge):[2, 4, 1, 3] Expected Groundtruth:[2,· · ·] Figure 5: LaMP-4 forward and backward judge prompts. Forward / Backward judge prompts — LaMP-5 (scholarly title generation). Forward instruction:You are an expert judge. Given a user summary, rank the following academic works (abs...

[27] [27]

Scalable Heterogeneous GNNs for Web- Scale Recommendation

Title: “Scalable Heterogeneous GNNs for Web- Scale Recommendation”; Abstract:

[28] [28]

A Randomized Controlled Trial of

Title: “A Randomized Controlled Trial of. . . ”; Ab- stract: clinical trial methodology

[29] [29]

Subgraph Sampling for Trillion-Edge Rep- resentation Learning

Title: “Subgraph Sampling for Trillion-Edge Rep- resentation Learning”; Abstract:

[30] [30]

On the Stability of Tokamak Plasma

Title: “On the Stability of Tokamak Plasma. . . ”; Abstract:

[31] [31]

Graph Attention for Cross-Domain Cold Start

Title: “Graph Attention for Cross-Domain Cold Start”; Abstract: . . . Output (judge):[1, 3, 5, 4, 2] Expected Groundtruth:[1, 3, 5,· · ·] Backward instruction:You are an expert judge. Given an academic work (abstract + title), rank the follow- ing user summaries from most likely to least likely to describe its author. Input — Work h(p) u :Title: “Subgraph...

[32] [32]

Clinical-trial methodologist; titles emphasize study design

“Clinical-trial methodologist; titles emphasize study design. . . ”

[33] [33]

Graph ML researcher favoring ‘X for Y’ titles on scalable representation learning

“Graph ML researcher favoring ‘X for Y’ titles on scalable representation learning. . . ”

[34] [34]

Plasma physicist; titles foreground stability and confinement

“Plasma physicist; titles foreground stability and confinement. . . ”

[35] [35]

NLP researcher focused on instruction tuning and alignment

“NLP researcher focused on instruction tuning and alignment. . . ” Output (judge):[2, 4, 1, 3] Expected Groundtruth:[2,· · ·] Figure 6: LaMP-5 forward and backward judge prompts. Forward / Backward judge prompts — LaMP-7 (tweet paraphrasing). Forward instruction:You are an expert judge. Given a user summary, rank the following tweets from most likely to l...

[36] [36]

Excited to announce my new role @MegaCorp! #blessed #newchapter

“Excited to announce my new role @MegaCorp! #blessed #newchapter”

[37] [37]

ate cold pizza standing up at 1am, peak fine din- ing

“ate cold pizza standing up at 1am, peak fine din- ing”

[38] [38]

URGENT: Sign the petition NOW #ActOnCli- mate

“URGENT: Sign the petition NOW #ActOnCli- mate” [5]“halftime score is rigged i cant do this anymore” Output (judge):[1, 3, 5, 2, 4] Expected Groundtruth:[1, 3, 5,· · ·] Backward instruction:You are an expert judge. Given a tweet, rank the following user summaries from most likely to least likely to describe its author. Input — Tweet h(p) u :“ate cold pizz...

[39] [39]

Corporate-comms-style tweeter; capitalized, hash- tagged announcements

“Corporate-comms-style tweeter; capitalized, hash- tagged announcements. . . ”

[40] [40]

Casual lowercase NBA + late-night-food tweeter with dry self-deprecating humor

“Casual lowercase NBA + late-night-food tweeter with dry self-deprecating humor. . . ”

[41] [41]

Activist account; urgent calls-to-action with peti- tions and hashtags

“Activist account; urgent calls-to-action with peti- tions and hashtags. . . ”

[42] [42]

Tech-industry thread writer; numbered lists and product takes

“Tech-industry thread writer; numbered lists and product takes. . . ” Output (judge):[2, 4, 1, 3] Expected Groundtruth:[2,· · ·] Figure 7: LaMP-7 forward and backward judge prompts. Summary-generation prompt for BUMP — in- struction given to πθ to elicit the profile su from a user’s visible historyH vis u . Instruction (LaMP-1, citation identification):Be...

1990