{"id":"8d48bdbb-a9aa-4609-a39e-1b52ef18b17f","arxiv_id":"2502.05822","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding pseudo-query pre-training and a hierarchical softmax loss to an ALBEF-style multimodal model improves query-video relevance for short video search ads.","lead":"HCMRM is a vision-language model for search ads that adds a pseudo-query pre-training task, where part of a video's own keyword list is treated as the search query, plus a hierarchical softmax loss that respects the four relevance levels. It was deployed in Kuaishou's search advertising system, and the paper reports a 6.1% drop in irrelevant ads and a 1.4% rise in ad revenue.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PQVM pre-training splits a video's own keyword sequence into a pseudo-query and a remaining suffix, so the text input seen in pre-training is not the full video text used in fine-tuning; the claimed 'high consistency' between pre-training and the downstream relevance task is therefore not actually…","rationale":"The reader's weakest assumption concerned whether a pseudo-query split from a video's own keywords behaves like a real user query. My stress-test identifies a related but more specific and internally checkable inconsistency: the PQVM pre-training procedure does not even use the same text-side input format as the downstream fine-tuning task. In pre-training, the video text is always the suffix of the keyword sequence after the pseudo-query prefix is removed; in fine-tuning, the video text is the full keyword sequence. This is not merely an external question about real-query distributions; it is a discrepancy visible directly from Equations 2-4 and the fine-tuning setup. The paper's stated motivation is 'high consistency' between pre-training and relevance tasks, so this input-mismatch undermines the central design rationale. That said, the paper provides some empirical support: PQVM improves multiple baselines, and pseudo-query pre-training outperforms click-based pre-training in their platform comparison. Those results are real evidence even if the mechanism is less clean than claimed. The concern is therefore not grounds for rejection, but it strengthens the conditional verdict: the offline and online gains attributed to PQVM need a direct test that separates genuine query-video semantic learning from a shortcut based on the artificial prefix/suffix split. The proposed overlap-partition experiment and the masked-full-text PQVM ablation would settle this. Because the reader already issued a conditional verdict and my concern reinforces that conditionality rather than moving it, the appropriate verdict remains UNCHANGED.","tokens_in":16320,"tokens_out":8744,"duration_ms":98263,"concrete_test":"Partition the 230K labeled query-video evaluation pairs by lexical overlap between the real query and the video's keyword sequence (e.g., high overlap: query tokens appear among the top-k keywords; low overlap: paraphrases or synonyms not present in the keyword list). Retrain HCMRM and ALBEF under identical settings and compare AUC/Spearman on each partition. If HCMRM's advantage is concentrated in the high-overlap partition and disappears or reverses in the low-overlap partition, then PQVM has learned prefix-overlap matching rather than robust query-video relevance. Additionally, run an ablation of PQVM that keeps the full video text but masks the pseudo-query tokens in T to match the downstream input format; if this variant does not retain the published offline gains, the reported benefit is an artifact of the split-input distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 defines PQVM positives as {pseudoQ_i, rem.T_i, I_i} and negatives as {pseudoQ_x, rem.T_i, I_i}, where pseudoQ_i is a prefix of the video's own keyword sequence (Eq. 2). The authors explicitly remove the pseudo-query prefix from the video text to avoid 'information leakage.' But the downstream relevance task in Section 3.5 feeds the model the full video text T, including those high-importance keywords. Thus the text-side input distribution differs between pre-training and fine-tuning: PQVM always sees only the tail of the keyword sequence as T, while fine-tuning always sees the whole sequence. The model is never pre-trained on the actual downstream input configuration (real query plus full T), so the claimed consistency between PQVM and QVM is at best approximate. Moreover, because the pseudo-query is a prefix of the same video's own text, positive pairs can be solved by lexical/statistical association with the removed prefix rather than by learning a general query-video relevance function. The paper's only defense (Section 4.7.1) compares against noisy click-derived queries on a single platform; it does not test generalization to clean real-user queries whose vocabulary may not overlap the video keywords. This makes the central mechanism's benefit, and hence the attribution of the reported gains to PQVM, insecure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HCMRM, an ALBEF-based multimodal relevance model for query-to-video matching in Kuaishou's search advertising system. The two main contributions are (i) a pre-training task called Pseudo-Query-Video Matching (PQVM), which synthesizes pseudo-queries by splitting a video's importance-ranked keyword sequence into a query prefix and a remaining text suffix, and (ii) a symmetric hierarchical softmax fine-tuning loss, defined in Equation 10, that combines three binary classifiers with an MSE term. Offline experiments on 2.5M labeled query-video pairs report improvements in AUC, Spearman, and Pearson over BERT, BERT-VL, ViLT, Query-LIFE, ALBEF, Qwen-VL, and MiniCPM-V-2.6. Online A/B tests report reductions in the irrelevant-ad ratio and increases in conversions and ad revenue, and the paper states the method has been deployed for over a year.","tokens_in":16613,"tokens_out":8954,"duration_ms":91406,"significance":"If the reported gains are reliable, this is a valuable industrial case study: it shows that a lightweight modification to ALBEF can improve query-video relevance without relying on noisy click logs, and it demonstrates a practical ordinal relevance loss for ranking. The paper's strengths include the scale of the pre-training data (200M videos), the 2.5M labeled query-video fine-tuning set, the breadth of baselines including multimodal LLMs, and the evidence from a real deployed advertising system. However, the incremental offline gains over ALBEF are small (0.003 AUC, 0.005 Spearman), no confidence intervals or significance tests are reported, the PQVM pre-training and downstream QVM input distributions differ in a way that weakens the 'high consistency' claim, and the abstract's aggregate online numbers appear to be obtained by summing two separate A/B tests. These issues currently leave the attribution of the reported gains to the two proposed components less secure than the presentation suggests.","major_comments":[{"comment":"The PQVM pre-training task feeds [CLS] pseudoQ_i [SEP] rem.T_i to the text encoder, while the downstream QVM task uses the full video keyword sequence T. Thus the model is never pre-trained on the exact downstream input configuration (real query plus full video text). Moreover, because pseudoQ_i and rem.T_i are contiguous pieces of the same original keyword sequence, positive PQVM pairs can be solved by lexical/structural continuity between the prefix and suffix rather than by a general query-video relevance function. The comparison in Section 4.7.1 against click-derived queries does not resolve this, since it does not test with clean real-user queries or quantify lexical overlap. Please report how often pseudo-query terms appear in rem.T, compare with a variant that uses the full T on the video side during PQVM, and consider a pseudo-query generation mechanism that is not a prefix of the same video's own text.","section":"Section 3.4, Eq. (2); Section 3.5"},{"comment":"The proposed loss is the sum of a hierarchical softmax negative log-likelihood and an MSE term (r - l)^2. The paper never ablates these two terms. Table 2 compares 'hierarchical softmax' (i.e., the combined loss) against BCE, MSE, and ordinal regression, so the reported gains could be driven partly or wholly by the MSE regression component rather than by the hierarchical structure. Please report hierarchical softmax NLL alone, the MSE term alone, and the combination, and justify why both are needed.","section":"Section 3.5, Eq. (10); Table 2"},{"comment":"No confidence intervals, standard deviations, or significance tests are reported for any offline metric. The headline improvement over ALBEF is 0.003 in AUC and 0.005 in Spearman, which is the same order as typical run-to-run variation for large-scale fine-tuning. Please provide results over multiple random seeds (or a paired significance test on the evaluation set) for at least the ALBEF vs HCMRM comparison and the loss-function comparison, so the reader can judge whether the proposed components improve over the baselines beyond noise.","section":"Section 4.5, Tables 1 and 5"},{"comment":"The abstract and conclusion state a 6.1% reduction in the irrelevant-ad ratio and a 1.4% increase in ad revenue, but the online tables report separate A/B experiments: -4.0%/+1.0% for the model change versus BERT-VL and -2.1%/+0.4% for the loss change versus BCE. Simply adding percentages from independent tests is not a valid estimate of the combined effect. Please report a single combined A/B test, or clearly state how the aggregate figures were computed and whether the two effects were actually measured jointly.","section":"Abstract; Section 4.6, Tables 3 and 4"},{"comment":"Appendix A says the model deployed online is a lightweight BERT-VL-based student distilled from the offline HCMRM teacher, yet Section 4.6.1 states the A/B test compares HCMRM with BERT-VL. Please clarify what exactly differs between the online experimental buckets: is the online comparison between students distilled from HCMRM and BERT-VL teachers, or between the full teachers? If the former, the online results validate the distillation pipeline and the teacher's scores, not HCMRM itself, and this should be stated explicitly.","section":"Section 4.6.1; Appendix A"},{"comment":"The comparison intended to show that pseudo-queries 'surpass real queries' confounds data source with data size: row 2 uses 60M click-derived query-video pairs, while row 4 uses 200M video materials. The two-stage row 3 partially addresses this, but the total amount of pre-training data still differs. Please add a matched comparison, e.g., ALBEF pre-trained on a 60M subset of the material data with a QVM task on the click data, or HCMRM with pseudo-queries on the 60M click-derived video subset, so the source of the gain is identifiable.","section":"Section 4.7.1, Table 5"}],"minor_comments":[{"comment":"The range [a, b] for the pseudo-query length is never specified in the paper; please provide the value used in the experiments and, ideally, a small sensitivity analysis.","section":"Section 3.4, Eq. (2)"},{"comment":"The field importance weights alpha_i are defined but their values are not reported; please state the weights used for title, description, OCR, ASR, and the other fields.","section":"Section 3.2, Algorithm 1"},{"comment":"The AUC metric is computed on binary labels derived by collapsing 0/1 and 2/3, while the model outputs a continuous score in [0, 3]; please clarify exactly how AUC is computed over the evaluation set.","section":"Section 4.2"},{"comment":"The description 'balanced binary tree' is informal; Equation 8 does follow from the tree, but the sentence should say 'a two-level hierarchical decomposition' rather than implying a standard binary search tree over labels.","section":"Section 3.5, Figure 3"},{"comment":"The input formatting for Qwen-VL and MiniCPM-V-2.6 is not described; since these models have different tokenizers and image processors, please specify how the query, video text, and video frames were packaged for fine-tuning.","section":"Section 4.7.2, Table 6"}],"recommendation":"major_revision","confidential_remarks":"This is a credible industrial systems paper with clear practical value and real deployment evidence. The main issues are statistical substantiation, missing ablations, and the attribution of online gains, all of which are addressable. I do not recommend rejection, but the current manuscript overstates the certainty of the offline and aggregate online results. If the authors can supply the requested ablations and significance evidence, the paper could be acceptable as a companion-paper-level contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper’s real novelty is PQVM: during pre-training, it splits a video’s keyword sequence into a pseudo-query prefix and a remaining suffix, then trains a query-video matching task on the triplet. That lets you get query awareness without click logs. The second useful piece is a symmetric hierarchical softmax loss that respects the ordinal nature of four-level relevance labels and improves ranking correlation. The method is deployed in Kuaishou’s search ads, with a reported 6.1% reduction in irrelevant ads and 1.4% revenue lift.\n\nThe paper does what an industrial systems paper should: it compares against sensible baselines, ablates PQVM on several backbones, and reports both offline and online results. The comparison in Table 5 is probably the strongest point—pseudo-queries from video materials beat click-derived real queries on their platform, which directly supports the design choice. The hierarchical softmax loss also shows a clear Spearman/Pearson improvement over BCE, ordinal regression, and MSE.\n\nSoft spots, in proportion. The offline gains over ALBEF are small: 0.003 AUC, 0.005 Spearman, and there are no confidence intervals or significance tests. That matters because the online gains come from a distilled student model, so the attribution is indirect. The compound loss in Equation 10 mixes hierarchical softmax with an MSE term and is not ablated. The stress-test concern about the input distribution gap is real but not a killer: pre-training always truncates the video text to the tail after removing the pseudo-query, while fine-tuning uses the full text. So the model is never trained on the exact downstream input, and the positive pairs are easier than real queries because the pseudo-query comes from the same video’s own text. The paper’s defense—that pseudo-queries outperform click queries—is platform-specific and doesn’t prove generalization to clean real queries with non-overlapping vocabulary. I’d like to see the authors discuss that gap honestly.\n\nOverall: this is a solid industrial contribution, not a research breakthrough. Worth serious refereeing. A referee should push for significance tests, a loss ablation, and a more nuanced discussion of the synthetic-to-real query gap.","headline":"A useful industrial contribution with a clever pseudo-query pre-training task and a sensible ordinal loss, backed by a year of deployment; the main weakness is that the offline gains are small and the synthetic-to-real query gap is only partially addressed.","tokens_in":17148,"tokens_out":3004,"would_cite":true,"duration_ms":29138,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-training a search-ad relevance model on pseudo-queries cut from a video's own keywords, then fine-tuning with a hierarchical softmax loss, reduced irrelevant ads by 6.1% and raised ad revenue by 1.4% in the deployed system.","keywords":["multimodal relevance","search advertising","query-video matching","vision-language pre-training","pseudo-query","hierarchical softmax","short video ads","relevance ranking"],"falsifier":"A controlled experiment where PQVM is replaced with real user queries from the same platform during pre-training, holding the model and all other objectives fixed: if the real-query variant matches or exceeds HCMRM's offline AUC and Spearman on the 230K-sample evaluation set, the paper's claim that pseudo-queries are sufficient—and better than click-derived queries—would be called into question. Alternatively, removing PQVM from HCMRM while keeping the hierarchical softmax loss and all other settings identical should measurably lower Spearman on the same evaluation set if the mechanism carries the reported ranking gain.","tokens_in":16118,"feed_emoji":"🎯","tokens_out":6681,"duration_ms":58405,"temperature":0.7,"pith_summary":"The paper claims that query-to-video relevance in search advertising improves when the model is pre-trained to match a synthetic query—made by cutting the video's own keyword list at a random point—against the video's remaining text and frames. This pseudo-query-video matching (PQVM) task makes pre-training mirror the downstream relevance task without needing click logs, which the paper argues are noisy and sparse. On top of that, the paper replaces binary cross-entropy fine-tuning with a symmetric hierarchical softmax loss that learns the order among the four relevance labels (Bad, Less, Good, Excellent). Deployed in the Kuaishou search advertising system, the combined method reduced the proportion of irrelevant ads by 6.1% and raised ad revenue by 1.4% over the previous BERT-VL model.","feed_headline":"Pseudo-queries from video text trim irrelevant search ads by 6.1%","feed_subtitle":"A relevance model pre-trains on synthetic queries and a hierarchical softmax loss, then beats the deployed baseline in live A/B tests.","key_machinery":"Two mechanisms carry the argument. PQVM: during pre-training, the video keyword sequence is split at a random index into a pseudo-query and the remaining video text; positive and hard-negative triplet pairs are trained with a binary cross-entropy classifier on the fusion encoder's [CLS] embedding, making pre-training structurally identical to the downstream query-video relevance task. Symmetric hierarchical softmax: the four relevance labels form a two-level binary tree, so the label distribution factors into three sigmoid probabilities ($p_{pos}$, $p_{le}$, $p_{ex}$); the expected label value (Equation 9) is the relevance score used for ranking, and the loss (Equation 10) combines the hierarchical softmax likelihood with an MSE regression term. The architecture itself is reused from ALBEF with minimal modification: a text encoder, a vision encoder, and a fusion encoder with cross-attention.","core_discovery":"HCMRM is an ALBEF-style dual-stream vision-language model—a text encoder, an image encoder, and a fusion encoder—pre-trained with four objectives. Three are standard (image-text contrastive learning, image-text matching, masked language modeling); the fourth, PQVM, is the paper's contribution. For each video, its keyword sequence is split at a random length in [a,b]: the left part becomes the pseudo-query and the right part remains the video text, with the full keyword sequence excluded to avoid information leakage. The model must classify whether a pseudo-query matches a video, using hard negatives sampled from the in-batch similarity distribution. For fine-tuning, the four relevance labels are organized as a balanced binary tree ('Is Positive?', then 'Is Less?' among negatives and 'Is Excellent?' among positives), producing three binary classifiers whose probability products give the label distribution and an expected relevance score in [0,3]. The paper reports that this architecture outperforms unimodal and multimodal baselines and both Qwen-VL and MiniCPM-V-2.6 on offline AUC, Spearman, and Pearson metrics, and beats the deployed BERT-VL online.","pith_inferences":["Inference: The pseudo-query synthesis assumes the head of the keyword list is a faithful proxy for user intent; this is plausible for title-like texts but may fail for videos where the query targets an aspect not captured by top keywords, and varying the split range or weighting keywords by field type could be a testable refinement.","Inference: The same synthetic-query pre-training idea could transfer to other retrieval domains with rich item-side text, such as product search or news video search, wherever a query is a short text about the item.","Inference: The symmetric hierarchical softmax loss is a general ordinal-classification technique that could improve other graded-relevance ranking systems beyond ads, such as answer ranking or recommendation explanations.","Inference: The paper's comparison against click-based pre-training (Table 5) suggests that synthetic queries from item text may be a cheaper and less noisy alternative to behavior logs; verifying this on a public dataset would be a direct next step."],"forward_implications":["Relevance models for search ads can be made query-aware without any query logs, using only video self-text.","The PQVM pre-training task transfers to other base architectures: BERT, BERT-VL, and ViLT all improve when PQVM is added (Table 1).","The hierarchical softmax loss improves Spearman and Pearson rank correlation substantially over binary cross-entropy, aligning relevance scores better with the downstream ad-ranking objective.","Multimodal large language models fine-tuned on the same data do not beat the domain-pre-trained HCMRM, suggesting that domain-specific pre-training still matters.","The deployed impact is a 6.1% relative reduction in irrelevant ads and a 1.4% ad revenue increase."],"supporting_citations":[{"why":"Supplies the base ALBEF architecture and the ITC/ITM/MLM pre-training objectives that HCMRM extends with PQVM.","marker":"[15]"},{"why":"Introduces leaf-smoothed hierarchical softmax for ordinal prediction, the basis of the paper's symmetric hierarchical softmax loss.","marker":"[25]"},{"why":"A query-aware multimodal video-search relevance model that uses click-derived query-video pairs; one of the two main query-aware baselines PQVM is compared against.","marker":"[33]"},{"why":"Query-LIFE, an e-commerce query-aware multimodal embedding method that also relies on click query-item pairs; a baseline PQVM is compared against.","marker":"[37]"},{"why":"Provides evidence that click data in search is noisy due to exposure bias and clickbait, supporting the paper's argument for pseudo-queries over click-derived queries.","marker":"[38]"},{"why":"BERT-base-chinese initializes the text and fusion encoders; the word-level masked language modeling follows BERT.","marker":"[5]"}],"fun_headline_variants":["Pseudo-queries from video text cut irrelevant search ads 6.1%","Multimodal model with pseudo-query pre-training lifts ad revenue 1.4%","Hierarchical softmax loss powers better ad relevance in search","HCMRM: Triplet modeling reduces irrelevant ads, boosts revenue"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that splitting a video's own keyword sequence at a random position produces a pseudo-query that behaves like a real user query for learning relevance; if real queries systematically target content outside the top keywords, the PQVM pre-training could teach the model the wrong kind of query-awareness.","fun_headline_variants_meta":{"raw":{"variants":["Pseudo-queries from video text cut irrelevant search ads 6.1%","Multimodal model with pseudo-query pre-training lifts ad revenue 1.4%","Hierarchical softmax loss powers better ad relevance in search","HCMRM: Triplet modeling reduces irrelevant ads, boosts revenue"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000519,"raw_usage":{"total_tokens":2582,"prompt_tokens":1080,"completion_tokens":1502,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":696,"completion_tokens_details":{"reasoning_tokens":1430}},"tokens_in":696,"tokens_out":1502,"duration_ms":10960,"temperature":1.0,"reasoning_tokens":1430,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:49:18.484114+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled experiment where PQVM is replaced with real user queries from the same platform during pre-training, holding the model and all other objectives fixed: if the real-query variant matches or exceeds HCMRM's offline AUC and Spearman on the 230K-sample evaluation set, the paper's claim that pseudo-queries are sufficient—and better than click-derived queries—would be called into question. Alternatively, removing PQVM from HCMRM while keeping the hierarchical softmax loss and all other settings identical should measurably lower Spearman on the same evaluation set if the mechanism carries the reported ranking gain.","supporting_citations":[{"cited_title":"Selvaraju, Akhilesh Deepak Gotmare, Shafiq R","cited_arxiv_id":null,"evidence_quote":"Supplies the base ALBEF architecture and the ITC/ITM/MLM pre-training objectives that HCMRM extends with PQVM."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces leaf-smoothed hierarchical softmax for ordinal prediction, the basis of the paper's symmetric hierarchical softmax loss."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"A query-aware multimodal video-search relevance model that uses click-derived query-video pairs; one of the two main query-aware baselines PQVM is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Query-LIFE, an e-commerce query-aware multimodal embedding method that also relies on click query-item pairs; a baseline PQVM is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence that click data in search is noisy due to exposure bias and clickbait, supporting the paper's argument for pseudo-queries over click-derived queries."}],"review_version":1}