{"id":"b8436c4b-5d27-4a4b-b301-b7033f03f227","arxiv_id":"2501.00330","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"LUSAR applies listwise sampling and ranking to multimodal LLMs for entity set expansion and reports improved MESED scores, though the gains are confounded with supervised fine-tuning.","lead":"This paper tests whether multimodal large language models can expand a small set of seed entities into a full semantic category, and proposes a listwise sampling-and-ranking method called LUSAR. The method shows large score gains on the MESED benchmark, but the experiments do not cleanly separate the new ranking mechanism from the fine-tuning used to train it.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Listwise mechanism is confounded with GPT-4 SFT: without a pointwise SFT baseline, the reported LUSAR gains cannot be attributed to listwise ranking.","rationale":"The paper's central claim is that LUSAR, a listwise sampling-and-ranking method, significantly improves MLLM performance on MESE. The most load-bearing threat to that claim is not a formal inconsistency in the sampling argument but an experimental confound: the LUSAR pipeline includes supervised fine-tuning on GPT-4-generated ranking data, and the reported baselines do not include an equally fine-tuned pointwise model. Without that control, the Table I gains could be caused by SFT-induced formatting compliance or GPT-4 knowledge distillation rather than by listwise comparison. The reader's strongest claim already conditions the result on excluding SFT and hyperparameter selection, and the rationale explicitly notes the missing SFT-only baseline, so this is not a new objection. However, the reader's stated weakest assumption focuses on uniform sampling representativeness and beam-search recall; I regard the SFT confound as more directly tied to the central causal claim. The proposed concrete test is a pointwise SFT baseline trained on the same data and evaluated with the same aggregation; this single experiment would settle whether listwise ranking is the operative mechanism. Because the concern is addressable and the paper is otherwise clearly described, the existing CONDITIONAL verdict remains appropriate; no verdict change is needed.","tokens_in":8154,"tokens_out":2748,"duration_ms":28513,"concrete_test":"Train a pointwise SFT baseline using the same backbone, the same roughly 4,000 GPT-4 entries converted into independent single-entity scoring examples, the same LoRA budget, and the same training schedule; at inference, score each candidate independently and aggregate with Equation 3 without any listwise comparison. Compare MAP@10/20/50/100 and P@10/20/50/100 on MESED against Table I. If this pointwise SFT baseline closes the gap to LUSAR, the central claim that listwise ranking drives the improvement is not supported; if LUSAR remains substantially ahead, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section II.B.3 fine-tunes the backbone with LoRA on roughly 4,000 GPT-4-generated ranking lists, while Section III.C's only non-listwise contrast is a single-MLLM 'absolute scoring' baseline with no indication that it received the same SFT. The headline comparison in Table I is therefore LUSAR (listwise prompts plus SFT) against raw Qwen/Deepseek models without SFT. Any component of the pipeline—prompt formatting, GPT-4 distillation, LoRA adaptation, or listwise aggregation—could produce the reported gains, so the specific contribution attributed to the listwise mechanism is not isolated. A second unmodeled degree of freedom is o=10 and n=5: Section III.E reports these as best but does not state that they were selected on a validation split rather than by peeking at the test queries; if tuned on test, part of the gain is selection. The representativeness assumption in Section II.B.2 is also informal, but it is secondary: even if uniform sampling is perfectly representative, the SFT confound prevents attributing the improvement to the listwise component.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LUSAR, a two-stage method for multimodal entity set expansion (MESE) with generative multimodal large language models. In the first stage, a prefix-constrained beam search makes the model generate candidate entities from a fixed vocabulary. In the second stage, candidate entities are repeatedly sampled into short lists and ranked by the MLLM; the ranks are aggregated by Eq. (3) into a global score for every candidate. The model is fine-tuned with LoRA on roughly 4,000 GPT-4-generated ranking lists. Experiments on the MESED benchmark report that LUSAR improves over text-based, vision-based, and generative MLLM baselines on MAP@K and P@K, with the best results for Qwen2-VL-7b. The paper claims that this is the first use of generative MLLMs for ESE and that listwise ranking unlocks implicit semantic reasoning in MLLMs.","tokens_in":8380,"tokens_out":3700,"duration_ms":34446,"significance":"If the reported gains are caused by the listwise sampling-and-ranking mechanism, LUSAR is a useful methodological contribution: it extends listwise ranking beyond retrieval-style tasks to entity-level semantic judgment in MLLMs, and it provides a concrete probe for studying implicit semantic inference. The paper's strengths are that it evaluates against several established baselines, uses a public benchmark (MESED), and reports standard metrics at multiple cutoffs. The main weakness is attribution: the headline comparison conflates the listwise mechanism with supervised fine-tuning on GPT-4 data, and no code or data release is mentioned, so the contribution is not yet fully reproducible. The ablation of o and n is informative but incomplete without a described model-selection protocol.","major_comments":[{"comment":"The main comparison in Table I is between LUSAR, which includes LoRA SFT on roughly 4,000 GPT-4-generated ranking lists (Section II.B.3), and raw generative MLLM baselines, including the single-MLLM absolute scoring baseline of Section III.C. The latter is the only non-listwise generative contrast, but the paper gives no indication that it received the same SFT. Consequently, the observed gains could be due to SFT, prompt formatting, or LoRA adaptation rather than to the listwise sampling and aggregation mechanism. Please add a pointwise scoring baseline that is fine-tuned with the same SFT data and a non-listwise SFT baseline for Qwen2-VL-7b.","section":"III.C and Table I"},{"comment":"The choices o=10 and n=5 are described as 'best' but the paper does not state whether these were selected on a held-out validation split or on the test queries. If selection used test queries, the reported gains include selection bias. Specify the selection procedure and, if possible, report performance across the sweep rather than only the best point.","section":"III.E"},{"comment":"The claim that uniform random sampling lets local list ranks reflect global ranking is informal. No formal representativeness condition, sample-size bound, or variance analysis is given, and with n=5 and o=10 it is not obvious that aggregate scores are stable. Provide a formal statement or an empirical stability analysis.","section":"II.B.2, Eq. (3)"},{"comment":"The prefix-constrained beam search must generate every true positive entity in Step 1; any positive entity absent from the candidate set is unrecoverable. The paper gives no analysis of recall of this candidate generation stage. Report the recall of the candidate set with respect to ground-truth positives, or justify the beam size.","section":"II.A"},{"comment":"The Deepseek-VL-7b generative baseline row reports P@10, P@20, P@50, and P@100 values (86.60, 79.15, 68.03, 61.12) identical to the ALBEF row. This likely indicates a copying error and undermines confidence in the baseline table; please correct and verify all baseline numbers.","section":"Table I"}],"minor_comments":[{"comment":"There are several typos and grammatical errors, e.g., 'we applies' should be 'we apply' and 'can serves' should be 'can serve'; please proofread.","section":"Abstract and Introduction"},{"comment":"The text contains misspellings and encoding issues, such as 'patial', 'canditate', and '¡EOS¿'; these should be cleaned before publication.","section":"II.A"},{"comment":"Figure 1 shows 'Llama 7b' in Step 1, but the experiments use Qwen-VL-Chat, Qwen2-VL-7b, and Deepseek-VL-7b; please make the figure consistent with the actual backbone models.","section":"Figure 1"},{"comment":"The variable M is used in Eq. (3) before its definition, which appears only in the following sentence; reorder the text so M is defined before the equation.","section":"II.B.2, Eq. (3)"},{"comment":"The dataset section does not state the candidate vocabulary size or the number of candidate entities per query; adding these statistics would help readers assess the difficulty of the ranking problem.","section":"III.A"},{"comment":"The conclusion contains a duplicated period and it repeats the introduction's claim about being 'the first to apply LLMs to the ESE task'; please reconcile this with the generative MLLM baselines listed in Section III.B, which are also LLM-based.","section":"IV"}],"recommendation":"major_revision","confidential_remarks":"The MESED benchmark is authored by overlapping authors (ref. [18] shares authors with this paper), so using it as the sole evaluation benchmark raises a mild novelty-disclosure concern, though the comparison against external baselines mitigates the circularity risk. The manuscript does not mention code or data release; for a pilot study, releasing the SFT data, the sampling hyperparameter sweep, and the exact fine-tuning recipe would materially improve reproducibility and would allow the field to verify the attribution of the gains to the listwise mechanism."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. The genuinely new piece is LUSAR: sample random n=5 entity lists from the candidate pool, have the MLLM rank each list, aggregate the ranks with Eq. 3, and use the aggregate to order the full set. It is a clean recombination of classic listwise LTR [21,22] and generative ESE [19,24], and it looks like the first generative MLLM for multimodal entity set expansion. The paper writes \"first use of generative MLLM\" in the abstract and \"first to apply LLM\" in the conclusion; the latter is contradicted by its own refs 19/24, so the claim should be narrowed to MLLM. The method section is readable, the aggregation is simple and reproducible, and the consistent gains across Qwen-VL-Chat, Qwen2-VL-7b, and Deepseek-VL-7b make the phenomenon worth checking.\n\nThe main problem is the comparison in Table I. The LUSAR rows include listwise prompts, LoRA SFT on about 4,000 GPT-4-generated lists, and LLaVA-instruct mixing. The generative MLLM baselines are zero-shot. The \"single MLLM absolute scoring\" baseline described in Section III.C is the right control, but there is no indication it received the same SFT, and it has no row in Table I. So the observed gains cannot be attributed to the listwise mechanism; they could come from SFT, from GPT-4 distillation, from LoRA, or from prompt formatting. This is a load-bearing confound, and it is fixable. A pointwise SFT baseline on the same data would settle it.\n\nSecond issue: hyperparameters o=10 and n=5 are reported as best in Section III.E, but the paper never says they were chosen on a validation split. If they were tuned against test queries, part of the gain is selection. Minor but needs stating. Third, the prefix-constrained beam search in Step 1 is the only candidate generator, and any true entity it misses is unrecoverable; the paper reports no candidate recall. That could matter more in practice than on MESED. Finally, evaluation is on MESED, built by overlapping authors; not disqualifying since they face external baselines, but no code or data are released, so independent checks are limited.\n\nThe stress-test note is correct, and the secondary concern about representativeness of random sampling is indeed secondary: even perfect sampling would not fix the SFT confound.\n\nThis paper is for readers working on set expansion or listwise prompting for MLLMs. It deserves a serious referee — I would send it to review, not desk reject, but the revision must include the pointwise SFT baseline, validation-set hyperparameter selection, and candidate recall. With those, it becomes a credible short paper.","headline":"A plausible listwise ranking method for multimodal entity set expansion, but the reported gains are confounded with GPT-4 SFT and the missing pointwise control prevents attribution.","tokens_in":8957,"tokens_out":3246,"would_cite":false,"duration_ms":31084,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LUSAR, a listwise sampling-and-ranking method, significantly improves multimodal large language models on entity set expansion by converting local list rankings into a global score per candidate.","keywords":["entity set expansion","multimodal large language models","listwise ranking","implicit semantic reasoning","LUSAR","MESED","constrained decoding","low-rank adaptation"],"falsifier":"Fine-tune the same MLLM on the same GPT-4 ranking data but replace the listwise prompt with a pointwise prompt that asks for an absolute relevance score per candidate; if MAP@10 on MESED does not drop below LUSAR's, then the listwise sampling and aggregation are not the source of the gains.","tokens_in":7950,"feed_emoji":"🤖","tokens_out":10831,"duration_ms":95626,"temperature":0.7,"pith_summary":"Generative multimodal large language models can expand a handful of seed entities into their hidden semantic class—solving the Multi-modal Entity Set Expansion task—if the model is asked to rank small candidate lists instead of assigning absolute scores. The paper proposes LUSAR, a listwise sampling-and-ranking method: it samples many short candidate lists, has the MLLM rank each list, and aggregates the local ranks into one global score per entity. On the MESED benchmark, LUSAR lifts Qwen-VL-Chat from 52.07 to 68.45 MAP@10 and lets Qwen2-VL-7b reach 82.15 MAP@10, with consistent gains across precision-at-k metrics. This matters because it is the first demonstration that a generative MLLM, not just retrieval-based models, can do entity set expansion, and it suggests a general recipe for eliciting implicit semantic knowledge from large models.","feed_headline":"Listwise ranking unlocks multimodal AI for entity-set expansion","feed_subtitle":"Small ranked lists, aggregated globally, let multimodal models infer hidden semantic classes from a few seeds.","key_machinery":"The load-bearing object is LUSAR (Listwise Multimodal Sampling And Ranking), a two-stage pipeline. Stage one generates candidate entities with prefix-constrained beam search, so the model only emits tokens that complete valid entities from the vocabulary. Stage two builds $M = o \\cdot |\\text{candidates}| / n$ random lists of $n=5$ candidates, each appearing $o=10$ times, prompts the MLLM to rank the list by its relevance to the seed class, and aggregates the results with the position-weighted scoring rule $\\operatorname{score}_{e_k} = \\sum_{m=1}^{M} \\sum_{i=1}^{n} i \\cdot \\mathbb{I}_{e_k = r_{mi}}$. A low-rank (LoRA) fine-tune on GPT-4-generated ranking data, mixed with LLaVA-Instruct-150K to preserve general abilities, teaches the model the output format and sharpens its discrimination between true positives and distractors. The identity that carries the argument is that a short list's local rank, aggregated over many random lists, approximates the entity's global rank in the full candidate set.","core_discovery":"The paper's central claim is that listwise comparison, not absolute scoring, is what unlocks implicit semantic reasoning in multimodal large language models for entity set expansion. Its LUSAR pipeline first uses prefix-constrained beam search over an entity vocabulary to generate a candidate set, then repeatedly samples five candidates at a time (each entity appears in ten lists), asks the MLLM to rank each list by relevance to the seed set, and converts the local ranks into a global score via $\\operatorname{score}_{e_k} = \\sum_{m=1}^{M} \\sum_{i=1}^{n} i \\cdot \\mathbb{I}_{e_k = r_{mi}}$. The ranked candidates are then used directly for evaluation on the MESED benchmark, where fine-grained classes and hard negative entities make the task genuinely difficult for models that rely on explicit prompts. The authors report that this approach outperforms pointwise MLLM scoring by large margins, and rivals or exceeds specialized ESE baselines, marking the first use of a generative MLLM for entity set expansion.","pith_inferences":["The reported gains are not cleanly attributed to the listwise mechanism alone, because the model is also fine-tuned on GPT-4-generated ranking data; an ablation that trains on the same data with a pointwise prompt would separate the two effects.","Because the candidate set comes from beam search, the method's ceiling is set by the first stage: any true positive that the generator never emits can never be recovered by ranking.","The same score-aggregation rule could be applied to any few-exemplar ranking task—recommendation, retrieval re-ranking, or taxonomy expansion—where the model cannot see all candidates in one context.","The paper's probabilistic justification for uniform sampling is heuristic; deriving a sample-complexity bound for how many lists are needed to recover a global order with high probability would make the method's reliability predictable."],"forward_implications":["Generative MLLMs become a viable tool for entity set expansion without corpus statistics or context-feature engineering, as long as the task is framed as ranking small lists.","The listwise sampling-and-aggregation recipe is portable: any LLM that can compare a handful of items can be turned into a global ranker over a large candidate set.","Multimodal inputs (images) contribute to disambiguation of synonyms, polysemy, and hard negatives, so the method's gains are largest when each entity has a representative image.","The operating point $n=5$, $o=10$ is reported as optimal on MESED, implying that listwise aggregation has tunable sample-complexity parameters that practitioners can adjust per task."],"supporting_citations":[{"why":"MESED; the benchmark dataset with 70 fine-grained classes and hard negatives used in all evaluations.","marker":"[18]"},{"why":"Listwise learning-to-rank theory; supplies the conceptual basis for deriving a global order from local list rankings.","marker":"[21]"},{"why":"Prefix-constrained decoding; provides the beam-search candidate-generation step of Stage 1.","marker":"[24]"},{"why":"LoRA; the fine-tuning method used to adapt the backbone MLLM to listwise ranking.","marker":"[25]"},{"why":"LLaVA-Instruct-150K; mixed into SFT to preserve general capabilities while learning the ranking format.","marker":"[26]"},{"why":"CLIP; a vision-language baseline that shows images alone do not solve the task.","marker":"[30]"},{"why":"ALBEF; the strongest multimodal ESE baseline, which LUSAR's best model surpasses on several metrics.","marker":"[31]"},{"why":"Qwen-VL-Chat; the primary backbone whose pointwise score (52.07 MAP@10) is the direct baseline for LUSAR.","marker":"[32]"},{"why":"Deepseek-VL-7b; a second MLLM backbone demonstrating that listwise gains transfer across model families.","marker":"[33]"},{"why":"Qwen2-VL-7b-Instruct; the strongest LUSAR backbone, reaching 82.15 MAP@10.","marker":"[34]"}],"fun_headline_variants":["Listwise ranking boosts multimodal entity set expansion","Ranking small lists beats absolute scores in multimodal LLMs","LUSAR uses listwise ranking to improve entity expansion","Multimodal LLMs expand entities better with listwise ranking","Entity expansion: listwise ranking outperforms pointwise scoring"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a few randomly assembled five-item lists, with each candidate appearing ten times, produce aggregate scores that faithfully reproduce the entity's true rank in the whole candidate set—and that the beam-search candidate generator has already found every entity that belongs to the target class.","fun_headline_variants_meta":{"raw":{"variants":["Listwise ranking boosts multimodal entity set expansion","Ranking small lists beats absolute scores in multimodal LLMs","LUSAR uses listwise ranking to improve entity expansion","Multimodal LLMs expand entities better with listwise ranking","Entity expansion: listwise ranking outperforms pointwise scoring"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1331,"prompt_tokens":905,"completion_tokens":426,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":347}},"tokens_in":521,"tokens_out":426,"duration_ms":4629,"temperature":1.0,"reasoning_tokens":347,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:53:07.672512+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune the same MLLM on the same GPT-4 ranking data but replace the listwise prompt with a pointwise prompt that asks for an absolute relevance score per candidate; if MAP@10 on MESED does not drop below LUSAR's, then the listwise sampling and aggregation are not the source of the gains.","supporting_citations":[{"cited_title":"Mesed: A multi-modal entity set expansion dataset with fine-grained semantic classes and hard negative entities,","cited_arxiv_id":null,"evidence_quote":"MESED; the benchmark dataset with 70 fine-grained classes and hard negatives used in all evaluations."},{"cited_title":"Listwise approach to learning to rank: theory and algorithm,","cited_arxiv_id":null,"evidence_quote":"Listwise learning-to-rank theory; supplies the conceptual basis for deriving a global order from local list rankings."},{"cited_title":"From retrieval to gen- eration: Efficient and effective entity set expansion,","cited_arxiv_id":null,"evidence_quote":"Prefix-constrained decoding; provides the beam-search candidate-generation step of Stage 1."}],"review_version":1}