{"id":"2b08c69c-54da-4516-976b-416c83744010","arxiv_id":"2608.07417","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A new person-centric video reasoning benchmark and 7B model that link a reference image of a person to their appearances and actions in a video.","lead":"This paper introduces a new video reasoning task where a model must use a reference photo of a person to find that person in a video and answer questions about them. It also contributes a benchmark, a 75K training set, and a 7B model that beats most open-source systems and approaches commercial models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unresolved overlap between ISYV-Benchmark and ISYV-75K, plus test-set-based hyperparameter selection, leave the 57% headline claim unvalidated.","rationale":"The reader's weakest assumption targets exactly the load-bearing issue: benchmark/training disjointness and lack of a held-out validation split. My reading of the full text confirms this. The benchmark is described as curated out of the same candidate pool as ISYV-75K, and the experimental section tunes architectural hyperparameters directly on ISYV-Benchmark accuracy. These are not merely stylistic omissions; they determine whether Table 3's central comparison is a valid test-set measurement or a possibly contaminated one. I considered whether another concern is more fundamental, such as the ambiguity of 'rivals closed-source MLLMs' given Gemini-2.5-Pro's 67.10%, but that is a framing issue, not a correctness risk. The data-split issue is more load-bearing because if overlap exists, the main empirical claim collapses. The reader's CONDITIONAL verdict is appropriate: the paper's contributions are plausible and the writing is clear, but the empirical claims need the split validation and data release before they can be accepted at face value. I found no additional independent objection beyond the reader's, and I do not see a basis to move the verdict further without running the proposed check.","tokens_in":15987,"tokens_out":2412,"duration_ms":25858,"concrete_test":"Release exact video-level identifiers or frame hashes for every ISYV-Benchmark and ISYV-75K sample, and report the overlap count. Then retrain ISYV-Model using the same recipe but with all benchmark-derived samples removed from ISYV-75K, and select ICQ token count and frame count on a held-out validation split instead of on the benchmark. If O.Avg on the untouched ISYV-Benchmark drops materially, or if any benchmark video appears in the training set, the headline 'rivals closed-source MLLMs' should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ISYV-Model-RFT rivals closed-source MLLMs rests on its 57.01% O.Avg on ISYV-Benchmark. That number is only meaningful if the benchmark is disjoint from the ISYV-75K training set and if model choices were not tuned on the benchmark. Section 3.2.3 states that ISYV-Benchmark is 'constructed through manual curation and verification based on ISYV-75K', and Section 3.3.1 describes selecting benchmark samples from a 'candidate dataset' filtered out of the training pool. The paper never states that benchmark videos, reference images, or QA pairs were excluded from ISYV-75K. The annotation pipeline's 'leakage detection' step targets label leakage into CoT rationales, not video-level overlap between benchmark and training data. In addition, Table 5 ablates the ICQ token count (16/32/64) and frame count (16/32/64) directly on ISYV-Benchmark O.Avg, and Table 4 uses the same benchmark to justify reward components. No held-out validation split is described. If benchmark samples were seen during SFT/RFT, or if token/frame counts were selected on the test set, the reported gains over baselines are not unbiased estimates of generalization. Given only 1,377 benchmark questions, even modest overlap or selection on test metrics could materially inflate the gap between ISYV-Model-RFT and baselines like Video-R1 (47.57%).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Identity-conditioned Queries (ICQ), a person-centric video reasoning task in which a model must jointly process a video, a reference image of a target person, and a text query. The authors contribute three components: (1) ISYV-Bench, a manually curated and verified benchmark of 1,377 real-world video clips with 1,377 question-answer pairs organized into six cognitive-hierarchy levels; (2) ISYV-75K, a 75K-sample training set built by an automated LLM-based pipeline with multi-stage quality checks; and (3) ISYV-Framework, which pairs an ICQ Module that compresses the reference image into learnable tokens with a two-stage SFT+RFT training strategy using a format reward, accuracy reward, caption semantic reward, and an Effective Shot Re-Reasoning (ESR) reward. The central empirical claim is that the resulting 7B-parameter ISYV-Model-RFT reaches 57.01% average accuracy on ISYV-Bench, outperforming open-source baselines and approaching closed-source models such as Gemini-2.5-Pro, while exposing a capability gap in existing MLLMs for identity-conditioned video reasoning.","tokens_in":16318,"tokens_out":3827,"duration_ms":35561,"significance":"If the benchmark is clean and the results hold, the paper makes several useful contributions. The ICQ task formulation is a natural and practical extension of video reasoning to multi-source inputs, and the six-level cognitive hierarchy provides a structured way to report model abilities beyond a single aggregate score. The data pipeline, with automated annotation plus multi-stage verification and manual review, is a concrete and scalable approach to building a person-centric video QA dataset. The ICQ Module's learnable-token compression of the reference image is a simple, plausible remedy for the semantic confusion and token overhead of naive video-image concatenation, and the ESR reward is an interesting attempt to learn effective-shot selection without shot-level ground truth. The paper also evaluates a broad set of open- and closed-source MLLMs, and the observation that several models either ignore the reference image or 'hack' the answer (e.g., VideoLLaMA3-7B's 33.92 point gap between O.Avg and ICQ-Q.Avg) is a genuinely informative finding for the community.","major_comments":[{"comment":"The manuscript never states that ISYV-Benchmark is disjoint from ISYV-75K. Section 3.2.3 says ISYV-Benchmark is 'constructed through manual curation and verification based on ISYV-75K', and Section 3.3.1 says benchmark samples are selected from a 'candidate dataset' after further filtering of data that went through the ISYV-75K quality pipeline. No sentence excludes benchmark videos, reference images, or QA pairs from the ISYV-75K training pool. If the benchmark samples were seen during SFT or RFT, the reported 57.01% O.Avg for ISYV-Model-RFT is not an unbiased measure of generalization, and the comparison against baselines (e.g., Video-R1 at 47.57%) would be materially inflated. The authors must add an explicit train/benchmark disjointness statement, describe how overlap was prevented, and ideally release per-sample IDs or an overlap-detection result.","section":"§3.2.3, §3.3.1, Table 3"},{"comment":"Hyperparameters and training choices are selected directly on ISYV-Benchmark, which is the test set. Table 5 ablates the ICQ token count (16/32/64) and input frame count (16/32/64) using O.Avg on ISYV-Benchmark, and Table 4 uses the same benchmark to justify the reward composition; no held-out validation split is described. This means the reported '32 is optimal' conclusion and the final 57.01% figure are test-set-tuned numbers, which overstates expected generalization performance and weakens the claim that the framework, rather than test-set selection, is responsible for the gains. The authors should introduce a validation split or, at minimum, clearly label these results as test-set-tuned and report the corresponding performance of baselines under the same selection procedure, or quantify the selection bias.","section":"§5.3, Tables 4 and 5"},{"comment":"The training labels and the benchmark question-answer pairs are both produced by the same LLM family (Qwen3-Max, Qwen3-VL-32B, with Gemini-2.5-Pro for annotations). Because ISYV-Model is trained on Qwen-generated rationales and answers, its strong performance on ISYV-Bench may partly reflect distributional familiarity with Qwen-style reasoning and phrasing rather than a general capability for identity-conditioned video reasoning. Human manual verification does not eliminate this concern, as it validates correctness rather than independence of generation style. The paper should acknowledge this risk and ideally provide an evaluation on an independently authored benchmark or report performance stratified by annotation source to show the result is not driven by model-family familiarity.","section":"§3.2.3, §5.1"}],"minor_comments":[{"comment":"The text refers to the appendix for reward hyperparameters, annotation prompts, evaluation details, and additional case analyses ('We will discuss these in the appendix', 'detailed settings are provided in the appendix'), but the provided manuscript contains no appendix or supplementary material. The authors should include the appendix or clearly state where the details are available.","section":"§4.3, §5.1"},{"comment":"The claim that the 7B model 'rivals closed-source MLLMs' is somewhat stronger than the evidence in Table 3, where ISYV-Model-RFT (57.01%) trails Gemini-2.5-Pro (67.10%) by about 10 points, though it does outperform other closed-source and open-source models on several levels. Consider tempering the phrasing to 'approaches closed-source performance on some levels' or reporting a significance/statistical comparison, especially since the benchmark has only 1,377 questions.","section":"Abstract and §1"},{"comment":"The 'Annotation' column uses the abbreviations 'M' and 'M+A' without a legend. Please define these in the table caption.","section":"Table 1"},{"comment":"The six-level hierarchy is described as corresponding to 'the developmental trajectory of human cognitive abilities' citing d'Andrade (1995), but the mapping between specific cognitive constructs (e.g., 'Object Permanence', 'Spatial Memory') and the designed question types is asserted rather than empirically validated. This is acceptable as a design choice, but the paper should be explicit that the levels are an analogy rather than a validated cognitive model.","section":"§3.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is potentially solid, but the two load-bearing issues—benchmark/training overlap and test-set-based hyperparameter selection—must be resolved before the headline result can be trusted. If the authors can supply a clear disjointness guarantee and re-run the ablations on a held-out split, the work could become acceptable. I would also encourage the editor to verify that the appendix and dataset URLs are present in the final version, as the arXiv version omits several promised supplementary details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the paper to know about if you work on person-centric video reasoning, but the headline \"7B model rivals closed-source MLLMs\" should not be quoted until the data-split question is answered.\n\nWhat is new and good: ICQ is a real gap. Existing video QA is video+text, and identity-conditioned work like IDA-VLM and PLVM stops at images. Putting a reference image, a long video with shot transitions, and a text query into one task is a sensible formulation, and the six-level cognitive hierarchy is more than packaging—it helps diagnose where models fail, especially cross-domain identity matching and long-horizon tracking. The 75K training set is a substantial resource. The ICQ module (learnable query tokens compressing the reference image) is simple and reasonable, and the ESR reward is a legitimate attempt to teach evidence selection without shot-level labels. The evaluation covers many MLLMs plus human performance, which is useful.\n\nWhere it goes soft: the reader's central worry is on target. I checked the full text, and there is no statement that ISYV-Benchmark is disjoint from ISYV-75K. Section 3.2.3 says the benchmark is \"constructed through manual curation and verification based on ISYV-75K,\" and Section 3.3.1 discusses selecting benchmark samples from a candidate dataset. No exclusion is stated. The \"leakage detection\" step in the annotation pipeline targets label leakage into chain-of-thought rationales, not video-level overlap between benchmark and training. On top of that, Table 5 tunes token count and frame count on ISYV-Benchmark O.Avg, and Table 4 uses the same benchmark to justify reward components. No held-out validation split is described. With only 1,377 questions, even modest overlap or test-set selection can move the 57.01% versus Video-R1's 47.57% gap. This is not a nitpick; it is load-bearing for the central empirical claim. The fix is straightforward: document the construction as a proper train/bench split, confirm no overlap, and re-report if needed.\n\nA softer concern: both the training labels and benchmark QAs are generated by the same Qwen model family, so models fine-tuned on Qwen-style reasoning may get a bump that is not really \"person-centric skill.\" That does not invalidate the benchmark, but it should be discussed.\n\nVerdict: this deserves a serious referee, but I would send it back for major revision on the split and validation protocol. If the split is clean, I would cite this for the task and benchmark. Until then, treat the model numbers as provisional.","headline":"The ICQ task and benchmark are a genuinely useful new resource, but the empirical headline is unvalidated because the benchmark appears to be carved out of the training set and is used to pick hyperparameters.","tokens_in":16919,"tokens_out":5206,"would_cite":false,"duration_ms":47673,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-10T04:55:02.670140+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}