{"id":"e5865349-00c2-4be5-bebf-b2eaf70fe741","arxiv_id":"2506.04983","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper builds a long-video (over three minutes) scene-text QA benchmark and shows that time-aware position encoding and temporal prompting improve model performance.","lead":"TextVidBench is a new benchmark that tests AI models on reading text in videos longer than three minutes, using three tasks: finding text needles, predicting when text appears, and describing how text such as scores changes. Early results show current multimodal models do poorly on it, and the paper's own modifications to a vision-language model improve long-video text performance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark ground truth is the load-bearing risk: GPT-4o writes QA pairs and answers, humans correct only ~20%, and no inter-annotator agreement or audit is reported, so Tables 2-5 inherit unknown machine-generated label noise.","rationale":"The central claim is that TextVidBench is a reliable long-video scene-text benchmark and that current MLLMs, including the proposed model, can be meaningfully compared on it. That claim requires trustworthy question-answer ground truth. The paper does not provide evidence of that trustworthiness: GPT-4o generates the overwhelming majority of QA pairs, and only one-fifth are human-corrected, with no agreement statistics. I considered two other concerns: the inconsistent duration figures (abstract says 2306 seconds average, Appendix C says the Text Needle task covers 20 hours, and Table 2 evaluates up to 20-minute segments) and the undisclosed NUPI search objective. Both are real and should be fixed, but they are secondary to ground-truth validation because every quantitative conclusion in the paper is computed against these labels. If the labels are noisy in a systematic way, then the low absolute scores, the ranking of models, and the proposed method's gains all become unreliable. The proposed audit is a standard sanity check for a benchmark paper and would settle whether the concern lands. Until it is run, the conditional verdict is the right one; no change to the reader's verdict is needed.","tokens_in":17739,"tokens_out":4487,"duration_ms":58537,"concrete_test":"Randomly sample 100 Text Needle QA pairs, 50 Temporal Grounding pairs, and 30 Text Dynamics outputs. Have two independent annotators re-derive answer strings and timestamps from the original 1-fps frames using the full videos, not the MiniCPM-V filtered subset. Compute label-error rate and Cohen's kappa. If the error rate exceeds 5% or kappa is below 0.8, correct all affected labels, recompute Tables 2-5, and re-examine whether Ours still outperforms MiniCPM-V 2.6; if the gap shrinks or disappears, the reported gains are not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1's annotation pipeline has a load-bearing gap. Step 2 filters text-rich frames with MiniCPM-V, Step 4 uses GPT-4o to generate both questions and standard answers, and Step 5 reports human revision of only 'approximately 20%' of pairs; no inter-annotator agreement or independent audit is reported. Because GPT-4o writes the question and the answer from the same frame context, systematic hallucination or overreading of scene text propagates directly into the ground truth that produces Tables 2-5. If GPT-4o consistently misreads a scoreboard or invents a phone number, every model's ANLS and accuracy are wrong in the same direction, and the comparison between 'Ours' and MiniCPM-V 2.6 can no longer be attributed to the proposed method. A second side-effect is selection bias: MiniCPM-V defines which frames are 'text-rich', so the benchmark's target distribution is partly defined by the very base model the proposed method builds on. Both issues are fixable by explicit audit, but without it the central claim is conditional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces TextVidBench, a benchmark for long-video scene text understanding. The benchmark contains over 5,000 QA pairs from 9 video categories totaling about 23 hours, with an average video length of 2,306 seconds. It defines three tasks: Text Needle-in-Haystack (frame-level text retrieval and QA), Text Temporal Grounding (predicting timestamps within 30/60/120s tolerance), and Text Dynamics Captioning (describing changes in text, scored by GPT-4 from 0 to 10). The authors also propose an improved MLLM built on MiniCPM-V 2.6, adding IT-RoPE temporal position encoding, NUPI non-uniform position interpolation, a Time Prompt, and fine-tuning on 50k video-instruction pairs. Experiments show the benchmark is challenging: the best average accuracy on Text Needle-in-Haystack is 16.0%, temporal grounding accuracies are low (e.g., 15.05% at 120s for the proposed model), and all models score below 5 on Text Dynamics Captioning. The proposed model outperforms its base model on all three tasks and achieves gains on the public M4-ViteVQA benchmark.","tokens_in":17909,"tokens_out":10496,"duration_ms":113298,"significance":"If the annotations and evaluation protocols are sound, TextVidBench is a valuable contribution: it is the first benchmark targeting long-video scene text, and its results show a clear capability gap in current MLLMs. The proposed IT-RoPE and Time Prompt are simple and potentially generalizable, and the paper includes a broad baseline evaluation across several strong models. However, the reliability of the benchmark's ground truth is not yet established, a random baseline is missing for the temporal grounding task, and the GPT-4 scoring of captions is unvalidated. These issues must be addressed before the benchmark's scores can be interpreted as reliable measures of model performance.","major_comments":[{"comment":"The benchmark ground truth is generated semi-automatically: MiniCPM-V filters text-rich frames (step 2), GPT-4o writes questions and answers (step 4), and human reviewers correct approximately 20% of pairs (step 5). No inter-annotator agreement, annotation guidelines, or independent audit are reported. Because all scores in Tables 2-5 are computed against these labels, systematic GPT-4o errors (e.g., misreading scene text) would bias every model comparison in the same direction, so the claim that Ours outperforms MiniCPM-V 2.6 could be an artifact of label noise. Please report the number of annotators, their agreement statistics, and a category-wise audit of the 20% corrections, and make the annotation protocol available.","section":"Section 3.1"},{"comment":"The Text Temporal Grounding task lacks a random baseline. For an average video length of 2,306 seconds, a uniform random timestamp is within ±30/60/120s of the ground truth with probability about 2.6/5.2/10.4%. Many reported values are at or below this chance level (e.g., LongVA 120s accuracies of 4.53%, 2.71%, and 0.0%; LongVILA 120s accuracies of 4.27%, 1.45%, and 0.0%). The proposed model's 120s average of 15.05% is modestly above chance. Please add random and constant-prediction baselines to Table 4 and discuss what the task measures in their light.","section":"Section 3.2, Table 4"},{"comment":"The GPT-4 evaluation protocol for captioning is not validated. The paper states that GPT-4 assigns a score from 0 to 10 by comparing model output with standard answers, but gives no scoring prompt, no rubric, and no evidence that GPT-4 scores agree with human judgments. Without this, the comparisons in Table 5 (e.g., Ours 4.59 vs. MiniCPM-V 3.44) and the claim that 'none achieve an average score above 5' are not interpretable. Please provide the exact scoring prompt and the results of a human-GPT-4 correlation study on a sample of outputs.","section":"Section 3.2, Text Dynamics Captioning"},{"comment":"The proposed model is fine-tuned on EliteSet, which is described as being created 'based on the video data collected in VideoTextCap' and using 'the same data production process as the benchmark.' VideoTextCap itself is built from YouTube videos in the same nine categories used by TextVidBench. If any benchmark videos or near-duplicates appear in the training data, the reported improvements (Tables 2-5) would be inflated. Please clarify whether the benchmark videos were excluded from VideoTextCap/EliteSet and, ideally, evaluate the model on a fully disjoint set of long videos to show the gains are not due to distributional overlap.","section":"Appendix F.1 (EliteSet)"},{"comment":"The NUPI scaling coefficients λ_i are found by evolutionary search, but the paper does not state which data were used for the search. If the search was performed on TextVidBench or on a validation subset derived from it, then the NUPI gains in Table 7 are not a fair comparison with baselines, because they reflect test-set adaptation. Please state the optimization objective and data used to select λ_i, and report experiments with coefficients chosen on a completely separate validation set.","section":"Section 4.4, Appendix E"},{"comment":"The validation on M4-ViteVQA uses the proposed model fine-tuned on VideoTextQA, which includes 'public datasets on video text comprehension, including M4-ViteVQA, RoadTextVQA, and NewsVideoQA.' The paper does not state whether the official training split or the full datasets (including the test split) were used. If the M4-ViteVQA test set was included in training, the reported accuracy/ANLS gains over MiniCPM-V 2.6 are invalid. Please specify the exact train/test protocol used for these public datasets.","section":"Section 5.2.1, Appendix F.1"}],"minor_comments":[{"comment":"The text contains several typos and formatting errors: 'enrichs' should be 'enriches', 'n recent years' should be 'In recent years', and 'LenthY' in Eq. (9) should be a product symbol.","section":"Throughout"},{"comment":"The accuracy metric (ACC) for Text Needle-in-Haystack is not defined; please state the normalization and matching rule used to convert model outputs to binary correct/incorrect.","section":"Section 3.2"},{"comment":"The rows for Ours are ambiguous because 'Ours' and 'Ours(acc)' are not clearly separated into the 30s, 60s, and 120s blocks; please label each row explicitly.","section":"Table 4"},{"comment":"In the IT-RoPE description, the rotation angle notation is inconsistent: the text says 'for even values of i, the corresponding rotation angle is given by kθi', but k is not defined; please clarify the indexing.","section":"Appendix D"},{"comment":"The abstract says the benchmark contains over 5,000 QA pairs, while Appendix C reports 2,190 pairs for Text Needle and 600 for Text Change; the total count should be reconciled and stated transparently.","section":"Section 3.3 / Abstract"},{"comment":"The claim of being the 'first' long-video text benchmark would be stronger with a systematic comparison of existing video-text datasets' duration ranges and task designs, rather than only qualitative statements.","section":"Section 2.3 / Section 3.3"},{"comment":"The caption lists 'Random' and 'Human' baselines but does not explain how they were constructed; please provide the reference or a footnote describing these baselines.","section":"Table 1"},{"comment":"The Limitations section focuses only on computational resources and does not mention the potential annotation errors or the distributional overlap between training and evaluation data; please expand it to discuss these limitations.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important gap and the proposed method is promising. However, the missing annotation audit, the absence of random baselines, and the unclear training/evaluation separation are serious issues that bear on the validity of the central claims. I recommend major revision and suggest the editor ask the authors to provide the annotation agreement statistics, random baseline numbers, and clear statements about the data used in the NUPI search and the public benchmark splits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the note on TextVidBench. I think the paper is worth taking seriously: it builds the first long-video scene-text QA benchmark, with 30-minute videos and three tasks that—together—cover retrieval, temporal grounding, and dynamic captioning. That's a real gap. The experiments are thorough, the baselines all score low, and the proposed model (MiniCPM-V 2.6 with IT-RoPE, NUPI, and time prompting) beats the base on all three tasks and also improves on M4-ViteVQA, which suggests the method isn't just overfit to the new benchmark.\n\nThe soft spots are mostly around the benchmark's ground truth. The annotation pipeline uses MiniCPM-V to select text-rich frames, GPT-4o to write questions and answers, and humans correct only ~20% with no inter-annotator agreement or audit. That's load-bearing. If GPT-4o hallucinates or misreads scene text, the errors are baked into every table, and the comparison between 'ours' and MiniCPM-V could be distorted. It's fixable—release an audited subset and report agreement—but without it, the central claim is conditional. There's also a lighter circularity concern: the EliteSet training data is generated with the same pipeline and video categories as the benchmark, so the proposed method is partly tuned toward the eval distribution. The M4-ViteVQA result mitigates that, but the NUPI search objective should be disclosed.\n\nOther issues are minor: no random baseline for temporal grounding (they do include one for M4-ViteVQA, just not on the new task), the GPT-4 judge for captioning is unvalidated, and the duration numbers are inconsistent (abstract says 2306s average, intro says 30 hours total for 9 categories—which roughly matches, but the paper also says '>3 minutes' and 'approximately 100x longer' while the M4-ViteVQA average is ~10s, so 100x would be 1000s not 2306s; these need cleanup). The paper also has a few typos and self-referential statements that suggest a rushed final revision, but nothing that undermines the core contribution.\n\nI'd send this to peer review. The benchmark addresses a gap, the method is plausible, and the flaws are fixable. I'd ask for dataset release, an annotation audit with agreement numbers, a random baseline for temporal grounding, and a validation of the GPT-4 judge. With those, this could become a standard reference. I'd also bring it to reading group—the annotation discussion is a good case study in benchmark construction.","headline":"A valuable long-video scene-text benchmark whose central claim is conditional on an unverified annotation pipeline; deserves peer review and careful revision.","tokens_in":18505,"tokens_out":2646,"would_cite":true,"duration_ms":30227,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TextVidBench, the first benchmark for scene-text question answering on videos longer than three minutes, finds that current multimodal models answer at best 16.0% of its text-retrieval questions.","keywords":["long-video scene text understanding","video visual question answering","text needle-in-haystack","temporal grounding","text dynamics captioning","rotary position embedding","position interpolation","multimodal large language models"],"falsifier":"Re-annotate a random sample of TextVidBench's question-answer pairs with several independent human annotators and measure agreement; if agreement is low, or if errors cluster in the GPT-4o-generated questions, the reported 16.0% top accuracy and the relative model rankings would need to be revised.","tokens_in":17511,"feed_emoji":"🎬","tokens_out":8609,"duration_ms":87608,"temperature":0.7,"pith_summary":"This paper argues that existing video text question-answering benchmarks, with clips averaging under ten seconds, reward models that can sample a few frames and therefore miss the long-horizon reasoning real applications need. To close the gap it builds TextVidBench, a benchmark of about 30 hours of video across nine domains with an average length of 2,306 seconds, and over 5,000 question-answer pairs organized into three tasks: retrieving a text-bearing frame, localizing that frame in time, and describing how on-screen text evolves. The central claim is that current multimodal large language models largely fail these tasks—the best average accuracy on Text Needle-in-Haystack is 16.0%, temporal grounding stays below 31% even at 120 seconds of tolerance, and Text Dynamics Captioning scores stay below 5 out of 10. The paper also claims that lightweight fine-tuning with inflatable temporal rotary position embeddings, non-uniform position interpolation, and explicit time prompts improves a strong base model on all three tasks.","feed_headline":"Best long-video text reader scores just 16%","feed_subtitle":"The new benchmark uses 38-minute videos; today's best models average 16% on text retrieval and under 5/10 on text-change captions.","key_machinery":"The load-bearing mechanism is the three-stage evaluation framework paired with three model-side additions. IT-RoPE (Inflated Temporal Rotary Position Embedding) modifies the standard 1D rotary position embedding so that all tokens belonging to the same video frame share one temporal position index while retaining an intra-frame flattened index; attention scores then depend on both which frame a token came from and where it sits inside that frame, and the scheme degenerates to ordinary RoPE for text tokens. NUPI (Non-uniform Position Interpolation) extends IT-RoPE by searching for non-uniform frequency scaling coefficients so the model can handle longer visual token sequences. The time prompt injects the total video duration, the number of sampled frames, and each frame's timestamp into the model input, which the ablation shows is almost a prerequisite for temporal grounding.","core_discovery":"TextVidBench is the first benchmark designed specifically for long-video scene-text question answering, with videos averaging 2,306 seconds—roughly 100 times longer than earlier ViteVQA datasets. It covers nine domains (driving, egocentric, entertainment, game, knowledge, life record, sports, talk shows, and video news) and its three-stage evaluation separates finding a text-bearing frame (Text Needle-in-Haystack), predicting the frame's timestamp (Text Temporal Grounding), and summarizing how text changes over time (Text Dynamics Captioning). Across these tasks the paper reports that existing multimodal large language models perform poorly: the best model, the authors' improved version of MiniCPM-V 2.6, reaches 16.0% average accuracy on Text Needle-in-Haystack, 15.05% average accuracy on temporal grounding at 120-second tolerance, and 4.59 out of 10 on Text Dynamics Captioning. The authors' method-level claim is that inflating 1D rotary position embeddings with a temporal dimension (IT-RoPE), extending the context with non-uniform position interpolation, and injecting frame timestamps through a time prompt each contribute measurable gains, with the time prompt being essential for temporal localization.","pith_inferences":["If the benchmark's difficulty profile is representative, the bottleneck for long-video text understanding may be temporal indexing rather than text recognition: a model that reads text well but cannot timestamp frames will fail grounding, so future training should include explicit timestamp supervision.","The annotation pipeline could be validated by measuring inter-annotator agreement on a held-out subset, since only about 20% of pairs were human-revised; low agreement or systematic GPT-4o errors would change the reported scores.","The three-task design could transfer to other sparse-text long-video domains, such as surveillance or lecture archives, offering a cheap way to test whether temporal-text reasoning learned on sports and news generalizes."],"forward_implications":["Short-video text benchmarks overstate model readiness: performance on long-video text retrieval declines steadily as video length grows, so a model that looks strong on M4-ViteVQA can still be near chance on 20-minute videos.","Explicit timestamp information is the difference between zero and nonzero temporal grounding: without the time prompt, localization accuracy collapses to near zero, even for the improved model.","Tracking dynamic text such as game scores remains an open problem: every tested 7B-scale model, including the proposed one, scores below 5 out of 10 on describing how scores and in-game currency change.","A base model can be upgraded for long-video text tasks without a full architecture replacement: the paper's lightweight fine-tuning with IT-RoPE, NUPI, and time prompting improves MiniCPM-V 2.6 on all three benchmark tasks while preserving short-video performance."],"supporting_citations":[{"why":"Supplies the M4-ViteVQA benchmark and baseline that TextVidBench extends and uses to validate short-video compatibility.","marker":"(Zhao et al., 2022)"},{"why":"The MiniCPM-V 2.6 model is both the base model for the proposed improvements and the frame filter in the annotation pipeline.","marker":"(Yao et al., 2024)"},{"why":"GPT-4o generates the benchmark's question-answer pairs in the semi-automatic annotation pipeline.","marker":"(OpenAI et al., 2024)"},{"why":"Provides the ANLS metric used to score Text Needle-in-Haystack outputs.","marker":"(Biten et al., 2019)"},{"why":"LongRoPE supplies the non-uniform position interpolation search that the paper adapts into NUPI.","marker":"(Ding et al., 2024)"},{"why":"LongVA is a long-video needle-in-a-haystack baseline that inserts irrelevant frames, which TextVidBench contrasts with.","marker":"(Zhang et al., 2024)"},{"why":"LongVILA is another long-video baseline compared on TextVidBench and on M4-ViteVQA.","marker":"(Xue et al., 2024)"},{"why":"LLaVA-OneVision's system prompt design inspires the time prompt used in the proposed model.","marker":"(Li et al., 2024)"}],"fun_headline_variants":["First benchmark for long-video text QA: models score 16%","Long-video text benchmark: best model hits just 16% accuracy","38-minute video texts stump even top multimodal models","New long-video text benchmark leaves models struggling at 16%","TextVidBench: why long-video text understanding is brutal"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth is produced by an automatic pipeline—a model selects text-rich frames, GPT-4o writes the questions, and humans correct only about 20% of them—so the reported scores inherit any systematic error in those generated pairs.","fun_headline_variants_meta":{"raw":{"variants":["First benchmark for long-video text QA: models score 16%","Long-video text benchmark: best model hits just 16% accuracy","38-minute video texts stump even top multimodal models","New long-video text benchmark leaves models struggling at 16%","TextVidBench: why long-video text understanding is brutal"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000312,"raw_usage":{"total_tokens":1843,"prompt_tokens":1084,"completion_tokens":759,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":670}},"tokens_in":700,"tokens_out":759,"duration_ms":7657,"temperature":1.0,"reasoning_tokens":670,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:28:49.246730+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of TextVidBench's question-answer pairs with several independent human annotators and measure agreement; if agreement is low, or if errors cluster in the GPT-4o-generated questions, the reported 16.0% top accuracy and the relative model rankings would need to be revised.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the M4-ViteVQA benchmark and baseline that TextVidBench extends and uses to validate short-video compatibility."}],"review_version":1}