{"id":"2f16b411-7f7d-441f-bad8-de4a026d86f8","arxiv_id":"2608.13210","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"NARU provides a human-verified benchmark of 1,481 multiple-choice questions over 155 Japanese long-form videos, and shows that even the best model scores only 76.2% while open-source models stay near chance.","lead":"A new benchmark called NARU tests AI models on understanding Japanese long-form videos, combining narrative tracking with culturally nuanced reasoning. It includes 1,481 questions from 155 videos and shows that current AI models, especially open-source ones, still struggle with these tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's video-grounding claim rests on the Solver-Critic debiasing loop, yet no final blind-solver accuracy is reported; if text-only shortcuts persist, Table II scores do not measure video understanding.","rationale":"The reader's weakest assumption identifies the same load-bearing gap: the Solver-Critic loop's effectiveness is unverified because final blind-solver accuracy is not reported, and the two-stage human verification lacks inter-annotator agreement. My stress-test places the heavier weight on the missing blind-solver numbers because they directly support the construct validity of the benchmark: if questions are solvable without video, then the central claim that NARU evaluates video-grounded narrative and cultural understanding fails, regardless of annotation quality. This is a missing-evidence concern rather than a demonstrated failure, so the appropriate disposition is the reader's CONDITIONAL verdict: the benchmark resource is plausibly useful and novel, but acceptance should require the authors to report the debiasing convergence statistics or run an independent blind evaluation on the final items. The proposed test would settle the concern in one pass: near-chance blind accuracy across diverse model families would validate the debiasing claim, while significantly above-chance accuracy would show that the headline evaluation numbers are inflated by text-only shortcuts. The lack of inter-annotator agreement is a related but secondary issue, since even a single careful cultural interpretation can define a usable ground truth if the benchmark explicitly scopes itself to that interpretation; text leakage, by contrast, would undermine the benchmark's stated purpose of measuring video understanding.","tokens_in":15813,"tokens_out":3830,"duration_ms":37616,"concrete_test":"Run the final 1,481 NARU items in their MCQ form with video and transcript removed (only stem plus four options) through at least five models: Gemini-2.5-Pro (the original blind solver), Gemini-3-Flash, Gemini-3-Pro, Qwen3-VL-8B, and a text-only LLM with no visual input; use temperature 0 and report overall and per-category (N.1-N.4, C.1-C.5) accuracy with binomial 95% confidence intervals. If any model's accuracy is statistically indistinguishable from or above the 25% chance level for a category, or if the aggregate blind accuracy exceeds roughly 30%, the debiasing loop has not removed text-only shortcuts for the evaluated model families, and Table II accuracies should be reinterpreted as contaminated by linguistic priors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-C2 (Step 2-2) defines the Iterative Debiasing Refinement loop and states it runs until the Blind Solver Agent's success rate draws close to natural random chance or a predefined iteration budget is reached. No result of this stopping condition is reported: the paper gives no final blind-solver accuracy, no per-category leakage rates, and no evidence that the iteration budget was not hit before convergence. Since the same Gemini 2.5 Pro serves as blind solver, diagnostic agent, revision agent, and question generator, convergence for that model only shows self-consistency, not absence of shortcuts for the models actually evaluated in Table II (e.g., Gemini-3-Flash, Qwen3-VL-8B). The paper itself admits at the start of Sec. IV-D that 'multiple-choice options can inadvertently leak contextual hints,' signaling residual leakage risk in the MCQ format that the final benchmark uses as its primary evaluation. If blind accuracy on the final 1,481 items is above chance, the central claim that NARU measures video-grounded narrative and cultural understanding is unsupported, and the headline 76.2% result would be partly a text-bias artifact. The post-refinement human verification additionally corrected 436 answer labels, so automated labels are demonstrably noisy and the missing convergence evidence becomes even more load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NARU, a benchmark of 1,481 multiple-choice questions grounded in 155 Japanese long-form videos totaling 146.8 hours, organized into four narrative subcategories (N.1–N.4) and five cultural subcategories (C.1–C.5). The benchmark is built with a hierarchical, memory-based annotation pipeline that chunks videos, produces narrative and cultural annotations, and then generates MCQs with an iterative Solver-Critic debiasing loop, followed by two native-speaker verification stages involving 68 annotators. The authors evaluate eight MLLM configurations in both MCQ and open-ended formats, reporting that Gemini-3-Flash achieves the highest MCQ accuracy at 76.2%, while open-source models range from 29.6% to 39.8%. They also present a frame-sampling sweep and an open-ended F ActScore evaluation, concluding that current models show substantial limitations in long-range narrative integration and culturally grounded reasoning.","tokens_in":16089,"tokens_out":4944,"duration_ms":44264,"significance":"If the benchmark's validity claims hold, NARU addresses a real gap: existing long-video benchmarks mostly separate temporal/narrative reasoning from culturally situated implicit understanding, and few operate in high-context non-English media. The construction pipeline is carefully designed, the taxonomy is grounded in Event Segmentation Theory and high-context communication research, and the two-stage verification with 68 native-speaker annotators is a concrete strength. The open-ended evaluation with a human-judge validation of the automated judge is also a positive methodological step. However, the paper's central video-grounding claim depends on debiasing and verification evidence that is currently missing, so the benchmark's validity as a measure of video-grounded, culturally nuanced understanding is not yet fully established.","major_comments":[{"comment":"The Solver-Critic debiasing loop is defined, but its stopping condition is never reported. The paper does not give the final Blind Solver Agent's accuracy on the final 1,481 items, per-category leakage rates, or the number of iterations relative to the iteration budget. Because the same model (Gemini 2.5 Pro) is used as blind solver, diagnostic agent, revision agent, and question generator, convergence for that model establishes self-consistency, not the absence of text-only shortcuts for the models actually evaluated in Table II, such as Gemini-3-Flash and Qwen3-VL-8B. The manuscript itself admits at the start of Sec. IV-D that 'multiple-choice options can inadvertently leak contextual hints,' so residual leakage in the primary MCQ format is an acknowledged risk. Please report the final blind-solver accuracy and per-category leakage rates; if blind accuracy remains above chance, remove or repair the leaked items and re-report Table II, because otherwise the central claim that NARU measures video-grounded understanding is unsupported.","section":"III-C2, Step 2-2"},{"comment":"The two-stage human verification is a strength, but no inter-annotator agreement statistics are reported for either the 40-annotator or 28-annotator stage, and the adjudication protocol is not described. This matters because the pipeline relies on labels that were demonstrably noisy: 177 label corrections in the first stage, 436 answer corrections in the second stage, and 107 annotator-authored answers. For culturally nuanced items, the claim of a single correct answer requires evidence of consensus (e.g., pairwise agreement, Cohen's kappa or Krippendorff's alpha, reported per category) and a documented rule for resolving disagreements; without this, NARU may encode one particular cultural interpretation rather than a verified ground truth.","section":"III-D"},{"comment":"All model-comparison claims are presented without confidence intervals, significance tests, or multiple-comparison corrections. Per-category sample sizes are only 143–187 items, so statements such as 'the sole category where Gemini-3-Pro yields higher accuracy' and 'open-source models show virtually no difference between narrative and cultural dimensions' are not yet supported. Please report binomial confidence intervals for overall and per-category accuracy and use paired tests (e.g., McNemar) for model and category comparisons, including for the frame-sweep results in Fig. 3 and the open-ended results in Table III.","section":"IV-B, Table II"},{"comment":"Using Gemini 2.5 Pro as the question generator, annotator, and blind solver while evaluating Gemini-family models creates a distributional overlap that may inflate Gemini results relative to open-source models. This is not circularity in the ground-truth definition, because the labels are human-verified, but it is a correctness risk for the cross-family comparison. A concrete test would be to report model accuracy separately on items whose stems or answers were authored or substantially rewritten by human annotators versus items left unchanged, or to run the blind-solver loop with a non-Gemini model; either analysis would show whether the reported ranking is robust to generator-family effects.","section":"III-C and IV-B"}],"minor_comments":[{"comment":"Please proofread for typographical errors, including 'T o' appearing in the abstract/footer text and 'question r answer pairs' in Sec. I.","section":"Throughout"},{"comment":"The Japanese gloss for 'kuuki wo yomu' appears garbled in the manuscript; ensure the Japanese script renders correctly in the camera-ready version.","section":"Fig. 1"},{"comment":"Define 'draws close to natural random chance' with a concrete threshold (e.g., within one percentage point of 25%) in the main text rather than only on the project website.","section":"III-C2"},{"comment":"Report the per-category distribution of the 500-question diagnostic subset and the atomic-fact decomposition statistics (e.g., average number of reference facts per answer) so the F ActScore recall comparison in Table III is interpretable.","section":"IV-D"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript is within scope for the journal and addresses a timely benchmark gap. My main concern is not the overall idea but the missing evidence on debiasing convergence and annotator agreement; these are fixable through additional analysis and reporting, which is why I recommend major revision rather than rejection. I also note that several references to 2026 preprints (e.g., [30], [31], [42]) should be verifiable before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"NARU is a real contribution. The idea of jointly testing narrative evolution and culturally situated reasoning in Japanese long-form video is new, and the benchmark is concrete: 1,481 human-verified MCQs over 155 videos, 146.8 hours, with a taxonomy that draws sensibly on event segmentation theory and Hall's high-context communication. The annotation pipeline is described in enough detail to be reproducible, and the two-stage native-speaker verification is a genuine strength—68 annotators, with concrete numbers on repairs and removals. The open-ended follow-up evaluation using FActScore, with a human-checked judge (90% acceptance), is the right kind of sanity check and gives the benchmark more credibility than most MCQ-only resources. The frame-budget sweep showing that narrative reasoning benefits from more frames while cultural reasoning does not is a useful empirical observation.\n\nThe soft spots are real but mostly about reporting, not the underlying design. The paper never reports the final blind-solver accuracy after the debiasing loop, so we do not know whether text-only shortcuts were actually brought down to chance on the final 1,481 items. The paper itself admits that MCQ options can leak contextual hints, and the N.2 reversal in the open-ended evaluation suggests the MCQ format was scaffolding sequential reasoning. That is worth knowing, but it is not a fatal blow: the open-ended results broadly preserve the model-ranking hierarchy, and the human verification provides an external anchor. Still, the missing convergence numbers should be added before publication.\n\nThree other gaps, in increasing order of seriousness: no inter-annotator agreement for the answer labels (the second cohort corrected 436 answer labels, so label noise is not trivial; we need to know how much is due to debiasing edits versus original annotation error); no confidence intervals or significance tests on the model comparison (the 76.2% vs 70.0% gap between Gemini-3-Flash and Pro could be within noise for all we know); and the diversity-sampling threshold τ is never given a value. The circularity concern about using Gemini for both construction and evaluation is minor given the human verification, though it would be nice to see at least one non-Gemini question generator in a future version.\n\nThis is a solid benchmark paper that deserves a serious referee. I would send it to review with a request for the missing statistics, not a desk reject. The resource is useful, the claims are mostly measured, and the limitations are acknowledged in the text. I would cite it if I were working on long-video or culturally grounded evaluation.","headline":"NARU is a genuinely useful new benchmark with a careful construction pipeline; the main soft spots are missing statistics (blind-solver convergence, inter-annotator agreement, error bars), but the core resource is solid and worth engaging with.","tokens_in":16594,"tokens_out":1655,"would_cite":true,"duration_ms":17532,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Best AI model scores 76 percent on long Japanese video benchmark","keywords":["long-form video understanding","multimodal large language models","video question answering","cultural understanding","narrative intelligence","Japanese high-context communication","benchmark construction","aizuchi"],"falsifier":"Run the final 1,481 items through the same blind solver with video access removed; if accuracy substantially exceeds 25 percent, text-only shortcuts remain in the benchmark. Independently, measure inter-annotator agreement on the culturally nuanced C.2/C.3 items; low agreement would show there is no single culturally correct answer for those questions.","tokens_in":15653,"feed_emoji":"🎬","tokens_out":5695,"duration_ms":51080,"temperature":0.7,"pith_summary":"This paper introduces NARU, a benchmark of 1,481 multiple-choice questions over 155 Japanese long-form videos totaling 146.8 hours, designed to test two abilities jointly: tracking a narrative as it evolves over hours and interpreting culturally implicit meaning such as 'reading the air' or the subtext of a polite offer. The authors argue that existing benchmarks keep these abilities separate, and that Japanese media is a strong testbed because meaning often depends on accumulated social context rather than literal content. To build the benchmark at scale, they use a hierarchical memory-based pipeline that chunks videos, annotates events, narratives, and cultural cues, generates questions with a multimodal model, and iteratively removes text-only shortcuts using a blind-solver loop, followed by two verification rounds with 68 native Japanese speakers. Evaluations across eight model configurations show the best model reaches 76.2 percent accuracy, while open-source models stay between 29.6 and 39.8 percent, with particular weakness in subtext interpretation and long-range thematic synthesis. If NARU is valid, it gives the field a metric for whether multimodal models genuinely integrate long-range, high-context video rather than retrieving isolated facts or exploiting option artifacts.","feed_headline":"Best AI model scores 76 percent on long Japanese video benchmark","feed_subtitle":"NARU pairs narrative tracking with cultural subtext like “reading the air,” where even leading models struggle most.","key_machinery":"The load-bearing mechanism is a hierarchical memory-based annotation pipeline that converts hours of video into structured, temporally connected evidence without requiring one model invocation to process the whole video. Videos are cut into five-minute chunks, each processed with a recap of prior chunks to maintain entity and event continuity; chunk records are merged into a global timeline, segmented into chapter-level units, and enriched by narrative and cultural annotation modules aligned with the nine-category taxonomy. Question generation is controlled by a Solver-Critic loop: a blind solver attempts each question with the video removed, a diagnostic agent identifies the shortcut (such as tone bias or process of elimination), and a revision agent rewrites the item until the blind solver approaches chance. The taxonomy itself, grounded in Event Segmentation Theory and high-context communication theory, defines what counts as narrative intelligence versus cultural understanding and shapes what evidence the annotation modules extract.","core_discovery":"NARU's central claim is that long-form video understanding should be evaluated as the joint problem of maintaining narrative state and interpreting culturally situated implicit meaning, and that in Japanese long-form video this joint competence remains far from solved. The benchmark operationalizes the claim through nine categories: four narrative dimensions (character/entity evolution, sequential/topical flow, plot/conflict progression, idea/thematic development) and five cultural dimensions (aizuchi, 'reading the air,' subtext interpretation, cultural context recognition, sentiment analysis). The paper reports that the strongest evaluated model reaches 76.2 percent multiple-choice accuracy, that all models gain more from added frames on narrative than on cultural tasks, and that removing the multiple-choice options reverses the relative difficulty of narrative versus cultural questions. The intended conclusion is that current multimodal large language models still show substantial limitations in both long-range narrative integration and culturally grounded reasoning, and that NARU provides a systematic testing ground for improving them.","pith_inferences":["If the construction pipeline generalizes, the same hierarchical-memory plus blind-solver-debiasing recipe could produce comparable benchmarks for other high-context cultures, enabling cross-cultural comparisons of multimodal models' pragmatic reasoning rather than treating Japanese as an isolated case.","The frame-sweep result implies that scaling context length and frame counts mainly helps narrative integration; improving cultural understanding may require training on explicitly annotated pragmatic and sociocultural cues rather than more raw video data.","The format-dependent reversal on sequential-flow questions suggests that benchmarks reporting only multiple-choice accuracy may overstate models' ability to reconstruct event order; adopting open-ended or ordering-free evaluation would give a more honest picture.","A testable extension would use NARU's per-category items to compare models of similar size but different pretraining data, isolating whether cultural failures reflect missing knowledge, weak pragmatic inference, or insufficient temporal integration."],"forward_implications":["Current multimodal AI models, even the strongest evaluated, remain far from ceiling performance on NARU: the top model reaches 76.2 percent while open-source models hover near or below 40 percent, with some dipping below the 25 percent random baseline on narrative tracking and cultural tasks.","Narrative errors are largely caused by missing dispersed events, since denser frame sampling consistently raises narrative accuracy across all models, whereas cultural understanding improves little with more frames and appears bottlenecked by pragmatic reasoning and domain knowledge.","Multiple-choice evaluation can overestimate temporal sequence understanding: N.2 (sequential/topical flow) is the easiest narrative category in multiple-choice form but becomes the weakest for seven of eight models in open-ended form, indicating that option lists scaffold chronological reconstruction.","The dominant narrative difficulty shifts with model capability: weaker models fail most at low-level entity continuity, while stronger models struggle most at high-level thematic development, implying different bottlenecks along the capability spectrum.","Scores on culturally nuanced categories such as subtext interpretation remain low even for the leading model, suggesting that cultural understanding will not be fixed simply by longer context windows or more frames."],"supporting_citations":[{"why":"Supplies Event Segmentation Theory, which grounds the narrative taxonomy's local versus global coherence distinction.","marker":"[35]"},{"why":"Provides the high-context communication framing that motivates the cultural understanding categories.","marker":"[36]"},{"why":"Defines aizuchi and supports the design of the C.1 interactional-signalling category.","marker":"[37]"},{"why":"Demonstrates that models can answer visual questions without visual input, motivating the blind-solver debiasing loop.","marker":"[42]"},{"why":"Supplies the FactScore atomic-fact recall metric used in the open-ended evaluation.","marker":"[49]"},{"why":"Sets the 30-minute duration criterion for long videos and serves as the long-video benchmark that NARU extends.","marker":"[14]"}],"fun_headline_variants":["New benchmark exposes AI's blind spot in Japanese video subtleties","AI models fail at reading between lines in long Japanese videos","NARU benchmark: 1,481 questions that trip up top AI on Japanese video","Cultural nuance benchmark stumps leading AI video models","Why AI still can't follow a story in Japanese long-form video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"NARU's validity depends on the blind-solver loop actually eliminating text-only shortcuts and on the two-stage native-speaker verification establishing a single correct answer for culturally nuanced items, yet the paper does not report final blind-solver accuracy or inter-annotator agreement.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark exposes AI's blind spot in Japanese video subtleties","AI models fail at reading between lines in long Japanese videos","NARU benchmark: 1,481 questions that trip up top AI on Japanese video","Cultural nuance benchmark stumps leading AI video models","Why AI still can't follow a story in Japanese long-form video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000811,"raw_usage":{"total_tokens":3549,"prompt_tokens":930,"completion_tokens":2619,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":2529}},"tokens_in":546,"tokens_out":2619,"duration_ms":17432,"temperature":1.0,"reasoning_tokens":2529,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:19:22.670990+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the final 1,481 items through the same blind solver with video access removed; if accuracy substantially exceeds 25 percent, text-only shortcuts remain in the benchmark. Independently, measure inter-annotator agreement on the culturally nuanced C.2/C.3 items; low agreement would show there is no single culturally correct answer for those questions.","supporting_citations":[{"cited_title":"Event segmentation,","cited_arxiv_id":null,"evidence_quote":"Supplies Event Segmentation Theory, which grounds the narrative taxonomy's local versus global coherence distinction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the high-context communication framing that motivates the cultural understanding categories."},{"cited_title":"Nodding, aizuchi, and final particles in japanese conversation: How conversation reflects the ideology of communication and social relationships,","cited_arxiv_id":null,"evidence_quote":"Defines aizuchi and supports the design of the C.1 interactional-signalling category."},{"cited_title":"Factscore: Fine- grained atomic evaluation of factual precision in long form text generation,","cited_arxiv_id":null,"evidence_quote":"Supplies the FactScore atomic-fact recall metric used in the open-ended evaluation."},{"cited_title":"Lvbench: An extreme long video understanding benchmark,","cited_arxiv_id":null,"evidence_quote":"Sets the 30-minute duration criterion for long videos and serves as the long-video benchmark that NARU extends."}],"review_version":1}