{"id":"9cff36c2-b468-4f1d-ae61-243945882b64","arxiv_id":"2607.22959","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A closed-loop image-to-video quality-control system reports 87–97% expert agreement on internal clips, with no released data, code, or thresholds.","lead":"HALLELUAI is an automated quality-control system for image-to-video AI generation: it checks each generated clip for blur, bad motion, and hallucinated objects, and automatically regenerates failed clips until they pass expert-style standards. It reports 87% expert agreement and 97% precision after regeneration, but the evaluation is small and entirely internal.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hallucination-aware claim rests on unvalidated GPT-4o judgments; per-category accuracy is never checked against expert annotations.","rationale":"The reader's weakest_assumption precisely identifies the unvalidated MLLM hallucination judgments as the load-bearing premise. I agree: the central claim that HALLELUAI is a reliable hallucination-aware gate depends on the correctness of the per-category severity scores and per-frame binary decisions produced by GPT-4o. The paper validates only the aggregate PASS/FAIL decision against expert judgment, which is insufficient because other deterministic modules (blur, contrast, brightness, noise, motion) can substantially determine the PASS/FAIL outcome. The reported precision could therefore be achieved even if the hallucination module is inaccurate, while the regeneration actions—the system's purported remediation capability—would be grounded in unreliable signals. The proposed concrete test directly addresses this gap by measuring per-category agreement and running an ablation to isolate the hallucination module's contribution. If the test fails, the hallucination-aware claim would be unsupported and the verdict should move toward REJECT; if it passes, the conditional acceptance is justified. Since the reader's verdict is already CONDITIONAL and my concern aligns with that condition, no change to the verdict is needed.","tokens_in":10049,"tokens_out":7955,"duration_ms":80039,"concrete_test":"Take a sample (e.g., the 146 shadow clips or a new set) and have at least two expert annotators label each clip for every hallucination subcategory in Fig. 2, including severity (0–10) and per-frame new-structure presence (natural entry, camera-induced entry, abrupt emergence). Then compute per-category agreement between GPT-4o's outputs and the expert majority (e.g., quadratically weighted kappa for severity, Cohen's kappa/F1 for binary labels). Additionally, ablate the moderation module: recompute PASS/FAIL using only §3.1 and §3.2 signals and compare agreement with experts against the full system. If per-category kappa is below ~0.5, or if the full system's agreement is not significantly higher than the non-hallucination ablation, the hallucination-aware claim is not supported by the evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The system's advertised novelty is fine-grained hallucination detection (§3.3). The MLLM (GPT-4o) outputs per-category severity scores (0–10) and per-frame binary new-structure labels, and these are thresholded to set PASS/FAIL and trigger category-specific regeneration actions (§3.4, §4). However, nowhere do the authors compare these intermediate outputs to independent human annotations for each hallucination category. The only validation is overall PASS/FAIL agreement with experts (§5.1), which conflates all signals. Because frame-level (§3.1) and motion (§3.2) modules are deterministic and explainable, the observed 86.9% agreement could be driven entirely by those components, with the hallucination module contributing noise. The reported 97% precision in Pseudo Production also cannot separate filtering from true hallucination correction. If GPT-4o's category scores are miscalibrated or biased, the regeneration policy (§4) will inherit that bias, yet the paper provides no evidence that per-category decisions are correct. This is the weakest load-bearing link for the central claim of a hallucination-aware, production-grade gate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes HALLELUAI, a closed-loop moderation-and-regeneration system for image-to-video (I2V) generation. The moderation module is a three-part gate: deterministic frame-level quality checks (blur, contrast, brightness, noise; Algorithm 1), deterministic motion checks based on RAFT and TAPIR (Algorithm 2), and an MLLM-based hallucination detector that rates seven object-hallucination categories and three new-structure cases (Section 3.3). The regeneration module translates structured diagnostics into prompt, camera, image, or model changes (Section 4). Evaluation is human-in-the-loop with creative experts: 86.9% PASS/FAIL agreement and 88% precision on 146 shadow clips, 97% expert precision on 1,158 pseudo-production clips, and a production deployment claimed at 70,000+ videos. The paper claims this is the first integrated system that provides per-asset, source-image-conditioned, machine-actionable quality control for I2V at scale.","tokens_in":10352,"tokens_out":11546,"duration_ms":109639,"significance":"The problem is real and the modular design is sensible. If the results hold, HALLELUAI would fill a practical gap by producing structured diagnostics that directly drive remediation rather than a scalar quality score. The deterministic algorithms are specified with explicit thresholds, and the use of external expert review in the loop is a strength. What is not yet demonstrated is the hallucination-aware core: the GPT-4o category scores and new-structure labels are never validated against independent human annotations, so the central claim that the system detects fine-grained hallucinations is unsupported. The evaluation also lacks confidence intervals, inter-annotator agreement, a regeneration-control comparison, and released data/code. These are remediable, but until they are supplied the paper should be treated as a system description with promising pilot validation rather than a validated production claim.","major_comments":[{"comment":"The hallucination-detection layer is the advertised differentiator, but its intermediate outputs are never validated. In §3.3.2, GPT-4o emits per-category severity scores S_o^i and per-frame labels d_new; §3.4 thresholds these to PASS/FAIL and §4 uses them to select regeneration actions. The only check is aggregate PASS/FAIL agreement (Table 1), which also includes the deterministic frame/motion modules; the 86.9%/88% figures could be produced by those modules while the MLLM adds noise. Calibration on 58 clips does not establish generalization for per-category hallucination judgments. Please report per-category human annotations, confusion matrices for object/new-structure categories, and an ablation with the hallucination module disabled.","section":"§3.3.2, §5.1"},{"comment":"The shadow evaluation is statistically thin. It uses 146 clips with no confidence intervals, no significance tests against baselines, and no inter-annotator reliability for the expert labels; expert PASS/FAIL is treated as ground truth. At this sample size the 86.9% agreement and 88% precision have wide intervals, and Table 3 reports no significance tests. Report confidence intervals, Cohen's kappa (or equivalent), the number of experts per clip, and the distribution of base I2V models and prompts so the reader can judge generalization.","section":"§5.1, Tables 1 and 3"},{"comment":"The 97% pseudo-production precision cannot be attributed to regeneration. Because only system-approved clips are reviewed, the experiment measures gate precision, not the causal effect of the regeneration loop. The sentence in §5.2 that the system is 'actively improving clips' is not supported without a control arm (e.g., regeneration disabled or random retries) and without reporting iteration counts, budget consumption, and per-action outcomes. Please supply a controlled comparison and the distribution of accepted clips by number of regeneration rounds.","section":"§5.2"},{"comment":"Table 2 and its surrounding text are internally ambiguous. The caption says 'contribution to Precision and False Positives' but the column is labeled 'Precision'; entries are 97–100% for hallucination categories, while the text says the majority of precision-impacting errors arise from object-level hallucinations. If these are per-signal precision values, the hallucination module is not the dominant error source; if they are error proportions, the column is mislabeled. Because diagnostics in Section 4 depend on failure codes, report per-signal error counts and conditional rates with confidence intervals.","section":"§5.1.1, Table 2"},{"comment":"Section 3.3.1 includes 'natural entry' and 'camera-induced entry' as new-structure hallucinations. In a camera pan, tilt, or dolly, previously off-screen content entering the frame is physically expected and is not an error relative to a coherent scene. Since d_new is an OR of all three cases and triggers FAIL, the system may systematically over-reject the camera-motion videos it explicitly claims to support. Please justify this definition with expert evidence or revise it.","section":"§3.3.1"}],"minor_comments":[{"comment":"There is a stray semicolon in 'automated; high-precision quality control'; use a comma or em dash.","section":"Abstract"},{"comment":"The caption uses 'VLLM' while the text uses 'MLLM'; unify the terminology.","section":"Fig. 2"},{"comment":"The alignment score s_align is defined as a 0/1 indicator, making thresholding by τ_align meaningless unless the classifier emits a probability. Provide a continuous alignment score or remove the threshold.","section":"Algorithm 2 (B5)"},{"comment":"Shadow Mode is described as 'calibration' although calibration was performed on a separate 58-clip set; rephrase to avoid confusion.","section":"§5.1"},{"comment":"The 70,000+ video production claim is not accompanied by sampled precision numbers or monitoring details; add the protocol and results.","section":"§5.3"},{"comment":"References [12] and [13] are incomplete author lists ('Liu et al.'); provide full citations.","section":"References"},{"comment":"The abstract promises end-user QoE and ultra-realism, but the evaluation measures expert pass/fail only; qualify this claim or add a user/QoE study.","section":"Abstract/§1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern raised by the reader is valid and is the main reason for major_revision. I would not reject because the architecture is plausible and the missing per-category validation and regeneration ablation are obtainable. If the authors cannot provide them, the paper should be reframed as an engineering case study with heuristic hallucination checks rather than a validated hallucination-aware system."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things to know: (1) HALLELUAI is a coherent industrial system that couples per-asset moderation with a regeneration loop, and the deterministic parts of the moderation module are sound enough to be useful today. (2) The paper's headline claim about hallucination-aware gating rests on a GPT-4o prompt that is never validated against independent per-category human labels, so the central evidence is thinner than the abstract suggests.\n\nThe genuinely new bit is the closed-loop coupling: structured diagnostics (failure codes, severities, remediation levers) directly parameterize the regeneration policy. That is a real contribution over existing evaluative benchmarks like VBench or VideoScore, which stop at scoring. The three-stage evaluation protocol (shadow, pseudo-production, production QA) is also sensible and reflects how such systems would be deployed. Frame-level signals (Laplacian blur, contrast, brightness) and the RAFT/TAPIR-based motion checks are standard but cleanly described.\n\nWhere it gets soft: the hallucination module is the load-bearing novelty, and the paper never compares its per-category severity scores or per-frame new-structure binaries to expert annotations. The only validation is overall PASS/FAIL agreement, which could be driven entirely by the deterministic frame and motion modules while the MLLM contributes noise. The prompt and thresholds were tuned on 58 clips from the same pipeline, and the 97% precision in pseudo-production is computed on system-approved clips after regeneration, so it can't separate filtering from genuine correction. No data, code, or exact thresholds are released, and the 146-clip shadow set has no confidence intervals. These are typical industrial-paper weaknesses, but they materially bound how much you can trust the central claim.\n\nThe stress-test note is right: per-category accuracy is never checked, and that is the weakest link. I don't see fatal flaws in the architecture itself. The paper is a decent engineering report, and the evaluation design is a step up from naive benchmark reporting.\n\nWho should read it: teams building AIGV pipelines for travel, real estate, or e-commerce will get practical value from the taxonomy and the loop design. Researchers looking for rigorous hallucination detection need to wait for independent validation. I probably won't cite it in the next year unless the authors release data. But it deserves a serious referee: the integration is meaningful, and the evaluation protocol, though imperfect, is exactly the kind of evidence we want from a production system. Send it out, with the expectation that hallucination-detection validation and artifact release are must-fix items.","headline":"Coherent industrial system paper whose closed-loop moderation-plus-regeneration design is genuinely useful, but the hallucination-aware claim rests on an unvalidated GPT-4o component and thin evidence.","tokens_in":10850,"tokens_out":2235,"would_cite":false,"duration_ms":23479,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HALLELUAI makes a closed loop of source-image-conditioned moderation and targeted regeneration the gate for production-grade image-to-video output.","keywords":["image-to-video generation","hallucination detection","video moderation","source-image fidelity","agentic regeneration","quality of experience","generative video quality control","multimodal LLM evaluation"],"falsifier":"Have independent annotators score the same 146 shadow-mode clips on the paper's seven object-hallucination and three new-structure categories, then compute per-category agreement with the system's 0–10 severity scores and binary flags. If per-category agreement is low, the reported 86.9% PASS/FAIL agreement and 97% approved-clip precision may be driven by non-hallucination failures, leaving the hallucination gate itself unvalidated.","tokens_in":9920,"feed_emoji":"🎬","tokens_out":5629,"duration_ms":55465,"temperature":0.7,"pith_summary":"The paper tries to establish that image-to-video generation can be made production-safe by pairing an automated moderation gate with an agentic regeneration loop. The moderation module scores each video relative to its source image on frame-level quality, camera motion, and hallucination, producing structured failure codes and a PASS/FAIL decision. The regeneration module turns those codes into targeted fixes — prompt refinement, camera adjustment, base-image substitution, or model switching — and iterates until the video passes or resources run out. In human-in-the-loop tests the system reached 86.9% PASS/FAIL agreement with experts in shadow mode and 97% precision on 1,158 system-approved clips, which the authors present as evidence of production-grade gating. The pith is that per-asset, source-image-conditioned moderation plus targeted regeneration can replace manual review at scale in high-trust domains like travel and real estate.","feed_headline":"Closed-loop gate pushes AI video approval to 97 percent","feed_subtitle":"System checks each clip against its source image, flags hallucinations and motion flaws, then regenerates until it passes expert review.","key_machinery":"The load-bearing object is the structured moderation report: a tuple of failure codes, category-wise severity scores (0–10), risk levels, and rationale text that serves both as a gate and as input to a planning agent. Hallucination detection rests on a taxonomy of seven object-hallucination categories and three new-structure entry types, applied by a multimodal language model that tracks source-image objects across sampled frames. Around it sit classical signal checks (Laplacian variance for blur and noise, grayscale mean and standard deviation for exposure and contrast), dense optical flow for motion intensity and smoothness, and point-trajectory alignment against prompt intent. The regener","core_discovery":"The central claim is that hallucination-aware moderation, keyed to the source image, acts as both gate and diagnostic engine for image-to-video output. The paper separates object hallucination (drift, fusion, deformation, disappearance, splitting, text corruption) from new-structure hallucination (natural, camera-induced, or abrupt entry), and asks a multimodal language model to score each category 0–10 and to make a per-frame binary call on new structures. These scores, with frame-signal and motion checks, drive a PASS/FAIL decision and action codes. The regeneration module converts codes into targeted interventions — prompt refinement, camera adjustment, base-image substitution, model swit","pith_inferences":["An extension left implicit: the same hallucination taxonomy could gate text-to-video output if the source image were replaced by a parsed scene graph from the prompt.","Per-category severity scores could be published as calibration curves per failure type, letting clients set their own risk thresholds; the paper reports only aggregate precision.","The 70,000+ videos produced in deployment, if audited per failure code, would reveal which regeneration levers actually fix each failure class — a benchmark no current evaluation suite offers.","The architecture is not limited to generative media; pointed at live-action footage, the same source-fidelity gate could screen for deepfakes or brand-safety violations."],"forward_implications":["Per-asset gating becomes feasible: each video is judged against its own source image, so a single failing asset can be blocked or fixed before release instead of only scoring a model's average output.","Failures become programmable: structured codes map to regeneration levers, so the loop replaces blind retries with targeted corrective actions.","The system can serve high-trust domains: at 97% precision on 1,158 approved clips, the authors position it as a practical gate for travel, real-estate, and e-commerce placements.","The taxonomy supplies a shared vocabulary for creative guidelines, letting expert standards be encoded as thresholds, risk levels, and remediation policies."],"fun_headline_variants":["AI video gate flags hallucinations and triggers auto-repair","Source-image fidelity enforced by iterative AI QC","Hallucination-aware moderation drives video regeneration","Closed-loop image-to-video QC with auto-correction","Image-to-video system checks and fixes hallucinations"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The system's quality gate rests on judgments made by a multimodal language model — the per-category severity scores and per-frame new-structure flags — and those judgments were tuned on 58 calibration clips and never checked against independent, per-category hallucination labels.","fun_headline_variants_meta":{"raw":{"variants":["AI video gate flags hallucinations and triggers auto-repair","Source-image fidelity enforced by iterative AI QC","Hallucination-aware moderation drives video regeneration","Closed-loop image-to-video QC with auto-correction","Image-to-video system checks and fixes hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000916,"raw_usage":{"total_tokens":3768,"prompt_tokens":740,"completion_tokens":3028,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":2970}},"tokens_in":484,"tokens_out":3028,"duration_ms":17345,"temperature":1.0,"reasoning_tokens":2970,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T04:00:47.963466+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent annotators score the same 146 shadow-mode clips on the paper's seven object-hallucination and three new-structure categories, then compute per-category agreement with the system's 0–10 severity scores and binary flags. If per-category agreement is low, the reported 86.9% PASS/FAIL agreement and 97% approved-clip precision may be driven by non-hallucination failures, leaving the hallucination gate itself unvalidated.","supporting_citations":[],"review_version":1}