{"id":"b30ed928-a56e-4f3c-8201-94a661d52a4e","arxiv_id":"2502.02610","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An automated pipeline generates personalized music videos from audio alone, using a CAPTCHA-style liveness check (CHARCHA) to collect and protect the user's facial identity.","lead":"This paper builds a fully automated pipeline that turns an audio file into a personalized music video, using speech-to-text, emotion detection, an AI image generator, and a face-verified user profile. Its main new piece is CHARCHA, a CAPTCHA-style check that confirms a real person on camera while collecting the images used to train their digital persona.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CHARCHA verifies liveness, not authorization: with no identity binding and a small fixed action set, replay and DeepFaceLive face-swap bypass the protocol, as Section 5 concedes, so the 'secure' claim in the title and abstract is currently unsupported.","rationale":"The reader's CONDITIONAL verdict is appropriate. The music-to-video pipeline is a plausible assembly of Whisper, a DEAM-trained emotion model, Stable Diffusion 1.5, checkpoint styles, and LoRA personalization, and I do not see an internal inconsistency in that part of the contribution. However, the central differentiator claimed by the title and abstract is 'secure,' and that claim depends entirely on CHARCHA. The protocol lacks identity binding and the Limitations section concedes that deepfake resistance is future work, so the security contribution is not merely under-evaluated; it is presently inadequate for the stated guarantee. The reader's weakest assumption points to the same load-bearing concern: CHARCHA's robustness against impersonation is not established. My analysis strengthens that concern by identifying a concrete protocol-level gap, not just missing experimental data: nothing in CHARCHA authenticates the person or binds the webcam face to a trusted identity. A replay or live face-swap attack can satisfy the liveness actions while using someone else's likeness. Given that the paper is a creative-track submission with honest limitations, I would not reject the whole pipeline, but the 'secure' wording should not stand unless the required adversarial evaluation is provided. The CONDITIONAL verdict remains unchanged, with the explicit condition that CHARCHA's false-accept rate under replay and face-swap attacks be reported and non-negligible rates addressed.","tokens_in":9416,"tokens_out":5847,"duration_ms":60188,"concrete_test":"Run the full CHARCHA Phase 2 protocol as an adversarial evaluation with n=20 trials per condition: (a) replay high-resolution video clips of a consenting individual performing each of the seven actions, presented to a webcam in the order CHARCHA requests; and (b) use DeepFaceLive so the attacker's webcam feed shows the same individual's face while the attacker performs the actions. Report the pass rate under the 6/10 score threshold for each condition. If either pass rate is materially above zero, the claim that CHARCHA 'protects people against unauthorized use of their face' is false as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's 'secure' framing rests entirely on CHARCHA, but Section 3.2's protocol verifies that a face is live and performing actions, not that the person shown is the person whose face is authorized. There is no binding to a trusted identity document, no pre-registered reference face, and no proof of consent. Since the action set is small and fixed (Figure 4 shows seven actions, while the text inconsistently says 'set of actions (4)'), an attacker can pre-record clips of a victim performing each action and replay them on demand; MediaPipe computes landmarks from the frames, so a replay can receive passing scores. A stronger variant is real-time face swap with DeepFaceLive: the attacker performs all required actions while the victim's face is overlaid, so the protocol collects and verifies the victim's face while the attacker remains in control. The authors' own Limitations section states that CHARCHA needs strengthening 'against deep fake technologies like Deep Face Live,' which concedes this failure mode is present in the current system. Additionally, the 6/10 threshold is said to be 'determined experimentally,' but no false-accept or false-reject rates are reported, so even the claimed separation between legitimate users and simple impostors is not established in the text.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MVP, a fully automated pipeline that takes a music audio file as the only input and produces a music video whose visuals are intended to follow the lyrics, emotion, and rhythm of the song. The pipeline combines Whisper for lyrics, a DEAM-trained valence/arousal model, ChatGPT 4o for prompt generation, Stable Diffusion 1.5 with checkpoint models for stylization, and spherical interpolation with onset-strength scheduling for synchronization. Personalization is achieved by training a DreamBooth/LoRA adapter on webcam-captured images of the user, and these images are collected through CHARCHA, a CAPTCHA-style protocol that asks the user to perform a set of facial actions while MediaPipe verifies them in real time. The paper evaluates the pipeline qualitatively with video stills and reports face-verification metrics (81% / 11.1% / 92%) for identity reproducibility, plus an appendix CLIP similarity analysis. The authors claim that CHARCHA provides secure facial identity verification that protects against unauthorized use of a person's face, and the title and abstract describe the framework as 'secure.'","tokens_in":9719,"tokens_out":4181,"duration_ms":41772,"significance":"If the security and personalization claims were established, this would be a useful demonstration of an end-to-end music-to-video system with a plausible consent-capture mechanism. The pipeline is concretely described and integrates several open-source components (Whisper, openSMILE, DEAM, Stable Diffusion, MediaPipe, DeepFace), and the qualitative results show that style transfer and lyric-driven prompting are feasible with current models. The CHARCHA idea of combining liveness verification with data collection for personalization is novel and worth exploring. However, the central 'secure' claim is currently unsupported: CHARCHA as described verifies only that a face is live and performing actions, not that the person shown is the authorized individual, and the authors' own limitations section concedes vulnerability to DeepFaceLive. The identity metrics are also circular because they compare generated frames to the very images used to train the LoRA, and they lack error bars or a reported baseline. With a reframed security scope and strengthened evaluation, the contribution could be a solid creative-AI systems paper; in its present form the headline claims outrun the evidence.","major_comments":[{"comment":"The 'secure' claim in the title and abstract is not supported by the CHARCHA protocol as described. Section 3.2 verifies liveness by having the user perform actions in front of a webcam, but there is no binding to a pre-registered reference face, no trusted identity document, and no mechanism to establish that the person acting is the person whose likeness is being authorized. Because the action set is small and fixed (Figure 4 shows seven actions; the text says four and later nine), an attacker can pre-record a victim performing each action and replay the clips, and MediaPipe would score the landmarks as valid. A real-time face-swap tool such as DeepFaceLive would be even more direct, and the Limitations section (Section 5) concedes that CHARCHA needs strengthening 'against deep fake technologies like Deep Face Live.' The paper must either remove 'secure' and 'identity verification' from the claims and scope CHARCHA as a liveness/consent-capture protocol, or add an identity-binding mechanism and demonstrate its security against replay and face-swap attacks.","section":"Abstract, §3.2, §5"},{"comment":"The pass threshold of 6/10 is asserted to be 'determined experimentally as shown in the results,' but no experimental threshold analysis is reported. Section 4.2 describes three phases, including a break-it attempt in Phase 3, but the only quantitative result shown is the survey in Figure 5; there are no false-accept rates, false-reject rates, or any accuracy numbers for the MediaPipe action verification. Without this data, the claimed separation between legitimate users and impostors is not established. The authors should report the threshold-tuning experiment with error rates and, ideally, an ROC curve or confusion matrix.","section":"§4.2"},{"comment":"The identity-reproducibility evaluation is partly circular. Table 1 compares generated video frames to the CHARCHA source images that were used to train the LoRA, so high similarity is expected by construction and does not measure generalization to unseen poses, expressions, or settings. The appendix's different-character baseline is described only qualitatively ('notably higher than would be expected') without reporting the numerical baseline scores or per-participant spread. Furthermore, the 81% / 11.1% / 92% figures have no error bars or per-subject variance for the seven participants. The evaluation should use held-out images not used in LoRA training, report baseline comparisons with variance, and state the number of frames and participants per metric.","section":"Table 1, Figure 9, §A.3"},{"comment":"The action set size is inconsistent across the paper: Section 3.2 says 'facial expressions and head poses (4)' and 'set of actions (4)', Figure 4 is captioned '7 CHARCHA Protocol Actions', and Section 4.2 Phase 1 refers to 'the list of nine actions.' This makes the protocol irreproducible. The authors must specify the exact action set, the total number of actions, and how the six actions in a test are sampled from that set.","section":"§3.2, §4.2"},{"comment":"The central claim that generated visuals are 'contextually aligned and synchronized with the music' is supported only by qualitative stills and a link to an external drive folder. No quantitative measure of lyric-to-visual alignment, emotion alignment, or beat synchronization is provided, and there is no user study of music-video quality. At a minimum, the authors should report an onset-alignment error metric (e.g., against the PLP beats described in Section A.5) or clearly scope the claim as a qualitative demonstration.","section":"§4.1, §A.2"}],"minor_comments":[{"comment":"Typo: 'emperically demonstrated' should be 'empirically demonstrated.'","section":"§4.2"},{"comment":"The protocol description says the test takes 60-90 seconds, but the stated timings (2-second calibration plus six actions of 10 seconds each with 5-second gaps) sum to about 92 seconds before any retry; please reconcile these numbers.","section":"§3.2"},{"comment":"The text says 'repeated for 6 actions' and later mentions 'seven screenshots'; it is unclear whether the calibration image is one of the seven or whether the action images alone are seven. Please clarify the exact number of images used for LoRA training.","section":"§3.2"},{"comment":"Figure 5 is described as a survey with n=16 participants, but the survey results are not summarized in the text; the reader cannot tell what the participants' comfort levels or pass rates were. Please add a brief description of the survey findings.","section":"§4.2, Figure 5"},{"comment":"The CLIP similarity analysis would be more convincing if the 'different character' baseline were plotted on the same axes in Figure 9, rather than described only in the text.","section":"§A.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is better suited as a systems/demonstration contribution than as a security contribution. The CHARCHA protocol is a creative idea, but the 'secure' framing is not defensible in its current form, and the evaluation of identity preservation needs de-circularization. I would encourage the editor to seek reviewers with security and human-factors expertise, since the liveness-vs-identity distinction is central to the verdict. The patent-pending note is not relevant to scientific evaluation and can be ignored."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is best read as a system description, not a security result. The music-to-video pipeline is concretely specified and plausibly produces the videos, and CHARCHA is a genuinely new combination—CAPTCHA-style liveness actions doubled as data collection for LoRA personalization. That dual-use idea is the real contribution. The paper is also transparent about its limitations, which is more than many.\n\nWhat works: the pipeline integrates Whisper, DEAM-based MER, GPT-4o prompting, SD1.5, spherical interpolation, and DreamBooth LoRA in a straightforward way. The authors ran a 16-person CHARCHA usability experiment and a 7-person identity-preservation evaluation. The CLIP similarity appendix correctly notes that fluctuations indicate the video does not merely replay training frames. That is a fair point, though the comparison to a different-character baseline is mentioned but not quantified.\n\nThe load-bearing issue is that CHARCHA verifies liveness, not identity authorization. There is no binding to a trusted identity document, no pre-registered reference face, and no proof of consent. A pre-recorded video of a victim performing the actions can pass MediaPipe, and the paper's own Limitations section concedes DeepFaceLive is an open attack. So 'secure' in the title and abstract is unsupported as it stands. The 6/10 threshold is said to be 'determined experimentally,' but no false-accept or false-reject rates are given, so even the basic separation between humans and simple replay is not established. The identity metrics (81%, 11.1%, 92%) have no error bars or per-subject variance, and there are no baselines. Also, the protocol description is internally inconsistent: the text says four actions, Figure 4 shows seven, and Phase 1 says nine. That is sloppy and needs fixing.\n\nNone of this means the pipeline is fake. It looks like a working prototype with a genuine consent-collection idea. But the presentation overclaims; it should be framed as a prototype with known weaknesses.\n\nWho is this for? People thinking about consent mechanics in generative media and anyone building music-to-video systems. It does not deserve a rigorous security referee as a security paper, but for a creative AI track or a workshop on generative media it could be accepted after major revision—primarily removing 'secure' from the headline and adding quantitative attack results or explicitly scoping the claim. I'd send it to a creative AI venue, not to a security conference.\n\nRecommendation: engage if you work on media personalization or consent; otherwise skim. It deserves a serious referee for the creative system, but the security claim should be withdrawn or heavily qualified.","headline":"A likeable prototype with a genuinely new dual-use liveness/consent idea, but the 'secure' claim is unsupported as written and needs to be scoped down before this should be accepted anywhere.","tokens_in":10231,"tokens_out":2617,"would_cite":false,"duration_ms":26237,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a fully automated pipeline that turns a single music audio file into a personalized music video, with visuals following lyrics, emotion, and rhythm, and a user's face inserted via LoRA trained on images collected by a…","keywords":["music video generation","personalization","low-rank adaptation","LoRA","face verification","CAPTCHA","liveness detection","multimodal generation"],"falsifier":"A direct adversarial test in which participants attempt to pass CHARCHA using a printed photo, a phone displaying a video, or a real-time face-swap tool such as DeepFaceLive; if any of these succeeds often enough to beat the 6/10 threshold, the security claim collapses. A second check would be to recover the experimental determination of the 6/10 threshold, since the paper states it was determined experimentally but provides no data.","tokens_in":12,"feed_emoji":"🎵","tokens_out":3318,"duration_ms":129204,"temperature":0.7,"pith_summary":"This paper claims to build a fully automated pipeline, MVP, that takes only a music audio file and produces a music video whose visuals track the lyrics, emotional arc, and rhythmic pulse of the song. A listener can be inserted into the video as a recurring character by training a LoRA adapter on seven webcam images of their face. To collect those images ethically, the paper introduces CHARCHA, a CAPTCHA-style liveness protocol that has a live user perform six randomized facial expressions and head poses, verified with MediaPipe, before their likeness can be used. If the pipeline works as described, an ordinary listener can turn any song into a personal music video with no manual editing and some protection against unauthorized use of their face.","feed_headline":"A song goes in; a personalized music video comes out","feed_subtitle":"The MVP pipeline adds your face via seven verified webcam shots and syncs visuals to lyrics, emotion, and beat.","key_machinery":"The load-bearing components are the CHARCHA protocol—a CAPTCHA-style live-user verification that records seven webcam images while a person performs six randomized facial expressions and head poses verified in real time by MediaPipe—and the LoRA fine-tuning step that turns those seven images into a reusable character adapter for Stable Diffusion. Around these sit a chain of pretrained translators: Whisper (lyrics with timestamps), a DEAM-trained arousal/valence network over openSMILE features (emotion quadrants), GPT-4o (lyric/emotion to image prompts), Stable Diffusion 1.5 with style checkpoints (frames), and onset-strength-weighted spherical interpolation (rhythm-synced transitions).","core_discovery":"On its own terms, the paper's central claim is that music-to-video generation can be made both context-aware and identity-aware in one zero-shot pipeline: Whisper extracts timestamped lyrics, a music emotion recognition model maps arousal and valence into four affect quadrants, GPT-4o turns lyric/emotion pairs into cinematic image prompts, and Stable Diffusion 1.5 renders frames that are stitched by spherical interpolation whose weights are modulated by the music's onset strength envelope. Personalization is achieved by fine-tuning a LoRA adapter on seven CHARCHA-collected webcam images, which the authors report reproduces the participant's face in 81% of frames (92% of face-containing frames). CHARCHA itself is the paper's security contribution: it pairs a calibration step with six randomized, MediaPipe-verified actions and a pass threshold of 6/10, intended to prove a live, consenting human is on camera while simultaneously gathering the images needed for the LoRA.","pith_inferences":["CHARCHA's design could be lifted out of music videos and applied to any generative service that uses a person's likeness—social avatars, advertising, or synthetic media—making it a potential template for consent-gated identity use.","The threshold of 6/10 and the specific set of six actions are presented as experimentally determined, but the paper does not report the experimental data; an independent replication of that threshold experiment would tell whether the liveness bar is actually calibrated against real spoofing attempts.","Because the pipeline relies on a commercial LLM for prompt generation, the visual narrative quality may shift with model updates, which means the system's consistency over time is not guaranteed by the paper's architecture alone.","The CLIP similarity metric used to argue that generated videos do not merely copy training images is acknowledged by the authors as sensitive and misaligned with human judgment, so a human-evaluation study would be a stronger test of personalization fidelity."],"forward_implications":["If the pipeline is correct, users can generate a complete music video from a single MP3 file, without any video editing skill, and see their own face rendered in multiple artistic styles.","The CHARCHA protocol, if it holds, turns the identity-collection step into a liveness check, so the images used for personalization double as evidence that the person consented to being captured.","Combining lyric timestamps, emotion quadrants, and onset strength provides a recipe for synchronizing generated frames to both semantic content and musical beats, which could generalize beyond music videos to other audio-driven animation tasks.","Because the LoRA adapter is trained on only seven images, the approach suggests a low-data path to personalized character consistency in diffusion-based video generation.","The reported face-verification numbers (81% of frames, 92% of face frames) imply that a small set of expressive webcam shots is enough to make a generated character recognizable, if the measurement is trusted."],"supporting_citations":[{"why":"Whisper provides the timestamped lyrics that drive the visual narrative in the pipeline.","marker":"[27]"},{"why":"The DEAM dataset is used to train the arousal/valence model that supplies emotional context for prompts.","marker":"[28]"},{"why":"openSMILE features are extracted from the audio and fed into the music emotion recognition model.","marker":"[10]"},{"why":"Stable Diffusion (latent diffusion) is the text-to-image backbone that renders each video frame.","marker":"[11]"},{"why":"DreamBooth methodology is used together with LoRA to fine-tune the character adapter on user images.","marker":"[33]"},{"why":"LoRA provides the low-rank adaptation mechanism that makes personalization efficient.","marker":"[36]"},{"why":"MediaPipe verifies each CHARCHA action in real time, forming the technical core of the liveness check.","marker":"[35]"},{"why":"DeepFaceLive is named in the limitations as the deepfake tool that CHARCHA is not yet hardened against.","marker":"[42]"}],"fun_headline_variants":["Your face, your music: CHARCHA makes videos just for you","Seven selfies and a verified face become your music video","Music that sees you: CHARCHA-backed personal videos","CHARCHA verifies, LoRA personalizes, music becomes visual","From consent to frames: a music video with your face"],"cache_read_input_tokens":12416,"weakest_assumption_plain":"The entire security claim rests on the assumption that a determined impostor cannot pass CHARCHA by holding a photo, playing a recording, or running a live deepfake in front of the webcam—the paper itself concedes it is not yet hardened against DeepFaceLive, and the 6/10 threshold is said to be determined experimentally without reporting the experiment.","fun_headline_variants_meta":{"raw":{"variants":["Your face, your music: CHARCHA makes videos just for you","Seven selfies and a verified face become your music video","Music that sees you: CHARCHA-backed personal videos","CHARCHA verifies, LoRA personalizes, music becomes visual","From consent to frames: a music video with your face"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1243,"prompt_tokens":875,"completion_tokens":368,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":281}},"tokens_in":491,"tokens_out":368,"duration_ms":4859,"temperature":1.0,"reasoning_tokens":281,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T17:01:14.433175+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct adversarial test in which participants attempt to pass CHARCHA using a printed photo, a phone displaying a video, or a real-time face-swap tool such as DeepFaceLive; if any of these succeeds often enough to beat the 6/10 threshold, the security claim collapses. A second check would be to recover the experimental determination of the 6/10 threshold, since the paper states it was determined experimentally but provides no data.","supporting_citations":[{"cited_title":"An Empirical Study & Evaluation of Modern CAPTCHAs","cited_arxiv_id":"2307.12108","evidence_quote":"Whisper provides the timestamped lyrics that drive the visual narrative in the pipeline."},{"cited_title":"Benchmarking music emotion recognition sys- tems,","cited_arxiv_id":null,"evidence_quote":"The DEAM dataset is used to train the arousal/valence model that supplies emotional context for prompts."},{"cited_title":"Opensmile: The munich versatile and fast open-source audio feature extractor,","cited_arxiv_id":null,"evidence_quote":"openSMILE features are extracted from the audio and fed into the music emotion recognition model."},{"cited_title":"Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation,","cited_arxiv_id":null,"evidence_quote":"DreamBooth methodology is used together with LoRA to fine-tune the character adapter on user images."},{"cited_title":"Available: https://civitai.com/","cited_arxiv_id":null,"evidence_quote":"MediaPipe verifies each CHARCHA action in real time, forming the technical core of the liveness check."},{"cited_title":"iperov/deepfacelive,","cited_arxiv_id":null,"evidence_quote":"DeepFaceLive is named in the limitations as the deepfake tool that CHARCHA is not yet hardened against."}],"review_version":1}