Pith. sign in

REVIEW 5 cited by

TalkCLIP: Talking Head Generation with Text-Guided Expressive Speaking Styles

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2304.00334 v4 pith:CCU3KPUK submitted 2023-04-01 cs.CV

Yifeng Ma , Suzhen Wang , Yu Ding , Bowen Ma , Tangjie Lv , Changjie Fan , Zhipeng Hu , Zhidong Deng
show 1 more author
Xin Yu
This is my paper
classification cs.CV
keywords expressionstalkingtalkclipdescriptionsheadtextfacialdataset
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Audio-driven talking head generation has drawn growing attention. To produce talking head videos with desired facial expressions, previous methods rely on extra reference videos to provide expression information, which may be difficult to find and hence limits their usage. In this work, we propose TalkCLIP, a framework that can generate talking heads where the expressions are specified by natural language, hence allowing for specifying expressions more conveniently. To model the mapping from text to expressions, we first construct a text-video paired talking head dataset where each video has diverse text descriptions that depict both coarse-grained emotions and fine-grained facial movements. Leveraging the proposed dataset, we introduce a CLIP-based style encoder that projects natural language-based descriptions to the representations of expressions. TalkCLIP can even infer expressions for descriptions unseen during training. TalkCLIP can also use text to modulate expression intensity and edit expressions. Extensive experiments demonstrate that TalkCLIP achieves the advanced capability of generating photo-realistic talking heads with vivid facial expressions guided by text descriptions.

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EmoteGPT: 3D Human Facial Expressions from Natural Language Descriptions

    cs.CV 2026-07 conditional novelty 6.5 of 10

    EmoteGPT regresses FLAME 3DMM expression parameters from explicit or implicit text using an MLLM with a dedicated <Expr> token, trained on the new Txt2Emote dataset plus image data, outperforming prior text-to-3D face...

  2. Think2Sing: Orchestrating Structured Motion Subtitles for Singing-Driven 3D Head Animation

    cs.GR 2025-09 conditional novelty 6.0 of 10

    Think2Sing uses LLM-generated, time-aligned motion subtitles and a motion-intensity proxy to guide diffusion-based 3D head animation from singing audio and lyrics.

  3. CEM-Net: Cross-Emotion Memory Network for Emotional Talking Face Generation

    cs.MM 2025-08 unverdicted novelty 6.0 of 10

    CEM-Net stores cross-emotion expression displacements in a memory bank so a generated talking face matches the emotion in the audio even when the reference image emotion conflicts.

  4. EDTalk++: Full Disentanglement for Controllable Talking Head Synthesis

    cs.CV 2025-08 conditional novelty 5.0 of 10

    EDTalk++ disentangles talking-head video into four orthogonal motion banks (mouth, pose, eyes, expression) and drives them from either video or audio inputs.

  5. MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A 3D facial animation framework that disentangles content and emotion and predicts frame-wise emotion intensity from audio plus text for dynamic expressions.

Pith tools