{"id":"174c9693-e2c7-4b2f-84cf-5e039e48e300","arxiv_id":"2501.07978","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper presents FDA, a manually annotated dataset, FaceTrack-MM, a face-tracking video MLLM, FEC-Bench, a benchmark, and TEM, a ChatGPT-based metric, all for dynamic facial expression captioning.","lead":"This paper introduces a new benchmark, dataset, and model for describing facial expressions in videos. The authors report that their face-tracking video language model outperforms existing video MLLMs on this new task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is confounded: FaceTrack-MM is fine-tuned on FDA while all 15 baselines are zero-shot, so the large Table 1 margins may reflect in-distribution tuning rather than the face-tracking architecture.","rationale":"The reader's stated weakest assumption is the TEM exact-string LCS flaw, which is real and should be fixed. However, that flaw is less load-bearing for the headline claim: even if the temporal component were removed, Table 1 still shows Ours best on AutoDQ, CIDEr, ROUGE-L, and VideoChatGPT scores. The missing fine-tuned baseline is more central because it is what would be needed to attribute the reported margins to the proposed face-tracking architecture rather than to in-domain instruction tuning. I partially agree with the reader because their rationale does list the unfair comparison as a concern, even though their formal weakest-assumption field focuses on TEM. The recommended verdict remains CONDITIONAL: the issue is addressable with a control experiment and does not undermine the dataset contribution, but the current evidence does not support the architecture-level superiority claim as stated.","tokens_in":26551,"tokens_out":5745,"duration_ms":59038,"concrete_test":"Fine-tune a strong open baseline, e.g. Qwen2-VL-7B, on the same 4,033-video FDA training split with the same LoRA recipe (r=64, alpha=128, batch size 32, 3 epochs, 16 frames, bf16) and evaluate it on the same 1,000-video FEC-Bench test split with the paper's metrics. If this fine-tuned baseline matches or exceeds FaceTrack-MM's AutoDQ 0.483 / TEM 0.364, the architecture-specific SOTA claim collapses; if it remains clearly below, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the evaluation protocol supporting the headline claim. Table 1 reports FaceTrack-MM after instruction tuning on the FDA training videos, while every comparator (GPT-4o, Claude-3.5, Qwen2-VL, LLaVA-OneVision, etc.) is evaluated zero-shot on a test set drawn from the same annotated pool and annotation style. The paper's own ablation in Table 2 shows that instruction tuning alone raises VideoLLaMA2's AutoDQ from 0.344 to 0.422 and TEM from 0.258 to 0.321; the full face-tracking plus FaceXFormer pipeline adds only +0.061 AutoDQ and +0.043 TEM. Therefore, the large margins over zero-shot baselines in Table 1 mostly measure the effect of in-distribution fine-tuning, not the proposed face-tracking module. Without a similarly fine-tuned strong baseline, 'significantly outperforms all other methods' does not establish FaceTrack-MM as the best architecture for DFEC; it only shows that fine-tuning on the benchmark domain helps. This is load-bearing because the abstract and introduction use the Table 1 margins to claim superior face tracking and main-character focus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Dynamic Facial Expression Captioning (DFEC), a new task of generating natural-language descriptions of a main character's facial changes in video. It contributes the FDA dataset (5,033 manually annotated clips), the FaceTrack-MM model (VideoLLaMA2 augmented with a dynamic face-tracking module and a FaceXFormer facial encoder), the TEM evaluation metric (event extraction, relation classification, and LCS), and the FEC-Bench benchmark comparing 15 MLLMs. The central empirical claim is that FaceTrack-MM 'significantly outperforms all other methods' on FEC-Bench, and that its face-tracking design is responsible for the improvement.","tokens_in":26792,"tokens_out":3836,"duration_ms":36867,"significance":"If the claims were fully supported, the paper would make a useful contribution: a new dataset and benchmark for a task (DFEC) that existing MLLMs handle poorly, a modular architecture for injecting facial priors into video LLMs, and a metric that tries to capture both event content and temporal order. The dataset and benchmark are potentially reusable resources, and the ablation study (Table 2) is a good start toward understanding where gains come from. However, the headline comparison is confounded by in-distribution fine-tuning, and the TEM metric has an internal inconsistency between its stated relation-classification step and its actual LCS implementation. These issues do not invalidate the dataset or the architectural idea, but they do undermine the paper's main comparative and metric claims as currently presented.","major_comments":[{"comment":"The headline comparison is confounded. FaceTrack-MM is fine-tuned on the FDA training split, while all 15 baselines in Table 1 are evaluated zero-shot. Table 2 shows that instruction tuning alone raises VideoLLaMA2's AutoDQ from 0.344 to 0.422 and TEM from 0.258 to 0.321, and the full face-tracking plus FaceXFormer pipeline adds only +0.061 AutoDQ and +0.043 TEM. Therefore, the large margins over zero-shot baselines in Table 1 mostly reflect in-distribution fine-tuning rather than the proposed face-tracking architecture. The claim in §5.2 that the method 'significantly outperforms all other methods on the FEC-Bench' is not supported without a comparably fine-tuned strong baseline (e.g., VideoLLaMA2 instruction-tuned on the same FDA training data) as the primary control.","section":"§5.2, Table 1 and Table 2"},{"comment":"The TEM metric is internally inconsistent. The text describes relation classification into Same Meaning, Opposite Meaning, and No Relation, but Algorithm 1 line 14 computes LCS using exact string equality (EG[i-1] == ER[j-1]). ChatGPT-extracted event strings will rarely be identical when they carry the same meaning, so the LCS temporal-order component does not measure semantic order consistency as claimed. The F-measure computation in line 24 is also unspecified: the paper does not state how the relation classifications are converted into precision and recall. Without a precise matching rule, TEM's numerical values are not reproducible, and the temporal-order component should be either implemented via the relation classifications or clearly described as an exact-string proxy.","section":"§3.4, Algorithm 1"},{"comment":"The ChatGPT-based reformatting of generated text before evaluation biases the benchmark in favor of the fine-tuned model. The paper states that all generated text is converted into a format similar to the reference 'without altering its content,' but no evidence is provided that content is preserved; for n-gram metrics such as CIDEr and ROUGE-L, even a paraphrase with the same meaning can change token overlap. Because FaceTrack-MM was fine-tuned on the reference style, it naturally matches the reformatted references better than zero-shot baselines. Please report scores both before and after reformatting, and show that the reformatting does not systematically change the relative ranking.","section":"§5.2, Fig. 8"},{"comment":"The claim of 'superior performance ... even in intricate multi-person scenarios' is not directly tested. No experiment isolates multi-person videos, and the qualitative example in Fig. 4 appears to be a single person. Please provide a breakdown of FEC-Bench results by number of detected faces per video, or a dedicated multi-person subset, to support this specific claim about multi-person robustness.","section":"Abstract and §5.4"}],"minor_comments":[{"comment":"The LCS score is normalized by m (the number of generated events), which is asymmetric and not justified; if a model generates many spurious events, the score is penalized even when all reference events are matched in order. Please state the normalization choice explicitly and consider reporting both order-aware recall and precision, or normalizing by max(m,n).","section":"§3.4, Algorithm 1 line 21"},{"comment":"The main-trajectory selection uses K-means with exactly two clusters, implicitly assuming that all videos contain only two roles (main and background). Many videos may have multiple salient characters; please justify this fixed choice or make the number of clusters adaptive.","section":"§4.2.2"},{"comment":"The table caption lists a 'Consistency' metric, but the table header shows only Correctness, Detail, Context, and Temporal. Please align the caption with the columns actually reported.","section":"Table 1"},{"comment":"The annotation pipeline involves three annotators and a final reviewer, but no inter-annotator agreement statistic is reported. A measure such as Cohen's kappa or a sample-level agreement rate would strengthen confidence in the dataset's quality.","section":"§3.2"},{"comment":"There are several typos and formatting inconsistencies, e.g., 'PLLaV A' in Table 1, 'VideoLLaMA2' vs 'VideoLLaMA2-7B' vs 'Mistral-7B-Instruct' in §4.2.1, and 'Fig. 1' used to refer to both the task-comparison figure and the architecture figure. Please copyedit throughout.","section":"Various"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claimed contribution—that FaceTrack-MM outperforms all existing MLLMs on DFEC—is currently supported only by a comparison against zero-shot baselines. The authors' own ablation shows that the proposed modules add modest gains relative to instruction tuning alone. The TEM metric also needs a clear algorithmic specification. These are fixable with additional experiments and text revisions, but the authors should temper the abstract and introduction claims unless they add the missing controls. The dataset and benchmark have standalone value, so I see this as a major-revision case rather than a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the DFEC task and FDA dataset are the real contribution; the model comparison in Table 1 is not fair as stated. The paper deserves a serious referee, but the load-bearing SOTA claim needs to be reworked.\n\nWhat's new: a new task (dynamic facial expression captioning), a manually corrected dataset of 5,033 clips with rich annotations, and a benchmark of 1,000 test clips with 15 baselines. FaceTrack-MM is a reasonable integration of face tracking and an auxiliary facial encoder on top of VideoLLaMA2. The motivation is legitimate: video MLLMs allocate few tokens per frame and lose facial detail. The ablation in Table 2 is honest and useful: it shows instruction tuning alone accounts for most of the gain, and the face-tracking modules add a modest but measurable increment.\n\nThe soft spots are in the evaluation protocol and the metric. The stress-test concern holds up. Table 1 compares FaceTrack-MM, fine-tuned on the FDA training split, against zero-shot baselines. The paper's own ablation shows that instruction tuning alone raises VideoLLaMA2's AutoDQ from 0.344 to 0.422 and TEM from 0.258 to 0.321; the full pipeline adds only +0.061 and +0.043. So the large margins over GPT-4o and Qwen2-VL mostly reflect in-distribution fine-tuning, not the face-tracking modules. \"Significantly outperforms all other methods\" is not established without at least one strong baseline fine-tuned on the same data.\n\nThe TEM metric has a concrete implementation flaw. Algorithm 1 computes the LCS by exact string equality of event strings, but the method text justifies the metric by classifying event relations into Same Meaning, Opposite Meaning, and No Relation. ChatGPT-extracted events that share meaning will rarely be string-identical, so the temporal LCS component does not measure what it claims. Either the algorithm should align events through the relation classification, or the event strings need to be canonicalized first.\n\nMinor issues: no code or data released yet, which makes the announced dataset and benchmark unverifiable; the K-means cluster count for main/background separation is a free parameter but not a big deal.\n\nThis is a conditional accept. The dataset and task are useful resources, and the ideas are sound enough to be fixed. A referee should ask for a properly matched fine-tuned baseline, a corrected TEM implementation (or a clear statement that the LCS is a crude proxy), and a release of the data and code. With those changes the paper would be a solid benchmark contribution.","headline":"The DFEC task and FDA dataset are the real contribution; the Table 1 SOTA claim is confounded by fine-tuning and needs rework.","tokens_in":27315,"tokens_out":2420,"would_cite":true,"duration_ms":23657,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces dynamic facial expression captioning with a 5,033-clip instruction dataset, a face-tracking multimodal model, a benchmark, and an event-order-aware metric, and claims the model beats every compared method on the…","keywords":["dynamic facial expression captioning","video multimodal large language models","instruction tuning","face tracking","temporal event matching","facial expression benchmark","multi-person video understanding","LoRA fine-tuning"],"falsifier":"Construct caption pairs that describe the same facial events in the same order but use synonymous wording, such as 'his eyes widen' versus 'his eyes become wide', run TEM on them, and check whether the order component stays unchanged; if the score drops, the metric is measuring surface wording rather than event order.","tokens_in":26366,"feed_emoji":"🎭","tokens_out":10161,"duration_ms":98200,"temperature":0.7,"pith_summary":"Video multimodal large language models can describe what happens in a clip, but they under-describe faces because every frame is compressed into a small number of visual tokens. The paper's proposal is that this can be fixed by giving the model a dedicated, tracked view of the main character's face: it introduces a new task, dynamic facial expression captioning; a 5,033-clip manually annotated instruction dataset (FDA); and FaceTrack-MM, a model that adds a face-tracking module and a face-specific encoder on top of an existing video MLLM. It also contributes FEC-Bench, a 1,000-clip benchmark, and a metric, TEM, that scores both event content and event order. The paper's central claim is that FaceTrack-MM outperforms all 15 compared open-source and proprietary models on FEC-Bench, with the largest gains on lexical and event-based metrics. If true, this gives downstream applications a compact recipe for turning video MLLMs into reliable, temporally ordered describers of facial behavior.","feed_headline":"Face-tracking video model tops all 15 rivals in facial captions","feed_subtitle":"A 5,033-clip annotated dataset plus a dedicated face pathway lets a 7B model capture subtle facial changes.","key_machinery":"The central mechanism is the dynamic video face tracking module. It downsamples the video to 16 fps, runs face detection (DaMOFD) and face-feature extraction (TransFace), links detections into trajectories with StrongSORT, and uses K-means on trajectory area and internal cosine similarity to pick the main character's trajectory; the chosen face is then encoded by FaceXFormer and projected into the LLM as extra tokens. The paper's other mechanism is the Temporal Event Matching (TEM) metric, which extracts face-related events with ChatGPT, classifies each generated-versus-reference event pair as Same Meaning, Opposite Meaning, or No Relation, and averages an event F-measure with a longest-common-subsequence score on the ordered event lists.","core_discovery":"On its own terms, the paper's discovery is that the bottleneck in facial-expression captioning is visual-token allocation rather than language understanding, and that the bottleneck is removable. FaceTrack-MM takes the VideoLLaMA2 architecture and inserts a dynamic face pathway: frames are downsampled to 16 fps, faces are detected and tracked into trajectories, K-means clustering separates the main character's trajectory from background faces, and FaceXFormer encodes that trajectory into compact 'facial prior' tokens that enter the LLM alongside the usual CLIP/STC video tokens. After LoRA instruction tuning on FDA, the model records the best score in every column of FEC-Bench, including ChatGPT correctness, detail, context, and temporal scores, CIDEr, ROUGE-L, AutoDQ, and the proposed TEM. The ablation in the paper attributes the gains specifically to the trajectory-based face selection and to the specialized face encoder, with plain face-crop concatenation giving almost no improvement.","pith_inferences":["The face-pathway recipe likely transfers to other small-region video understanding tasks such as hand, gaze, or gesture description, but the paper does not test this.","TEM could be made more robust by feeding the relation-classification results into the longest-common-subsequence matching instead of requiring identical event strings; the paper does not do this.","Because the benchmark uses a random split of one annotated video distribution, the reported gains may partly reflect learning the annotation style, and cross-domain generalization remains untested.","Expanding the 5,033-clip manual annotation pipeline with model-assisted pre-labeling plus human correction is a natural next step toward scaling this task."],"forward_implications":["A 7B open video MLLM fine-tuned only with LoRA can surpass GPT-4o and Claude-3.5-Sonnet on facial-expression captioning once the face pathway and instruction data are added.","The FDA dataset's separation of objective facial changes from marked subjective inferences offers a reusable annotation pattern for subtle video behavior.","TEM gives an automatic, LLM-based way to penalize caption errors that preserve content but scramble event order, which CIDEr, ROUGE-L, and AutoDQ do not do.","FaceTrack-MM's trajectory selection should let the model describe the intended person's face in multi-person video instead of blending or switching between faces."],"supporting_citations":[{"why":"Base architecture, STC visual projector, and the main baseline that FaceTrack-MM builds on.","marker":"[11]"},{"why":"Auxiliary facial visual encoder that turns the tracked face into tokens for the LLM.","marker":"[56]"},{"why":"Multi-object tracker that forms consistent face trajectories across frames.","marker":"[15]"},{"why":"Face feature extractor used to compute trajectory similarity for main-character selection.","marker":"[12]"},{"why":"Face detection and keypoint model used in the tracking module.","marker":"[47]"},{"why":"Supplies the event-extraction and F-measure machinery that TEM extends.","marker":"[75]"},{"why":"Performs event extraction and relation classification inside TEM, and generated preliminary annotations.","marker":"[58]"},{"why":"Generates preliminary annotations and serves as the strongest proprietary baseline in the benchmark.","marker":"[61]"},{"why":"The token-limited video MLLM used to motivate the need for a face-specific pathway and as a strong baseline.","marker":"[76]"},{"why":"The facial-prior-expert idea that motivates adding face features to a large language model.","marker":"[36]"}],"fun_headline_variants":["Face-tracking video model tops 15 rivals in facial captions","Token-efficient face pathway boosts video expression captions","Video LLM reads faces better with dedicated tracking tokens","FEC-Bench exposes and improves facial captioning in video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The metric's event-order score assumes that ChatGPT extracts identical strings for the same facial event in the generated and reference captions; if the wording differs, the order match fails even when the events and their sequence are correct.","fun_headline_variants_meta":{"raw":{"variants":["Face-tracking video model tops 15 rivals in facial captions","Token-efficient face pathway boosts video expression captions","Video LLM reads faces better with dedicated tracking tokens","FEC-Bench exposes and improves facial captioning in video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000321,"raw_usage":{"total_tokens":1828,"prompt_tokens":990,"completion_tokens":838,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":771}},"tokens_in":606,"tokens_out":838,"duration_ms":8578,"temperature":1.0,"reasoning_tokens":771,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:29:23.062027+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct caption pairs that describe the same facial events in the same order but use synonymous wording, such as 'his eyes widen' versus 'his eyes become wide', run TEM on them, and check whether the order component stays unchanged; if the score drops, the metric is measuring surface wording rather than event order.","supporting_citations":[{"cited_title":"DamoFD: Digging into backbone de- sign on face detection","cited_arxiv_id":null,"evidence_quote":"Face detection and keypoint model used in the tracking module."},{"cited_title":"Tarsier: Recipes for training and evaluating large video description models, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the event-extraction and F-measure machinery that TEM extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Performs event extraction and relation classification inside TEM, and generated preliminary annotations."},{"cited_title":"Gpt-4o system card, 2024","cited_arxiv_id":null,"evidence_quote":"Generates preliminary annotations and serves as the strongest proprietary baseline in the benchmark."},{"cited_title":"Facial affective behavior analysis with instruction tuning, 2024","cited_arxiv_id":null,"evidence_quote":"The facial-prior-expert idea that motivates adding face features to a large language model."}],"review_version":1}