Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

Social Caption: Evaluating Social Understanding in Multimodal Models

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multimodal models' social understanding is three separate abilities—answering questions, describing whole scenes, and extracting relevant details—and accuracy on the first does not predict the other two.

desk verdict New multidimensional social-understanding evaluation for MLLMs that is worth engaging; the judge-alignment headline overstates what the data support. read the letter →

arxiv 2601.14569 v2 pith:XSR2IGQP submitted 2026-01-21 cs.CL cs.LG

Leena Mathur , Bhaavanaa Thumu , Youssouf Kebe , Louis-Philippe Morency This is my paper
classification cs.CLcs.LG
keywords socialunderstandingmultimodallargelanguagemodelsevaluationframeworkvideoquestionansweringholisticanalysisdirectedMLLMjudgesAPRACEtaxonomy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that social understanding in multimodal large language models (MLLMs) is not a single ability and cannot be read off from multiple-choice question-answering accuracy. It introduces SOCIALCAPTION, an evaluation framework with three dimensions: Social Inference (choosing the correct answer about an interaction), Holistic Social Analysis (writing a comprehensive description of the scene), and Directed Social Analysis (writing a description containing only the information relevant to a given question). On one-minute real interaction videos, the authors find that a model can be strong at inference while weak at holistic or directed description; that adding spoken-context transcriptions improves inference for every model; and that small open-source models can match or exceed much larger closed-source models on the generative dimensions. They further report that open-source MLLM judges rate HSA and DSA generations in close binary-F1 agreement with human raters, which they offer as a proof-of-concept for scaling automated evaluation of social understanding.

What carries the argument

The load-bearing instrument is the SOCIALCAPTION rubric: HSA and DSA outputs are each rated on six 1-5 Likert sub-dimensions—scene, individuals, topic/context, socio-emotional analysis, answer detail, and prompt adherence for HSA; relevant scene details, key individuals, relevant interactions, relevant context, plus the same two meta-criteria for DSA—giving a maximum score of 30. The sub-dimensions are grounded in the APRACE taxonomy of social interactions (actors, partners, relations, activities, context, evaluation). SI is measured separately by multiple-choice accuracy. Alignment between human and MLLM-judge ratings is computed by binarizing Likert scores at a threshold of ≥3 and reportin

What would settle it

Take a larger sample (100+ videos) and compute agreement on the original 5-point Likert scale (e.g., quadratic weighted kappa or average absolute difference) between each MLLM judge and human raters, then compare with human-human agreement. If InternVL3's strict ordinal agreement drops below human-human agreement, or if including lower-performing models as judges reverses the ranking between Gemini-2.5-Pro and InternVL3, the claim that open-source judges can scale human evaluation is not supported.

Watch

Extended reading notes

Core claim

The central claim is that social understanding should be measured along three separate axes, and that the two generative axes reveal competencies that QA accuracy hides. In the paper's experiments, Gemini-2.5-Pro leads all models on Social Inference but receives lower human scores than Gemini-1.5-Pro on holistic socio-emotional description; Qwen2.5-Omni is competent on inference yet produces the worst HSA and DSA generations of any evaluated model; and InternVL3-8B, an 8-billion-parameter open model, matches or beats several closed-source models on Directed Social Analysis. The paper also claims that MLLM judges can stand in for human raters: InternVL3-8B and InternVL3-78B show binary-F1 agr

Load-bearing premise

The load-bearing premise is that binary-F1 agreement at a hand-set threshold (Likert ≥3 = high quality) on a 20-video sample faithfully represents 'alignment with human scores'; if the threshold, the video sample, or the judge selection changed, the proof-of-concept for automated evaluation could weaken.

Editorial extensions

If this is right

  • Evaluation of social understanding should treat QA accuracy, holistic description, and directed relevance as distinct report cards; a single QA score can mis-rank models.
  • Providing spoken context (transcriptions) is a reliable lever for improving social inference across all tested models, with gains up to 13 accuracy points.
  • Scale is not the main driver of generative social understanding: 7-8B open models can match or exceed much larger closed models on HSA and DSA.
  • MLLM judges—especially InternVL3 variants—can serve as scalable evaluators and potential reward models for social-understanding training, provided their score inflation and self-preference are accounted for.
  • Architectural choices such as temporal video modeling versus treating video as extended images, and low-latency streaming designs, materially change generative social understanding.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the judge-alignment result is used to build reward models, the observed self-preference and absolute-score inflation are likely to distort preference learning; using relative rankings or calibrated thresholds may be necessary.
  • The alignment claim rests on only 20 videos and binary F1 at a threshold of 3; re-running on a larger sample with ordinal agreement metrics (e.g., weighted kappa on the 5-point scale) would test whether the proof-of-concept survives stricter measurement.
  • The HSA/DSA rubrics could be turned into training signal: a model fine-tuned to produce structured, rubric-scored social narratives might develop more usable social understanding than one trained solely on QA.
  • Because the human annotations are English-only and US-based, the framework's sub-dimensions may encode Western interaction norms; a cross-cultural stress test would show whether the dimensions are universal or culturally specific.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SOCIALCAPTION, a three-dimensional evaluation framework for multimodal social understanding: Social Inference (multiple-choice QA accuracy), Holistic Social Analysis, and Directed Social Analysis (open-ended generation scored with six Likert sub-dimensions). Using SOCIAL-IQ 2.0 validation videos and a range of open- and closed-source MLLMs, it reports SI accuracy with and without transcripts, human and MLLM-judge HSA/DSA scores, analyses of model scale/architecture/spoken-context effects, and judge-alignment results. The central positive claim is that MLLM judges—especially InternVL3-8B and InternVL3-78B—align strongly with human HSA/DSA ratings, thereby enabling scalable automated evaluation of social understanding.

Significance. If the judge-alignment result holds, the paper would provide a useful, theory-grounded complement to QA-only social benchmarks. The framework's HSA/DSA sub-dimensions are anchored in an external taxonomy (APRACE), the prompts are given in full, annotator agreement is reported transparently, and the mismatch test is a reasonable control against indiscriminate high scoring. The central claim is falsifiable. However, the evidence currently does not support the abstract's unqualified 'strong alignment' statement: the agreement metric is a binary F1 at a single lenient threshold, the human-judge comparison may not be matched to the human-human baseline, and the HSA/DSA evidence base is 20 videos with judge models selected from the top-SI performers. These issues are load-bearing for the scalability conclusion.

major comments (4)
  1. [Section 4.3 / Appendix E / Table 4] The headline 'strong alignment with human scores' rests on binary F1 at a single lenient threshold (Likert ≥3 = high). Appendix G shows all MLLM judges inflate absolute scores, so most real responses are 'high'; a high-labeling judge can achieve high F1 without human-like discrimination. Table 4 itself shows H-G F1 (86.10/81.99) below H-H (88.85/86.69), and Table 2 shows Gemini-2.5-Pro below H-H in 8 of 12 sub-dimensions (e.g., HSA Individuals 75.09 vs 84.24; DSA Relevant Interactions 75.37 vs 87.04; DSA Answer Detail 71.76 vs 85.06). Please qualify the claim, vary the threshold (e.g., ≥4, =5, ordinal/rank agreement), and report a matched single-annotator baseline rather than an averaged human score (Appendix G states human scores are averaged across annotators).
  2. [Section 3.2.4 / Appendix D / Table 1] All HSA/DSA and judge-agreement conclusions rest on 20 videos and 10 models (top-7 SI performers plus 3 closed-source models). The power analysis (dz≥0.66) addresses paired model-mean differences, not precision of per-subdimension F1; with 20 videos the subdimension differences in Table 2 have wide confidence intervals. Models with low generation quality are omitted from HSA/DSA (Table 1 footnote), which can bias agreement upward. Please provide bootstrap CIs, analyze or at least characterize excluded models, and release outputs/annotations so the agreement numbers are independently checkable.
  3. [Section 4.3 / Appendix G] The three judges were selected as the highest-SI models (Gemini-2.5-Pro, InternVL3-8B, InternVL3-78B). The scalability conclusion ('open-source models such as InternVL3 can serve as evaluators') is therefore based on a favorable subset. Report judge-alignment for additional MLLMs (e.g., Qwen2.5-VL or GPT-4o) or justify why SI ranking is a valid criterion for judge selection; absent that, the claim that 'MLLM judges' align with humans is overgeneralized.
  4. [Section 4.3 / Table 7] The paper's own ranking results reveal judge-human disagreement. Human annotators rank Gemini-1.5-Pro highest in HSA (24.95), while all three MLLM judges rank Gemini-2.5-Pro highest (29.95/30.00/29.90). Binary F1 at a per-item threshold cannot capture this kind of rank-order disagreement. The mismatch test (Appendix H) shows judges penalize unrelated pairs, but it does not show that their relative ordering of valid responses is human-like. Report rank correlation (Spearman/Kendall) between human and judge totals/sub-dimensions.
minor comments (5)
  1. [General] Typographical and formatting issues: 'Y oussouf' in the author list, 'V olume' in the Hoppler reference, 'LLaV A-NeXT' spacing, 'stucture' in Figure 10, and 'adher .' in Table 8 headers.
  2. [Table 1] The ↑/↓ arrows in the HSA/DSA columns are ambiguous: each model row has four judge columns (H, G, I8B, I78B), and it is unclear whether arrows mark the highest/lowest within each column, within each category, or across the whole row. Please define this in the caption.
  3. [Table 1 / Section 3.2.4] The footnote says '-' indicates models not included 'due to low generation quality,' while Section 3.2.4 says the top 7 standard-scale models were selected by SI performance. Clarify whether exclusion was based on SI ranking, generation quality, or both.
  4. [Appendix D] The text says the 20 videos were 'randomly selected to ensure representation of videos across contexts.' Random selection does not by itself ensure representation; if a stratified procedure was used, describe it. Figure 11 validates SI representativeness but not HSA/DSA representativeness.
  5. [General] No data or code availability statement is included. Given that the judge-alignment and HSA/DSA claims rely on human annotations and model outputs, releasing these artifacts (or providing a clear reason not to) would substantially strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: judge alignment is anchored to external human ratings and the framework taxonomy is external.

full rationale

The paper's central empirical claims are benchmark measurements, not derivations from assumed conclusions. HSA/DSA rubrics are grounded in APRACE (Hoppler et al., 2022), an external taxonomy derived from 5000+ open-ended interaction descriptions, and human ratings are collected from independent Prolific annotators. The judge-alignment result compares MLLM ratings to these external human ratings; even though judges receive the same instructions as humans and were selected after SI performance, this design can affect agreement but does not make agreement true by construction. The self-citations (Social Genome, Social-IQ 2.0, Morency 2010) are used for dataset continuity, task alignment, and research positioning, not to justify the paper's conclusions. SI is standard QA accuracy and is independently evaluated. Concerns about binary F1 threshold, human-judge vs human-human comparison, and the 20-video sample are validity/statistical issues, not circular reductions. No quoted equation, fitted parameter, or definitional identity was found that makes any reported 'prediction' equivalent to its inputs.

Assumptions & free parameters 4 free parameters · 6 assumptions · 2 invented entities

No model parameters are fitted in this evaluation paper; the central claims rest on hand-chosen evaluation design decisions. The binary high/low threshold, equal rubric weighting, 20-video sample, and 1 FPS sampling are the main hand-set choices that influence headline numbers. The framework also assumes that SOCIAL-IQ 2.0 labels, APRACE-derived sub-dimensions, crowd ratings, and binary-F1 agreement are valid operationalizations of social understanding.

free parameters (4)
  • Binary quality threshold (Likert ≥3 = high) = 3
    All human-human and human-judge agreement numbers (e.g., 88.85% HSA H-H) are computed after collapsing 1-5 ratings to binary labels at this threshold; no sensitivity analysis is reported (Appendix E).
  • Equal sub-dimension weighting in HSA/DSA totals = each of 6 criteria weighted equally, total /30
    The 30-point totals used for model rankings assume each of the six rubric items contributes equally; no justification or ablation for this weighting is provided (Section 3.1.2).
  • HSA/DSA evaluation sample size = 20 videos
    All generative and judge-alignment conclusions rest on 20 randomly sampled videos, chosen for 80% power at dz≈0.66; representativeness is supported only by SI accuracy matching, not by full-set HSA/DSA scores (Appendix D).
  • Video frame sampling rate = 1 FPS
    All models receive 1 frame per second; conclusions about temporal architectures (LongVA vs Qwen2-VL) are conditional on this sampling choice (Section 3.2.2).
assumptions (6)
  • domain assumption SOCIAL-IQ 2.0 questions and options are valid ground truth for social inference
    SI accuracy treats the dataset's multiple-choice labels as correct; used as the base benchmark (Section 3).
  • domain assumption The APRACE taxonomy is a valid decomposition of social interactions
    HSA and DSA sub-dimensions are derived from APRACE; the framework inherits APRACE's validity (Section 3.1.2).
  • domain assumption Crowd annotators' Likert ratings are a reliable measure of response quality
    Human ratings are treated as ground truth for HSA/DSA and for judge alignment (Section 3.2.4).
  • ad hoc to paper Binary mapping of Likert scores preserves the quality distinction relevant to agreement
    The authors choose ≥3 as high quality without sensitivity analysis; headline IAA and judge-alignment values depend on this choice (Appendix E).
  • domain assumption The 20-video subset is representative of the full SOCIAL-IQ 2.0 validation set for HSA/DSA
    Representativeness is checked only via SI accuracy on the subset vs full set; HSA/DSA full-set scores are unavailable (Appendix D, Figure 11).
  • domain assumption MLLM judges using the same instructions as humans can be compared to humans via binary F1
    Judge-vs-human agreement assumes scale usage differences and prompt-following are not confounds; the paper partially addresses this with the mismatch test but not with calibration (Section 4.3, Appendix H).
invented entities (2)
  • Holistic Social Analysis (HSA) dimension
    purpose: Measures whether a model's free-form scene description is comprehensive across six sub-dimensions (scene, individuals, topic/context, socio-emotional, detail, prompt adherence).
    A new evaluation construct derived from APRACE; validated only through internal consistency (IAA) and judge alignment, not against an independent behavioral outcome.
  • Directed Social Analysis (DSA) dimension
    purpose: Measures whether a model extracts relevant social information for a targeted question.
    A new evaluation construct; same internal-validation situation as HSA.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Social Caption: Evaluating Social Understanding in Multimodal Models." pith.science (2026). https://pith.science/paper/XSR2IGQP

@misc{pith2026260114569,
  author       = {Pith},
  title        = {Pith review of: Social Caption: Evaluating Social Understanding in Multimodal Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XSR2IGQP}},
  note         = {Machine review of arXiv:2601.14569}
}
read the original abstract

Social understanding abilities are crucial for multimodal large language models (MLLMs) to interpret human social interactions. We introduce SOCIAL CAPTION, a framework grounded in interaction theory to evaluate social understanding abilities of MLLMs along three dimensions: Social Inference (SI), the ability to make accurate inferences about interactions; Holistic Social Analysis (HSA), the ability to generate comprehensive descriptions of interactions; Directed Social Analysis (DSA), the ability to generate relevant information from interactions. We analyze factors influencing model performance in social understanding, such as scale, architectural design, and spoken context. Experiments with MLLM judges demonstrate a path towards scaling automated evaluation of multimodal social understanding.

Figures

Figures reproduced from arXiv: 2601.14569 by the authors.

Figure 1
Figure 1. SOCIAL CAPTION, a framework that eval￾uates social understanding of MLLMs, going beyond social inference to evaluate holistic social analysis and directed social analysis abilities. 2022; Kong et al., 2025) or measure the quality of model reasoning traces when making QA infer￾ences (Mathur et al., 2025). Beyond performing QA (selecting an answer from given options), so￾cial understanding involves generating holistic… view at source ↗
Figure 2
Figure 2. Social inference performance across model sizes, with and without spoken context. Higher scores indicate stronger social inference capabilities. grids during training (Zhang et al., 2024a); these findings indicate that LongVA’s visual processing architecture struggles to extract meaningful tempo￾ral social signals from videos. In contrast, Qwen2- VL builds upon the same backbone as LongVA and, instead, uses 3D convo… view at source ↗
Figure 3
Figure 3. HSA and DSA performance of models across their six evaluation sub-dimensions, visual￾ized as heat maps. Each cell shows the mean human evaluation score (1–5 scale) for a model–dimension pair, where higher values indicate stronger performance. Color intensity corresponds to score magnitude. InternVL3-8B performance did not significantly differ from the substantially larger closed-source models (padj ≥ 0.05); in addit… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Qualitative example of MiniCPM-V2.6 responses evaluated by MLLM judges. The top section shows a representative video frame, the question, and the SI responses from both MiniCPM-V2.6 and MLLM judges. The middle section presents the DSA response of MiniCPM-V2.6, includin…
Figure 5
Figure 5. Figure 5: Prompt template to evaluate the SI dimension without providing transcriptions. SI Prompts With Transcriptions For this vari￾ant, the model is provided with the video and spo￾ken content in the form of transcriptions with times￾tamps and speaker turns. This additional c…
Figure 6
Figure 6. Figure 6: Prompt template to evaluate the SI dimension with transcriptions [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Prompt template for HSA generations. The model generates a detailed description of the video using both visual content and spoken dialogue, guided by the structured elements specified in the prompt. B HSA and DSA Prompts B.1 Holistic Social Analysis (HSA) Generations T…
Figure 8
Figure 8. Figure 8: Instructions for Humans and MLLM Judges to evaluate model generations on [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Prompt template for DSA generations. The model generates a detailed description of relevant infor￾mation in the video, guided by the structured elements specified in the prompt [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Instructions for Humans and MLLM Judges to evaluate model generations on [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Model SI Performance on Sampled Set vs Full Dataset. Performance of MLLMs on the ran￾domly sampled subset of 20 videos and the full valida￾tion dataset [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    GRASP is a large-scale dataset and benchmark for social reasoning grounded in gaze and gesture events in multi-person videos, with Social Grounding Reward (SGR) proposed to improve model performance on GRASP-Bench.

  2. Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models

    cs.CV 2026-05 unverdicted novelty 5.0 of 10

    VLMs are evaluated on gaze following and social gaze prediction using existing datasets in zero-shot and fine-tuned settings, revealing they currently lack precise capabilities compared to visual models.

  3. Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models

    cs.CV 2026-05 unverdicted novelty 5.0 of 10

    EyeVLM benchmark finds that current VLMs underperform specialized visual models on gaze following and social gaze prediction, with fine-tuning narrowing but not closing the gap.

Reference graph

Works this paper leans on

27 extracted references · 6 linked inside Pith · cited by 2 Pith papers

  1. [2]

    -Facial Expressions:Describe any notable facial expressions or emotions displayed by the individual throughout the video

    Individuals: For each person in the video, provide the following details: -Appearance:Describe their physical appearance, including age, gender, ethnicity, clothing, and distinctive features. -Facial Expressions:Describe any notable facial expressions or emotions displayed by the individual throughout the video. -Actions:Describe any significant actions, ...

  2. [4]

    Include the following information in your description: Transcription:{Transcription}

    Rate the model’s response using the 6 criteria and 5-point scale provided # Original prompt to the vision-language model STARTS here # Please watch the provided video and provide a detailed description of the scene, focusing on the socio-emotional context, dialogue, and topic. Include the following information in your description: Transcription:{Transcription}

  3. [5]

    Answer Detail (1-5):Is the description detailed and nuanced, offering in-depth social and emotional insights compared to what you see in the video?

  4. [6]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin

    Auto-rubric: Learning to extract generaliz- able criteria for reward modeling.arXiv preprint arXiv:2510.17314. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025. Qwen2.5-omni technical report. Preprint, arXiv:2503.20215. An Yang, Baosong Y...

    arXiv 2025

  5. [7]

    w/o trans

    Videorewardbench: Comprehensive evalua- tion of multimodal reward models for video under- standing.arXiv preprint arXiv:2509.00484. Xuhui Zhou, Hao Zhu, Akhila Yerukola, Thomas David- son, Jena D. Hwang, Swabha Swayamdipta, and Maarten Sap. 2023. COBRA frames: Contextual reasoning about effects and harms of offensive state- ments. InFindings of the Associ...

    arXiv 2023

  6. [14]

    -Time of Day:Mention the time of day if it is evident from the video (e.g., morning, afternoon, evening)

    Scene Description: -Setting:Describe the location and environment where the video takes place. -Time of Day:Mention the time of day if it is evident from the video (e.g., morning, afternoon, evening). -Mood:Describe the overall mood or atmosphere of the scene (e.g. cheerful, tense, relaxed)

  7. [15]

    -Facial Expressions:Describe any notable facial expressions or emotions displayed by the individual throughout the video

    Individuals: For each person in the video, provide the following details: -Appearance:Describe their physical appearance, including age, gender, ethnicity, clothing, and distinctive features. -Facial Expressions:Describe any notable facial expressions or emotions displayed by the individual throughout the video. -Actions:Describe any significant actions, ...

  8. [16]

    Scene Description (1-5):Does the response provide a good description of the setting, time of day, and mood compared to what you see in the video?

Show all 27 references
  1. [17]

    Individuals (1-5):Does the response describe the appearance, facial expressions, and actions of the individuals accurately compared to what you see in the video?

  2. [18]

    Topic and Context (1-5):Does the response capture the main topic and provide relevant context, including relationship dynamics compared to what you see in the video?

  3. [19]

    Socio-Emotional Analysis (1-5):How well does the response analyze emotions, power dynamics, and conflicts or harmony in the scene compared to what you see in the video?

  4. [21]

    scene_description

    Adherence to the Prompt (1-5):Does the description follow the instructions and is the description following the answer structure provided in the prompt compared to what you see in the video? For each of these categories, rate from 1 (Poor) to 5 (Excellent), with a total possib...

  5. [22]

    First, watch the provided video carefully to understand the actual content, context, and details

  6. [23]

    Read the model’s response below

  7. [24]

    Compare the model’s description against what you actually observe in the video

  8. [25]

    In your description, focus on the elements of the video that are most relevant to answering the question

    Rate the model’s response using the 6 criteria and 5-point scale provided # Original prompt to the vision-language model STARTS here # Please watch the provided video and generate a description that helps answer the given question. In your description, focus on the elements of...

  9. [26]

    Relevant Scene Details (1-5):Does the answer accurately describe the setting, time of day, and any objects that are crucial for answering the question compared to what you see in the video?

  10. [27]

    Key Individuals (1-5):Does the answer provide detailed and relevant descriptions of the individuals’ appearance, actions, and gestures that are significant to the question compared to what you see in the video?

  11. [28]

    Relevant Interactions (1-5):Does the answer capture and analyze the interactions between individuals that are most pertinent to answering the question compared to what you see in the video?

  12. [29]

    Relevant Context (1-5):Does the answer include necessary background or contextual information from the video that contributes to answering the question compared to what you see in the video?

  13. [30]

    Answer Detail (1-5):Is the answer detailed and nuanced, offering in-depth social and emotional insights compared to what you see in the video?

  14. [31]

    relevant_scene_details

    Adherence to the Prompt (1-5):Does the answer follow the instructions and is the answer following the stucture provided in the prompt compared to what you see in the video? For each of these categories, rate from 1 (Poor) to 5 (Excellent), with a total possible score of 30 poi...

  15. [2014]

    Version: Current month(s) and year(s) of use

    Copyright 2024. Version: Current month(s) and year(s) of use. Location: London, UK. Shu Pu, Yaochen Wang, Dongping Chen, Yuhang Chen, Guohao Wang, Qi Qin, Zhongyi Zhang, Zhiyuan Zhang, Zetong Zhou, Shuang Gong, Yi Gui, Yao Wan, and Philip S. Yu. 2025. Judge anything: Mllm as a...

    2024 arXiv

  16. [2018]

    InProceedings of the 2018 con- ference on empirical methods in natural language processing, pages 1369–1379

    Tvqa: Localized, compositional video ques- tion answering. InProceedings of the 2018 con- ference on empirical methods in natural language processing, pages 1369–1379. Paul Pu Liang, Akshay Goindani, Talha Chafekar, Leena Mathur, Haofei Yu, Ruslan Salakhutdinov, and Louis- Phi...

    2018 arXiv

  17. [2022]

    Aaron Hurst, Adam Lerer, Adam P

    The six components of social interactions: Ac- tor, partner, relation, activities, context, and evalua- tion.Frontiers in Psychology, V olume 12 - 2021. Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, and et al. 2024. Gpt-4o system card. Preprint, arXiv:2410.21276. De...

    2021 arXiv

  18. [2024]

    Renjie Pi, Felix Bai, Qibin Chen, Simon Wang, Jiu- long Shan, Kieran Liu, and Meng Cao

    Llm evaluators recognize and favor their own generations.Advances in Neural Information Pro- cessing Systems, 37:68772–68802. Renjie Pi, Felix Bai, Qibin Chen, Simon Wang, Jiu- long Shan, Kieran Liu, and Meng Cao. 2025. Mr. judge: Multimodal reasoner as a judge.Preprint, arXiv...

    2025 arXiv

  19. [2025]

    Sture Holm

    Rubrics as rewards: Reinforcement learn- ing beyond verifiable domains.arXiv preprint arXiv:2507.17746. Sture Holm. 1979. A simple sequentially rejective mul- tiple test procedure.Scandinavian Journal of Statis- tics, 6:65–70. Sarah Susanna Hoppler, Robin Segerer, and Jana Nikitin

    1979 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.