Pith. sign in

REVIEW 4 major objections 6 minor 147 references

FATE: Frame-Level Audio-Visual Temporal Embedding

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FATE claims that frame-level embeddings, scored over physically aligned frame pairs, put semantic and temporal synchronization into one similarity measure.

desk verdict FATE's frame-level design and ablations are the real contribution; the 'physical timeline' claim needs a calibration check, but the core result is solid. read the letter →

arxiv 2608.01310 v1 pith:5ZGFS6OP submitted 2026-08-02 cs.MM

Kaisi Guan , Bingzi Zhang , Xihua Wang , Ying Ba , Xin Cheng , Yijing Chen , Ruihua Song This is my paper · ORCID
classification cs.MM
keywords audio-visualembeddingframe-levelrepresentationtemporalsynchronizationcontrastivelearningcross-modalretrievaleventlocalizationgenerationevaluationsoftlabels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FATE aims to establish that one embedding space can carry both what a sound is and when it happens, without needing a task-specific prediction head. Instead of pooling each modality into a single vector, FATE keeps the full sequence of frame-level tokens, aligns video tokens to the audio timeline, and defines cross-modal similarity as the average inner product of corresponding frame pairs. That single score then does double duty: it identifies the semantically correct clip and also peaks only at zero temporal offset. The paper reports that this design outperforms both pooling-based embedding models and dedicated synchronization models on temporal retrieval, matches supervised methods on event localization without seeing that dataset's labels, and agrees better with human ratings than existing generation metrics.

What carries the argument

The load-bearing mechanism is the frame-level diagonal similarity with nearest-neighbor temporal alignment. After discarding the global pooling head, the video encoder outputs $T_v$ tokens per clip and the audio encoder outputs $T_a$ tokens; the paper uses audio as the temporal anchor and downsamples video by nearest-neighbor selection to length $T_a$. Cross-modal similarity is then the mean diagonal of the pairwise inner-product matrix, $S_{AV} = \frac{1}{T_a}\sum_i (\tilde{F}_V^{(i)})^\top F_A^{(i)}$. This diagonal trace enforces temporal order by construction: a shift moves true correspondences off the diagonal, so the score decays with offset. The second piece is the training objective, which combines semantic InfoNCE across videos with a raised-cosine soft label over candidate offsets within a video, shaping a similarity landscape that peaks at zero offset and rolls off smoothly.

What would settle it

Take a controlled recording of a single percussive event with known audio and video timestamps, encode it with FATE, and shift the audio by known offsets before computing similarity. If the similarity peak appears at the true recorded offset rather than at the coordinate-defined zero, the alignment is physical; if it peaks at zero regardless of the true offset, Eq. (3) is measuring coordinate alignment rather than real-world simultaneity.

Watch

Extended reading notes

Core claim

FATE's central claim is that retaining frame-level embeddings and scoring similarity over temporally aligned frame pairs encodes semantic and temporal discrimination in one number. Concretely, given aligned visual features $\tilde{F}_V$ and audio features $F_A$, similarity is $S_{AV} = \frac{1}{T_a}\sum_i (\tilde{F}_V^{(i)})^\top F_A^{(i)}$. A temporally shifted audio clip lowers $S_{AV}$ because the $i$-th video token no longer corresponds to the same physical moment, while a semantically different clip lowers $S_{AV}$ because the frame features do not match in content. Trained with a symmetric cross-video contrastive loss plus a within-video soft contrastive loss whose labels decay with absolute offset, FATE produces a similarity that peaks sharply at zero offset. The paper's empirical case is that this one representation surpasses the strongest baselines on temporal and semantic retrieval, matches fully supervised methods on zero-shot event localization, and is the best automatic proxy for human synchronization judgments among the metrics compared.

Load-bearing premise

The alignment step assumes that after nearest-neighbor downsampling, the $i$-th video token and the $i$-th audio token refer to the same physical instant for every $i$; if the two encoders' time origins or token centers are offset, the 'aligned' pairs are not simultaneous and the reported temporal discrimination would be an artifact of the coordinate mapping.

Editorial extensions

If this is right

  • Temporal retrieval can be done with a reusable embedding: FATE reports gains of more than 13 points in R@3 over the strongest synchronization baseline while simultaneously raising semantic retrieval accuracy by over 20 points.
  • Zero-shot event localization reaches supervised-level accuracy: on the AVE dataset, FATE's 48.3% average exceeds the strongest supervised baseline's 47.8%, without using any AVE training labels.
  • Frame-level structure, not the encoder or training data alone, drives the gain: the controlled ablation that pools tokens instead of keeping frames loses roughly half of retrieval performance with everything else held fixed.
  • As an automatic evaluation metric for generated audio-video, frame-level synchronization scoring correlates better with human ratings (sample-level $\rho=17.24$, model-ranking $\rho=44.41$) than offset predictors or pooling-based similarity.
  • Semantic and temporal capabilities are complementary rather than in tension: ablations show each loss alone underperforms the joint objective, and the combined gain is super-additive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the physical-timeline alignment is valid, the same diagonal-similarity construction could turn any pair of encoders with known token rates into a synchronization scorer, with no prediction head needed.
  • The soft-label kernel is anchored to a 0.5-second grid, so a finer test would probe whether FATE's similarity peak tracks human asynchrony thresholds below half a second, where the paper's chosen 1-second support begins to roll off.
  • Because FATE is initialized from a pretrained audio-visual encoder and trained on VGGSound, a transfer experiment to out-of-domain videos with independent ground-truth timestamps would test whether the learned alignment generalizes beyond the training distribution.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes FATE, a dual-encoder audio-visual model that keeps frame-level token sequences instead of pooling to a global vector. It aligns the higher-rate video tokens to the audio token grid via nearest-neighbor downsampling (Eq. 2) and defines cross-modal similarity as the mean inner product over the aligned diagonal (Eq. 3). Training combines a cross-video semantic InfoNCE loss with a within-video temporal soft-contrastive loss using a raised-cosine target (Eqs. 4-7). The authors evaluate FATE on temporal retrieval (AVSync15, VGG-Sync), zero-shot event localization (AVE), and as a generation metric correlated with human ratings, reporting large gains over pooling-based embeddings and synchronization models, with ablations isolating the frame-level design, loss components, label softness, and alignment strategy.

Significance. If the results hold, FATE is a simple and potentially influential design: it shows that a general-purpose embedding space can carry synchronization without a dedicated offset prediction head, and the controlled 'w/o Frame-level' ablation, the loss-component ablation, and the code release are concrete strengths. The temporal discrimination is explicitly built into Eq. (3) and Eq. (6), so the sharp peak in Fig. 3 is a property of the design rather than an emergent discovery; the conclusion's 'emerge' wording should be softened. The central claims are not yet fully supported because the physical-timeline alignment in Eq. (2) is uncalibrated, the headline comparisons lack variance estimates, and the human-annotation protocol is described inconsistently.

major comments (4)
  1. [Method, 'Temporal alignment' (Eq. 2)] The mapping phi(i)=min(round(i*Tv/Ta),Tv) assumes that the i-th video token and the i-th audio token are uniformly spaced from a common origin, with token rates exactly Tv/T and Ta/T for the same clip duration. The paper reports neither token timestamps nor receptive-field centers for the PE-AV encoders, so the 'aligned' pairs in Eq. (3) may be offset by a constant physical delay. Because the temporal loss in Eq. (7) labels candidates by index shift in the same coordinate system, training is blind to a constant bias, and the sensitivity curve in Fig. 3 cannot reveal it. This does not invalidate the relative retrieval results, but it means the central claim that synchronization is encoded 'on the physical timeline' is unverified; please measure or calibrate the token timestamps, or at least analyze sensitivity to a constant offset in phi.
  2. [Joint Audio-Video Generation Evaluation / Appendix] The main text states that 'ten independent annotators rate the temporal synchronization of each video on a 1-5 scale, and we average their ratings into a Mean Opinion Score (MOS),' while the supplementary 'Human Annotation Protocol' describes a different procedure: annotators assign a full ranking from 1 (best) to 5 (worst) with no ties, the rankings are averaged into a mean rank, and the rank is negated before Spearman correlation. These are incompatible protocols, and Table 3 depends on which one was actually used. Please state the exact score used in Eq. (10), report inter-annotator agreement (e.g., Krippendorff's alpha), and make the main-text description consistent with the appendix.
  3. [Abstract / Table 2] The abstract's 'surpasses supervised methods' claim rests on a 0.5-point average-accuracy margin over DAM (48.3% vs. 47.8% in Table 2), while FATE is actually worse on V2A (44.7% vs. 47.1%). With single runs and no significance tests, this margin cannot support 'surpasses'; please report variance over seeds or a paired test and, in the meantime, describe the result as matching or being on par with supervised methods.
  4. [Experimental Results / Tables 1-3] All tables report a single run without error bars, confidence intervals, or significance tests. This is particularly limiting for Table 3, where the best sample-level correlation is 17.24 and the model-ranking correlation is computed over only five generation models; the paper acknowledges the latter limitation but not the absence of variance. Please report at least three seeds for the main experiments and a significance test for the comparisons that underlie the headline claims.
minor comments (6)
  1. [Experimental Results, Fig. 3] In Fig. 3, the shaded region is labeled ±1 std, but the text does not state whether this is across videos or across runs; please specify.
  2. [Training Strategy, temporal loss] The phrase 'yields K=17 candidate audio clips' from a 10s video with a 2s window and 0.5s stride should show the arithmetic (10-2)/0.5+1.
  3. [Training Strategy, Eq. (4)] The loss weights in Eq. (4) are λ1=0.5 and λ2=1.0; a sentence justifying the 2:1 ratio or reporting sensitivity to it would help.
  4. [Introduction / Appendix] The abstract and introduction both mention Omni LLMs as baselines, but the quantitative MLLM comparison is only in the appendix and is not summarized in a main-text table; please either add the numbers or move the claim to the appendix.
  5. [Table 1 caption] The table caption for Table 1 includes the ablation row 'w/o Frame-level' among the methods; please mark it explicitly as a controlled ablation rather than a baseline.
  6. [Eq. (6) and psychophysical threshold] Eq. (6) cites Vatakis and Spence (2006) for the 0.5 s half-maximum point; please give the specific threshold reported there and note that the threshold may differ across stimulus types.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FATE's frame-level similarity and temporal soft-contrastive objective are openly specified, and the central claims are validated on held-out benchmarks.

full rationale

FATE derives its temporal discrimination from an explicit design: Eq. (3) computes similarity as the mean diagonal after index-based nearest-neighbor alignment (Eq. 2), and Eq. (7) trains the model to match soft offset labels (Eq. 6). The paper states this transparently: 'This built-in penalty is what FATE exploits for synchronization-aware retrieval and evaluation, without any task-specific prediction head.' The sharp peak in Fig. 3 is therefore an expected property of the trained model, not a hidden input renamed as a discovery. None of the paper's predictive claims reduces to a fitted parameter: temporal retrieval is evaluated on held-out AVSync15 and VGG-Sync test splits against official baseline checkpoints; event localization on AVE is zero-shot with no AVE supervision; and the generation-metric correlation in Table 3 uses human MOS not seen during training. The only substantive concern is whether Eq. (2)'s index-proportional mapping equals physical-time alignment—a correctness or calibration risk, not circularity—because both the training labels and the similarity function operate in the same index coordinate system. No load-bearing self-citation or author-imported uniqueness appears: PE-AV, LoRA, and VGGSound are external resources, and the authors' own prior works are cited only as generation models or related evaluation tools.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

FATE introduces no new physical entities. Its central claim rests on the token-index-to-time assumption, the correctness of VGGSound labels, and the chosen temporal kernel and loss weights. These are reasonable engineering assumptions but are not independently grounded beyond the paper's experiments.

free parameters (4)
  • Temporal kernel width T0 = 1s
    Hand-set in Eq. 6 to shape the soft temporal labels; the half-maximum point 0.5s is matched to a psychophysical threshold, but the exact kernel shape and width are chosen, not derived. Ablation Table 4(i) shows performance varies with width.
  • Loss weights lambda1, lambda2 = 0.5, 1.0
    Weights in Eq. 4 were chosen without a reported search; the temporal loss is weighted twice the semantic loss.
  • Training clip segmentation (window, stride) = 2s window, 0.5s stride
    Determines the K=17 candidate offsets in Eq. 6 and the graded label structure; an evaluation and training design choice that affects temporal supervision granularity.
  • InfoNCE temperature tau = 0.07
    Standard contrastive temperature but still a hand-set hyperparameter affecting similarity sharpness.
assumptions (4)
  • domain assumption Token index in the video encoder and audio encoder corresponds linearly to physical time, so Eq. (2)'s nearest-neighbor mapping aligns the two modalities on a common timeline.
    Invoked in the Temporal alignment paragraph and Eq. (2). If the encoders have different temporal offsets or nonlinear time coverage, the 'aligned' pairs in Eq. (3) are not physically simultaneous.
  • domain assumption VGGSound clips are correctly synchronized and provide trustworthy positive pairs for contrastive learning.
    Training uses the VGGSound training split as ground truth for both semantic and temporal alignment.
  • ad hoc to paper The raised-cosine kernel in Eq. (6) is a valid target distribution for perceived synchronization.
    The shape is motivated by psychophysics but is not derived from a perception model; Table 4(i) shows the choice matters.
  • domain assumption Human judgments used as the ground truth metric correlate monotonically with true synchronization quality.
    The generation evaluation in Eq. (10) treats human MOS (or mean rank) as the gold standard; the appendix protocol uses relative ranking, which may not be comparable across prompts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FATE: Frame-Level Audio-Visual Temporal Embedding." pith.science (2026). https://pith.science/paper/5ZGFS6OP

@misc{pith2026260801310,
  author       = {Pith},
  title        = {Pith review of: FATE: Frame-Level Audio-Visual Temporal Embedding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ZGFS6OP}},
  note         = {Machine review of arXiv:2608.01310}
}
read the original abstract

When a dog opens its mouth and barks, humans naturally recognize what the sound is and when it occurs. Building audio-visual models with this same ability requires representations that capture both semantic and temporal alignment. Current approaches fall short on one side or the other: embedding models match semantic but lose temporal information; synchronization models capture temporal offsets but lack semantic understanding. To bridge this gap, we propose FATE, Frame-level Audio-visual Temporal Embedding. Unlike prior embedding models that pool each modality into a single embedding and discard temporal information, FATE retains frame-level sequences, aligns them on the physical timeline, and computes similarity over strictly aligned frame pairs. Unlike synchronization models that output only an offset prediction, FATE encodes synchronization in a reusable embedding space, trained with a joint objective combining cross-video semantic and within-video temporal contrastive learning to capture both what sounds and when it occurs. Across three tasks, FATE surpasses the strongest baseline on temporal and semantic retrieval by a large margin, matches fully supervised methods on event localization in a zero-shot setting, and achieves the best correlation with human judgments as a generation evaluation metric. The source code can be found at \texttt{https://github.com/guankaisi/FATE}.

Figures

Figures reproduced from arXiv: 2608.01310 by the authors.

Figure 1
Figure 1. Semantic (what) vs. temporal (when) discrimina [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of FATE. (a) Architecture: Conventional audio-visual models compress encoder outputs into single global [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Similarity score vs. temporal offset. PE-AV’s global [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative visualization of FATE as a generation evaluation metric. For each case, we compare outputs from two [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

147 extracted references · 26 canonical work pages

  1. [1]

    Abril and Robert Plant

    Patricia S. Abril and Robert Plant. The patent holder's dilemma: Buy, sell, or troll?. Communications of the ACM. doi:10.1145/1188913.1188915

    arXiv

  2. [2]

    Deciding equivalances among conjunctive aggregate queries

    Sarah Cohen and Werner Nutt and Yehoshua Sagic. Deciding equivalances among conjunctive aggregate queries. doi:10.1145/1219092.1219093

    arXiv

  3. [3]

    Special issue: Digital Libraries. 1996

    1996

  4. [4]

    Understanding Policy-Based Networking

    David Kosiur. Understanding Policy-Based Networking

  5. [7]

    doi:10.1007/3-540-09237-4

    The title of book two. doi:10.1007/3-540-09237-4

    doi:10.1007/3-540-09237-4

  6. [8]

    Asad Z. Spector. Achieving application requirements. Distributed Systems. doi:10.1145/90417.90738

    arXiv

  7. [9]

    Douglass and David Harel and Mark B

    Bruce P. Douglass and David Harel and Mark B. Trakhtenbrot. Statecarts in use: structured analysis and object-orientation. Lectures on Embedded Systems. doi:10.1007/3-540-65193-4_29

    doi:10.1007/3-540-65193-4_29

  8. [10]

    Donald E. Knuth. The Art of Computer Programming, Vol. 1: Fundamental Algorithms (3rd. ed.)

Show all 147 references
  1. [11]

    Donald E. Knuth. The Art of Computer Programming

  2. [12]

    Structured Variational Inference Procedures and their Realizations (as incol)

    Dan Geiger and Christopher Meek. Structured Variational Inference Procedures and their Realizations (as incol). Proceedings of Tenth International Workshop on Artificial Intelligence and Statistics, The Barbados

  3. [13]

    Stan W. Smith. An experiment in bibliographic mark-up: Parsing metadata for XML export. Proceedings of the 3rd. annual workshop on Librarians and Computers

  4. [14]

    Catch me, if you can: Evading network signatures with web-based polymorphic worms

    Matthew Van Gundy and Davide Balzarotti and Giovanni Vigna. Catch me, if you can: Evading network signatures with web-based polymorphic worms. Proceedings of the first USENIX workshop on Offensive Technologies

  5. [15]

    Predicate Path expressions

    Sten Andler. Predicate Path expressions. Proceedings of the 6th. ACM SIGACT-SIGPLAN symposium on Principles of Programming Languages. doi:10.1145/567752.567774

  6. [16]

    LOGICS of Programs: AXIOMATICS and DESCRIPTIVE POWER

    David Harel. LOGICS of Programs: AXIOMATICS and DESCRIPTIVE POWER

  7. [17]

    Anisi , title =

    David A. Anisi , title =

  8. [18]

    Clarkson

    Kenneth L. Clarkson. Algorithms for Closest-Point Problems (Computational Geometry)

  9. [19]

    Introduction to Bayesian Statistics

    Harry Thornburg. Introduction to Bayesian Statistics. 2001

    2001

  10. [20]

    CLIFFORD: a Maple 11 Package for Clifford Algebra Computations, version 11

    Rafal Ablamowicz and Bertfried Fauser. CLIFFORD: a Maple 11 Package for Clifford Algebra Computations, version 11. 2007

    2007

  11. [21]

    Stats and Analysis

    Poker-Edge.Com. Stats and Analysis. 2006

    2006

  12. [22]

    A more perfect union

    Barack Obama. A more perfect union

  13. [23]

    The fountain of youth

    Joseph Scientist. The fountain of youth

  14. [24]

    Solder man

    Dave Novak. Solder man. ACM SIGGRAPH 2003 Video Review on Animation theater Program: Part I - Vol. 145 (July 27--27, 2003). doi:10.945/woot07-S422

    2003

  15. [25]

    Interview with Bill Kinder: January 13, 2005

    Newton Lee. Interview with Bill Kinder: January 13, 2005. Comput. Entertain. doi:10.1145/1057270.1057278

    2005

  16. [26]

    The Enabling of Digital Libraries

    Bernard Rous. The Enabling of Digital Libraries. Digital Libraries

  17. [28]

    (new) Finding minimum congestion spanning trees , journal =

    Werneck, Renato and Setubal, Jo\. (new) Finding minimum congestion spanning trees , journal =. doi:10.1145/351827.384253 , acmid = 384253, publisher =

  18. [30]

    and Mei, Alessandro , title =

    Conti, Mauro and Di Pietro, Roberto and Mancini, Luigi V. and Mei, Alessandro , title =. Inf. Fusion , volume =. 2009 , issn =. doi:10.1016/j.inffus.2009.01.002 , acmid =

    2009 doi

  19. [31]

    and Hutchful, David K

    Li, Cheng-Lun and Buyuktur, Ayse G. and Hutchful, David K. and Sant, Natasha B. and Nainwal, Satyendra K. , title =. CHI '08 extended abstracts on Human factors in computing systems , year =. doi:10.1145/1358628.1358946 , acmid =

  20. [32]

    , title =

    Hollis, Billy S. , title =. 1999 , isbn =

    1999

  21. [33]

    Goossens, Michel and Rahtz, S. P. and Moore, Ross and Sutor, Robert S. , title =. 1999 , isbn =

    1999

  22. [34]

    and Rosenberg, Arnold L

    Buss, Jonathan F. and Rosenberg, Arnold L. and Knott, Judson D. , title =. 1987 , source =

    1987

  23. [35]

    CHI '08: CHI '08 extended abstracts on Human factors in computing systems , year =

    , note =. CHI '08: CHI '08 extended abstracts on Human factors in computing systems , year =

  24. [36]

    Algorithms for Closest-Point Problems (Computational Geometry) , year =

    Clarkson, Kenneth Lee , advisor =. Algorithms for Closest-Point Problems (Computational Geometry) , year =

  25. [37]

    SIGCOMM Comput. Commun. Rev. , year =

  26. [38]

    2004 , isbn =

    IEEE TCSC Executive Committee , booktitle =. 2004 , isbn =. doi:http://dx.doi.org/10.1109/ICWS.2004.64 , acmid =

    2004 doi

  27. [39]

    Distributed systems (2nd Ed.) , year =

  28. [40]

    , title =

    Petrie, Charles J. , title =. 1986 , source =

    1986

  29. [41]

    Donald E. Knuth. Seminumerical Algorithms. 1981

    1981

  30. [42]

    E-commerce and cultural values , year =

    Kong, Wei-Chang , Title =. E-commerce and cultural values , year =

  31. [43]

    E-commerce and cultural values , year =

    Kong, Wei-Chang , type =. E-commerce and cultural values , year =

  32. [44]

    Chapter 9 , booktitle =

    Kong, Wei-Chang , editor =. Chapter 9 , booktitle =

  33. [45]

    E-commerce and cultural values , editor =

    Kong, Wei-Chang , title =. E-commerce and cultural values , editor =. 2003 , isbn =

    2003

  34. [46]

    E-commerce and cultural values - (InBook-num-in-chap) , chapter =

    Kong, Wei-Chang , editor =. E-commerce and cultural values - (InBook-num-in-chap) , chapter =. 2004 , address =

    2004

  35. [47]

    E-commerce and cultural values (Inbook-text-in-chap) , chapter =

    Kong, Wei-Chang , editor =. E-commerce and cultural values (Inbook-text-in-chap) , chapter =. 2005 , address =

    2005

  36. [48]

    E-commerce and cultural values (Inbook-num chap) , chapter =

    Kong, Wei-Chang , editor =. E-commerce and cultural values (Inbook-num chap) , chapter =. 2006 , address =

    2006

  37. [49]

    Microelectron

    Mehdi Saeedi and Morteza Saheb Zamani and Mehdi Sedighi , title =. Microelectron. J. , volume =. 2010 , pages =

    2010

  38. [50]

    Mehdi Saeedi and Morteza Saheb Zamani and Mehdi Sedighi and Zahra Sasanian , title =. J. Emerg. Technol. Comput. Syst. , volume =

  39. [51]

    Kirschmer, Markus and Voight, John , title =. SIAM J. Comput. , issue_date =. 2010 , issn =. doi:https://doi.org/10.1137/080734467 , acmid =

    2010 doi

  40. [52]

    Hoare, C. A. R. , title =. Structured programming (incoll) , editor =. 1972 , isbn =

    1972

  41. [53]

    History of programming languages I (incoll) , editor =

    Lee, Jan , title =. History of programming languages I (incoll) , editor =. 1981 , isbn =. doi:http://doi.acm.org/10.1145/800025.1198348 , acmid =

    1981

  42. [54]

    , title =

    Dijkstra, E. , title =. Classics in software engineering (incoll) , year =

  43. [55]

    , title =

    Wenzel, Elizabeth M. , title =. Multimedia interface design (incoll) , year =. doi:10.1145/146022.146089 , acmid =

  44. [56]

    , title =

    Mumford, E. , title =. Critical issues in information systems research (incoll) , year =

  45. [57]

    and Golden, Donald G

    McCracken, Daniel D. and Golden, Donald G. , title =. 1990 , isbn =

    1990

  46. [58]

    The analysis of linear partial differential operators

    H. The analysis of linear partial differential operators. 1985 , PAGES =

    1985

  47. [59]

    IEEE", address =

    A. Adya and P. Bahl and J. Padhye and A.Wolman and L. Zhou , title =. Proceedings of the IEEE 1st International Conference on Broadnets Networks (BroadNets'04) , publisher = "IEEE", address = "Los Alamitos, CA", year =

  48. [60]

    I. F. Akyildiz and W. Su and Y. Sankarasubramaniam and E. Cayirci , title =. Comm. ACM , volume = 38, number = "4", year =

  49. [61]

    I. F. Akyildiz and T. Melodia and K. R. Chowdhury , title =. Computer Netw. , volume = 51, number = "4", year =

  50. [62]

    ACM", address =

    P. Bahl and R. Chancre and J. Dungeon , title =. Proceeding of the 10th International Conference on Mobile Computing and Networking (MobiCom'04) , publisher = "ACM", address = "New York, NY", year =

  51. [63]

    8 (Special Issue on Sensor Networks)

    D. Culler and D. Estrin and M. Srivastava , title =. IEEE Comput. , volume = 37, number = "8 (Special Issue on Sensor Networks)", publisher = "IEEE", address = "Los Alamitos, CA", year =

  52. [64]

    Natarajan and M

    A. Natarajan and M. Motani and B. de Silva and K. Yap and K. C. Chua , title =. Network Architectures , editor =. 960935712

  53. [65]

    Tzamaloukas and J

    A. Tzamaloukas and J. J. Garcia-Luna-Aceves , title =

  54. [66]

    Zhou and J

    G. Zhou and J. Lu and C.-Y. Wan and M. D. Yarvis and J. A. Stankovic , title =

  55. [67]

    Mapping Powerlists onto Hypercubes

    Jacob Kornerup. Mapping Powerlists onto Hypercubes. 1994

    1994

  56. [68]

    Automatic Parallelization for Distributed-Memory Multiprocessing Systems

    Michael Gerndt. Automatic Parallelization for Distributed-Memory Multiprocessing Systems

  57. [69]

    J. E. Archer, Jr. and R. Conway and F. B. Schneider. User recovery and reversal in interactive systems. ACM Trans. Program. Lang. Syst

  58. [70]

    D. D. Dunlop and V. R. Basili. Generalizing specifications for uniformly implemented loops. ACM Trans. Program. Lang. Syst

  59. [71]

    Heering and P

    J. Heering and P. Klint. Towards monolingual programming environments. ACM Trans. Program. Lang. Syst

  60. [72]

    Donald E. Knuth. The book

  61. [73]

    Korach and D

    E. Korach and D. Rotem and N. Santoro. Distributed algorithms for finding centers and medians in networks. ACM Trans. Program. Lang. Syst

  62. [74]

    : A Document Preparation System

    Leslie Lamport. : A Document Preparation System

  63. [75]

    F. Nielson. Program transformations in a denotational setting. ACM Trans. Program. Lang. Syst

  64. [76]

    Brian K. Reid. A high-level approach to computer document formatting. Proceedings of the 7th Annual Symposium on Principles of Programming Languages

  65. [77]

    and Abdelzaher, Tarek F

    Zhou, Gang and Wu, Yafeng and Yan, Ting and He, Tian and Huang, Chengdu and Stankovic, John A. and Abdelzaher, Tarek F. , title =. ACM Trans. Embed. Comput. Syst. , issue_date =. doi:10.1145/1721695.1721705 , acmid = 1721705, publisher =

  66. [78]

    Institutional members of the Users Group

  67. [79]

    Boris Veytsman , title =

  68. [80]

    Robin Schneider , title =

  69. [81]

    and Peterson, Larry L

    Bowman, Mic and Debray, Saumya K. and Peterson, Larry L. , title =. ACM Trans. Program. Lang. Syst. , volume =. 1993 , doi =

    1993

  70. [82]

    TUGboat , volume =

    Braams, Johannes , title =. TUGboat , volume =

  71. [83]

    Post Congress Tristesse

    Malcolm Clark. Post Congress Tristesse. TeX90 Conference Proceedings

  72. [84]

    ACM Trans

    Herlihy, Maurice , title =. ACM Trans. Program. Lang. Syst. , volume =. 1993 , doi =

    1993

  73. [85]

    Salas and Einar Hille

    S.L. Salas and Einar Hille. Calculus: One and Several Variable. 1978

    1978

  74. [86]

    Publication quality tables in

    Simon Fear , month =. Publication quality tables in

  75. [87]

    Using the amsthm Package , organization =

  76. [88]

    2019 , url =

    R: A Language and Environment for Statistical Computing , author =. 2019 , url =

    2019

  77. [89]

    Sam Anzaroot and Andrew McCallum , title =

  78. [90]

    Brad and Haunschild, Robin , title =

    Bornmann, Lutz and Wray, K. Brad and Haunschild, Robin , title =

  79. [91]

    2014 , archivePrefix =

    Sam Anzaroot and Alexandre Passos and David Belanger and Andrew McCallum , title =. 2014 , archivePrefix =. 1403.1349 , doi =

    2014 arXiv

  80. [92]

    Proceedings of the 20th International Colloquium on Automata, Languages and Programming , series =

    Maintaining Discrete Probability Distributions Optimally , author =. Proceedings of the 20th International Colloquium on Automata, Languages and Programming , series =

  81. [93]

    Reiser , year = 2014, title =

    Brian J. Reiser , year = 2014, title =

    2014

  82. [94]

    Baggett and M

    R. Baggett and M. Simecek and C. Chambellan and K. Tsui and M. Fraune , year = 2025, title =

    2025

  83. [95]

    The Computational Geometry Algorithms Library , author =

  84. [96]

    Menelaos Karavelas , subtitle =

  85. [97]

    The Computational Geometry Algorithms Library , subtitle =

    Menelaos Karavelas , editor =. The Computational Geometry Algorithms Library , subtitle =

  86. [98]

    The Parmap library , author =

  87. [99]

    Christopher Anderson and Sophia Drossopoulou , title =

  88. [100]

    International conference on machine learning , pages=

    Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  89. [101]

    2021 , eprint=

    AudioCLIP: Extending CLIP to Image, Text and Audio , author=. 2021 , eprint=

    2021

  90. [102]

    CVPR , year=

    ImageBind: One Embedding Space To Bind Them All , author=. CVPR , year=

  91. [103]

    2023 , eprint=

    LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment , author=. 2023 , eprint=

    2023

  92. [104]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Cav-mae sync: Improving contrastive audio-visual mask autoencoders via fine-grained alignment , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  93. [105]

    arXiv 2023 , author=

    Contrastive Audio-Visual Masked Autoencoder. arXiv 2023 , author=. arXiv preprint arXiv:2210.07839 , year=

    2023 arXiv

  94. [106]

    Advances in Neural Information Processing Systems , volume=

    Mavil: Masked audio-video learners , author=. Advances in Neural Information Processing Systems , volume=

  95. [107]

    ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=

    Synchformer: Efficient synchronization from sparse cues , author=. ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=. 2024 , organization=

    2024

  96. [108]

    arXiv preprint arXiv:2512.19687 , year=

    Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning , author=. arXiv preprint arXiv:2512.19687 , year=

  97. [109]

    Advances in Neural Information Processing Systems , volume=

    Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models , author=. Advances in Neural Information Processing Systems , volume=

  98. [110]

    European Conference on Computer Vision , pages=

    Perceptual evaluation of audio-visual synchrony grounded in viewers’ opinion scores , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  99. [111]

    arXiv preprint arXiv:2210.07055 , year=

    Sparse in space and time: Audio-visual synchronisation with trainable selectors , author=. arXiv preprint arXiv:2210.07055 , year=

    arXiv

  100. [112]

    Computer Vision -- ACCV 2016 Workshops , pages=

    Out of Time: Automated Lip Sync in the Wild , author=. Computer Vision -- ACCV 2016 Workshops , pages=. 2017 , organization=

    2016

  101. [113]

    Communications of the ACM , volume=

    ImageNet classification with deep convolutional neural networks , author=. Communications of the ACM , volume=. 2017 , publisher=

    2017

  102. [114]

    2021 , eprint=

    LoRA: Low-Rank Adaptation of Large Language Models , author=. 2021 , eprint=

    2021

  103. [115]

    VGGSound: A Large-scale Audio-Visual Dataset

    Honglie Chen and Weidi Xie and Andrea Vedaldi and Andrew Zisserman. VGGSound: A Large-scale Audio-Visual Dataset. International Conference on Acoustics, Speech, and Signal Processing (ICASSP). 2020

    2020

  104. [116]

    Proceedings of the European conference on computer vision (ECCV) , pages=

    Audio-visual event localization in unconstrained videos , author=. Proceedings of the European conference on computer vision (ECCV) , pages=

  105. [117]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Dual attention matching for audio-visual event localization , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  106. [118]

    Proceedings of the 30th International Conference on Machine Learning , pages =

    Deep Canonical Correlation Analysis , author =. Proceedings of the 30th International Conference on Machine Learning , pages =. 2013 , editor =

    2013

  107. [119]

    2023 , eprint=

    Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation , author=. 2023 , eprint=

    2023

  108. [120]

    arXiv preprint arXiv:2510.01284 , year=

    Ovi: Twin backbone cross-modal fusion for audio-video generation , author=. arXiv preprint arXiv:2510.01284 , year=

    arXiv

  109. [121]

    2025 , eprint=

    Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction , author=. 2025 , eprint=

    2025

  110. [122]

    arXiv preprint arXiv:2601.03233 , year=

    LTX-2: Efficient Joint Audio-Visual Foundation Model , author=. arXiv preprint arXiv:2601.03233 , year=

    arXiv

  111. [123]

    arXiv preprint arXiv:2503.23377 , year=

    Javisdit: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchronization , author=. arXiv preprint arXiv:2503.23377 , year=

  112. [124]

    Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

    Animate and sound an image , author=. Proceedings of the Computer Vision and Pattern Recognition Conference , pages=

  113. [125]

    International Conference on Learning Representations (ICLR) , year=

    LoRA: Low-Rank Adaptation of Large Language Models , author=. International Conference on Learning Representations (ICLR) , year=

  114. [126]

    Brain Research , volume=

    Audiovisual synchrony perception for music, speech, and object actions , author=. Brain Research , volume=

  115. [127]

    arXiv preprint arXiv:2112.04432 , year=

    Audio-visual synchronisation in the wild , author=. arXiv preprint arXiv:2112.04432 , year=

    arXiv

  116. [128]

    European Conference on Computer Vision , pages=

    Audio-synchronized visual animation , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    2024

  117. [129]

    European Conference on Computer Vision , pages=

    Temporal and cross-modal attention for audio-visual zero-shot learning , author=. European Conference on Computer Vision , pages=. 2022 , organization=

    2022

  118. [130]

    AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization , year=

    Mahmud, Tanvir and Marculescu, Diana , booktitle=. AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization , year=

  119. [131]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Audio-visual contrastive learning with temporal self-supervision , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  120. [132]

    IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP , year =

    Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation , author =. IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP , year =

  121. [133]

    arXiv preprint arXiv:2507.02915 , year=

    Audio-JEPA: Joint-Embedding Predictive Architecture for Audio Representation Learning , author=. arXiv preprint arXiv:2507.02915 , year=

    arXiv

  122. [134]

    arXiv preprint arXiv:2506.09985 , year=

    V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning , author=. arXiv preprint arXiv:2506.09985 , year=

    arXiv

  123. [135]

    arXiv preprint arXiv:1807.03748 , year=

    Representation learning with contrastive predictive coding , author=. arXiv preprint arXiv:1807.03748 , year=

    arXiv

  124. [136]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Vision transformers are parameter-efficient audio-visual learners , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  125. [137]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Sound and visual representation learning with multiple pretraining tasks , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  126. [138]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Masked autoencoders are scalable vision learners , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  127. [139]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Etva: Evaluation of text-to-video alignment via fine-grained question generation and answering , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  128. [140]

    Huang, Ziqi and He, Yinan and Yu, Jiashuo and Zhang, Fan and Si, Chenyang and Jiang, Yuming and Zhang, Yuanhan and Wu, Tianxing and Jin, Qingyang and Chanpaisit, Nattapol and Wang, Yaohui and Chen, Xinyuan and Wang, Limin and Lin, Dahua and Qiao, Yu and Liu, Ziwei , booktitle=

  129. [141]

    2025 , doi=

    Huang, Ziqi and Zhang, Fan and Xu, Xiaojie and He, Yinan and Yu, Jiashuo and Dong, Ziyue and Ma, Qianli and Chanpaisit, Nattapol and Si, Chenyang and Jiang, Yuming and Wang, Yaohui and Chen, Xinyuan and Chen, Ying-Cong and Wang, Limin and Lin, Dahua and Qiao, Yu and Liu, Ziwei...

    2025

  130. [142]

    Zheng, Dian and Huang, Ziqi and Liu, Hongbo and Zou, Kai and He, Yinan and Zhang, Fan and Zhang, Yuanhan and He, Jingwen and Zheng, Wei-Shi and Qiao, Yu and Liu, Ziwei , journal=

  131. [143]

    arXiv preprint arXiv:2512.13507 , year=

    Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Model , author=. arXiv preprint arXiv:2512.13507 , year=

    arXiv

  132. [144]

    arXiv preprint arXiv:1711.05101 , year=

    Decoupled weight decay regularization , author=. arXiv preprint arXiv:1711.05101 , year=

    arXiv

  133. [145]

    2024 , isbn =

    Wang, Xihua and Wang, Yuyue and Wu, Yihan and Song, Ruihua and Tan, Xu and Chen, Zehua and Xu, Hongteng and Sui, Guodong , title =. 2024 , isbn =. doi:10.1145/3664647.3681027 , booktitle =

    2024

  134. [146]

    LoVA: Long-form Video-to-Audio Generation , year=

    Cheng, Xin and Wang, Xihua and Wu, Yihan and Wang, Yuyue and Song, Ruihua , booktitle=. LoVA: Long-form Video-to-Audio Generation , year=

  135. [147]

    2026 , eprint=

    VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learning , author=. 2026 , eprint=

    2026

  136. [148]

    Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , month =

    Wang, Xihua and Cheng, Xin and Wang, Yuyue and Song, Ruihua and Wang, Yunfeng , title =. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , month =. 2025 , pages =

    2025

  137. [149]

    2025 , howpublished=

    LoRA Without Regret , author=. 2025 , howpublished=

    2025

  138. [150]

    arXiv preprint arXiv:2509.17765 , year=

    Qwen3-Omni Technical Report , author=. arXiv preprint arXiv:2509.17765 , year=

    arXiv

  139. [151]

    2023 , eprint=

    Gemini: A Family of Highly Capable Multimodal Models , author=. 2023 , eprint=

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.