Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

A six-step, largely automated pipeline converts raw recorded videos into millions of audio-image-text triples, with the paper's own run producing 2,240,231 pairs from 282,081 videos.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 04:56 UTC pith:WRKWD2RU

load-bearing objection A large, transparently-built audio-image-text dataset with a reproducible pipeline, but the 'robust semantic connection' claim is unvalidated; worth serious review, not a pass. the 3 major comments →

arxiv 2509.05786 v1 pith:WRKWD2RU submitted 2025-09-06 cs.MM cs.SDeess.AS

Effectively obtaining acoustic, visual and textual data from videos

Jorge E. Le\'on , Miguel Carrasco This is my paper
classification cs.MM cs.SDeess.AS
keywords multimodal datasetaudio-image-text datadata generationmiddle-frame extractionaudio-visual alignmentimage captioningvideo processing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that a systematic, openly described recipe can convert ordinary videos into large, aligned audio-image-text datasets, and it demonstrates the recipe at scale: 2,240,231 audio-image pairs extracted from 282,081 public videos, each pair extended with an automatically generated caption. The load-bearing idea is that in a continuous, unedited recording, a one-second slice of audio and the frame at the middle of that slice refer to the same event, so temporal co-occurrence can substitute for expensive manual annotation. This matters because existing multimodal corpora are either small, missing audio, or assembled by stitching several single-purpose datasets together, and audio-visual-text data in particular is scarce. The authors also take a methodological stand: minimize conversions between modalities (no audio-to-text transcription), because each conversion risks adding noise; hence text is produced by captioning the image. If the alignment assumption holds, the pipeline gives any researcher a low-cost route to custom multimodal training data, which is the stronger claim the whole paper rests on.

Core claim

On the paper's own terms, the central claim is that a fully specified, easily automated pipeline can turn ordinary videos into audio-image-text observations whose three modalities describe the same situation, and that it scales: 282,081 public videos from MUSIC, AudioSetZSL and SoundNet yielded 2,240,231 audio-image pairs, each later paired with a BLIP-generated caption and published as open datasets. The defining move is to treat temporal co-occurrence as alignment: cut the video into one-second segments, take each segment's middle frame, discard segments failing silence, darkness, or scene-cut filters, subsample for diversity, caption each surviving image. Support comes from an Acoustic Di

What carries the argument

The mechanism that carries the argument is the middle-frame extraction heuristic: for each one-second audio segment cut from the video, take the frame that falls approximately at the midpoint of that interval as the segment's visual counterpart. This single rule converts raw temporal co-occurrence into the claimed strict semantic connection between audio and image. Around it, the pipeline adds protective filters (cropping black borders, splitting at abrupt pixel changes, discarding segments with 0.5 seconds of continuous silence or overly dark frames, keeping one pair in three for diversity), a fixed storage format (16kHz, 16-bit, mono, one-second WAV audio; 512x512 RGB24 JPG/PNG images), an

Load-bearing premise

The method assumes that whatever is audible in a one-second clip of a video is also what is visible in the frame taken at the middle of that second, so that simply cutting the video at that instant creates a true audio-visual match; no check verifies that the sound and the picture actually correspond.

What would settle it

Take a random sample of a few hundred released pairs; for each, play the audio and show a human annotator the paired frame alongside three frames drawn from other pairs, and count how often the correct frame is chosen. Chance is 25 percent; accuracy near chance would mean the middle-frame heuristic is not delivering the claimed semantic connection. A computational analogue measures a pretrained audio-visual agreement score for true pairs and for the same frames paired with audio shifted in time from the same video.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Any research group can replicate the pipeline on its own video collection and obtain a custom audio-image-text dataset without manual annotation or specialized hardware; the paper's captioning run was done on a laptop.
  • The released corpus (over 2.2 million pairs with captions, in multiple public datasets) gives the community training and evaluation material for tasks that currently lack data: audio-conditioned image generation, audio-to-text captioning, and any-to-any multimodal models.
  • Datasets built this way inherit the character of their source videos: continuous unedited recordings from MUSIC, AudioSetZSL and SoundNet yield mostly non-artificial, non-abstract audio-visual relations, which the authors argue simplifies training and convergence.
  • Because text is generated from the image, caption failures are localized to image quality; switching to a stronger image-to-text model or ensembling several captioners can upgrade the textual modality without re-running audio-image extraction.
  • The reported yield (about 8 pairs per video after filtering and one-in-three subsampling) is a concrete benchmark figure that future extraction pipelines can compare against.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The alignment assumption is testable and is the first thing to check: a retrieval-style probe on the released pairs (does each audio clip identify its own frame among distractors?) would quantify how much semantic signal the middle-frame heuristic preserves; the paper itself only runs silence, darkness, and scene-cut filters, not this check.
  • The paper's minimize-modality-conversions principle has a corollary the authors only gesture at: if audio-to-image models are trained on this corpus, their generations could be evaluated by whether they recreate the source video's frames from the audio alone, turning the dataset into a closed-loop benchmark.
  • The one-second window is a design choice, not a necessity: sweeping the window length (e.g., 0.5s versus 2s) would map the trade-off between temporal alignment precision and semantic completeness, and could adapt the recipe to other video types such as tutorials, dialogue, or music.
  • The pipeline generalizes beyond video: any temporally co-occurring sensor pair (egocentric video with inertial or physiological signals, for instance) could use the same co-occurrence-as-alignment trick to generate paired data at scale, extending the paper's reach well beyond audio-image-text.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a three-phase pipeline for constructing large audio-image-text datasets from videos: selecting suitable videos, extracting 1-second audio clips paired with the middle frame from each second, and generating textual captions from the frames with BLIP. The authors apply the pipeline to 282,081 videos from MUSIC, AudioSetZSL, and SoundNet, yielding 2,240,231 audio-image pairs and, after captioning, a set of audio-image-text triples released on Kaggle. The paper's central claim is that the extraction 'ensures a robust semantic connection between modalities,' making the dataset useful for training and evaluating multimodal models, particularly audio-conditioned image-to-image generation.

Significance. If the claimed robust semantic connection holds, the contribution is a large, openly released multimodal resource that addresses a genuine gap: existing datasets rarely combine audio, image, and text modalities at scale. The pipeline is described with concrete, reproducible thresholds (black-border threshold, scene-cut threshold, silence duration, subsampling factor, BLIP configuration), and the data are publicly available. The authors also provide useful summary statistics and an Acoustic Diversity Index estimate. These are real strengths. However, the manuscript's central quality claim is not validated by any direct measurement of audio-image or audio-text correspondence, which limits the confidence one can place in the dataset's utility for the stated downstream tasks.

major comments (3)
  1. [§4.2, steps 3–4; abstract] The paper's central claim—'robust semantic connection between modalities'—rests entirely on temporal co-occurrence: a 1-second audio segment is paired with the frame approximately at its middle, after splitting on scene cuts and filtering silence/dark frames. No quantitative or human evaluation is provided to show that the middle frame is semantically representative of the audio content, or that the two are aligned beyond being drawn from the same raw footage. Given that the stated purpose of the dataset is to train/evaluate audio-conditioned image models, this is a load-bearing gap. I recommend adding a validation study, e.g., a forced-choice human test on a random sample of pairs (audio plus one correct and one incorrect frame from the same video/different video), or a retrieval-style metric (e.g., CLAP/ImageBind embeddings) demonstrating that paired audio and images are more similar t
  2. [§4.3 and Figure 8] The text modality is generated by BLIP from the image alone, not from the audio. The paper itself acknowledges in Figure 8 that captions are sometimes wrong, and in Section 5 admits there is 'room for improvement' in the filters. Since the text is derived from the image, any audio-image mismatch automatically corrupts the audio-text link as well. The quantitative text statistics (word counts, frequency table) describe surface properties, not semantic correctness. The authors should either present a targeted evaluation of text-image and text-audio alignment (even a modest human-annotated subset) or clearly reposition the dataset as 'weakly aligned' rather than 'robustly semantically connected.'
  3. [§5, ADI and waveform analysis] The two quantitative tests reported—the aggregate waveform plot and the Acoustic Diversity Index—measure global statistical properties of the audio collection, not the semantic correspondence between paired modalities. The ADI value of ~3.05 indicates diversity across the whole audio set, and the Gaussian-like amplitude distribution is consistent with generic audio; neither says anything about whether a given 1-second clip matches the paired frame. These analyses therefore do not address the central alignment claim and should not be presented as evidence of dataset quality in that regard.
minor comments (6)
  1. [Abstract and §1] The phrase 'more than 2,000,000 audio-image pairs' appears in the abstract, while Section 5 reports 2,240,231. The counts are consistent, but the abstract's wording could be made exact. Also, 'audio-image-text observations' is used in the abstract, but the text is generated only from the image; the authors may wish to clarify this dependency early.
  2. [§4, introductory sentence] Typo: 'In this this section' should be 'In this section.'
  3. [§5, comparison paragraph] Typo: 'researchers need to arduousness search' should likely be 'researchers need to arduously search.'
  4. [§4.2, step 2] The threshold of 90 for 'abrupt change' is presented without a sensitivity analysis or reference; since this threshold controls scene-cut detection and thus the duration of each fragment, a brief justification or a check on a few videos would strengthen reproducibility.
  5. [§4.2, step 4] The silence threshold (absolute sample value < 100 for 0.5 s) and dark-frame threshold (mean pixel intensity < 10) are stated as fixed values. It would be helpful to note whether these were tuned or adopted from prior work, and how sensitive the final dataset size is to these choices.
  6. [§5, Table 3] For the comparison with AudioSetZSL and SoundNet, the table lists audio and video as present, but the 'Ours' row lists audio, image, and text. Since the underlying video source for our dataset is not distributed, it may be clearer to mark 'V' as not directly included in the released Kaggle datasets.

Circularity Check

0 steps flagged

No significant circularity: the dataset pipeline is self-contained; the semantic-connection claim rests on an unvalidated heuristic assumption, not on a fitted or self-referential derivation.

full rationale

The paper's derivation chain is a data-construction pipeline: select continuous videos (§4.1), split on scene cuts, pair 1-second audio segments with middle frames (§4.2), filter silence/dark frames, subsample, and generate captions with BLIP (§4.3). No parameter is fitted to a subset of the output and then 'predicted' on a closely related quantity; the reported 2,240,231 audio-image pairs are the direct result of applying the stated rules, not a model prediction. The middle-frame heuristic is justified by external citations ([65,38,93,132]) and is not a self-citation; BLIP is an external image-to-text model; the only self-citations are to the released datasets themselves, which are outputs, not load-bearing inputs. The paper's claim that the approach 'ensures a robust semantic connection between modalities' is supported only by the assumption that temporal co-occurrence in a continuous video implies semantic correspondence. That is an empirical proxy assumption, and the paper does not quantitatively validate it; indeed it acknowledges remaining quality issues in Figure 8 and §5. However, an unsupported heuristic is not circularity under the criteria here: the claim is not made true by definition of the extraction rule, nor is any 'prediction' equivalent to its fitted inputs, nor does the argument reduce to a self-citation chain. The acknowledged limitations are correctness/quality concerns, not circularity. The derivation is therefore self-contained, with no circular step to flag.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The method is an engineering pipeline; it introduces no new mathematical formalism or physical entities. Its claims rest on hand-chosen thresholds and on the assumption that temporal co-occurrence implies semantic correspondence.

free parameters (7)
  • abrupt change detection threshold = 90
    Average squared pixel difference between consecutive frames above which the video is split (Section 4.2, step 2). Hand-chosen; affects segmentation and hence pair semantics.
  • black border intensity threshold = 15
    Pixel channel value below/above which a row/column is considered black and cropped (Section 4.2, step 1).
  • silence detection amplitude threshold = 100
    Absolute sample value below which a sample is considered silent; continuous 0.5s silence discards a pair (Section 4.2, step 4). Value relative to 16-bit range.
  • continuous silence duration = 0.5 s
    Minimum length of continuous silence to discard an audio-image pair (Section 4.2, step 4).
  • mean pixel intensity threshold = 10
    Images whose mean pixel value does not surpass 10 are discarded as too dark (Section 4.2, step 4).
  • skip factor = 1/3
    Only every third pair is kept to increase diversity (Section 4.2, step 5).
  • BLIP beam/token settings = 2 beams, min 10, max 20 tokens
    Used for text generation; chosen for speed, not quality (Section 4.3).
axioms (5)
  • domain assumption The middle frame of a 1-second video segment is semantically representative of the segment's content.
    Used in Section 4.2 step 3 to pair audio with a single image; cited to [65,38,93,132] but treated as given. If false, the audio-image correspondence is weakened.
  • domain assumption Audio and visual content co-occurring in a raw video are semantically related.
    The core assumption behind the dataset's claimed 'robust semantic connection' (Section 4.1). It is inherent to real-world recordings but not validated here.
  • domain assumption BLIP image captioning produces descriptions that adequately represent the image content.
    BLIP is used to generate the text modality (Section 4.3); the paper acknowledges quality issues (Section 5, Figure 8), so the text portion may be noisy.
  • domain assumption The source datasets (MUSIC, AudioSetZSL, SoundNet) contain continuous, high-quality recordings without copyright conflicts.
    Section 5: videos are drawn from these public datasets; the paper does not re-license or verify content.
  • domain assumption The waveform distribution approximating a Gaussian indicates no bias in the audio data.
    Section 5 uses the central-limit-like waveform histogram as evidence of 'no evident biases'; this is a weak proxy for dataset bias.

pith-pipeline@v1.3.0-alltime-deepseek · 27430 in / 9492 out tokens · 93440 ms · 2026-08-05T04:56:58.304754+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Effectively obtaining acoustic, visual and textual data from videos." pith.science (2026). https://pith.science/paper/WRKWD2RU

@misc{pith2026250905786,
  author       = {Pith},
  title        = {Pith review of: Effectively obtaining acoustic, visual and textual data from videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WRKWD2RU}},
  note         = {Machine review of arXiv:2509.05786}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The increasing use of machine learning models has amplified the demand for high-quality, large-scale multimodal datasets. However, the availability of such datasets, especially those combining acoustic, visual and textual data, remains limited. This paper addresses this gap by proposing a method to extract related audio-image-text observations from videos. We detail the process of selecting suitable videos, extracting relevant data pairs, and generating descriptive texts using image-to-text models. Our approach ensures a robust semantic connection between modalities, enhancing the utility of the created datasets for various applications. We also discuss the challenges encountered and propose solutions to improve data quality. The resulting datasets, publicly available, aim to support and advance research in multimodal data analysis and machine learning.

Figures

Figures reproduced from arXiv: 2509.05786 by Jorge E. Le\'on, Miguel Carrasco.

Figure 1
Figure 1. Figure 1: Text-to-image generation example. Text-to-image is a technique that generates [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Types of data modalities. To address this last challenge, one of the strategies that has been adopted is to increase the number of data modalities that the models receive (i.e. the types of data that are taken as input; e.g. text, image, audio, etc.) [105, 107, 123, 88]. It is pertinent to comment that this increase in the number of modalities not only allows for greater control on the respective tasks, bu… view at source ↗
Figure 3
Figure 3. Figure 3: Summary of the whole method. 1. Video Selection: This initial phase involves identifying and selecting high-quality, continuous video recordings that feature synchronized audio and frames, ideally ensuring a strong semantic connection between the modalities (i.e. both audio and image in each pair are extracted from and related to the same situation). 2. Audio-Image Pair Extraction: In this step, audio segm… view at source ↗
Figure 4
Figure 4. Figure 4: Summary of the audio-image pair extraction procedure. 1) Removal of black bor [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Application example of steps 3, 4 and 5 of the audio-image pairs generation process, [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Example of the final data preprocessing. Audio-image pairs are expanded to [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Visualization of the sources of our data, with the approximate percentage for each dataset and some real examples resulting from each one. An illustration of how this preprocessing would look like is shown in [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: (a.) A small selection of what we label sufficient and (b.) insufficient quality [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Average of all the waveforms in our observations. The horizontal axis contains the timestamps, while the vertical one is for the instantaneous amplitudes. deal with millions of parameters and therefore require larger datasets to be properly trained. Currently, researchers need to arduousness search for multiple datasets and artfully come up with ways to utilize them in audio-image-text tasks; as they not o… view at source ↗
Figure 10
Figure 10. Figure 10: Summary of our ADI calculation. 1) We take all of our audios in their raw form. 2) We apply the Hann function [15] (visible on the top) over each audio signal, so all of them loop smoothly and we avoid spectral leakage. 3) We compute the fast Fourier transform of each signal from the previous step, we get the magnitude of each resulting complex number and square them; now these new values are proportional… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Testing chatbots on the creation of encoders for audio conditioned image generation

    cs.SD 2025-09 conditional novelty 6.0

    All chatbot-designed audio encoders failed to align with CLIP text embeddings and produced incoherent images, while showing a surprising architectural similarity across chatbots.

Reference graph

Works this paper leans on

139 extracted references · 25 canonical work pages · cited by 1 Pith paper · 4 internal anchors

  1. [1]

    Denk, Zal´ an Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank

    Andrea Agostinelli, Timo I. Denk, Zal´ an Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank. MusicLM: Generating Music From Text. ArXiv, 2301.11325, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Don’t Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering

    Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don’t Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering. InPro- ceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, pages 4971–4980, 2018

    2018

  3. [3]

    Mistral Models, 2024

    Mistral AI. Mistral Models, 2024

    2024

  4. [4]

    Transcripter- Generation of the transcript from audio to text using Deep Learning.International Journal of Computer Sciences and Engineering, 7(1):770–773, 2019

    Fatima Ansari, Ramsakal Gupta, Uday Singh, and Fahimur Shaikh. Transcripter- Generation of the transcript from audio to text using Deep Learning.International Journal of Computer Sciences and Engineering, 7(1):770–773, 2019

    2019

  5. [5]

    The Claude 3 Model Family: Opus, Sonnet, Haiku, 2024

    Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku, 2024

    2024

  6. [6]

    SoundNet: Learning Sound Repre- sentations from Unlabeled Video

    Yusuf Aytar, Carl Vondrick, and Antonio Torralba. SoundNet: Learning Sound Repre- sentations from Unlabeled Video. InProceedings of the 30th International Conference on Neural Information Processing Systems, page 892–900, 2016

    2016

  7. [7]

    Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph

    AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis- Philippe Morency. Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2236–2246, 2018

    2018

  8. [8]

    AudioSetCaps: An Enriched Audio-Caption Dataset using Auto- mated Generation Pipeline with Large Audio and Language Models

    Jisheng Bai, Haohe Liu, Mou Wang, Dongyuan Shi, Mark Plumbley, Woon-Seng Gan, and Jianfeng Chen. AudioSetCaps: An Enriched Audio-Caption Dataset using Auto- mated Generation Pipeline with Large Audio and Language Models. InAudio Imag- ination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation, 2024

    2024

  9. [9]

    Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

    Max Bain, Arsha Nagrani, Gul Varol, and Andrew Zisserman. Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval. InProceedings of the 2021 IEEE International Conference on Computer Vision, pages 1708–1718, 2021

    2021

  10. [10]

    Are Mod- els Biased on Text without Gender-related Language? InProceedings of the 12th International Conference on Learning Representations, 2024

    Catarina G Bel´ em, Preethi Seshadri, Yasaman Razeghi, and Sameer Singh. Are Mod- els Biased on Text without Gender-related Language? InProceedings of the 12th International Conference on Learning Representations, 2024

    2024

  11. [11]

    Ballester

    Marcelo Bertalm ´ ıo, Guillermo Sapiro, Vicent Caselles, and C. Ballester. Image in- painting. InProceedings of the 27th Internationl Conference on Computer Graphics and Interactive Techniques Conference, pages 417–424, 2000. 20

    2000

  12. [12]

    Improving Image Generation with Better Captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving Image Generation with Better Captions. 2023

    2023

  13. [13]

    Clifton, Yuxiong He, Dacheng Tao, and Shuaiwen Leon Song

    Fengxiang Bie, Yibo Yang, Zhongzhu Zhou, Adam Ghanem, Minjia Zhang, Zhewei Yao, Xiaoxia Wu, Connor Holmes, Pareesa Golnari, David A. Clifton, Yuxiong He, Dacheng Tao, and Shuaiwen Leon Song. RenAIssance: A Survey into AI Text-to- Image Generation in the Era of Large Model.ArXiv, 2309.00810, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Birhane and V

    A. Birhane and V. Prabhu. Large image datasets: A pyrrhic win for computer vision? InProceedings of the 2021 IEEE Winter Conference on Applications of Computer Vision, pages 1536–1546, 2021

    2021

  15. [15]

    R. B. Blackman and J. W. Tukey. The measurement of power spectra from the point of view of communications engineering — Part I.The Bell System Technical Journal, 37(1):185–282, 1958

    1958

  16. [16]

    Using acoustic indices in ecology: Guidance on study design, analyses and interpretation.Methods in Ecology and Evolution, 14(9):2192–2204, 2023

    Tom Bradfer-Lawrence, Camille Desjonqueres, Alice Eldridge, Alison Johnston, and Oliver Metcalf. Using acoustic indices in ecology: Guidance on study design, analyses and interpretation.Methods in Ecology and Evolution, 14(9):2192–2204, 2023

    2023

  17. [17]

    Bar- nett, Amy Beeston, Jennifer Darby, Benedict Dell, Nick Gardner, Amandine Gasc, Becky Heath, Nia Howells, Magnus Janson, Maria-Viktoria Kyoseva, Thomas Luy- paert, Oliver C

    Tom Bradfer-Lawrence, Brad Duthie, Carlos Abrahams, Maty´ aˇ s Adam, Ross J. Bar- nett, Amy Beeston, Jennifer Darby, Benedict Dell, Nick Gardner, Amandine Gasc, Becky Heath, Nia Howells, Magnus Janson, Maria-Viktoria Kyoseva, Thomas Luy- paert, Oliver C. Metcalf, Anna E. Nousek-McGregor, Frederica Poznansky, Samuel R. P.-J. Ross, Sarab Sethi, Siobhan Smyt...

    2025

  18. [18]

    Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-Ann Heng, and Stan Z. Li. A Survey on Generative Diffusion Models.IEEE Transactions on Knowledge and Data Engineering, 36(7):2814–2830, 2024

    2024

  19. [19]

    Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Con- cepts

    Soravit Changpinyo, Piyush Kumar Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Con- cepts. InProceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, pages 3557–3567, 2021

    2021

  20. [20]

    Phantom-Data: Towards a General Subject-Consistent Video Generation Dataset.ArXiv, 2506.18851, 2025

    Zhuowei Chen, Bingchuan Li, Tianxiang Ma, Lijie Liu, Mingcong Liu, Yi Zhang, Gen Li, Xinghui Li, Siyu Zhou, Qian He, and Xinglong Wu. Phantom-Data: Towards a General Subject-Consistent Video Generation Dataset.ArXiv, 2506.18851, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Veo, 2024

    Google DeepMind. Veo, 2024. 21

    2024

  22. [22]

    Central Limit Theorem in the Functional Approach.IEEE Transactions on Signal Processing, 61(16):4025–4037, 2013

    Dominique Dehay, Jacek Leskow, and Antonio Napolitano. Central Limit Theorem in the Functional Approach.IEEE Transactions on Signal Processing, 61(16):4025–4037, 2013

    2013

  23. [23]

    A Survey of On-Device Machine Learning: An Algorithms and Learning Theory Perspective.ACM Transactions on Internet of Things, 2(3), 2021

    Sauptik Dhar, Junyao Guo, Jiayi (Jason) Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah. A Survey of On-Device Machine Learning: An Algorithms and Learning Theory Perspective.ACM Transactions on Internet of Things, 2(3), 2021

    2021

  24. [24]

    Jukebox: A Generative Model for Music.ArXiv, 2005.00341, 2020

    Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A Generative Model for Music.ArXiv, 2005.00341, 2020

    Pith/arXiv arXiv 2005

  25. [25]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethan...

    Pith/arXiv arXiv 2024

  26. [26]

    Image Generation: A Review.Neural Processing Letters, 54(5):4609–4646, 2022

    Mohamed Elasri, Omar Elharrouss, Somaya Al-Maadeed, and Hamid Tairi. Image Generation: A Review.Neural Processing Letters, 54(5):4609–4646, 2022

    2022

  27. [27]

    Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M¨ uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. ArXiv, 2403.03206, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Creativity and Machine Learning: A Survey

    Giorgio Franceschelli and Mirco Musolesi. Creativity and Machine Learning: A Survey. ArXiv, 2104.02726, 2022

    Pith/arXiv arXiv 2022

  29. [29]

    The Pile: An 800GB Dataset of Diverse Text for Language Modeling.ArXiv, 2101.00027, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling.ArXiv, 2101.00027, 2020. 24

    Pith/arXiv arXiv 2020

  30. [30]

    Listen to Look: Action Recognition by Previewing Audio

    Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen to Look: Action Recognition by Previewing Audio.ArXiv, 1912.04487, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 1912

  31. [31]

    Gemmeke, Daniel P

    Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio Set: An ontology and human-labeled dataset for audio events. InProceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 776–780, 2017

    2017

  32. [32]

    ImageBind: One Embedding Space To Bind Them All.ArXiv, 2305.05665, 2023

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Al- wala, Armand Joulin, and Ishan Misra. ImageBind: One Embedding Space To Bind Them All.ArXiv, 2305.05665, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Vincent Cartillier, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Devansh Kukreja, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Meng- meng Xu, Eric Zhongcong Xu, Chen Zhao, Sidd...

    2024

  34. [34]

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces.ArXiv, 2312.00752, 2024

    Albert Gu and Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces.ArXiv, 2312.00752, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Temporal Alignment Networks for Long-term Video

    Tengda Han, Weidi Xie, and Andrew Zisserman. Temporal Alignment Networks for Long-term Video.ArXiv, 2204.02968, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  36. [36]

    Rae, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre...

    2024

  37. [37]

    Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model

    Joanna Hong, Se Park, and Yong Ro. Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model. InFindings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4886–4890, 2023

    2023

  38. [38]

    Key Frame Selection for Temporal Graph Opti- mization of Skeleton-Based Action Recognition.Applied Sciences, 14(21), 2024

    Jingyi Hou, Lei Su, and Yan Zhao. Key Frame Selection for Temporal Graph Opti- mization of Skeleton-Based Action Recognition.Applied Sciences, 14(21), 2024

    2024

  39. [39]

    NLIP: Noise-Robust Language-Image Pre-training

    Runhui Huang, Yanxin Long, Jianhua Han, Hang Xu, Xiwen Liang, Chunjing Xu, and Xiaodan Liang. NLIP: Noise-Robust Language-Image Pre-training. InProceedings of the 37th AAAI Conference on Artificial Intelligence, pages 926–934, 2023

    2023

  40. [40]

    Imagen-Team-Google, :, Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brich- tova, Andrew Bunner, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuqing Du, Zach Eaton-Rosen, Hongliang Fei, Nando de Freitas, Yilin Gao, Evgeny Gladchenko, Sergio G´ omez Colmenarejo, Mandy Guo, Alex Haig, Will Hawkins, Hexiang Hu, Huil- ian Huang, Tobenna Peter Igwe, Chris...

    arXiv 2024

  41. [41]

    Dennis L. Jackson. Revisiting Sample Size and Number of Parameter Estimates: Some Support for the N:q Hypothesis.Structural Equation Modeling: A Multidisciplinary Journal, 10(1):128–141, 2003

    2003

  42. [42]

    LL VIP: A Visible- infrared Paired Dataset for Low-light Vision

    Xinyu Jia, Chuang Zhu, Minzhen Li, Wenqi Tang, and Wenli Zhou. LL VIP: A Visible- infrared Paired Dataset for Low-light Vision. InProceedings of the 2021 IEEE Inter- national Conference on Computer Vision Workshops, pages 3489–3497, 2021

    2021

  43. [43]

    Nicolas Jonason and Bob L. T. Sturm. TimbreCLIP: Connecting Timbre to Text and Images.ArXiv, 2211.11225, 2022

    Pith/arXiv arXiv 2022

  44. [44]

    Noise-Aware Learning from Web-Crawled Image-Text Data for Image Captioning

    Wooyoung Kang, Jonghwan Mun, Sungjun Lee, and Byungseok Roh. Noise-Aware Learning from Web-Crawled Image-Text Data for Image Captioning. InProceedings of the 2023 IEEE International Conference on Computer Vision, pages 2942–2952, 2023

    2023

  45. [45]

    MMIS: Multimodal Dataset for Interior Scene Visual Generation and Recognition

    Hozaifa Kassab, Ahmed Mahmoud, Mohamed Bahaa, Ammar Mohamed, and Ali Hamdi. MMIS: Multimodal Dataset for Interior Scene Visual Generation and Recog- nition.ArXiv, 2407.05980, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  46. [46]

    Zahra Khanjani, Gabrielle Watson, and Vandana P. Janeja. Audio deepfakes: A survey. Frontiers in Big Data, 5, 2023

    2023

  47. [47]

    AudioCaps: Generating Captions for Audios in The Wild

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. AudioCaps: Generating Captions for Audios in The Wild. InProceedings of the 2019 North Amer- ican Chapter of the Association for Computational Linguistics, pages 119–132, 2019

    2019

  48. [48]

    Benchmarking Cognitive Biases in Large Language Models as Evaluators.ArXiv, 2309.17012, 2023

    Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking Cognitive Biases in Large Language Models as Evaluators.ArXiv, 2309.17012, 2023. 27

    Pith/arXiv arXiv 2023

  49. [49]

    AudioGen: Textually Guided Audio Generation.ArXiv, 2209.15352, 2023

    Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre D´ efossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. AudioGen: Textually Guided Audio Generation.ArXiv, 2209.15352, 2023

    Pith/arXiv arXiv 2023

  50. [50]

    BindDiffusion: One Diffusion Model to Bind Them All, 2024

    Sea AI Lab. BindDiffusion: One Diffusion Model to Bind Them All, 2024

    2024

  51. [51]

    FLUX, 2024

    Black Forest Labs. FLUX, 2024

    2024

  52. [52]

    Jorge E. Le´ on. A VT Multimodal Dataset, 2024

    2024

  53. [53]

    Jorge E. Le´ on. Image-audio pairs (1 of 3), 2024

    2024

  54. [54]

    Jorge E. Le´ on. Image-audio pairs (2 of 3), 2024

    2024

  55. [55]

    Jorge E. Le´ on. Image-audio pairs (3 of 3), 2024

    2024

  56. [56]

    Jorge E. Le´ on. Text-audio pairs (1 of 4), 2024

    2024

  57. [57]

    Jorge E. Le´ on. Text-audio pairs (2 of 4), 2024

    2024

  58. [58]

    Jorge E. Le´ on. Text-audio pairs (3 of 4), 2024

    2024

  59. [59]

    Jorge E. Le´ on. Text-audio pairs (4 of 4), 2024

    2024

  60. [60]

    OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation

    Hui Li, Mingwang Xu, Yun Zhan, Shan Mu, Jiaye Li, Kaihui Cheng, Yuxuan Chen, Tan Chen, Mao Ye, Jingdong Wang, and Siyu Zhu. OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation. InProceedings of the 2025 IEEE Conference on Computer Vision and Pattern Recognition, pages 7752–7762, 2025

    2025

  61. [61]

    BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Mod- els.ArXiv, 2301.12597, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Mod- els.ArXiv, 2301.12597, 2023

    Pith/arXiv arXiv 2023

  62. [62]

    BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Gen- eration.ArXiv, 2201.12086, 2022

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Gen- eration.ArXiv, 2201.12086, 2022

    Pith/arXiv arXiv 2022

  63. [63]

    Word-Level Explanations for Analyzing Bias in Text-to-Image Models.ArXiv, 2306.05500, 2023

    Alexander Lin, Lucas Monteiro Paes, Sree Harsha Tanneru, Suraj Srinivas, and Himabindu Lakkaraju. Word-Level Explanations for Analyzing Bias in Text-to-Image Models.ArXiv, 2306.05500, 2023

    Pith/arXiv arXiv 2023

  64. [64]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ra- manan, Piotr Doll´ ar, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. InProceedings of the 13th European Conference on Computer Vision, pages 740–755, 2014. 28

    2014

  65. [65]

    A Comparison Between KeyFrame Extraction Methods for Clothing Recognition, 2023

    Gabriel Lindgren. A Comparison Between KeyFrame Extraction Methods for Clothing Recognition, 2023

    2023

  66. [66]

    Plumbley

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. InProceedings of the 40th International Conference on Machine Learning, pages 21450–21474, 2023

    2023

  67. [67]

    Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models.ArXiv, 2402.17177, 2024

    Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, Lifang He, and Lichao Sun. Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models.ArXiv, 2402.17177, 2024

    Pith/arXiv arXiv 2024

  68. [68]

    Z. Liu, P. Luo, S. Qiu, X. Wang, and X. Tang. DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations. InProceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, pages 1096–1104, 2016

    2016

  69. [69]

    BLAP: Bootstrapping Language-Audio Pre-training for Music Captioning

    Nathana¨ el Perraudin Luca A Lanzend¨ orfer, Constantin Pinkl and Roger Wattenhofer. BLAP: Bootstrapping Language-Audio Pre-training for Music Captioning. InAudio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Genera- tion, 2024

    2024

  70. [70]

    Stable Diffusion Akashic Records, 2023

    Maks-s. Stable Diffusion Akashic Records, 2023

    2023

  71. [71]

    GenRL: Multimodal-foundation world models for generalization in embodied agents

    Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Aaron Courville, and Sai Rajeswar. GenRL: Multimodal-foundation world models for generalization in embodied agents. ArXiv, 2406.18043, 2024

    Pith/arXiv arXiv 2024

  72. [72]

    Mustango: Toward Controllable Text-to-Music Generation

    Jan Melechovsky, Zixun Guo, Deepanway Ghosal, Navonil Majumder, Dorien Herre- mans, and Soujanya Poria. Mustango: Toward Controllable Text-to-Music Generation. InProceedings of the 2024 North American Chapter of the Association for Computa- tional Linguistics, page 8293–8316, 2024

    2024

  73. [73]

    Mukhamediev, Adilkhan Symagulov, Yan Kuchin, Kirill Yakunin, and Ma- rina Yelis

    Ravil I. Mukhamediev, Adilkhan Symagulov, Yan Kuchin, Kirill Yakunin, and Ma- rina Yelis. From Classical Machine Learning to Deep Neural Networks: A Simplified Scientometric Review.Applied Sciences, 11(12), 2021

    2021

  74. [74]

    DALL·E 3 System Card, 2023

    OpenAI. DALL·E 3 System Card, 2023

    2023

  75. [75]

    Video generation models as world simulators, 2024

    OpenAI. Video generation models as world simulators, 2024

    2024

  76. [76]

    Douglas O’Shaughnessy.Speech Communications: Human and Machine, volume 2. 2000

    2000

  77. [77]

    Image-to-Image Translation: Methods and Applications.IEEE Transactions on Multimedia, 24:3859–3881, 2022

    Yingxue Pang, Jianxin Lin, Tao Qin, and Zhibo Chen. Image-to-Image Translation: Methods and Applications.IEEE Transactions on Multimedia, 24:3859–3881, 2022. 29

    2022

  78. [78]

    AudioSetZSL, 2019

    Kranti Kumar Parida. AudioSetZSL, 2019

    2019

  79. [79]

    Coordi- nated Joint Multimodal Embeddings for Generalized Audio-Visual Zero-shot Classifi- cation and Retrieval of Videos

    Kranti Kumar Parida, Neeraj Matiyali, Tanaya Guha, and Gaurav Sharma. Coordi- nated Joint Multimodal Embeddings for Generalized Audio-Visual Zero-shot Classifi- cation and Retrieval of Videos. InProceedings of the 2020 IEEE Winter Conference on Applications of Computer Vision, pages 3240–3249, 2020

    2020

  80. [80]

    Pijanowski, Luis J

    Bryan C. Pijanowski, Luis J. Villanueva-Rivera, Sarah L. Dumyahn, Almo Farina, Bernie L. Krause, Brian M. Napoletano, Stuart H. Gage, and Nadia Pieretti. Sound- scape Ecology: The Science of Sound in the Landscape.BioScience, 61(3):203–216, 2011

    2011

Showing first 80 references.