pith. sign in

arxiv: 1906.11759 · v1 · pith:AD6KAVBFnew · submitted 2019-06-20 · 🧬 q-bio.NC · cs.IR· cs.LG· cs.SD· eess.AS· stat.ML

Low-dimensional Embodied Semantics for Music and Language

Pith reviewed 2026-05-25 18:58 UTC · model grok-4.3

classification 🧬 q-bio.NC cs.IRcs.LGcs.SDeess.ASstat.ML
keywords embodied semanticsfMRI embeddingsmusic semanticslanguage semanticslow-dimensional representationsjoint subject modelingneural representationssemantic classification
0
0 comments X

The pith

Joint modeling of fMRI from multiple subjects produces low-dimensional embeddings that outperform high-dimensional voxel spaces in music genre and language topic classification.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that semantics for music and language is represented in brain activity patterns, but individual brains add idiosyncratic noise. By jointly modeling data from several human subjects, low-dimensional vector embeddings can capture the shared semantics across people. These embeddings are shown to perform better than the original high-dimensional fMRI data in tasks that classify music genres and language topics. The joint modeling also makes the latent spaces semantically richer. This approach offers an efficient way to extract common semantic representations from noisy brain data.

Core claim

We propose to represent shared semantics using low-dimensional vector embeddings by jointly modeling several brains from human subjects. We show these unsupervised efficient representations outperform the original high-dimensional fMRI voxel spaces in proxy music genre and language topic classification tasks. We further show that joint modeling of several subjects increases the semantic richness of the learned latent vector spaces.

What carries the argument

Low-dimensional vector embeddings from joint modeling of multi-subject fMRI data, which extracts shared semantics by reducing idiosyncratic noise.

If this is right

  • Low-dimensional embeddings can replace raw fMRI voxels for semantic classification tasks.
  • Joint modeling across subjects yields richer semantic representations than single-subject modeling.
  • The method applies to both music and language modalities.
  • Unsupervised learning suffices to derive these shared semantic representations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The embeddings might extend to other semantic tasks such as retrieval or alignment across modalities.
  • Scaling to larger groups of subjects could further reduce noise and improve generalization.
  • This joint modeling approach could help align brain data for cross-individual semantic studies.

Load-bearing premise

The chosen proxy tasks of music genre and language topic classification accurately reflect semantic richness, and joint modeling captures shared semantics rather than just averaging noise or task artifacts.

What would settle it

If low-dimensional embeddings fail to outperform high-dimensional fMRI voxel spaces on the same classification tasks when evaluated on held-out data or new subjects, the central claim would be falsified.

Figures

Figures reproduced from arXiv: 1906.11759 by David Martins de Matos, Francisco Afonso Raposo, Ricardo Ribeiro.

Figure 1
Figure 1. Figure 1: Classification accuracies (red, green, and blue lines relative [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
read the original abstract

Embodied cognition states that semantics is encoded in the brain as firing patterns of neural circuits, which are learned according to the statistical structure of human multimodal experience. However, each human brain is idiosyncratically biased, according to its subjective experience history, making this biological semantic machinery noisy with respect to the overall semantics inherent to media artifacts, such as music and language excerpts. We propose to represent shared semantics using low-dimensional vector embeddings by jointly modeling several brains from human subjects. We show these unsupervised efficient representations outperform the original high-dimensional fMRI voxel spaces in proxy music genre and language topic classification tasks. We further show that joint modeling of several subjects increases the semantic richness of the learned latent vector spaces.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes representing shared semantics for music and language via low-dimensional vector embeddings obtained by jointly modeling fMRI data across multiple human subjects. It claims these unsupervised embeddings outperform the original high-dimensional fMRI voxel spaces on proxy music-genre and language-topic classification tasks and that joint multi-subject modeling increases the semantic richness of the learned latent spaces.

Significance. If the central claims hold after proper validation, the approach could offer a practical route to extracting shared semantic structure from noisy, idiosyncratic brain recordings, with relevance to embodied cognition models. The unsupervised and multi-subject framing is a potential strength for noise reduction, though the proxy-task grounding remains the key untested link.

major comments (2)
  1. [Abstract] Abstract: the claim that the low-dimensional embeddings 'outperform the original high-dimensional fMRI voxel spaces' in proxy classification tasks is presented without reference to data splits, cross-validation procedure, statistical tests, or baseline comparisons; this information is load-bearing for the outperformance assertion.
  2. [Abstract] Abstract: the further claim that joint modeling 'increases the semantic richness' of the latent spaces rests on the untested assumption that superior proxy-task performance isolates semantic content rather than non-semantic cues (e.g., low-level acoustic statistics for genre or syntactic regularities for topic); no ablation or control experiment is referenced to rule out the latter.
minor comments (1)
  1. [Abstract] The abstract would benefit from stating the number of subjects, the target embedding dimensionality, and the specific fMRI preprocessing steps to allow immediate assessment of the experimental scale.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments on the abstract. We address each major comment below and indicate planned revisions to improve clarity while preserving the manuscript's claims.

read point-by-point responses
  1. Referee: [Abstract] Abstract: the claim that the low-dimensional embeddings 'outperform the original high-dimensional fMRI voxel spaces' in proxy classification tasks is presented without reference to data splits, cross-validation procedure, statistical tests, or baseline comparisons; this information is load-bearing for the outperformance assertion.

    Authors: The evaluation details, including data splits, cross-validation, statistical tests, and baseline comparisons, are provided in the Methods and Results sections. We will revise the abstract to briefly reference the validation protocol supporting the outperformance claim. revision: yes

  2. Referee: [Abstract] Abstract: the further claim that joint modeling 'increases the semantic richness' of the latent spaces rests on the untested assumption that superior proxy-task performance isolates semantic content rather than non-semantic cues (e.g., low-level acoustic statistics for genre or syntactic regularities for topic); no ablation or control experiment is referenced to rule out the latter.

    Authors: The proxy tasks target semantic categories (genre and topic), but we acknowledge the abstract does not explicitly rule out non-semantic confounds. We will revise the abstract to qualify the semantic richness claim and expand the discussion section to address potential alternative explanations and limitations of the proxy tasks. revision: partial

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper learns low-dimensional embeddings via unsupervised joint modeling of multi-subject fMRI data and evaluates them on separate proxy classification tasks (music genre, language topic). These tasks are external to the embedding construction and serve as independent benchmarks rather than being defined by or equivalent to the fitted representations. No equations, self-citations, or steps are quoted that reduce the performance claims or 'semantic richness' increase to the inputs by construction (e.g., no fitted parameter renamed as prediction, no self-definitional loop). The derivation remains self-contained against the stated proxies.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no explicit free parameters, axioms, or invented entities; all such elements remain unknown.

pith-pipeline@v0.9.0 · 5661 in / 1120 out tokens · 24218 ms · 2026-05-25T18:58:50.591619+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages

  1. [1]

    Kiefer, F

    M. Kiefer, F. Pulverm¨ uller, Conceptual Representations in Mind and Brain: Theoretical Developments, Current Evidence and Future Directions, Cortex 48 (2012) 805–825. doi: 10.1016/ j.cortex.2011.04.006

  2. [2]

    Lakoff, Mapping the Brain’s Metaphor Circuitry: Metaphor- ical Thought in Everyday Reason, Frontiers in Human Neuro- science 8 (2014) 958

    G. Lakoff, Mapping the Brain’s Metaphor Circuitry: Metaphor- ical Thought in Everyday Reason, Frontiers in Human Neuro- science 8 (2014) 958. doi: 10.3389/fnhum.2014.00958

  3. [3]

    Pulverm¨ uller, Neurobiological Mechanisms for Semantic Fea- ture Extraction and Conceptual Flexibility, Topics in Cognitive Science 10 (2018) 590–620

    F. Pulverm¨ uller, Neurobiological Mechanisms for Semantic Fea- ture Extraction and Conceptual Flexibility, Topics in Cognitive Science 10 (2018) 590–620. doi: 10.1111/tops.12367

  4. [4]

    M. A. L. Ralph, E. Jefferies, K. Patterson, T. T. Rogers, The Neural and Computational Bases of Semantic Cognition, Na- ture Reviews Neuroscience 18 (2017) 42–55. doi: 10.1038/nrn. 2016.150

  5. [5]

    R. H. Desai, J. R. Binder, L. L. Conant, Q. R. Mano, M. S. Seidenberg, The Neural Career of Sensory-motor Metaphors, Journal of Cognitive Neuroscience 23 (2011) 2376–2386. doi:10. 1162/jocn.2010.21596

  6. [6]

    Lakoff, Explaining Embodied Cognition Results, Topics in Cognitive Science 4 (2012) 773–785

    G. Lakoff, Explaining Embodied Cognition Results, Topics in Cognitive Science 4 (2012) 773–785. doi:10.1111/j.1756-8765. 2012.01222.x

  7. [7]

    P. H. Thibodeau, L. Boroditsky, Natural Language Metaphors Covertly Influence Reasoning, PLOS ONE 8 (2013) e52961. doi:10.1371/journal.pone.0052961

  8. [8]

    Korsakova-Kreyn, Two-level Model of Embodied Cognition in Music, Psychomusicology: Music, Mind, and Brain 28 (2018) 240–259

    M. Korsakova-Kreyn, Two-level Model of Embodied Cognition in Music, Psychomusicology: Music, Mind, and Brain 28 (2018) 240–259. doi:10.1037/pmu0000228

  9. [9]

    Virtala, M

    P. Virtala, M. Huotilainen, E. Partanen, V. Fellman, M. Ter- vaniemi, Newborn Infants’ Auditory System is Sensitive to Western Music Chord Categories, Frontiers in Psychology 4 (2013) 492. doi: 10.3389/fpsyg.2013.00492

  10. [10]

    G. M. Bidelman, A. Krishnan, Neural Correlates of Consonance, Dissonance, and the Hierarchy of Musical Pitch in the Human Brainstem, Journal of Neuroscience 29 (2009) 13165–13171. doi:10.1523/jneurosci.3900-09.2009

  11. [11]

    Leman, An Embodied Approach to Music Seman- tics, Musicae Scientiae 14 (2010) 43–67

    M. Leman, An Embodied Approach to Music Seman- tics, Musicae Scientiae 14 (2010) 43–67. doi: 10.1177/ 10298649100140S104

  12. [12]

    Koelsch, P

    S. Koelsch, P. Vuust, K. Friston, Predictive Processes and the Peculiar Case of Music, Trends in Cognitive Sciences 23 (2019) 63–77. doi:10.1016/j.tics.2018.10.006

  13. [13]

    Pereira, B

    F. Pereira, B. Lou, B. Pritchett, S. Ritter, S. J. Gersh- man, N. Kanwisher, M. Botvinick, E. Fedorenko, Toward a Universal Decoder of Linguistic Meaning from Brain Acti- vation, Nature Communications 9 (2018) 963. doi: 10.1038/ s41467-018-03068-4

  14. [14]

    A. E. Hoerl, R. W. Kennard, Ridge Regression: Biased Esti- mation for Nonorthogonal Problems, Technometrics 12 (1970) 55–67. doi:10.1080/00401706.1970.10488634

  15. [15]

    Pennington, R

    J. Pennington, R. Socher, C. D. Manning, GloVe: Global Vec- tors for Word Representation, in: Proceedings of the 2014 Con- ference on Empirical Methods in Natural Language Processing, 2014, pp. 1532–1543. doi: 10.3115/v1/d14-1162

  16. [16]

    M. A. Casey, Music of the 7Ts: Predicting and Decoding Mul- tivoxel fMRI Responses with Acoustic, Schematic, and Cate- gorical Music Features, Frontiers in Psychology 8 (2017) 1179. doi:10.3389/fpsyg.2017.01179

  17. [17]

    Cortes, V

    C. Cortes, V. Vapnik, Support-vector Networks, Machine Learning 20 (1995) 273–297. doi: 10.1007/BF00994018

  18. [18]

    Hanke, R

    M. Hanke, R. Dinga, C. H¨ ausler, J. S. Guntupalli, M. Casey, F. R. Kaule, J. Stadler, High-resolution 7-Tesla fMRI Data on the Perception of Musical Genres - An Extension to the Study- forrest Dataset, F1000Research 4 (2015) 174. doi: 10.12688/ f1000research.6679.1

  19. [19]

    Abraham, F

    A. Abraham, F. Pedregosa, M. Eickenberg, P. Gervais, A. Mueller, J. Kossaifi, A. Gramfort, B. Thirion, G. Varo- quaux, Machine learning for neuroimaging with scikit-learn, Frontiers in Neuroinformatics 8 (2014) 14. doi: 10.3389/fninf. 2014.00014

  20. [20]

    Hotelling, Relations Between Two Sets of Variates, Biometrika 28 (1936) 321–377

    H. Hotelling, Relations Between Two Sets of Variates, Biometrika 28 (1936) 321–377. doi: 10.2307/2333955

  21. [21]

    Horst, Generalized Canonical Correlations and their Appli- cations to Experimental Data, Journal of Clinical Psychology 17 (1961) 331–347

    P. Horst, Generalized Canonical Correlations and their Appli- cations to Experimental Data, Journal of Clinical Psychology 17 (1961) 331–347. doi:10.1002/1097-4679(196110)17:4<331:: aid-jclp2270170402>3.0.co;2-d

  22. [22]

    J. R. Kettenring, Canonical Analysis of Several Sets of Vari- ables, Biometrika 58 (1971) 433–451. doi: 10.1093/biomet/58. 3.433

  23. [23]

    F. A. Raposo, D. M. de Matos, R. Ribeiro, Learning Embodied Semantics via Music and Dance Semiotic Correlations, CoRR abs/1903.10534 (2019)

  24. [24]

    Y. Yu, S. Tang, F. Raposo, L. Chen, Deep Cross-modal Corre- lation Learning for Audio and Lyrics in Music Retrieval, ACM Transactions on Multimedia Computing, Communications, and Applications 15 (2019) 20. doi: 10.1145/3281746

  25. [25]

    Bestgen, Exact Expected Average Precision of the Ran- dom Baseline for System Evaluation, The Prague Bulletin of Mathematical Linguistics 103 (2015) 131–138

    Y. Bestgen, Exact Expected Average Precision of the Ran- dom Baseline for System Evaluation, The Prague Bulletin of Mathematical Linguistics 103 (2015) 131–138. doi: 10.1515/ pralin-2015-0007. 6