pith. sign in

arxiv: 1906.09264 · v2 · pith:FQJIBPMKnew · submitted 2019-06-21 · 🧬 q-bio.NC · cs.AI· cs.LG· cs.NE· stat.ML

Visualizing Representational Dynamics with Multidimensional Scaling Alignment

Baihan Lin , Marieke Mur , Tim Kietzmann , Nikolaus Kriegeskorte This is my paper

Pith reviewed 2026-05-25 18:44 UTC · model grok-4.3

classification 🧬 q-bio.NC cs.AIcs.LGcs.NEstat.ML
keywords representational similarity analysismultidimensional scalingProcrustes alignmentrepresentational dynamicsobject categorizationinferotemporal cortexneural recordingsRDM movies
0
0 comments X

The pith

Procrustes-aligned MDS on successive RDMs visualizes the time course of object representations in monkey IT cortex.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a pipeline that turns sequences of representational dissimilarity matrices into movies and then aligns their low-dimensional embeddings across time using Procrustes analysis. Applied to neural recordings from monkey inferotemporal cortex, the aligned embeddings track how the geometry of responses to different object categories changes from stimulus onset onward. The results indicate that these visualizations capture genuine category-specific dynamics and point to object categorization as a process that unfolds in stages, possibly with recurrence. A reader would care because the time evolution of representational geometry can distinguish competing accounts of how the brain achieves visual recognition.

Core claim

The authors formulate a pipeline using RDM movies and Procrustes-aligned Multidimensional Scaling (pMDS) to analyze representational geometry as it changes over the time course from stimulus onset to offset. When this pipeline is applied to neural recording data from monkey IT cortex, the multidimensional scaling alignment captures the dynamics of the category-specific representation spaces and indicates that object categorization may be hierarchical, multi-staged, and oscillatory or recurrent.

What carries the argument

Procrustes-aligned Multidimensional Scaling (pMDS) applied to time-resolved RDMs, which aligns successive low-dimensional embeddings to maintain continuity while revealing changes in representational geometry.

If this is right

  • RDM movies can be visualized in multiple ways once their embeddings are aligned across time.
  • Category representations in IT cortex follow a hierarchical structure.
  • The categorization process occurs in multiple stages.
  • Representational changes can include oscillatory or recurrent components.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same alignment procedure could be applied to data from earlier visual areas to test whether similar dynamics appear at different processing stages.
  • If the method reliably tracks dynamics, it could serve as a benchmark for comparing the time course of representations in biological and artificial networks.
  • Extensions to human EEG or fMRI time series would allow similar visualization of representational trajectories in non-invasive recordings.

Load-bearing premise

Aligning MDS embeddings from one time point to the next with Procrustes analysis preserves the true underlying representational dynamics rather than creating artificial continuity.

What would settle it

Demonstrating that the visualized trajectories change substantially when different random seeds are used for the MDS step or when a different alignment method is substituted would show that the observed dynamics are not robust.

Figures

Figures reproduced from arXiv: 1906.09264 by Baihan Lin, Marieke Mur, Nikolaus Kriegeskorte, Tim Kietzmann.

Figure 1
Figure 1. Figure 1: 3D plot of Procrustes-aligned MDS over time. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Changing areas of convex hulls over time. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: MDS embeddings over time (dim0, dim1, the dis [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 6
Figure 6. Figure 6: After offset (300-800ms) [PITH_FULL_IMAGE:figures/full_fig_p004_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: RDM and MDS at example time points (onset at [PITH_FULL_IMAGE:figures/full_fig_p004_7.png] view at source ↗
read the original abstract

Representational similarity analysis (RSA) has been shown to be an effective framework to characterize brain-activity profiles and deep neural network activations as representational geometry by computing the pairwise distances of the response patterns as a representational dissimilarity matrix (RDM). However, how to properly analyze and visualize the representational geometry as dynamics over the time course from stimulus onset to offset is not well understood. In this work, we formulated the pipeline to understand representational dynamics with RDM movies and Procrustes-aligned Multidimensional Scaling (pMDS), and applied it to neural recording of monkey IT cortex. Our results suggest that the the multidimensional scaling alignment can genuinely capture the dynamics of the category-specific representation spaces with multiple visualization possibilities, and that object categorization may be hierarchical, multi-staged, and oscillatory (or recurrent).

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces a pipeline combining time-binned representational dissimilarity matrices (RDMs), classical multidimensional scaling (MDS), and Procrustes alignment (pMDS) to visualize representational dynamics in neural data. Applied to monkey IT cortex recordings, it claims that pMDS captures genuine category-specific dynamics with multiple visualization options and that object categorization is hierarchical, multi-staged, and oscillatory/recurrent.

Significance. If the alignment step can be shown not to fabricate continuity, the method would supply a practical visualization extension to RSA for time-resolved data; the absence of any quantitative validation, baseline comparison, or artifact test currently limits its contribution to an untested applied tool.

major comments (2)
  1. [Abstract] Abstract: the claim that pMDS 'can genuinely capture the dynamics' is load-bearing for all downstream interpretations, yet no ground-truth simulation, permutation test, or comparison against unaligned MDS is described to demonstrate that Procrustes alignment recovers rather than imposes continuity when underlying geometries may change discontinuously.
  2. [Abstract] Abstract: the suggestion that categorization 'may be hierarchical, multi-staged, and oscillatory' rests on visual inspection of the aligned embeddings, but the manuscript provides neither error bars, statistical tests on the observed stages/oscillations, nor quantitative comparison to simpler time-resolved RSA measures that would be required to support these interpretations.
minor comments (1)
  1. [Abstract] Abstract contains a repeated word ('the the').

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. We address each major comment below and commit to revisions that directly respond to the concerns raised.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that pMDS 'can genuinely capture the dynamics' is load-bearing for all downstream interpretations, yet no ground-truth simulation, permutation test, or comparison against unaligned MDS is described to demonstrate that Procrustes alignment recovers rather than imposes continuity when underlying geometries may change discontinuously.

    Authors: We agree that the manuscript does not currently include ground-truth simulations, permutation tests, or explicit comparisons to unaligned MDS to demonstrate that alignment recovers rather than imposes continuity. While Procrustes alignment is a standard method that minimizes squared distances between configurations without altering internal geometries, the absence of such validation is a valid limitation. In the revised manuscript we will add a dedicated simulation section using synthetic RDM movies with controlled discontinuous and continuous changes, include permutation tests for alignment quality, and directly compare aligned versus unaligned MDS trajectories on both synthetic and real data. revision: yes

  2. Referee: [Abstract] Abstract: the suggestion that categorization 'may be hierarchical, multi-staged, and oscillatory' rests on visual inspection of the aligned embeddings, but the manuscript provides neither error bars, statistical tests on the observed stages/oscillations, nor quantitative comparison to simpler time-resolved RSA measures that would be required to support these interpretations.

    Authors: The claims about hierarchical, multi-staged, and oscillatory categorization are presented as interpretive suggestions arising from the pMDS visualizations rather than as statistically tested conclusions. We acknowledge that visual inspection alone, without error bars, formal tests, or comparisons to baseline time-resolved RSA metrics, is insufficient to support these interpretations robustly. In revision we will add bootstrap-derived error bars or confidence regions to the embeddings, apply permutation-based statistical tests to evaluate the significance of identified stages and oscillations, and include quantitative comparisons (e.g., time-series correlations) against simpler time-resolved RSA measures such as evolving RDM correlations. revision: yes

Circularity Check

0 steps flagged

No circularity: applied visualization pipeline with standard components

full rationale

The paper describes an analysis pipeline that computes RDMs per time bin from neural recordings, embeds each via MDS, and applies Procrustes alignment to successive embeddings for visualization. No derivation, equation, or claim reduces a result or 'prediction' to its own inputs by construction. Conclusions about hierarchical or oscillatory categorization are presented as observations from the visualizations rather than forced by any fitted parameter or self-citation. The method relies on established techniques (RSA, MDS, Procrustes) without uniqueness theorems or ansatzes imported from the authors' prior work in a load-bearing way. This is a self-contained applied tool whose outputs are data-driven and externally falsifiable against the raw recordings.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no explicit free parameters, axioms, or invented entities; insufficient detail to populate ledger entries.

pith-pipeline@v0.9.0 · 5679 in / 943 out tokens · 22401 ms · 2026-05-25T18:44:54.092037+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

11 extracted references · 11 canonical work pages

  1. [1]

    write newline

    " write newline " cite write " FUNCTION editor.postfix editor num.names #1 > "( )" "( )" if FUNCTION editor.trans.postfix editor num.names #1 > "( )" "( )" if FUNCTION trans.postfix translator num.names #1 > "( )" "( )" if FUNCTION authors.editors.reflist.apa5 'field := 'dot := field num.names 'numnames := numnames 'format.num.names := format.num.names na...

    work page

  2. [2]

    , Malecek, N J

    bell2011relationship APACrefauthors Bell, A H. , Malecek, N J. , Morin, E L. , Hadj-Bouziane, F. , Tootell, R B. \ Ungerleider, L G. APACrefauthors \ 2011 . Relationship between functional magnetic resonance imaging-identified regions and neuronal category selectivity Relationship between functional magnetic resonance imaging-identified regions and neuron...

    work page 2011

  3. [3]

    , Swayne, D F

    buja2008data APACrefauthors Buja, A. , Swayne, D F. , Littman, M L. , Dean, N. , Hofmann, H. \ Chen, L. APACrefauthors \ 2008 . Data visualization with multidimensional scaling Data visualization with multidimensional scaling . Journal of Computational and Graphical Statistics 17 2 444--472

    work page 2008

  4. [4]

    \ Tsao, D Y

    freiwald2010functional APACrefauthors Freiwald, W A. \ Tsao, D Y. APACrefauthors \ 2010 . Functional compartmentalization and viewpoint generalization within the macaque face-processing system Functional compartmentalization and viewpoint generalization within the macaque face-processing system . Science 330 6005 845--851

    work page 2010

  5. [5]

    APACrefauthors \ 1975

    gower1975generalized APACrefauthors Gower, J C. APACrefauthors \ 1975 . Generalized procrustes analysis Generalized procrustes analysis . Psychometrika 40 1 33--51

    work page 1975

  6. [6]

    , Kreiman, G

    hung2005fast APACrefauthors Hung, C P. , Kreiman, G. , Poggio, T. \ DiCarlo, J J. APACrefauthors \ 2005 . Fast readout of object identity from macaque inferior temporal cortex Fast readout of object identity from macaque inferior temporal cortex . Science 310 5749 863--866

    work page 2005

  7. [7]

    \ Kriegeskorte, N

    khaligh2014deep APACrefauthors Khaligh-Razavi, S M. \ Kriegeskorte, N. APACrefauthors \ 2014 . Deep supervised, but not unsupervised, models may explain IT cortical representation Deep supervised, but not unsupervised, models may explain it cortical representation . PLoS computational biology 10 11 e1003915

    work page 2014

  8. [8]

    , Spoerer, C J

    kietzmann2019recurrence APACrefauthors Kietzmann, T C. , Spoerer, C J. , S \"o rensen, L. , Cichy, R M. , Hauk, O. \ Kriegeskorte, N. APACrefauthors \ 2019 . Recurrence required to capture the dynamic computations of the human ventral visual stream Recurrence required to capture the dynamic computations of the human ventral visual stream . arXiv preprint ...

    work page arXiv 2019

  9. [9]

    APACrefauthors \ 2015

    kriegeskorte2015deep APACrefauthors Kriegeskorte, N. APACrefauthors \ 2015 . Deep neural networks: a new framework for modeling biological vision and brain information processing Deep neural networks: a new framework for modeling biological vision and brain information processing . Annual review of vision science 1 417--446

    work page 2015

  10. [10]

    \ Kievit, R A

    kriegeskorte2013representational APACrefauthors Kriegeskorte, N. \ Kievit, R A. APACrefauthors \ 2013 . Representational geometry: integrating cognition, computation, and the brain Representational geometry: integrating cognition, computation, and the brain . Trends in cognitive sciences 17 8 401--412

    work page 2013

  11. [11]

    \ DiCarlo, J J

    yamins2016using APACrefauthors Yamins, D L. \ DiCarlo, J J. APACrefauthors \ 2016 . Using goal-driven deep learning models to understand sensory cortex Using goal-driven deep learning models to understand sensory cortex . Nature neuroscience 19 3 356

    work page 2016