REVIEW 3 major objections 6 minor 298 references
Multimodal image-tabular rankings flip with the tabular encoder; a simple fusion with a strong encoder matches a heavy engineered method.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-10 19:00 UTC pith:BZJBWIGP
load-bearing objection Solid empirical re-ranking study: tabular encoder choice is a real confound in image-tabular fusion, and a simple bilinear baseline with strong ICL features matches a heavy engineered method. the 3 major comments →
The Importance of Encoder Choice:A Tabular-Image Study
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On likely-multimodal image-tabular datasets, multimodal rankings are not stable across tabular encoders, so conclusions drawn from a single encoder do not generalize. A simple bilinear fusion paired with a strong tabular encoder (especially non-partitioned extraction from TabPFN) performs on par with a carefully engineered multimodal method that uses on average 13.2 times more parameters and a dedicated pretraining stage. The observed fusion lift itself shrinks as the unimodal tabular baseline improves, and the context-query representation shift of in-context learning tabular models is not limited to TabPFN.
What carries the argument
Three ICL feature-extraction schemes (Vanilla context-role, leave-one-fold-out query-role, and non-partitioned query-role on the full training set) plus Excess Distribution Shift (EDS), an MMD-based gap that isolates context-query divergence from ordinary data split variation; these let the authors place train and test embeddings in a shared latent space and measure whether that space is usable for fusion.
Load-bearing premise
The paper assumes that a stronger unimodal tabular predictor also yields better frozen representations for multimodal fusion; if predictive accuracy and representation quality diverge, the encoder-sensitivity story collapses.
What would settle it
Re-run the same bilinear-versus-TIP comparison on the same likely-multimodal datasets with an additional strong tabular encoder whose unimodal F1 is high but whose frozen embeddings yield no multimodal lift, or show that Vanilla extraction no longer degrades fusion once a different image encoder or fusion head is used.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that tabular encoder choice is a critical and previously neglected confound in image-tabular multimodal learning. Prior work largely used plain MLPs; the authors systematically replace them with SOTA tabular models (TabPFNv2, TabDPT, TabICLv2 under three extraction schemes, TARTE, TabM/TabM-SSL, TabVec) while freezing a ViT-B/16 image encoder and comparing a simple bilinear fusion baseline against the engineered TIP pipeline and unimodal linear heads. They introduce Relative Percentile Rank (RPR) and Excess Distribution Shift (EDS via multi-bandwidth MMD) and, on seven datasets, show that (i) method rankings are unstable across encoders, (ii) fusion can hurt on modality-dominated data, (iii) the multimodal lift shrinks with stronger unimodal tabular performance (OLS slope β̂=0.434 on likely-multimodal sets), (iv) bilinear+strong encoder matches TIP at ~13 imes fewer parameters, and (v) the context-query representation shift of ICL TFMs is not limited to TabPFNv2 and is best mitigated by non-partitioned (NP) extraction.
Significance. If the empirical pattern holds, it is a useful methodological correction for multimodal tabular-image work: many published fusion gains may be partly encoder artifacts, and simple bilinear fusion with a modern tabular foundation model can be competitive with carefully engineered pipelines. The paper also supplies the first multi-model quantification of the ICL context-query shift (EDS, Table 2, Fig. 5) and a practical recommendation (prefer NP over Vanilla). Strengths include nested 4-inner/2-outer CV, TPE HPO, 95% CIs, critical-difference diagrams, an explicit OLS null of constant lift (β=1), and released code/per-trial logs. The contribution is primarily empirical and cautionary rather than a new architecture; its value lies in re-calibrating how the community benchmarks multimodal methods when a tabular modality is present.
major comments (3)
- [Introduction / §4.3] The central premise stated in the Introduction—that a stronger unimodal tabular predictor necessarily yields better frozen representations for multimodal fusion—is used to motivate the encoder sweep but is never tested directly. Representation quality and predictive accuracy can diverge (e.g., via label leakage into the embedding or via features that are predictive alone but redundant with the image). A short diagnostic that correlates unimodal F1 with a pure representation metric (linear probe on held-out tabular labels after freezing, or mutual information with the target after residualizing the image) would make the premise falsifiable and would strengthen the interpretation of the OLS slope in §4.3 / Fig. 3.
- [§3 / §5 Limitations] All image representations come from a single frozen ViT-B/16 CLS token (§3). The authors correctly flag this in Limitations, yet the dataset typology (image-dominated / tabular-dominated / likely multimodal) and the claim that fusion lift shrinks with encoder quality are both conditioned on this choice. A stronger or architecturally different image encoder could reclassify datasets and alter the OLS slope. At minimum the paper should report a sensitivity check on one likely-multimodal dataset with a second image backbone (e.g., ResNet-50 or a larger ViT) so that the encoder-sensitivity conclusion is not itself image-encoder-specific.
- [§4.3 / Fig. 3] The OLS analysis in §4.3 treats each (encoder, dataset) point as independent when fitting Y=βX+b on likely-multimodal data and reports a one-sided p=2.5e-57 for β=1. With only three datasets and multiple extraction schemes of the same ICL model, the effective sample size is small and points are correlated. A mixed-effects or clustered bootstrap that respects dataset and model family would give a more credible confidence interval on β and on the claimed 0.566 reduction in multimodal advantage per unit unimodal F1.
minor comments (6)
- [Abstract] Abstract and Introduction list five numbered findings while the body has five correspondingly numbered subsections; the abstract’s fifth finding is the ICL extraction recommendation, which is fine, but the numbering in the abstract jumps from 4 to 5 without an explicit “5.” in the printed text of some renderings—please check consistency.
- [Fig. 2] Figure 2 is dense (filled vs hatched bars, two CI styles, many encoders). A short legend sentence in the caption that explicitly maps star/dagger/wrenches to Vanilla/LOFO/NP would help readers who do not keep §2.3 open.
- [§3.1 / Eq. (4)] Eq. (4) for bilinear fusion uses W∈R^{d imes C imes d}; a one-line note that this is the standard multi-class bilinear form of Jayakumar et al. (and that setting one modality to zero recovers the unimodal linear head) would make the reduction claim fully self-contained.
- [Table 1] Table 1 reports image size quantiles as √(W imes H); the notation “Img √W imes H” is slightly ambiguous at first glance—consider “√(area)” or “side length of equivalent square”.
- [§4.1] In §4.1 the win/tie/loss counts for TIP vs baselines are given as (W/T/L) without stating whether ties are defined by overlapping 95% CIs or by exact F1 equality; a one-sentence definition would remove ambiguity.
- [References / header] References: TabICLv2 is cited as Qu et al. 2026 in the text and 2025 in the bibliography in places; unify the year. Also, the arXiv ID in the header (2607.07756) is future-dated relative to the content—confirm the intended citation key.
Circularity Check
No significant circularity: pure empirical encoder sweep with independently defined metrics (RPR, EDS, F1 ranks, OLS slope) that do not reduce to inputs by construction.
full rationale
The paper is an empirical study of tabular encoder choice in image-tabular fusion. Its load-bearing claims (encoder-dependent re-ranking of TIP vs. bilinear, fusion lift shrinking with unimodal strength, context-query shift across ICL TFMs) are established by nested CV F1, RPR relative to a fixed image baseline, EDS (MMD gaps), critical-difference diagrams, and an OLS regression of multimodal vs. unimodal F1 whose null is the external hypothesis eta=1. None of these quantities is defined in terms of the result being claimed, fitted to the target and then re-presented as a prediction, or forced by a self-citation uniqueness theorem. Extraction schemes (Vanilla/LOFO/NP) follow Ye et al. and are evaluated, not assumed. Self-citations are ordinary background (TabPFN, TabDPT, TIP, TabM) and do not underwrite the encoder-sensitivity finding. The study is therefore self-contained against its own experimental protocol; circularity score is zero.
Axiom & Free-Parameter Ledger
free parameters (3)
- shared embedding dimension d =
selected per trial from the discrete set
- learning rate, weight decay, label smoothing, dropout rates =
per-dataset, per-encoder via TPE
- LOFO fold count K =
10
axioms (4)
- ad hoc to paper A stronger unimodal tabular predictor yields better frozen representations for multimodal fusion.
- domain assumption Frozen pretrained encoders plus a linear LayerNorm projection isolate the effect of the downstream fusion module.
- domain assumption Bilinear product (Jayakumar et al.) is a fair naïve multimodal baseline that reduces to the unimodal linear classifier when one modality is zero.
- standard math Maximum Mean Discrepancy with multi-bandwidth RBF kernel quantifies representation shift beyond ordinary train-val variation.
invented entities (2)
-
Relative Percentile Rank (RPR)
independent evidence
-
Excess Distribution Shift (EDS)
independent evidence
read the original abstract
Multimodal learning usually requires a dedicated encoder per modality. When a tabular modality is involved, prior work has been mostly using a \emph{plain MLP} as the encoder. Yet if it were a strong encoder, the tabular domain would not be ``the last unconquered castle for deep learning''. This study evaluates state-of-the-art tabular models as encoders in the image-tabular setting for the first time. An obstacle stands out. In-Context Learning models, among the best performing methods in the tabular domain, require labels to process instances, making it non-trivial to embed training and test instances the same way. We addressed this problem across multiple models of this family. With this study, we would like to highlight the importance of encoder factor in the multimodal learning.
Figures
Reference graph
Works this paper leans on
-
[1]
[Neo4j-Tut] Jimwebber/Neo4j-Tutorial -
- [2]
-
[3]
Adaptive Boosting for Transfer Learning Using Dynamic Updates
-
[4]
Google Docs , urldate =
-
[5]
Bayesian Dark Knowledge , urldate =
-
[6]
ACM Conferences , doi =
- [7]
- [8]
- [9]
-
[10]
Diagrams.Net , urldate =
-
[11]
EUR-Lex - 31995L0046 - EN , urldate =
Directive 95/46/. EUR-Lex - 31995L0046 - EN , urldate =
-
[12]
Hella Provides Lane-Change Warning for
-
[13]
Index of File:///Home/Lst/Books/ , urldate =
-
[14]
An Integrated Growing-Pruning Method for Feedforward Network Training -
-
[15]
Jimwebber/Neo4j-Tutorial -
-
[16]
Log-Based Predictive Maintenance , urldate =
- [17]
- [18]
- [19]
-
[20]
Random Permutations Fix a Worst Case for Cyclic Coordinate Descent
-
[21]
Chemical & Engineering News , urldate =
Simulations. Chemical & Engineering News , urldate =
-
[22]
Teaching , urldate =
-
[23]
Ukraine Dekret 117 -
-
[24]
Zotero - Ismllwiki , urldate =
-
[25]
doi:10.1109/ICADIWT.2008.4664413 , urldate =
Realizing Process Interoperability Using. doi:10.1109/ICADIWT.2008.4664413 , urldate =
-
[26]
Wiley. Statistical. doi:10.1002/9781119013563.scard , urldate =
-
[27]
Partial Sorting , year = 2013, month = may, journal =
work page 2013
- [28]
- [29]
- [30]
-
[31]
Bundesministerium f\"ur Bildung und Forschung - BMBF , urldate =
-
[32]
Price Elasticity of Demand , year = 2025, month = mar, journal =
work page 2025
- [33]
-
[34]
User Modeling and User-Adapted Interaction , volume =
Modeling. User Modeling and User-Adapted Interaction , volume =
-
[35]
User Modeling and User-Adapted Interaction , volume =
Knowledge. User Modeling and User-Adapted Interaction , volume =
-
[36]
Aalst, Wil M. P. and Song, Minseok , editor =. Mining. Business
-
[37]
Brendan and Mironov, Ilya and Talwar, Kunal and Zhang, Li , year = 2016, series =
Abadi, Martin and Chu, Andy and Goodfellow, Ian and McMahan, H. Brendan and Mironov, Ilya and Talwar, Kunal and Zhang, Li , year = 2016, series =. Deep. Proceedings of the 2016. doi:10.1145/2976749.2978318 , urldate =
-
[38]
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, Mart. Tensorflow:. arXiv preprint arXiv:1603.04467 , eprint =
work page internal anchor Pith review Pith/arXiv arXiv
-
[39]
Large Scale Tag Recommendation Using Different Image Representations , booktitle =
Abbasi, Rabeeh and Grzegorzek, Marcin and Staab, Steffen , year = 2009, pages =. Large Scale Tag Recommendation Using Different Image Representations , booktitle =
work page 2009
-
[40]
Rajendra and Makarenkov, Vladimir and Nahavandi, Saeid , year = 2021, month = dec, journal =
Abdar, Moloud and Pourpanah, Farhad and Hussain, Sadiq and Rezazadegan, Dana and Liu, Li and Ghavamzadeh, Mohammad and Fieguth, Paul and Cao, Xiaochun and Khosravi, Abbas and Acharya, U. Rajendra and Makarenkov, Vladimir and Nahavandi, Saeid , year = 2021, month = dec, journal =. A Review of Uncertainty Quantification in Deep Learning:. doi:10.1016/j.inff...
-
[41]
Adaptive Optimization of Hyperparameters in
-
[42]
Sparse-posterior Gaussian Processes for general likelihoods
Sparse-Posterior. arXiv preprint arXiv:1203.3507 , eprint =
work page internal anchor Pith review Pith/arXiv arXiv
-
[43]
Applying Convolutional Neural Networks Concepts to Hybrid. 2012
work page 2012
-
[44]
Exploring Convolutional Neural Network Structures and Optimization Techniques for Speech Recognition. , booktitle =
-
[45]
IEEE/ACM Transactions on audio, speech, and language processing , volume =
Convolutional Neural Networks for Speech Recognition , author =. IEEE/ACM Transactions on audio, speech, and language processing , volume =
-
[46]
Abdelmalak, Ibram and Madhusudhanan, Kiran and Choi, Jungmin and Kl. Channel. Advances in. doi:10.1007/978-981-92-1462-4_46 , urldate =
-
[47]
Abdennadher, K. and Venet, P. and Rojat, G. and Retif, J. and Rosset, C. , year = 2010, month = jul, journal =. A. doi:10.1109/TIA.2010.2049972 , abstract =
-
[48]
Explainable Matrix Factorization for Collaborative Filtering , booktitle =
Abdollahi, Behnoush and Nasraoui, Olfa , year = 2016, pages =. Explainable Matrix Factorization for Collaborative Filtering , booktitle =
work page 2016
-
[49]
Using Explainability for Constrained Matrix Factorization , booktitle =
Abdollahi, Behnoush and Nasraoui, Olfa , year = 2017, pages =. Using Explainability for Constrained Matrix Factorization , booktitle =
work page 2017
-
[50]
Advances in Neural Information Processing Systems , volume =
A Definition of Continual Reinforcement Learning , author =. Advances in Neural Information Processing Systems , volume =
-
[51]
Abid, Lobna and Masmoudi, Afif and Ghorbel, Sonia Zouari , year = 2016, journal =. The. doi:10.18488/journal.aefr/2016.6.1/102.1.27.42 , urldate =
work page doi:10.18488/journal.aefr/2016.6.1/102.1.27.42 2016
- [52]
-
[53]
Abouzeid, Azza and Pawlikowski, Kamil Bajda and Abadi, Daneil , year = 2009, month = aug, journal =
work page 2009
-
[54]
Abouzeid, A. and. Proceedings of the VLDB Endowment , volume =
-
[55]
DNF-Net: A Neural Architecture for Tabular Data
Abutbul, Ami and Elidan, Gal and Katzir, Liran and. doi:10.48550/arXiv.2006.06465 , urldate =. arXiv , keywords =:2006.06465 , primaryclass =
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2006.06465 2006
-
[56]
Acar, E. and Dunlavy, D. M. and Kolda, T. G. , year = 2009, pages =. Link Prediction on Evolving Data Using Matrix and Tensor Factorizations , booktitle =
work page 2009
-
[57]
All-at-once Optimization for Coupled Matrix and Tensor Factorizations
Acar, Evrim and Kolda, Tamara G. and Dunlavy, Daniel M. , year = 2011, month = may, journal =. All-at-Once. arXiv , keywords =:1105.3422 , primaryclass =
work page internal anchor Pith review Pith/arXiv arXiv 2011
- [58]
-
[59]
International Journal of Production Research , volume =
Solving Inventory Routing with Transshipment and Substitution under Dynamic and Stochastic Demands Using Genetic Algorithm and Deep Reinforcement Learning , author =. International Journal of Production Research , volume =. doi:10.1080/00207543.2021.1987549 , urldate =
-
[60]
Acharya, Deepak Bhaskar and Kuppan, Karthigeyan and Bhaskaracharya, Divya , year = 2025, journal =. Agentic. doi:10.1109/ACCESS.2025.3532853 , file =
-
[61]
Achiam, Josh and Adler, Steven and Agarwal, Sandhini and Ahmad, Lama and Akkaya, Ilge and Aleman, Florencia Leoni and Almeida, Diogo and Altenschmidt, Janko and Altman, Sam and Anadkat, Shyamal and Avila, Red and Babuschkin, Igor and Balaji, Suchir and Balcom, Valerie and Baltescu, Paul and Bao, Haiming and Bavarian, Mohammad and Belgum, Jeff and Bello, I...
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2303.08774
-
[62]
Acikalin, Utku Umur and Ferber, Aaron M. and Gomes, Carla P. , year = 2024, month = oct, urldate =. Learning to. The
work page 2024
-
[63]
Information Diffusion in Online Social Networks , author =
-
[64]
Adar, Eytan and Weld, Daniel S. and Bershad, Brian N. and Gribble, Steven S. , year = 2007, pages =. Why We Search , booktitle =. doi:10.1145/1242572.1242595 , urldate =
-
[65]
Adavanne, Sharath and Politis, Archontis and Virtanen, Tuomas , year = 2018, month = jan, journal =. Multichannel. arXiv , keywords =:1801.09522 , primaryclass =
work page internal anchor Pith review Pith/arXiv arXiv 2018
-
[66]
Addanki, Ravichandra and Nair, Vinod and Alizadeh, Mohammad , langid =. Neural
-
[67]
Global Optimization Using Local Searches , author =. Unpublished doctoral dissertation, Global Optimization Laboratory, University of Florence , urldate =
-
[68]
Adomavicius, G. and Tuzhilin, A. , year = 2001, journal =. Multidimensional Recommender Systems: A Data Warehousing Approach , shorttitle =
work page 2001
-
[69]
Using Data Mining Methods to Build Customer Profiles , author =. Computer , volume =. doi:10.1109/2.901170 , urldate =
-
[70]
ACM Transactions on Information Systems (TOIS) , volume =
Incorporating Contextual Information in Recommender Systems Using a Multidimensional Approach , author =. ACM Transactions on Information Systems (TOIS) , volume =
-
[71]
Adomavicius, G. and Tuzhilin, A. , year = 2005, journal =. Toward the next Generation of Recommender Systems:
work page 2005
-
[72]
IEEE Intelligent Systems , pages =
New Recommendation Techniques for Multicriteria Rating Systems , author =. IEEE Intelligent Systems , pages =
-
[73]
Adomavicius, G. and Tuzhilin, A. , year = 2008, volume =. Context-Aware Recommender Systems , booktitle =
work page 2008
-
[74]
Recommender Systems Handbook , pages =
Multi-Criteria Recommender Systems , author =. Recommender Systems Handbook , pages =
- [75]
-
[76]
Proceedings of the VLDB Endowment , volume =
Building a High-Level Dataflow System on Top of. Proceedings of the VLDB Endowment , volume =
-
[77]
Afrati, Foto N. and Ullman, Jeffrey D. , year = 2010, pages =. Optimizing Joins in a Map-Reduce Environment , booktitle =
work page 2010
-
[78]
doi:10.5445/KSP/1000058749/23 , urldate =
Aga, Rosa Tsegaye and Wartena, Christian and Lange, Otto and Aders, Nelleke , year = 2017, journal =. doi:10.5445/KSP/1000058749/23 , urldate =
-
[79]
Agarwal, Deepak and Chen, Bee-Chung and Elango, Pradheep , year = 2009, month = dec, pages =. Explore/. Proceedings of. doi:10.1109/ICDM.2009.52 , urldate =
-
[80]
Agarwal, D. and Chen, B. C , year = 2009, pages =. Regression-Based Latent Factor Models , booktitle =
work page 2009
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.