REVIEW 4 major objections 5 minor 1 cited by
Not Every AI Problem is a Data Problem: We Should Be Intentional About Data Scaling
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper's thesis is that data scaling should be task-specific: the shape of a task's data, not its volume, determines whether more data will help.
desk verdict A clear, well-cited position essay whose central TDA-based prediction is a promising hypothesis but is never operationalized or tested; worth reading, not yet worth citing as a result. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is topological data analysis (TDA), defined as the use of topological methods to identify intrinsic dimensions, compositional patterns, and structural patterns in datasets. In this paper TDA supplies a vocabulary for 'shape of data,' specifically the persistence and stability of structures across multiple scales, which the authors use to separate tasks that will benefit from data scaling from tasks that will not. A second mechanism is the data-acquisition feasibility check: availability, verifiability, cost, and benchmark quality determine whether scaling is viable in practice. Together the two mechanisms form the paper's proposed framework for intentional data acquisition.
What would settle it
Compute persistent-homology summaries of the training data for a diverse set of task families, then measure the performance gain per additional training token for each family; if families with stable, persistent multi-scale structure do not show systematically larger gains than families with fragmented structure, the paper's central claim is refuted.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that the shape of data, understood through topological data analysis, carries predictive clues about where data scaling will succeed. The authors maintain that stable features appearing across multiple scales of granularity indicate that a task is well suited to more data, because the model can keep exploiting recurring structure; where structure is unstable, fragmented, or effectively unbounded, extra data yields diminishing or negligible gains. They illustrate this contrast with machine translation, robotics, and drug discovery on the favorable side and journalistic fact-checking and misinformation detection on the unfavorable side. The paper also asserts that the practical feasibility of acquiring quality data—and the ability to evaluate progress—are independent predictors of scaling viability, so the scaling decision should be made per use case rather than applied uniformly.
Load-bearing premise
The load-bearing premise is that topological descriptions of data shape—persistence and stability of structure across scales—actually predict which tasks gain from more data, but the paper gives no definition of scale, no measurement recipe, and no empirical evidence for that link.
Editorial extensions
If this is right
- Data budgets should be concentrated on task families whose data shows stable, persistent structure across scales, such as translation, robotics, and drug discovery.
- For tasks with unstable or fragmented data structure, continued data collection is unlikely to close the gap; research should shift to architecture and algorithm changes.
- Synthetic data will help mainly in domains with automatic verification, not as a general replacement for human or sensor data.
- Evaluation must become more use-case-specific, since current single-turn benchmarks cannot show whether extra data creates real user value.
- Even without new collection, intentional filtering of existing training mixtures should improve efficiency and reduce the volume of data needed.
Reading between the lines
- Editorial extension: a direct test of the paper's logic would be to compute persistent-homology descriptors for many task families and correlate them with measured gains per additional training token; a positive correlation would turn the heuristic into a quantitative predictor.
- Editorial extension: if the link holds, data valuation could be augmented by topological descriptors, flagging which data points change the multi-scale structure of the training set rather than merely adding tokens.
- Editorial extension: the same stability criterion may predict early saturation, so it could guide compute allocation by identifying tasks that will plateau quickly.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper is a short position/opinion essay arguing that the AI community should be 'intentional' about data scaling rather than assuming that more data always helps. The authors propose that the 'shape of data'—interpreted through topological data analysis concepts such as compositional and structural patterns and their stability across scales—can predict which tasks will benefit from data scaling. They illustrate the idea with claimed successes (machine translation, robotics, drug discovery) and failures (misinformation detection, robust reasoning), and they add a second, more practical predictor based on data-acquisition feasibility, verifiability, and evaluation quality. The paper concludes with a call for a research program centered on operationalizing data shape and using it to guide data collection and model-training decisions.
Significance. If the paper's central claim could be operationalized and empirically supported, it would offer a useful decision principle for allocating scarce data-collection resources and for identifying tasks where scaling alone is insufficient. The framing is timely and the literature synthesis is broad, drawing on scaling laws, synthetic data, and topological data analysis. The authors are appropriately cautious in presenting their proposal as an argument rather than a settled result. However, the paper currently provides no quantitative definition of 'data shape', no measurement procedure, and no empirical evidence that shape predicts scaling outcomes. As a result, its predictive claim is not yet testable, and the illustrative cases are post hoc and confounded with other predictors. The paper's strengths are its clear writing and its emphasis on a potentially important but underdeveloped research direction.
major comments (4)
- [Predictive Power of Data Shape] The central predictive variable, 'shape of data', is never operationalized. The paper states that topological features and their stability across scales 'provide insights into whether certain applications are suitable for data-driven scaling,' but it does not define how to compute such features on real datasets, what 'multiple scales' means for text or sensor data, or which persistence thresholds or filtrations should be used. Without a concrete definition, the claim cannot be tested or falsified, and the argument reduces to analogy. The citation of Wu et al. (2021) about multi-scale topology optimization does not supply a method for characterizing the shape of training data.
- [Predictive Power of Data Shape] The characterizations of the two main examples are post hoc assertions rather than analyses. The paper claims that machine translation data exhibits 'regular and persistent patterns at different scales' and that misinformation data is 'highly fragmented or unstable over time,' but it provides no persistence diagrams, no stability computations, and no comparison of topological summaries across these datasets. As written, these examples are equally consistent with explanations based on data availability, verifiability, or evaluation quality, so the paper has not demonstrated that data shape is the operative factor.
- [Data Acquisition as Another Predictor] The additional predictors listed in this section—availability, verifiability, evaluation quality, and the presence of procedural knowledge—are plausibly sufficient to explain the success/failure examples without invoking data shape. For instance, machine translation has abundant parallel corpora, well-established quality metrics, and clear verification, whereas misinformation is inherently scarce, hard to verify, and lacks reliable benchmarks. The paper does not control for these factors or argue why data shape is needed in addition to them; without such an argument, the central claim is underdetermined by the cited cases.
- [Where Data-Driven Scaling Thrives and Stumbles] The statement that field experts 'almost unanimously agreed' that reasoning failures are rooted in architecture rather than data is presented as evidence, but no methodology, number of experts, or interview protocol is given. This is an anecdotal appeal and does not meet the evidentiary standard of the surrounding literature, especially given that the paper elsewhere relies on published scaling-law results. This passage should either be removed or replaced with citations to systematic studies on reasoning limitations.
minor comments (5)
- [Figure 1] The text references Figure 1 and its caption describes a table of questions, but the figure itself appears to be missing from the manuscript; please ensure it is included in the final submission.
- [Predictive Power of Data Shape] The reference to Wu et al. (2021) is a topology-optimization review, not a source on topological data analysis of datasets; consider citing a TDA methodology paper, such as a guide to persistent homology, to support the claimed framework.
- [Abstract and Section 2] The terms 'shape of data', 'structural patterns', and 'stability across multiple scales' are used repeatedly without precise definitions; a short glossary or concrete example of a topological summary (e.g., a persistence diagram) would greatly improve clarity.
- [Data Acquisition as Another Predictor] This section mixes data-acquisition constraints with a broader discussion of evaluation benchmarks; the final two paragraphs on evaluation frameworks would fit better as a separate subsection or in the conclusion.
- [Where Data-Driven Scaling Thrives and Stumbles] The drug-discovery paragraph is vague, stating that 'stable features that emerge across experiments highlight critical research directions' without specifying which features or which experiments; a concrete example would strengthen the argument.
Circularity Check
No significant circularity: the paper is a position argument with no fitted parameters, no equations, and no self-citation chain; its data-shape claim is under-specified but not circular.
full rationale
This is a position paper rather than a derivation. It makes no predictions from fitted parameters, has no equations that reduce to inputs, and does not invoke any uniqueness theorem or self-citation as load-bearing evidence. The central claim in 'Predictive Power of Data Shape' is that topological features of data indicate whether data scaling will help, but the paper does not define how to measure those features or provide quantitative TDA results; the examples of machine translation and misinformation are asserted rather than computed. This is an empirical-support gap and an operationalization gap, not circularity: the argument could in principle be tested if the data-shape measure were specified, and the paper itself acknowledges the notion is still being developed ('As we continue to learn how to define the shape of data'). The references cited (Carlsson 2009, Uchendu and Le 2024, Wu et al. 2021) are independent prior work, and none of the authors' own prior results are used as a premise. Because no step reduces to its own input, the honest finding is no significant circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption Data has a measurable topological shape whose features, such as persistent structures across scales, are meaningful for language and sensor datasets.
- ad hoc to paper Tasks with stable, persistent topological structure benefit from data scaling, while tasks with fragmented or unstable structure do not.
- domain assumption High-quality human-generated data is finite and increasingly scarce.
- domain assumption Current evaluation benchmarks do not reflect real-world user value.
Cite this review
Pith. "Pith review of Not Every AI Problem is a Data Problem: We Should Be Intentional About Data Scaling." pith.science (2026). https://pith.science/paper/K2VXUGNL
@misc{pith2026250113779,
author = {Pith},
title = {Pith review of: Not Every AI Problem is a Data Problem: We Should Be Intentional About Data Scaling},
year = {2026},
howpublished = {\url{https://pith.science/paper/K2VXUGNL}},
note = {Machine review of arXiv:2501.13779}
}
read the original abstract
While Large Language Models require more and more data to train and scale, rather than looking for any data to acquire, we should consider what types of tasks are more likely to benefit from data scaling. We should be intentional in our data acquisition. We argue that the shape of the data itself, such as its compositional and structural patterns, informs which tasks to prioritize in data scaling, and shapes the development of the next generation of compute paradigms for tasks where data scaling is inefficient, or even insufficient.
Figures
Forward citations
Cited by 1 Pith paper
-
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
MesaNet uses conjugate-gradient-optimal test-time regression in a chunkwise-parallelizable recurrent layer, achieving strong language modeling and benchmark performance at up to 1B scale.
Reference graph
Works this paper leans on
-
[3]
ISSN 0001-0782. doi: 10.1145/3647631. URLhttps://doi.org/10.1145/3647631. T. Evans, N. Parthasarathy, H. Merzic, and O. J. Henaff. Data curation via joint example selection further accelerates multimodal learning.arXiv preprint arXiv:2406.17711,
-
[4]
URLhttps://arxiv. org/abs/2411.04872. A. Gulati, B. Miranda, E. Chen, E. Xia, K. Fronsdal, B. de Moraes Dumont, and S. Koyejo. Putnam- axiom: A functional and static benchmark for measuring higher level mathematical reasoning. In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24,
-
[5]
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556,
-
[8]
URLhttps://arxiv.org/abs/2404. 07503. S. Longpre, G. Yauney, E. Reif, K. Lee, A. Roberts, B. Zoph, D. Zhou, J. Wei, K. Robinson, D. Mimno, et al. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity.arXiv preprint arXiv:2305.13169,
-
[10]
L. Ruis, M. Mozes, J. Bae, S. R. Kamalakara, D. Talupuru, A. Locatelli, R. Kirk, T. Rockt¨ aschel, E. Grefenstette, and M. Bartolo. Procedural knowledge in pretraining drives reasoning in large language models.arXiv preprint arXiv:2411.12580,
-
[11]
L. Soldaini, R. Kinney, A. Bhagia, D. Schwenk, D. Atkinson, R. Authur, B. Bogin, K. Chandu, J. Dumas, Y. Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research.arXiv preprint arXiv:2402.00159,
-
[12]
A. Uchendu and T. Le. Unveiling topological structures in text: A comprehensive survey of topological data analysis applications in nlp.arXiv preprint arXiv:2411.10298,
-
[14]
P. Yu, J. Xu, J. Weston, and I. Kulikov. Distilling system 2 into system 1.arXiv preprint arXiv:2407.06023, 2024a. Y. Yu, Y. Zhuang, J. Zhang, Y. Meng, A. J. Ratner, R. Krishna, J. Shen, and C. Zhang. Large language model as attributed training data generator: A tale of diversity and bias.Advances in Neural Information Processing Systems, 36, 2024b. W. Zh...
Show all 14 references
-
[2017]
Villalobos, J
P. Villalobos, J. Sevilla, L. Heim, T. Besiroglu, M. Hobbhahn, and A. Ho. Will we run out of data? an analysis of the limits of scaling datasets in machine learning.arXiv preprint arXiv:2211.04325, 1,
-
[2018]
URLhttps://www.atanet.org/client-assistance/ata-position- paper-machine-translation-a-clear-approach-to-a-complex-topic. N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang. Quantifying memorization across neural language models.arXiv preprint arXiv:2202.07646,
-
[2020]
K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini. Deduplicating training data makes language models better.arXiv preprint arXiv:2107.06499,
-
[2021]
Kaplan, S
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,
2001 arXiv
-
[2022]
Mirzadeh, K
I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models.arXiv preprint arXiv:2410.05229,
-
[2024]
URLhttps://multilingual.com/magazine/june- 2024/neural-machine-translation-versus-large-language-models/. A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,
2024 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.