Pith. sign in

REVIEW 2 major objections 5 minor 18 references

No single model family wins for scarce-label crowd-state classification: foundation models lead with few labels, while tuned conventional models overtake them as labels grow.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

For Hajj-style crowd-state tables, foundation models lead at very few labels on clean density tasks, while tuned gradient boosting catches up and surpasses them on geometry and full-data regimes.

T0 review reviewed 2026-07-11 challenge →

load-bearing objection Careful multi-budget bake-off: TabPFN-3 wins low-label clean density; tuned boosting wins full-budget geometry; the map is the result. the 2 major comments →

arxiv 2607.04013 v1 pith:YF6HOQSC submitted 2026-07-04 cs.LG cs.AI

When Does Small Data Work? Accuracy and Efficiency Trade-offs Between Tabular Foundation Models and Conventional Methods for Crowd-State Classification at Hajj and Umrah

AlJawharh S. AlOtaibi , Mohamed Eltahir , Jude AlSubaie This is my paper
classification cs.LG cs.AI
keywords tabular foundation modelslabel efficiencycrowd-state classificationHajj and Umrahin-context learninggradient boostingfew-shot tabular learningdata leakage
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether tabular foundation models help when labels are scarce for classifying crowd safety states relevant to Hajj and Umrah, and how they trade off against ordinary machine-learning methods. Using three real datasets—controlled trajectories, a noisy field gathering, and real Hajj wearable sensors—it compares three foundation models against classical and gradient-boosting baselines, both at defaults and after careful tuning, across label budgets from 16 examples up to the full set. The result is conditional rather than absolute: foundation models give the strongest accuracy with no tuning when labels are very few, especially on clean density targets; as labels increase, tuned conventional models catch up and significantly surpass them on the more structural flow-geometry target. Cost profiles also diverge—foundation models avoid expensive tuning but reprocess their context at every prediction—so the practical answer is a map of which family to prefer under a given label and compute budget. That map matters for operators who must detect dangerous crowd regimes before a season produces abundant labels.

Core claim

No single family of methods is best everywhere for crowd-state classification from tabular movement and wearable signals. When labels are very few, tabular foundation models—especially TabPFN-3—lead on label efficiency and significantly outperform even tuned gradient boosting on the clean density regime; as the label budget grows, well-tuned conventional models, particularly gradient boosting, close the gap and significantly surpass the foundation models on the structural flow-geometry target. The preferable approach therefore depends on the label budget, the task structure, and computational constraints.

What carries the argument

The conditional preference map (accuracy, integrated low-label efficiency AULC, and wall-clock cost) obtained under group-aware, leakage-controlled splits, permutation signal tests, and paired significance tests: it shows which model family to prefer as a function of available labels and compute.

Load-bearing premise

The controlled lab and field trajectory datasets are good enough stand-ins for real Hajj crowd regimes that the label-budget preference map will transfer to live monitoring at the holy sites.

What would settle it

Collect live labeled streams from Tawaf, Jamarat, or Mina under the same group-aware protocol: if foundation models lose their significant low-label density edge (or tuned gradient boosting never overtakes on geometry at full budget), the operational map is falsified for holy-site use.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • When labels are very few and tuning is not feasible, start with a strong tabular foundation model at defaults for clean sensor-derived density classification.
  • For noisier field conditions or structural geometry targets, and once larger labeled sets exist, prefer tuned tree ensembles and gradient boosting.
  • Foundation models deliver low-label accuracy without a one-time tuning search, at cost comparable to untuned boosting, but their per-prediction cost grows with context size at scale.
  • Results vary enough across foundation models that operators should pick a specific model (e.g., TabPFN-3 for the low-label win), not the family label.
  • The value of foundation models is concentrated in the low-label, clean-data corner of the operating space and shrinks or reverses as either condition is relaxed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar conditional maps may apply to other high-stakes tabular settings where labels arrive slowly, such as industrial sensor faults or medical triage tables.
  • If live holy-site streams prove noisier than the controlled proxies, the crossover where tuned conventional models become preferable may arrive earlier in the label budget.
  • Continuous high-throughput monitoring could invert the cost advantage once enough labels exist, favoring a once-trained conventional model over repeated in-context reprocessing.
  • A hybrid seasonal policy—foundation models for cold-start, then switch to tuned conventional models after mid-season labels accumulate—follows directly from the map but is not tested here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper compares tabular foundation models (TabPFN-3, TabICL-v2, LimiX-2M) against conventional and gradient-boosting baselines, both default and group-aware-tuned, for three crowd-state classification targets (density regime, flow geometry, pilgrim fatigue) drawn from Jülich, Lyon, and real-Hajj wearable data. Under a leakage-controlled protocol with group-aware time-window splits, 20 paired repeats, permutation signal tests, and Holm-corrected paired tests, it reports a conditional map: foundation models lead at very low label budgets on clean density (significant TabPFN-3 advantage at 16/64/256 labels and on AULC), while tuned conventional models catch up and significantly surpass foundation models on structural geometry at full budget (losses of 0.022–0.096). Fatigue is correctly discarded after failing the signal test. Efficiency is also mapped: foundation models avoid tuning cost but reprocess context at prediction. The authors frame the result as an operator-facing preference map under label and compute budgets rather than a universal ranking.

Significance. If the reported map holds, the paper supplies concrete, operator-facing guidance for a high-stakes, label-scarce domain where practitioners currently lack evidence on when tabular foundation models are worth deploying versus tuned gradient boosting. Strengths include explicit leakage controls (density features withheld from density prediction; labels from source experiment type or tiredness field), permutation nulls, pre-registered low-label comparisons with Holm correction, AULC and full-budget paired tests, robustness to three preprocessing variants, and public data with a frozen-environment notebook. The contribution is empirical and conditional rather than theoretical, but the rigor of the bake-off and the practical map are useful for both the tabular-foundation-model literature and Hajj/Umrah safety tooling.

major comments (2)
  1. [§III / Fig. 1 / Limitations] §III, Fig. 1, and Limitations: the central operational claim is that the Jülich/Lyon proxy regimes transfer to recurring Hajj regimes (Tawaf counterflow, Jamarat bottleneck, Mina/Mashaer corridors). The paper already frames results as conditional and discards fatigue after the signal test, so this does not invalidate the internal map, but it remains the load-bearing premise for any deployment recommendation. A short quantitative sensitivity check—e.g., domain-shift diagnostics or a leave-one-regime-out analysis on the controlled experiments—would make the transfer claim more falsifiable without expanding scope.
  2. [Abstract / §V / Conclusion] §V-B–V-D and Tables V–VIII: the foundation family is correctly shown to be non-monolithic (TabPFN-3 carries the only significant low-label win; LimiX-2M loses on geometry). The abstract and conclusion still sometimes speak of “foundation models” as a class leading at few labels. Tightening those sentences to name the winning model (or to state explicitly that only TabPFN-3 is supported) would keep the headline claim aligned with the corrected tests.
minor comments (5)
  1. [Table IV] Table IV: the p-value floor of 0.0099 is correctly explained as an artifact of 100 permutations, but a one-sentence note that the discriminating quantity is the real–null F1 gap (not the p-value) would help readers who stop at the table.
  2. [Fig. 2] Fig. 2 averages macro-F1 across tasks of differing difficulty; the caption already cautions that ordering, not level, carries the message. Adding per-task curves (or a pointer to the notebook) would reduce the risk of over-reading the average.
  3. [Table IX / §V-F] §IV-E / Table IX: wall-clock times are reported for a single scale. A brief remark that foundation-model cost grows with context size (already noted in Discussion) could be echoed next to the table so cost claims are not read as scale-invariant.
  4. [Throughout] Minor typography: “BACKGROUND ANDRELATEDWORK”, “CROWD-STATETARGETS”, “DATASETS SPANNING CLEAN,NOISY,AND…” and similar missing spaces in section/table titles should be fixed for production.
  5. [References] References [9], [11], [12] are arXiv preprints with future-looking dates (2025–2026); confirm final citations or mark as preprints consistently.

Circularity Check

0 steps flagged

No circularity: empirical bake-off with leakage controls, not a derivation that reduces to its inputs.

full rationale

This paper is a controlled empirical comparison of tabular foundation models versus conventional classifiers under varying label budgets. Its central claim is a conditional map (foundation models lead at very few labels on clean density; tuned conventional models catch up and significantly surpass them on structural geometry at full budget), supported by paired Wilcoxon tests, Holm correction, AULC, permutation nulls, and group-aware time-window splits. Density labels are binned from measured local density while density-derived features are withheld from the feature set, so the model cannot read its own label; flow-geometry and fatigue labels come from source experiment type and the recorded tiredness field. No equation, uniqueness theorem, or fitted constant is re-presented as a prediction. Citations (TabPFN, TabICL, LimiX, XGBoost, etc.) are external baselines, not load-bearing self-citations. The only soft premise is external validity of Jülich/Lyon as Hajj proxies, which the authors themselves flag in Limitations and which does not make the internal results circular. Score 0 is therefore the correct finding.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

Empirical ML comparison paper. Load-bearing content is experimental design choices and domain proxy assumptions rather than free physical constants or invented particles. The central map rests on (i) treating Jülich/Lyon as Hajj-regime proxies, (ii) the chosen label budgets and caps, (iii) macro-F1 with group-aware evaluation as the decision metric, and (iv) default foundation-model settings versus group-aware inner-CV tuning for classical models.

free parameters (4)
  • label_budgets = 16, 64, 256, 1024, full
    Discrete budgets {16, 64, 256, 1024, full} define the x-axis of the entire map; different choices would shift where families cross.
  • row_and_eval_caps = 300000 / 30000
    Datasets capped at 300k rows and model inputs stratified-capped at 30k; evaluation set also stratified-capped. These hand choices affect absolute F1 and cost but are shared across models.
  • n_repeated_group_splits = 20
    Twenty repeated group-disjoint splits drive all means, Wilcoxon tests, and bootstrap CIs.
  • permutation_null_count = 100
    100 label shuffles set the p-value floor (0.0099) used to retain or discard tasks.
axioms (5)
  • domain assumption Controlled Jülich and field Lyon trajectory regimes are valid proxies for recurring Hajj crowd regimes (counterflow/Tawaf, bottleneck/Jamarat, corridor/Mina).
    Stated in §III and Fig. 1; without this, the operator-facing map does not transfer to holy sites.
  • domain assumption Group-aware splits by short time window (and pilgrim) prevent train–test leakage for these streaming crowd tables.
    §IV-C evaluation protocol; if groups still leak, reported F1 and significance are inflated.
  • domain assumption Density regime can be predicted honestly from kinematic features alone after deterministic binning of local density, with density-derived inputs withheld.
    §IV-A/B; this is the anti-circularity design for the density task.
  • standard math Macro-F1 averaged over paired group-disjoint splits, with Holm-corrected paired Wilcoxon tests, is an adequate decision criterion for which family to prefer.
    §IV-E and Results tables; standard ML statistics, not proved optimal for operational risk.
  • ad hoc to paper Foundation models may be evaluated at published defaults while conventional models receive group-aware hyperparameter search inside the label budget.
    §IV-D; this asymmetry is intentional (no-tuning operational story) but shapes the cost and accuracy comparison.

reviewed 2026-07-11 · how reviews work

0 comments
Cite this review

Pith. "Pith review of When Does Small Data Work? Accuracy and Efficiency Trade-offs Between Tabular Foundation Models and Conventional Methods for Crowd-State Classification at Hajj and Umrah." pith.science (2026). https://pith.science/paper/YF6HOQSC

@misc{pith2026260704013,
  author       = {Pith},
  title        = {Pith review of: When Does Small Data Work? Accuracy and Efficiency Trade-offs Between Tabular Foundation Models and Conventional Methods for Crowd-State Classification at Hajj and Umrah},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YF6HOQSC}},
  note         = {Machine review of arXiv:2607.04013}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Learning from few labeled examples is a central challenge in tabular machine learning, and it becomes the binding constraint in domains where labeling is costly, such as crowd monitoring during Hajj and Umrah. Tabular foundation models, which predict from only a handful of examples without task-specific training, were recently introduced to address this very-few-label regime. In this study we test them on crowd-state classification to assess how much they help when labels are scarce, and we compare them against standard machine learning methods to characterize the accuracy and efficiency trade-offs between the two approaches. Using three real datasets we evaluate different machine learning models, in untuned and tuned forms, against three foundation models. Results show that no single family is best everywhere. The right choice depends on the label budget. When labels are very few, foundation models lead. As labels grow, tuned conventional models catch up and significantly surpass the foundation models on the more structural geometry target. Efficiency separates them further where tuned machine learning models incur a large tuning cost that foundation models avoid, although foundation models reprocess their context at every prediction. We summarize these results as a practical map of which approach to prefer under a given label budget and computational budget.

Figures

Figures reproduced from arXiv: 2607.04013 by AlJawharh S. AlOtaibi, Jude AlSubaie, Mohamed Eltahir.

Figure 1
Figure 1. Figure 1: Relationship between the evaluated datasets and their Hajj counter [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Macro-F1 as a function of labeled examples, averaged across tasks. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Cost against accuracy at the largest budget, averaged across tasks, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 1 linked inside Pith

  1. [1]

    Open dataset for predicting pilgrim activities for crowd management during hajj using wearable sensors,

    A. M. Al-Shaeryet al., “Open dataset for predicting pilgrim activities for crowd management during hajj using wearable sensors,”IEEE Access, vol. 12, pp. 72 828–72 846, 2024

    2024

  2. [2]

    An intelligent system for hajj crowd management using data mining techniques,

    B. Fahad, I. R. Abdelmaksoud, and H. El-Bakry, “An intelligent system for hajj crowd management using data mining techniques,”J. Theor. Appl. Inf. Technol., vol. 103, no. 14, 2025

    2025

  3. [3]

    From crowd dynamics to crowd safety: A video-based analysis,

    A. Johansson, D. Helbing, H. Z. Al-Abideen, and S. Al-Bosta, “From crowd dynamics to crowd safety: A video-based analysis,”Adv. Complex Syst., vol. 11, no. 4, pp. 497–527, 2008

    2008

  4. [4]

    A machine learning model for crowd density classification in hajj video frames,

    A. A. Shah, “A machine learning model for crowd density classification in hajj video frames,”Int. J. Adv. Comput. Sci. Appl., vol. 15, no. 12, 2024

    2024

  5. [5]

    Crowd management intelligence framework: Umrah use case,

    W. Halboobet al., “Crowd management intelligence framework: Umrah use case,”IEEE Access, vol. 12, pp. 6752–6767, 2024

    2024

  6. [6]

    Dense crowd dynamics and pedestrian trajectories: A multiscale field dataset from the festival of lights in lyon,

    O. Dufouret al., “Dense crowd dynamics and pedestrian trajectories: A multiscale field dataset from the festival of lights in lyon,”Scientific Data, vol. 12, 2025

    2025

  7. [7]

    Accurate predictions on small data with a tabular foundation model,

    N. Hollmannet al., “Accurate predictions on small data with a tabular foundation model,”Nature, vol. 637, no. 8045, pp. 319–326, 2025

    2025

  8. [8]

    Transformers can do bayesian inference,

    S. Mülleret al., “Transformers can do bayesian inference,” inInt. Conf. Learning Representations (ICLR), 2022

    2022

  9. [9]

    Tabpfn-3: Technical report,

    L. Grinsztajn, K. Flöge, O. Key, F. Birkel, P. Jundet al., “Tabpfn-3: Technical report,”arXiv:2605.13986, 2026

    Pith/arXiv arXiv 2026

  10. [10]

    Tabicl: A tabular foundation model for in-context learning on large data,

    J. Qu, D. Holzmüller, G. Varoquaux, and M. L. Morvan, “Tabicl: A tabular foundation model for in-context learning on large data,” inInt. Conf. Machine Learning (ICML), 2025

    2025

  11. [11]

    Tabiclv2: A better, faster, scalable, and open tabular foundation model,

    J. Quet al., “Tabiclv2: A better, faster, scalable, and open tabular foundation model,”arXiv:2602.11139, 2026

    arXiv 2026

  12. [12]

    Limix: Unleashing structured-data modeling capability for generalist intelligence,

    X. Zhanget al., “Limix: Unleashing structured-data modeling capability for generalist intelligence,”arXiv:2509.03505, 2025

    arXiv 2025

  13. [13]

    Leakage in data mining: Formulation, detection, and avoidance,

    S. Kaufman, S. Rosset, C. Perlich, and O. Stitelman, “Leakage in data mining: Formulation, detection, and avoidance,”ACM Trans. Knowl. Discov. Data, vol. 6, no. 4, pp. 1–21, 2012

    2012

  14. [14]

    Permutation tests for studying classifier performance,

    M. Ojala and G. C. Garriga, “Permutation tests for studying classifier performance,”J. Mach. Learn. Res., vol. 11, pp. 1833–1863, 2010

    2010

  15. [15]

    Scikit-learn: Machine learning in python,

    F. Pedregosaet al., “Scikit-learn: Machine learning in python,”J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011

    2011

  16. [16]

    Xgboost: A scalable tree boosting system,

    T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” inACM SIGKDD, 2016

    2016

  17. [17]

    Lightgbm: A highly efficient gradient boosting decision tree,

    G. Keet al., “Lightgbm: A highly efficient gradient boosting decision tree,” inNeurIPS, 2017

    2017

  18. [18]

    Catboost: Unbiased boosting with categorical features,

    L. Prokhorenkovaet al., “Catboost: Unbiased boosting with categorical features,” inNeurIPS, 2018

    2018

This paper was first reviewed by grok-4.5 on July 11, 2026.