Pith. sign in

REVIEW 4 major objections 6 minor 3 references

Automatic Demonstration Selection for LLM-based Tabular Data Classification

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes that the number of demonstrations for in-context tabular classification can be read off from the spectral gap of a token-overlap similarity graph, and reports that the resulting demonstration count performs close to the…

desk verdict A genuinely new angle on ICL demonstration-count selection, but the spectral-gap-to-count link is asserted rather than shown, and the paper's own tables show the count often misses the best random choice by a wide margin. read the letter →

arxiv 2506.20451 v1 pith:EQQWFIHV submitted 2025-06-25 cs.LG cs.AI

Shuchu Han , Wolfgang Bruckner This is my paper
classification cs.LGcs.AI MSC 68T0705C5062H30
keywords in-contextlearningdemonstrationselectiontabulardataclassificationspectralgraphtheorygaplargelanguagemodelsfew-shotJaccardsimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how many demonstrations—labeled example rows—should be placed in a prompt when using an LLM to classify tabular data, and proposes an automatic answer instead of asking users to guess and check. The proposed algorithm renders each row as a natural-language demonstration, tokenizes it with the chosen LLM, builds a k-nearest-neighbor graph from Jaccard similarity of token-ID lists, and reads the number of demonstrations $d$ off the spectral gap of the graph's normalized Laplacian. The authors report that this automatically chosen $d$ gives classification performance close to the best result obtained by sweeping random demonstration counts, across eight OpenML datasets and three LLMs. A sympathetic reader would take the contribution as a parameter-light heuristic that turns demonstration-count selection into a graph-spectrum computation.

What carries the argument

The load-bearing object is the spectral gap of the normalized Laplacian of a k-nearest-neighbor similarity graph. Nodes are demonstrations, each represented by the token-ID list produced by the selected LLM's tokenizer; edge weights are Jaccard similarities between token lists (Eq. (1)); after keeping only the top-k neighbors and resetting weights to 1, the normalized Laplacian is formed (Eqs. (3)-(5)). The index at which successive eigenvalues are farthest apart, $k = \arg\max_i (\lambda_{i+1}-\lambda_i)$, is taken to be the number of clusters and therefore the minimum number of demonstrations. Spectral clustering on the corresponding eigenvectors then picks one representative per cluster.

What would settle it

A direct test: for a dataset with known classes, compute $d$ from the spectral gap and compare it with the count in $\{0,2,4,6,8,10\}$ that maximizes Macro-F1. The paper's own eigenvalue plots show no visible gap for tae and cmc, yet the algorithm outputs $d=1$ and $d=42/45$ for them; measuring whether those counts are near the accuracy-optimal counts, or far from them, would settle the claim.

Watch

Extended reading notes

Core claim

The central claim is that the number of demonstrations needed for in-context tabular classification is recoverable from the intrinsic representation space of the LLM itself. Each row is turned into a text demonstration, tokenized into a list of token IDs; the overlap structure of those lists is summarized by a similarity graph, and the largest gap between consecutive eigenvalues of the normalized Laplacian (Eq. (6)) estimates the number of clusters, which the algorithm takes as the minimum number of demonstrations $d$. One demonstration is then drawn from each cluster. The paper reports that this $d$ varies across LLMs and datasets (for example, cmc yields 45 for Llama but 42 for Mistral and Qwen, while iris yields 3 for all three), and that classification with these selected demonstrations is stable and close to the best random-selection result rather than dominant.

Load-bearing premise

The whole pipeline rests on treating the largest eigenvalue gap of a token-overlap graph as the correct number of demonstration examples; if that graph's cluster structure does not match the number of examples that actually helps the LLM classify, the returned $d$ has no principled connection to classification performance.

Editorial extensions

If this is right

  • Users no longer need to sweep demonstration counts: the algorithm returns a single $d$ per LLM, prompt template, and dataset.
  • The selected count is tied to the LLM's tokenizer and the prompt template, so the same data can lead to different $d$ for different models, and the method can react to prompt changes.
  • Because only token IDs are needed, the procedure is cheap relative to embedding-based selection.
  • Performance is expected to be stable and near the best random-selection result, not necessarily better than a well-tuned random count.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit is to replace the raw token-ID Jaccard overlap with a task-aware similarity (for example, distance in the LLM's embedding space after dimensionality reduction) and check whether the spectral gap then tracks accuracy-optimal counts more closely on datasets like tae and cmc.
  • Another testable extension is cluster-aware sampling: the paper draws one random row per cluster; selecting the row closest to each cluster centroid, or sampling proportional to cluster size, could reduce variance in the reported Macro-F1.
  • Since tokenization depends on the prompt template, the same pipeline could be used to compare templates: templates that produce clearer spectral gaps would be predicted to yield better classification, a hypothesis the paper does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes an automatic method for choosing the number of in-context learning demonstrations for LLM-based tabular classification. The method represents each table row by the token IDs of a templated natural-language demonstration, builds a k-NN graph using Jaccard similarity between token-ID sets, computes the normalized graph Laplacian, and sets the number of demonstrations d to the index of the largest gap among the first 50 eigenvalues. It then selects one demonstration per spectral cluster. Experiments on eight OpenML datasets and three LLMs compare the chosen d with random demonstration counts in {0,2,4,6,8,10} using Macro-F1. The authors report that their method does not outperform random selection but is intended to give 'stable' performance close to the best tuned count.

Significance. If the central claim were established, the paper would address a practical pain point: choosing how many demonstrations to put into an ICL prompt for tabular data without a guess-and-check loop. The proposed estimator is cheap (token IDs only, no embeddings) and the idea of linking demonstration count to the structure of the token-ID similarity graph is novel. However, the paper provides no derivation or prior result connecting spectral gaps of this graph to ICL accuracy, and its own experiments contain direct counterexamples to the core claim. The significance of the work therefore depends entirely on an unsupported and empirically contradicted premise.

major comments (4)
  1. [Section 2.4, Eq. (6)] The central claim is that the index of the largest eigenvalue gap of the normalized Laplacian of a k-NN graph on Jaccard-overlapping token-ID lists equals the number of demonstrations needed for ICL classification. No derivation, theorem, or prior empirical result is offered to justify this mapping. The number of clusters in a token-ID similarity graph is not the same as the number of demonstrations that optimizes classification performance, and the manuscript does not define what 'minimum number of demonstrations' means formally. This connection is load-bearing and is never established.
  2. [Table 4] The paper's own experimental results contradict the claim that the algorithm selects a 'reasonable' number of demonstrations. For example, under Llama: cmc, d=45 while the best random count is 2 (Macro-F1 0.12 vs 0.19); Customers, d=1 while the best is 10 (0.26 vs 0.86); tae, d=1 while the best is 4 (0.28 vs 0.36). Similar large misses occur under Mistral and Qwen (e.g., Qwen iris d=3 vs best 2, Macro-F1 0.06 vs 0.37). Section 3.6 explicitly states that tae and cmc show no obvious spectral gap, yet the algorithm outputs d=1 and d=45/42 for these datasets. These results directly undermine the paper's main claim.
  3. [Algorithm 1, line 9; Section 2.3] The algorithm resets all k-NN edge weights to 1, so the Laplacian reflects only unweighted k-NN topology, and the Jaccard similarity values are discarded. Because every demonstration shares the fixed template tokens ('input: given', 'is', 'and', 'class:', punctuation), the Jaccard overlap is inflated by prompt-template text and may be dominated by it. The manuscript provides no analysis of how template tokens affect the graph spectrum, so the estimated d may reflect the template rather than the data distribution. This is a serious problem for the validity of Eq. (6) as a data-driven estimate.
  4. [Section 1, contribution 2; Section 3.1] The claim that the algorithm is 'almost parametric-free except the construction of the sparse graph' is inaccurate. The algorithm has at least two free parameters: the neighborhood size k in the k-NN graph (set to 10) and the number of eigenvalues inspected (50 in the experiments, as stated in Section 3.6). No sensitivity analysis is provided for either parameter, and the choice of 50 is especially consequential because Eq. (6) searches over a fixed prefix of the spectrum; for the cmc dataset, d=45 is close to the boundary of this search window, so the result may be an artifact of the truncation.
minor comments (6)
  1. [Throughout] There are numerous typos and grammatical errors, including 'contribuions' (Section 1), 'resonable' (Section 1), 'stablely' (Section 1), 'per-trained' (Algorithm 1), and 'potion' (Section 3.5). These should be corrected.
  2. [Equation (6) and Section 3.1] The symbol k is used both for the k-NN hyperparameter and for the index variable in Eq. (6), which is confusing. Consider renaming one of them.
  3. [Figure 3] The caption says 'First 50 eigenvalues (sorted)' but the axes are unlabeled and it is unclear whether eigenvalues are sorted ascending or descending, and whether the gap is λ_{i+1} - λ_i or λ_i - λ_{i+1}. Adding axis labels and a clear description of the gap would help the reader interpret the figure.
  4. [Table 1] The three rightmost columns are labeled only with model names (llama, Mistral, Qwen) and contain integers, but the caption does not explicitly explain that these are the estimated d values until the text in Section 3.5. This should be stated directly in the caption.
  5. [Section 3.5] The statement that zero-shot performance is zero for 'all other LLMs and datasets' is not tied to a specific figure or table, and the only exception mentioned (Qwen on wine and iris) is not quantified. Please provide the underlying numbers or refer to the appropriate panel in Figure 2.
  6. [References] The HuggingFace reference contains the placeholder '[Insert Date]' and should be completed or removed. Also, several cited works in Section 4 on demonstration selection are discussed only briefly; a short comparison with those methods would strengthen the positioning of the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the spectral-gap demonstration count is computed from token IDs without fitting to labels or test performance; the paper's weakness is an unsupported validity assumption, not a circular derivation.

full rationale

The paper's derivation chain is: tokenize each demonstration with the selected LLM's tokenizer, compute Jaccard similarity over token-ID lists (Eq. 1), build a k-NN similarity graph with edge weights reset to 1 (Algorithm 1, lines 8-9), form the normalized Laplacian (Eqs. 3-5), set d to the index of the largest eigenvalue gap (Eq. 6), cluster the demonstrations into d groups, and sample one demonstration per group. At no point are the dataset labels, the Macro-F1 scores, or the best-performing demonstration counts from the random-selection grid used as inputs to compute d. The comparison in Section 3.5 is a post-hoc evaluation on held-out test data. The paper explicitly calls the method a 'heuristic algorithm' and openly reports that it 'does not show dominant performance' and that tae and cmc show no obvious spectral gap, which are validity caveats rather than circularity. The quantity d is not forced by construction to equal the best random-selection count; Table 4 shows large mismatches (e.g., cmc d=45 vs best d=2 for Llama), which is evidence against the validity of the estimator but not evidence of circularity. The central weakness is an unsupported assumption linking spectral gaps of token-ID graphs to ICL demonstration counts, not a reduction of the result to its own inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new physical entities are introduced. The contribution is a heuristic algorithm whose central output d rests on three unproven assumptions: token-ID Jaccard overlap is a faithful similarity, the largest spectral gap identifies a meaningful cluster count, and one representative per cluster suffices for ICL. The only tuned inputs are k=10 and the implicit 50-eigenvalue window.

free parameters (2)
  • k for k-nearest-neighbor graph = 10
    Chosen by hand for all experiments; controls the sparsity of the similarity graph and therefore the spectral estimates, though Algorithm 1 lists it as an input hyperparameter.
  • Number of eigenvalues inspected = 50
    The spectral gap is computed over the first 50 eigenvalues (Figure 3); no criterion is given for this cutoff, and for datasets such as cmc the largest gap may lie outside this window.
assumptions (3)
  • domain assumption Jaccard similarity over LLM token IDs captures the similarity of demonstrations in the LLM's intrinsic representation space.
    Section 2.2 replaces embedding vectors with raw token IDs and assumes token overlap is a meaningful similarity for clustering; no evidence is given that token-ID overlap tracks the LLM's internal notion of similarity.
  • domain assumption The largest eigenvalue gap of the normalized Laplacian of the k-NN token-overlap graph equals the number of clusters, and this cluster count is the minimum number of demonstrations needed for accurate in-context classification.
    Section 2.4 uses the spectral gap directly as d via Eq. (6); the link from graph clusters to ICL classification performance is taken for granted. For cmc and tae the authors state there is no obvious spectral gap, yet a d is still output.
  • ad hoc to paper Discarding the Jaccard edge weights by resetting all k-NN edges to weight 1 still preserves the data distribution needed for clustering.
    Algorithm 1 line 9 changes the weighted similarity graph to an unweighted graph; the effect on the spectral estimate is not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automatic Demonstration Selection for LLM-based Tabular Data Classification." pith.science (2026). https://pith.science/paper/EQQWFIHV

@misc{pith2026250620451,
  author       = {Pith},
  title        = {Pith review of: Automatic Demonstration Selection for LLM-based Tabular Data Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EQQWFIHV}},
  note         = {Machine review of arXiv:2506.20451}
}
read the original abstract

A fundamental question in applying In-Context Learning (ICL) for tabular data classification is how to determine the ideal number of demonstrations in the prompt. This work addresses this challenge by presenting an algorithm to automatically select a reasonable number of required demonstrations. Our method distinguishes itself by integrating not only the tabular data's distribution but also the user's selected prompt template and the specific Large Language Model (LLM) into its estimation. Rooted in Spectral Graph Theory, our proposed algorithm defines a novel metric to quantify the similarities between different demonstrations. We then construct a similarity graph and analyze the eigenvalues of its Laplacian to derive the minimum number of demonstrations capable of representing the data within the LLM's intrinsic representation space. We validate the efficacy of our approach through experiments comparing its performance against conventional random selection algorithms on diverse datasets and LLMs.

Figures

Figures reproduced from arXiv: 2506.20451 by the authors.

Figure 1
Figure 1. An overview of our proposed algorithm. The prompt template and the LLM are chosen by [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Classification performance of different number of demonstrations that selected randomly. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. First 50 eigenvalues (sorted) of experiment datasets. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

3 extracted references · 1 canonical work pages

  1. [1]

    & Kasneci, G

    Borisov, V ., Seßler, K., Leemann, T., Pawelczyk, M. & Kasneci, G. (2022), ‘Language models are realistic tabular data generators’, arXiv preprint arXiv:2210.06280 . Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. et al. (2020), ‘Language models are few-shot learners’, Advances i...

    arXiv 2022

  2. [3]

    Hu, J., Liu, W. & Du, M. (2024), ‘Strategic demonstration selection for improved fairness in llm in-context learning’, arXiv preprint arXiv:2408.09757 . HuggingFace (2023), ‘Hugging face – the ai community building the future.’, https:// huggingface.co/. Accessed: [Insert Date]. Lehoucq, R. B., Sorensen, D. C. & Yang, C. (1998),ARPACK users’ guide: soluti...

    work page Pith review arXiv 2024

  3. [30]

    & Berant, J

    10 Rubin, O., Herzig, J. & Berant, J. (2022), Learning to retrieve prompts for in-context learning, in M. Carpuat, M.-C. de Marneffe & I. V . Meza Ruiz, eds, ‘Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies’, Association for Computational Linguistics, Seattle, U...

    arXiv 2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.