{"id":"f16bc5ea-bdbc-4477-8f15-b457ac81bca0","arxiv_id":"2506.20451","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A spectral-gap heuristic on a token-ID similarity graph automatically chooses the number of in-context demonstrations for LLM-based tabular classification, with mixed empirical results against random selection.","lead":"This paper proposes an algorithm that uses spectral graph theory on token overlaps to pick how many example rows to put in a prompt when asking a large language model to classify tabular data. The goal is to save users from hand-tuning the number of examples while keeping classification accuracy reasonable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is unsupported: Eq. (6) is never shown to map the spectral gap of a token-ID Jaccard graph to the demonstration count that maximizes ICL accuracy, and the paper's own Table 4 shows large misses (cmc d=45 vs best d=2; Customers d=1 vs best d=10).","rationale":"The reader's weakest assumption identifies the same load-bearing point: the spectral gap of the normalized Laplacian is not established as the right number of demonstrations for ICL, and the paper's own results contradict it. My read confirms this. Table 4 is not a minor discrepancy; it shows the algorithm's d is frequently far from the best random d, sometimes by more than an order of magnitude (cmc: 45 vs 2). The paper's internal admission that tae and cmc show no obvious spectral gap further undercuts the graph-theoretic interpretation. The additional concern about template-dominated Jaccard similarity, and the reset of edge weights to 1 in Algorithm 1, suggests the graph may not even encode the intended data structure. This is a correctness risk, not a disagreement with consensus: the paper does not provide a derivation, a benchmark correlation, or a reproducible implementation that would connect the spectral gap to classification performance. The author's own characterization as a 'heuristic' with 'stable' rather than dominant performance is honest but does not support the stronger 'minimum number of demonstrations' claim. The verdict should remain reject.","tokens_in":9090,"tokens_out":8475,"duration_ms":96303,"concrete_test":"Run a controlled experiment on at least 30 dataset/LLM/template combinations: for each, compute d_algo from Eq. (6) and d_best as the count with highest average Macro-F1 over repeated random selections from the grid {0,2,4,6,8,10,20,30,45}; also run each dataset with a second prompt template that adds three content-free tokens to every demonstration. Report (i) Spearman correlation between d_algo and d_best and (ii) the distribution of |d_algo(template1) - d_algo(template2)|. The central claim is supported only if the correlation is significantly positive and the template perturbation changes d only when it changes the best count.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the index of the largest eigenvalue gap of the normalized Laplacian of a k-NN graph built from Jaccard overlap of raw token-ID lists (Eq. (6), Algorithm 1) equals the number of demonstrations a user should put in the prompt. No derivation, prior result, or benchmark links this spectral quantity to ICL accuracy, and the paper's own Table 4 provides counterexamples. For Llama, cmc: d=45 while the best random count in {0,2,4,6,8,10} is 2 (Macro-F1 0.12 vs 0.19); Customers: d=1 while the best is 10 (0.26 vs 0.86); tae: d=1 while the best is 4 (0.28 vs 0.36). Similar misses occur under Mistral and Qwen. Section 3.6 admits tae and cmc show no obvious spectral gap, yet the algorithm outputs 1 and 45/42 for these datasets. The graph itself is also questionable: Algorithm 1 line 9 resets Jaccard weights to 1, so the Laplacian reflects only unweighted k-NN topology, and because every demonstration shares the fixed template tokens ('input:', 'given', 'is', 'and', 'class:', punctuation), Jaccard overlap is inflated by template text and may be dominated by it. Thus the graph may represent the prompt template rather than the data distribution, and the output d has no demonstrated connection to the number of demonstrations needed for ICL. The paper's own caveats ('heuristic', 'does not show dominant performance', absence of a gap for tae/cmc) strengthen rather than weaken this concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automatic method for choosing the number of in-context learning demonstrations for LLM-based tabular classification. The method represents each table row by the token IDs of a templated natural-language demonstration, builds a k-NN graph using Jaccard similarity between token-ID sets, computes the normalized graph Laplacian, and sets the number of demonstrations d to the index of the largest gap among the first 50 eigenvalues. It then selects one demonstration per spectral cluster. Experiments on eight OpenML datasets and three LLMs compare the chosen d with random demonstration counts in {0,2,4,6,8,10} using Macro-F1. The authors report that their method does not outperform random selection but is intended to give 'stable' performance close to the best tuned count.","tokens_in":1686,"tokens_out":1881,"duration_ms":53485,"significance":"If the central claim were established, the paper would address a practical pain point: choosing how many demonstrations to put into an ICL prompt for tabular data without a guess-and-check loop. The proposed estimator is cheap (token IDs only, no embeddings) and the idea of linking demonstration count to the structure of the token-ID similarity graph is novel. However, the paper provides no derivation or prior result connecting spectral gaps of this graph to ICL accuracy, and its own experiments contain direct counterexamples to the core claim. The significance of the work therefore depends entirely on an unsupported and empirically contradicted premise.","major_comments":[{"comment":"The central claim is that the index of the largest eigenvalue gap of the normalized Laplacian of a k-NN graph on Jaccard-overlapping token-ID lists equals the number of demonstrations needed for ICL classification. No derivation, theorem, or prior empirical result is offered to justify this mapping. The number of clusters in a token-ID similarity graph is not the same as the number of demonstrations that optimizes classification performance, and the manuscript does not define what 'minimum number of demonstrations' means formally. This connection is load-bearing and is never established.","section":"Section 2.4, Eq. (6)"},{"comment":"The paper's own experimental results contradict the claim that the algorithm selects a 'reasonable' number of demonstrations. For example, under Llama: cmc, d=45 while the best random count is 2 (Macro-F1 0.12 vs 0.19); Customers, d=1 while the best is 10 (0.26 vs 0.86); tae, d=1 while the best is 4 (0.28 vs 0.36). Similar large misses occur under Mistral and Qwen (e.g., Qwen iris d=3 vs best 2, Macro-F1 0.06 vs 0.37). Section 3.6 explicitly states that tae and cmc show no obvious spectral gap, yet the algorithm outputs d=1 and d=45/42 for these datasets. These results directly undermine the paper's main claim.","section":"Table 4"},{"comment":"The algorithm resets all k-NN edge weights to 1, so the Laplacian reflects only unweighted k-NN topology, and the Jaccard similarity values are discarded. Because every demonstration shares the fixed template tokens ('input: given', 'is', 'and', 'class:', punctuation), the Jaccard overlap is inflated by prompt-template text and may be dominated by it. The manuscript provides no analysis of how template tokens affect the graph spectrum, so the estimated d may reflect the template rather than the data distribution. This is a serious problem for the validity of Eq. (6) as a data-driven estimate.","section":"Algorithm 1, line 9; Section 2.3"},{"comment":"The claim that the algorithm is 'almost parametric-free except the construction of the sparse graph' is inaccurate. The algorithm has at least two free parameters: the neighborhood size k in the k-NN graph (set to 10) and the number of eigenvalues inspected (50 in the experiments, as stated in Section 3.6). No sensitivity analysis is provided for either parameter, and the choice of 50 is especially consequential because Eq. (6) searches over a fixed prefix of the spectrum; for the cmc dataset, d=45 is close to the boundary of this search window, so the result may be an artifact of the truncation.","section":"Section 1, contribution 2; Section 3.1"}],"minor_comments":[{"comment":"There are numerous typos and grammatical errors, including 'contribuions' (Section 1), 'resonable' (Section 1), 'stablely' (Section 1), 'per-trained' (Algorithm 1), and 'potion' (Section 3.5). These should be corrected.","section":"Throughout"},{"comment":"The symbol k is used both for the k-NN hyperparameter and for the index variable in Eq. (6), which is confusing. Consider renaming one of them.","section":"Equation (6) and Section 3.1"},{"comment":"The caption says 'First 50 eigenvalues (sorted)' but the axes are unlabeled and it is unclear whether eigenvalues are sorted ascending or descending, and whether the gap is λ_{i+1} - λ_i or λ_i - λ_{i+1}. Adding axis labels and a clear description of the gap would help the reader interpret the figure.","section":"Figure 3"},{"comment":"The three rightmost columns are labeled only with model names (llama, Mistral, Qwen) and contain integers, but the caption does not explicitly explain that these are the estimated d values until the text in Section 3.5. This should be stated directly in the caption.","section":"Table 1"},{"comment":"The statement that zero-shot performance is zero for 'all other LLMs and datasets' is not tied to a specific figure or table, and the only exception mentioned (Qwen on wine and iris) is not quantified. Please provide the underlying numbers or refer to the appropriate panel in Figure 2.","section":"Section 3.5"},{"comment":"The HuggingFace reference contains the placeholder '[Insert Date]' and should be completed or removed. Also, several cited works in Section 4 on demonstration selection are discussed only briefly; a short comparison with those methods would strengthen the positioning of the paper.","section":"References"}],"recommendation":"reject","confidential_remarks":"The empirical results in Table 4 are not merely a matter of the method being under-powered; they are direct counterexamples to the central claim. For Llama on Customers, the chosen d=1 yields Macro-F1 0.26 while the best random count 10 yields 0.86, and for cmc d=45 is far outside the evaluated range and performs worse than d=2. Given that the theoretical link is also absent, I do not see a minor revision that could fix the manuscript within its current scope. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is an honest, cheap heuristic for choosing how many ICL demonstrations to put in a tabular classification prompt, but the central claim—that the spectral gap of a token-ID Jaccard k-NN graph gives the 'minimum' number of demonstrations—is neither derived nor supported by the experiments. What is genuinely new: most prior work selects which demonstrations, not how many. The authors frame the count-selection problem and combine token-ID overlap, a k-NN graph, and the normalized Laplacian's eigengap in a way that is not in the cited literature, and the computational cost is low. I also give them credit for honesty: they call it a heuristic, say the performance is not dominant, and note that tae and cmc show no obvious spectral gap.\n\nThat said, the central claim does not hold up. There is no derivation or prior result connecting the Laplacian's eigenvalue gap to the number of demonstrations that helps an LLM classify. Algorithm 1 line 9 resets edge weights to 1, so the graph carries only unweighted k-NN topology, and because every demonstration shares the fixed template tokens ('input:', 'given', 'class:', etc.), Jaccard similarity is inflated by template text and may mostly reflect the prompt skeleton rather than the data. Table 4 undercuts the method: for Llama on Customers, d=1 gives Macro-F1 0.26 while the best random count (10) gives 0.86; on cmc, d=45 against best 2. That is not 'close to the best result.' Wine, iris, penguins, and LED also show large gaps.\n\nThe experiments compare only against random selection, not against any existing demonstration-selection method, and no code is released. The one hyperparameter, k, is not analyzed for sensitivity. The 'minimum number' phrasing is odd, since the algorithm outputs a cluster count from a graph, with no evidence that fewer demonstrations would fail.\n\nWho gets value from this? Someone working on tabular ICL prompts might try it as a cheap starting point, and the framing could seed a better method. As a scientific claim, though, it is not substantiated; I would not cite it in my own work yet.\n\nMy recommendation: send it to review with strong expectations. A serious referee can push the authors to justify the metric, add comparison methods, and report sensitivity to k. The problem is real and the idea is original enough to deserve referee time, even if heavy revision is needed.","headline":"A genuinely new angle on ICL demonstration-count selection, but the spectral-gap-to-count link is asserted rather than shown, and the paper's own tables show the count often misses the best random choice by a wide margin.","tokens_in":9984,"tokens_out":5064,"would_cite":false,"duration_ms":46297,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","05C50","62H30"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes that the number of demonstrations for in-context tabular classification can be read off from the spectral gap of a token-overlap similarity graph, and reports that the resulting demonstration count performs close to the…","keywords":["in-context learning","demonstration selection","tabular data classification","spectral graph theory","spectral gap","large language models","few-shot learning","Jaccard similarity"],"falsifier":"A direct test: for a dataset with known classes, compute $d$ from the spectral gap and compare it with the count in $\\{0,2,4,6,8,10\\}$ that maximizes Macro-F1. The paper's own eigenvalue plots show no visible gap for tae and cmc, yet the algorithm outputs $d=1$ and $d=42/45$ for them; measuring whether those counts are near the accuracy-optimal counts, or far from them, would settle the claim.","tokens_in":8901,"feed_emoji":"📊","tokens_out":6243,"duration_ms":54370,"temperature":0.7,"pith_summary":"The paper asks how many demonstrations—labeled example rows—should be placed in a prompt when using an LLM to classify tabular data, and proposes an automatic answer instead of asking users to guess and check. The proposed algorithm renders each row as a natural-language demonstration, tokenizes it with the chosen LLM, builds a k-nearest-neighbor graph from Jaccard similarity of token-ID lists, and reads the number of demonstrations $d$ off the spectral gap of the graph's normalized Laplacian. The authors report that this automatically chosen $d$ gives classification performance close to the best result obtained by sweeping random demonstration counts, across eight OpenML datasets and three LLMs. A sympathetic reader would take the contribution as a parameter-light heuristic that turns demonstration-count selection into a graph-spectrum computation.","feed_headline":"Spectral gap picks how many examples to show an LLM","feed_subtitle":"For tabular classification, a token-overlap graph finds a demonstration count close to the best tuned choice.","key_machinery":"The load-bearing object is the spectral gap of the normalized Laplacian of a k-nearest-neighbor similarity graph. Nodes are demonstrations, each represented by the token-ID list produced by the selected LLM's tokenizer; edge weights are Jaccard similarities between token lists (Eq. (1)); after keeping only the top-k neighbors and resetting weights to 1, the normalized Laplacian is formed (Eqs. (3)-(5)). The index at which successive eigenvalues are farthest apart, $k = \\arg\\max_i (\\lambda_{i+1}-\\lambda_i)$, is taken to be the number of clusters and therefore the minimum number of demonstrations. Spectral clustering on the corresponding eigenvectors then picks one representative per cluster.","core_discovery":"The central claim is that the number of demonstrations needed for in-context tabular classification is recoverable from the intrinsic representation space of the LLM itself. Each row is turned into a text demonstration, tokenized into a list of token IDs; the overlap structure of those lists is summarized by a similarity graph, and the largest gap between consecutive eigenvalues of the normalized Laplacian (Eq. (6)) estimates the number of clusters, which the algorithm takes as the minimum number of demonstrations $d$. One demonstration is then drawn from each cluster. The paper reports that this $d$ varies across LLMs and datasets (for example, cmc yields 45 for Llama but 42 for Mistral and Qwen, while iris yields 3 for all three), and that classification with these selected demonstrations is stable and close to the best random-selection result rather than dominant.","pith_inferences":["A natural extension the paper leaves implicit is to replace the raw token-ID Jaccard overlap with a task-aware similarity (for example, distance in the LLM's embedding space after dimensionality reduction) and check whether the spectral gap then tracks accuracy-optimal counts more closely on datasets like tae and cmc.","Another testable extension is cluster-aware sampling: the paper draws one random row per cluster; selecting the row closest to each cluster centroid, or sampling proportional to cluster size, could reduce variance in the reported Macro-F1.","Since tokenization depends on the prompt template, the same pipeline could be used to compare templates: templates that produce clearer spectral gaps would be predicted to yield better classification, a hypothesis the paper does not test."],"forward_implications":["Users no longer need to sweep demonstration counts: the algorithm returns a single $d$ per LLM, prompt template, and dataset.","The selected count is tied to the LLM's tokenizer and the prompt template, so the same data can lead to different $d$ for different models, and the method can react to prompt changes.","Because only token IDs are needed, the procedure is cheap relative to embedding-based selection.","Performance is expected to be stable and near the best random-selection result, not necessarily better than a well-tuned random count."],"supporting_citations":[{"why":"Supplies the spectral gap method and the Laplacian machinery that the algorithm uses to estimate the number of clusters.","marker":"Chung (1997)"},{"why":"Provides the spectral clustering background that motivates the k-NN sparse graph and the use of eigenvectors for clustering.","marker":"Von Luxburg (2007)"},{"why":"Establishes that demonstration labels are not essential for ICL performance, justifying a selection driven by token/text similarity; also supplies the Macro-F1 evaluation choice.","marker":"Min et al. (2022)"},{"why":"Contributes the natural-language row-to-sentence representation used to turn tabular rows into demonstrations.","marker":"Borisov et al. (2022)"},{"why":"Also informs the demonstration template and the LIFT-style framing of LLM tabular classification.","marker":"Dinh et al. (2022)"},{"why":"Provides the OpenML platform from which the eight evaluation datasets are drawn.","marker":"Van Rijn et al. (2013)"},{"why":"Supplies the eigenvalue computation and spectral clustering implementation used to calculate the spectral gap.","marker":"Pedregosa et al. (2011)"}],"fun_headline_variants":["Spectral gap picks LLM demonstration count","Automatic demo count via spectral gap","Spectrum tells how many demos LLMs need","Eigenvalues pick demo count for LLM tables"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline rests on treating the largest eigenvalue gap of a token-overlap graph as the correct number of demonstration examples; if that graph's cluster structure does not match the number of examples that actually helps the LLM classify, the returned $d$ has no principled connection to classification performance.","fun_headline_variants_meta":{"raw":{"variants":["Spectral gap picks LLM demonstration count","Automatic demo count via spectral gap","Spectrum tells how many demos LLMs need","Eigenvalues pick demo count for LLM tables"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000784,"raw_usage":{"total_tokens":3413,"prompt_tokens":853,"completion_tokens":2560,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":2502}},"tokens_in":469,"tokens_out":2560,"duration_ms":16796,"temperature":1.0,"reasoning_tokens":2502,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:47:43.938863+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test: for a dataset with known classes, compute $d$ from the spectral gap and compare it with the count in $\\{0,2,4,6,8,10\\}$ that maximizes Macro-F1. The paper's own eigenvalue plots show no visible gap for tae and cmc, yet the algorithm outputs $d=1$ and $d=42/45$ for them; measuring whether those counts are near the accuracy-optimal counts, or far from them, would settle the claim.","supporting_citations":[],"review_version":1}