{"id":"b8ced9c0-8a8a-4f77-b058-37a14285fd62","arxiv_id":"2505.05744","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LLM-generated post hoc explanations improve demonstration selection and few-shot accuracy of a smaller surrogate language model on four tabular classification datasets.","lead":"This paper describes a system that asks a large language model to explain why certain data rows get certain labels, then uses those explanations to pick examples and prompt a smaller model that runs locally. The reported benefit is roughly 5% higher average accuracy across four tabular datasets at a fraction of the cost of querying the large model at inference time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 3 with n=5 and p=0.85 selects no features, so the method as written cannot produce the reported results.","rationale":"The reader's weakest assumption concerned explanation quality and the transfer of GPT-3.5 attributions to Llama2-7B, focused on Eqs. 3 and 4. My stress-test found a more fundamental issue in the same equations: the importance score as defined cannot exceed 0.2 when n=5, yet the threshold is set to 0.85, so W is necessarily empty. This makes the method as described internally inconsistent, which is a correctness risk distinct from the untested quality of the explanations. It is the single most load-bearing concern because the explanation-guided filter is the core novel component: if this stage cannot operate as written, the reported empirical improvements cannot be traced to the specified mechanism. A simple typographical fix (e.g., g(w_j) = (1/M) * sum_i 1{w_j in e_i}) would make p=0.85 plausible, so the paper might still be salvageable. That is why I keep the verdict at CONDITIONAL rather than REJECT: the concern is decisive only in the absence of a correction or code. The concrete test would settle it by forcing a precise statement of the actual filtering rule and checking whether the headline numbers survive that rule. I am not questioning the authors' honesty; the issue is purely that the mathematical definitions and hyperparameters are mutually inconsistent as published, and no implementation is provided to disambiguate them.","tokens_in":11608,"tokens_out":12174,"duration_ms":135365,"concrete_test":"Implement Eq. 3 exactly as written on any of the four datasets with n=5 and M=100; compute max_j g(w_j) and then W from Eq. 4 with p=0.85. This will confirm W is empty. Then request or inspect the released code: if g is actually normalized by M, or if p is treated as a percentile/quantile threshold, rerun the full pipeline under each interpretation and check whether the Table 3 numbers (e.g., average 65.06% and the 5.31% gain) reproduce. If the implementation uses an undocumented normalization, the paper must state it; if the corrected formula changes the results, the formal definition is load-bearing for the central claim.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central novelty is the explanation-guided feature filter of Section 3.3, but the formal definition is inconsistent with the experimental setting. In Eq. 3, g(w_j) = (1/(n*M)) * sum_i 1{w_j in e_i}. Because each explanation e_i contains n words, the maximum possible value is g(w_j) = 1/n; with n=5, this maximum is 0.2. The paper sets the filtering threshold to p=0.85 (Section 4.2), and Eq. 4 defines W as the features whose importance passes this threshold. Since 0.85 > 0.2, W is empty. Under Eq. 5, filtering with an empty W either removes every feature sentence from the serialized demonstrations or is undefined. Therefore, the pipeline described by Eqs. 3-5 with the stated hyperparameters cannot yield the results in Tables 3 and 5. The empirical claim is thus tied to an undocumented alternative rule: either g is normalized differently (e.g., by M rather than n*M, which would make max g = 1), or p is interpreted as a quantile rather than an absolute threshold, or another filtering criterion is used. Without code or a corrected formal description, the central claim is not reproducible as written. This is an internal inconsistency in the core mechanism, not merely a disagreement with external consensus.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a three-stage in-context learning framework for tabular classification. In stage 1, GPT-3.5 generates post hoc feature-attribution explanations (n words) for M=100 candidate demonstration question-answer pairs. In stage 2, explanations are aggregated into feature importance scores g(w_j) (Eq. 3), a threshold p and top-q selection define a feature set W (Eq. 4), and serialized demonstrations are filtered to W (Eq. 5), embedded with SentenceBERT, and scored by cluster/cosine/Euclidean/Manhattan methods to select k-shot demonstrations. In stage 3, selected demonstrations are augmented with their explanations and fed to Llama2-7B, whose outputs are parsed into answer and explanation. Experiments on bank, creditg, heart, and income report 4-shot accuracy of 65.06% on average, a 5.31% improvement over the best average baseline, lower API usage than AO(ChatGPT), and ablations attributing gains to both explanation augmentation and guided selection.","tokens_in":11831,"tokens_out":6799,"duration_ms":68929,"significance":"The core idea is reasonable and of practical interest: use a strong LLM once to generate explanations that guide a locally deployable SLM, thereby reducing API calls while retaining interpretable rationales. The paper explicitly compares API usage and includes ablations and hyperparameter analysis. However, the current evidence is thin: four small-to-medium tabular datasets, one explanation generator (GPT-3.5), one surrogate (Llama2-7B), no statistical significance testing, and no code or data release. More importantly, the formal definition of the filtering step is inconsistent with the stated hyperparameters, so the reported numbers cannot be reproduced as written. If that inconsistency is resolved and the claims are re-validated, the framework would be a useful contribution; in its current form, the central empirical claim needs substantial revision.","major_comments":[{"comment":"Eq. (3) defines g(w_j) = (1/(n*M)) * sum_i 1{w_j in e_i}. Since each explanation e_i contains exactly n words, the maximum possible value of g(w_j) is 1/n. With n=5 and threshold p=0.85 (Section 4.2), no feature can satisfy the threshold, so W is empty. Eq. (5) then removes every feature sentence from S, leaving degenerate demonstrations; the selection and prediction stages of Eqs. (6)-(13) cannot yield the nondegenerate accuracies in Tables 3 and 5. Either g is normalized differently (e.g., by M), p is interpreted as a quantile, or an undocumented filtering criterion was used. The authors must correct the formal definition and provide code or a detailed algorithm trace showing a non-empty W for all datasets and seeds.","section":"Section 3.3, Eqs. (3)-(5), and Section 4.2"},{"comment":"The sentence 'achieving a more than 5% performance improvement for all settings' is contradicted by the table. Per dataset, the improvements over the best baseline are 0.26 points on bank (55.26 vs SCARF 55.00), 0.00 on creditg (64.33 vs STUNT 64.33), 0.06 on heart (73.06 vs RF 73.00), and 0.58 on income (67.58 vs LR 67.00). The 5.31% figure refers only to the difference between the average accuracy and the average of the LR baseline. Moreover, several of these gaps are smaller than the reported standard deviations; no significance test is provided. The performance advantage claim must be restated and supported with error bars or statistical tests.","section":"Section 4.4, Table 3"},{"comment":"The text states that our approach 'achieves a higher average accuracy (6.18% improvement over AO(ChatGPT))', but Table 4 reports 65.06 vs 59.90, a difference of 5.16 percentage points. This numerical inconsistency must be corrected.","section":"Section 4.5, Table 4"},{"comment":"The paper claims interpretability as a core contribution (Challenge 3 in the Introduction), yet no evaluation of explanation quality, faithfulness, or stability is reported. Since the whole framework depends on GPT-3.5 explanations being informative and stable across the M=100 candidate pool, the absence of any analysis of explanation correctness or seed sensitivity leaves a central assumption unverified. Please add quantitative or qualitative evaluation of explanations and an analysis of how W varies across candidate pools and seeds.","section":"Section 3.2 and Section 4"}],"minor_comments":[{"comment":"The notation Topq((w_i, g(w_i), p)) is ambiguous: it should specify whether p is an absolute cutoff on g, a rank quantile, or something else; this ambiguity is directly related to Major Comment 1.","section":"Eq. (4)"},{"comment":"Standard deviations are given as subscripts, but in the plain-text rendering they run into the values (e.g., '51.0013.00'), making the tables hard to read. Please format them with clear separators.","section":"Tables 3 and 5"},{"comment":"There are typos in the figure labels and captions: 'Eistance' should be 'Distance', 'credigt' should be 'creditg', and 'demonstrates' appears where 'demonstrations' is intended.","section":"Figures 3 and 4"},{"comment":"Please clarify how the API usage counts are computed for each row, e.g., 100 explanation-generation calls for ours(Llama2-7B), 2400 inference calls for AO(ChatGPT), and 2500 for ours(ChatGPT).","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"I recommend requesting code and data release before further review; without them, the filtering inconsistency cannot be resolved and the results are not independently verifiable. The authors should also report exact prompts and parsed explanations, since the pipeline depends on proprietary API outputs. In its current state the paper is not ready for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the paper's central mechanism cannot work as written. Eq.3 defines g(w_j) = (1/(n*M)) * sum_i 1{w_j in e_i}. Since each explanation e_i has exactly n words, the maximum possible value is 1/n. They set n=5 and the filtering threshold p=0.85 in Section 4.2, so the maximum importance any feature can get is 0.2. Eq.4 then selects features whose importance passes 0.85, which means W is empty, and Eq.5 filters everything out. The pipeline as formally described cannot produce the numbers in Tables 3 and 5. Either the implementation normalizes g differently, or p is treated as a quantile, or some other rule was used. No code or data is released to check. This is a load-bearing inconsistency in the core method, not a minor typo.\n\nWhat is worth saving: the idea is reasonable. Use a strong LLM once to write post hoc explanations for a small candidate pool, then use those explanations to filter features for demonstration selection and as rationales for a small local model. That is a sensible cost-shifting and interpretability story. The ablations in Table 5 move in the expected direction, and the 100 vs 2400 API-call comparison is the most concrete benefit.\n\nBut the evidence is thin. Four datasets, one SLM backbone (Llama2-7B), one explanation generator (GPT-3.5), no code, no release. Some baseline numbers look implausible — sub-50% accuracy on a binary dataset like creditg is hard to believe — so the claimed 5.31% average improvement over a mixed baseline set is not credible as presented. There is also no analysis of explanation quality or stability across seeds, datasets, or test rows, which matters because the whole pipeline trusts GPT-3.5's attributions.\n\nVerdict: I would not trust the numbers as they stand. I would still send this to peer review, because the idea has practical value and a referee can demand the corrected formal description, proper baseline checks, and code. But I would expect major revision and, ideally, code/data release before anything like acceptance. As is, I would not cite it.","headline":"The idea is sensible, but the method as written cannot produce the reported results — Eq.3 with n=5 and p=0.85 selects no features.","tokens_in":12395,"tokens_out":4197,"would_cite":false,"duration_ms":42081,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Explanations generated once by GPT-3.5 can be turned into filtered demonstrations and rationale text that let a 7-billion-parameter surrogate model beat all baselines, including ChatGPT-driven in-context learning, on four tabular…","keywords":["in-context learning","tabular data classification","surrogate language model","post hoc explanations","demonstration selection","few-shot learning","large language models","interpretability"],"falsifier":"Replace GPT-3.5's explanations with uniformly random feature subsets of the same average size, keep the same p=0.85 filter and cosine selection, and rerun the four datasets; if average 4-shot accuracy stays near 65.06%, the explanation content is not carrying the gain, whereas a large drop would confirm it is.","tokens_in":11367,"feed_emoji":"📊","tokens_out":15541,"duration_ms":155338,"temperature":0.7,"pith_summary":"The paper tries to establish that a small, locally deployable 7-billion-parameter language model can beat both classical tabular learners and direct few-shot prompting of a much larger API-only model, provided it is fed demonstrations selected and filtered by that larger model's post hoc explanations. The proposed framework generates feature-attribution explanations for 100 candidate rows, aggregates them into per-feature importance scores, keeps only the features that clear a threshold, and uses the cleaned rows to choose four demonstrations. Each selected demonstration has its explanation appended as a rationale, and the surrogate model is prompted to answer with a label and an explanation. Across the bank, creditg, heart, and income datasets, the paper reports 65.06% average 4-shot accuracy, 5.31 points above the strongest baseline, and higher accuracy than ChatGPT-based in-context learning while using 100 API calls instead of 2,400. The result matters because it suggests the expensive model can be consulted once, offline, while a cheap open model carries the repeated inference load.","feed_headline":"A 7B model with LLM explanations outscores ChatGPT on tables","feed_subtitle":"GPT-3.5 attributions filter features, pick examples, and lift 4-shot accuracy to 65% with only 100 API calls.","key_machinery":"The load-bearing object is the post hoc explanation set $e_i$, a short list of feature names an LLM produces for candidate row $i$. From $M=100$ such sets the framework computes $g(w_j)=\\frac{1}{nM}\\sum_{i=1}^{M} \\mathbf{1}\\{w_j\\in e_i\\}$ (Eq.~3), the frequency with which feature $w_j$ is named; the top-$q$ features whose importance clears threshold $p=0.85$ form the allowed vocabulary $W$ (Eq.~4). Every candidate's serialized text is filtered down to sentences mentioning $W$ (Eq.~5), then embedded with SentenceBERT (Eq.~6). The demonstration score $s_i$ comes from one of four selection rules—K-Means distance, cosine similarity, Euclidean distance, or Manhattan distance (Eqs.~7--10)—and the top-$k$ rows are kept (Eq.~11). The mechanism does two jobs at once: filtering removes spurious features before embedding, so similarity is computed on a cleaner representation, and appending $e_i$ to each selected demonstration (Eq.~12) gives the SLM a rationale to imitate when it outputs a prediction plus explanation (Eqs.~13--14).","core_discovery":"On the paper's own terms, the central discovery is that the bottleneck in few-shot tabular classification with language models is not the size of the predictor but the information that reaches it. The authors build a three-stage loop: GPT-3.5 writes, for each of $M=100$ candidate rows, the $n=5$ feature names it judges most important for the answer; those explanations are aggregated into per-feature importance scores; features whose aggregate importance clears the threshold $p=0.85$ are kept and all other feature sentences are stripped from the serialized rows; and the cleaned rows are embedded and scored against the test row to choose the $k=4$ demonstrations. Those demonstrations are shown to Llama2-7B with their explanations appended as rationales, and the surrogate replies with both a class label and an explanation. The reported result is 65.06% average 4-shot accuracy on bank, creditg, heart, and income, 5.31 points above the strongest baseline, and higher accuracy than ChatGPT-based in-context learning while using 100 API calls instead of 2,400.","pith_inferences":["[Editorial inference] The filter-and-select stage is effectively a feature-selection regularizer; comparing explanation-guided filtering against equal-sized random or variance-based feature deletion would show whether the explanatory content itself, rather than simple dimensionality reduction, drives the gain.","[Editorial inference] Since the expensive model is queried only 100 times up front, the same loop could plausibly be run with a small local explainer in place of GPT-3.5; if accuracy held, the method would need no commercial API after the warm-up stage.","[Editorial inference] The reported average covers four binary datasets, so the next natural test is whether the margin survives multi-class or high-cardinality categorical tables, where attribution quality is harder to judge."],"forward_implications":["A 7B surrogate running this pipeline reaches 65.06% average 4-shot accuracy on bank, creditg, heart, and income, topping every baseline tested.","Each component matters: removing the appended explanations drops the average by about 8 points, removing the explanation-guided selection drops it by about 8 points, and removing both returns to the 53.94% plain in-context-learning baseline.","The framework moves almost all API cost to a one-time offline explanation stage: the Llama2-7B variant uses 100 GPT-3.5 calls, versus 2,400 for ChatGPT-based in-context learning, and still reports higher average accuracy (65.06% vs 59.90%).","Predictions arrive with an explanation string from the surrogate, so the interpretability is part of the output rather than a separate post-processing step.","Under cosine-similarity selection, the method beats random demonstration selection on all four datasets, with or without the feature filter."],"supporting_citations":[{"why":"Provides the Llama2-7B surrogate model whose boosted 4-shot predictions are the reported result.","marker":"[33]"},{"why":"TABLET is the closest baseline that also enriches tabular prompts with external rule/prototype information; the framework is compared against it.","marker":"[30]"},{"why":"The source of the AO in-context-learning baseline used in the main comparison, including the ChatGPT API-cost analysis.","marker":"[35]"},{"why":"Chain-of-thought prompting established that appended rationales change model reasoning, motivating the explanation-as-rationale step.","marker":"[36]"},{"why":"Provides the Bank marketing dataset on which the reported 55.26% accuracy is measured.","marker":"[25]"},{"why":"Cited source for the creditg benchmark used in the reported average.","marker":"[17]"},{"why":"Cited as the data source for the Income benchmark used in the reported average.","marker":"[2]"}],"fun_headline_variants":["Small LM beats ChatGPT on tables with GPT-3.5 explanations","100 API calls: GPT-3.5 explanations make a 7B model beat ChatGPT","Explanations from LLMs turn a 7B surrogate into a ChatGPT-beating table classifier","GPT-3.5 explanations lift a 7B model 5.31% past ChatGPT on tabular tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline rests on the assumption that the feature names GPT-3.5 writes for 100 candidate rows form a reliable map of which features drive the label, so that deleting every feature absent from that map (threshold p=0.85) removes noise rather than signal; the paper does not measure the quality or seed-to-seed stability of those explanations.","fun_headline_variants_meta":{"raw":{"variants":["Small LM beats ChatGPT on tables with GPT-3.5 explanations","100 API calls: GPT-3.5 explanations make a 7B model beat ChatGPT","Explanations from LLMs turn a 7B surrogate into a ChatGPT-beating table classifier","GPT-3.5 explanations lift a 7B model 5.31% past ChatGPT on tabular tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000274,"raw_usage":{"total_tokens":1683,"prompt_tokens":1031,"completion_tokens":652,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":647,"completion_tokens_details":{"reasoning_tokens":556}},"tokens_in":647,"tokens_out":652,"duration_ms":6436,"temperature":1.0,"reasoning_tokens":556,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:56:35.012308+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace GPT-3.5's explanations with uniformly random feature subsets of the same average size, keep the same p=0.85 filter and cosine selection, and rerun the four datasets; if average 4-shot accuracy stays near 65.06%, the explanation content is not carrying the gain, whereas a large drop would confirm it is.","supporting_citations":[{"cited_title":"Advances in neural information processing systems34, 23928– 23941 (2021)","cited_arxiv_id":null,"evidence_quote":"Cited source for the creditg benchmark used in the reported average."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited as the data source for the Income benchmark used in the reported average."}],"review_version":1}