{"id":"50e481cb-3c03-45df-a70f-fd6a7aaeaa29","arxiv_id":"2607.22969","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"On AG News (n=200), five LLMs show four distinct few-shot regimes (monotonic gain, one-shot rescue, zero-shot-only, and worse-then-partial-recovery), and a substring parser under-scored Llama 3.3 70B by up to 206%.","lead":"Five LLMs respond very differently to few-shot prompting: one improves slightly, one fails without an example and recovers with one, one is best with zero examples, and one gets worse then partly recovers, a U-shaped curve. The paper also shows a substring parser under-measured Llama 3.3 70B by up to 206% and proposes a regex fix.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Regime curves rest on a single demonstration configuration; untested sensitivity to demo selection/order undermines the central claim.","rationale":"The Reader's weakest assumption correctly identifies the single most load-bearing gap: all conclusions about shot-count behavior are computed from one demonstration configuration, and the paper's own references document that this configuration space can reverse few-shot conclusions. The internal parser inconsistency (Listing 1 vs. Section 5.3) is a concrete correctness issue, but it is secondary to the external validity problem: even if every F1 number is arithmetically correct, the central claim about non-universality and non-predictability from scale cannot be established without showing robustness across demonstration configurations. A CONDITIONAL verdict is therefore appropriate, and my stress-test does not change that verdict; it strengthens the reader's rationale for requesting additional evidence.","tokens_in":16597,"tokens_out":5699,"duration_ms":63187,"concrete_test":"Fix the corrected parser, then re-run the full 5-model × 6-shot grid with at least 5 independently sampled demonstration pools, and within each pool test both class-ordered and random presentation at every k. If any equally valid configuration changes the regime label for Llama 3.3 70B (U-curve) or Llama 4 Scout (zero-shot specialist with oscillation), the taxonomy and scale-irrelevance conclusion do not generalize.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The four behavioral regimes—and the central claim that shot-count effects are not monotonic, not universal, and not predictable from scale—are derived from one fixed demonstration configuration: proportional sampling from a single pool, class-ordered presentation, and one seed (Section 4.1). The paper's own cited literature ([7], [8]) shows that demonstration selection and ordering can shift few-shot accuracy by up to ~30%, an effect larger than several of the reported regime differences. Section 8.1 concedes the results 'reflect one specific prompt realization.' This is not a minor caveat: if the Llama 3.3 70B U-curve or the Llama 4 Scout oscillation flattens or changes shape under another equally valid demonstration set, the four-regime taxonomy and the 'not predictable from scale alone' conclusion lose their empirical footing. The F1 arithmetic may be internally consistent, but the generalization of the central claim depends on an untested stability assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a controlled empirical study of five LLMs (Gemini Flash Lite, GPT-4o-mini, Llama 3.1 8B, Llama 3.3 70B, Llama 4 Scout 17B) on AG News (n=200) across six shot counts k ∈ {0,1,2,3,5,8}. It reports macro-F1, 95% bootstrap CIs, permutation-test p-values, and Cohen's d, and claims to identify four behavioral regimes: calibrated performers (Regime I), format-dependent learners (Regime II), zero-shot specialists (Regime III), and a U-shaped curve for Llama 3.3 70B (Regime IV). A second central contribution is the diagnosis and correction of a parsing artifact that allegedly deflated Llama 3.3 70B F1 by up to 206%. The paper concludes that few-shot shot-count effects are not monotonic, not universal across models, and not predictable from model scale alone, and proposes a two-axis taxonomy of few-shot behavior. It also provides practical guidance for model and shot-count selection.","tokens_in":16686,"tokens_out":7898,"duration_ms":79394,"significance":"If the empirical claims hold, the paper would be a useful corrective to the common assumption that more demonstrations are always better or that shot-count effects are scale-driven. The paper has real strengths: it reports point estimates for all 30 configurations, uses bootstrap CIs with B=10,000, gives permutation p-values, and includes an explicit limitations section. The central concept that parser choice can be as consequential as shot count is important for LLM evaluation practice. However, the paper's central qualitative claims rest on a single demonstration configuration and a parser exhibit that is internally inconsistent with its own code, and the statistical machinery for Cohen's d and permutation tests is not clearly valid for macro-F1. These issues must be resolved before the four-regime taxonomy and the U-curve claim can be accepted as established.","major_comments":[{"comment":"The claimed parsing failure is contradicted by the provided code. Listing 1 uses `if label.lower() in raw.lower()`, so the representative output 'I would classify this as: World.' would successfully match 'world' and return 'World'. The paper says the parser fails because of a preceding colon/formatting, but the substring parser does not require exact match. As written, the 19.5–76% PFR table and the '206% deflation' calculation are not supported by the exhibit. Either the deployed parser differs from Listing 1 or the actual raw outputs have a different structure; please clarify and provide a reproducible failure example.","section":"§5.3 / Listing 1"},{"comment":"The four-regime taxonomy and the central 'not monotonic, not universal, not predictable from scale' conclusion are curve-shape claims, but all curves come from one fixed demonstration configuration: one pool, proportional sampling, class-ordered presentation, and one seed (Section 4.1). Section 8.1 concedes the results 'reflect one specific prompt realization,' and the paper's own citations [7], [8] show demonstration selection/order can shift accuracy by up to ~30%. A sensitivity analysis over at least several independently drawn demonstration sets and orders is needed. Without it, the Llama 3.3 70B U-curve and Llama 4 Scout oscillation may be configuration-specific rather than model-general behaviors.","section":"§4.1 / §8.1"},{"comment":"The statistical foundation for Cohen's d is not well-defined. Eq. (12) computes d from 'instance-level F1 contributions,' but macro-F1 is an aggregate class-level statistic and is not decomposable into independent per-instance contributions. Similarly, the permutation test in Eq. (11) appears to 'randomly permute predictions across both conditions' in an unpaired manner, which ignores the pairing of test instances across shot conditions. This undermines the validity of d=10.98 and the permutation p-values in Table 5. Please provide a valid paired permutation scheme (e.g., swapping the two predicted labels within each instance) or explicitly justify the unpaired treatment.","section":"§3.4.3 / Eq. (12) / §6.3"},{"comment":"Regime III is described as 'monotonically degrade' in the abstract and conclusion, but the reported Llama 4 Scout data are not monotonic: 0.877 → 0.695 → 0.820 → 0.704 → 0.746 → 0.695. Section 5.2.4 itself calls the curve 'non-monotonic and high-variance.' This is an internal contradiction in the paper's key categorization and should be corrected throughout.","section":"Abstract / §7.1 / §5.2.4"},{"comment":"The corrected F1 estimator defined in Eq. (9) is computed only over the valid instances with successful parses, and Section 5.3 reports residual PFR=2% at k=5,8 for Llama 3.3 70B. Table 2's 'Llama 3.3 70B (fixed)' row therefore mixes F1 over 200 instances for k=0–3 with F1 over ~196 instances for k=5,8. The U-curve's recovery at k=5 (F1=0.785) may not be directly comparable to the zero-shot baseline or to other models. Please report the corrected F1 with residual failures counted as errors as well, or clearly state that the effective sample sizes differ.","section":"§3.3.3 / Tables 2 and 3"},{"comment":"The 'demonstration disruption' and 'format re-anchoring' account is presented as a 'mechanistic analysis' (contribution 2, Section 1), but no attention, routing, or generation-level evidence is provided; the paper's own Section 8.2 lists attention analysis as future work. This is a post hoc verbal mechanism. Please either support it with measurements or explicitly label it as a hypothesis and soften the contribution claim.","section":"§7.2 / §8.2"}],"minor_comments":[{"comment":"The regex pattern in Eq. (13) is shown as `Science[\\/\\s&]*Technology`, while Listing 2 has `Science[\\/\\s&] *Technology` (with a space before `*`). Please make the canonical pattern consistent.","section":"Eq. (13) / Listing 2"},{"comment":"GPT-4o-mini is called the 'Monotonic Improver,' but k=1 (0.8248) is below zero-shot (0.8446). This is not strictly monotonic; please qualify the description.","section":"§5.2.2"},{"comment":"The text reports 'd for 70B zero-shot vs. two-shot (trough) is 8.1' and 'for Scout zero-shot vs. one-shot is 5.3' but these values and their standard errors are not shown. Please include them or omit.","section":"§6.3 / Table 6"},{"comment":"There are several typos and spacing artifacts, e.g., 'JUL Y 2026' in the header and 'Y et' in the abstract. A careful proofreading pass is needed.","section":"Global"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful core idea and honestly reports limitations, but the parser exhibit is internally inconsistent and the demonstration-sensitivity issue is central to the paper's broad claims. The statistical treatment of macro-F1 also needs to be made rigorous. I did not find evidence of fabrication, but the '206% deflation' and 'mechanistic analysis' claims are currently overstated relative to the evidence shown. The paper would be substantially improved by adding a demonstration-selection sensitivity analysis and by reconciling the parser code with the reported parse-failure rates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The measurements look real and the statistical hygiene is good: fixed seed, temperature 0, bootstrap CIs, permutation tests, per-class breakdown, and a limitations section that actually admits the big caveats. The finding that shot-count effects are model-dependent and non-universal is useful, and the Llama 3.3 70B U-curve is a genuinely interesting datapoint, even if the paper's own citations already show optimal-k varies widely. The PFR analysis is a good idea, and the corrected parser is a practical contribution.\n\nBut there are two soft spots that matter. First, the parsing-artifact story contradicts its own code. Listing 1 uses `label.lower() in raw.lower()`, and the quoted \"corrupted\" output contains \"World\" — it would parse successfully. The paper claims the substring parser fails because the label is preceded by a colon, but substring matching doesn't care about preceding characters. So the mechanism described for the 206% deflation is wrong as written. That doesn't mean parse failures didn't happen; it means the explanation needs to be rewritten, and the example is misleading.\n\nSecond, the abstract and conclusion call Llama 4 Scout's behavior \"monotonic degradation,\" but Table 2 shows oscillation: 0.877 -> 0.695 -> 0.820 -> 0.704 -> 0.746 -> 0.695. That's a plain inconsistency.\n\nThe bigger structural concern is the one the stress-test flags: all regime curves come from one demonstration pool, one ordering, one seed. The paper acknowledges this in Section 8.1, which is more than many papers do, but the acknowledgment is a paragraph, not a sensitivity analysis. If the 70B U-curve or the Scout oscillation flattens under a different equally valid demonstration set, the four-regime taxonomy and the \"not predictable from scale alone\" conclusion lose their empirical footing. For a measurement paper, this is manageable as long as the conclusions are framed as single-configuration observations, not general laws.\n\nThe mechanistic story about \"demonstration-induced reasoning shift\" and \"format re-anchoring\" is speculative, but the paper does label it as a hypothesis and lists attention analysis as future work. That's acceptable.\n\nWho is this for? People who build evaluation pipelines for LLM classification, and anyone doing few-shot benchmarking who wants a cautionary example of how parser choice and demonstration selection can change conclusions. It deserves a serious referee and probably a conditional accept after rewriting the parser section, fixing the monotonic label, and adding a robustness check on demonstration selection or explicitly narrowing the claims.","headline":"Honest, well-run measurement study whose headline claims are undercut by an internal parser contradiction and a single-prompt-configuration design it frankly acknowledges but never stress-tests.","tokens_in":17340,"tokens_out":2254,"would_cite":true,"duration_ms":26631,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Few-shot prompting helps some LLMs, hurts others: optimal shot count is model-specific, not predictable from scale.","keywords":["few-shot prompting","in-context learning","shot count","text classification","large language models","evaluation methodology","output parsing","AG News"],"falsifier":"Re-run the same 30 model–shot configurations with demonstrations selected by similarity or with shuffled ordering instead of the fixed class-ordered pool; if the Llama 3.3 70B U-shape (0.907→0.635→0.785) flattens or vanishes under an equally valid demonstration set, the four-regime taxonomy loses its empirical footing.","tokens_in":16291,"feed_emoji":"📊","tokens_out":2772,"duration_ms":28490,"temperature":0.7,"pith_summary":"This paper tests five large language models at six different shot counts on a four-class news classification task and finds that adding demonstrations helps, hurts, or does nothing depending on the model. It identifies four behavioral regimes: already-calibrated models with negligible gains, one model that jumps from near-random to strong performance with a single example, a model that degrades monotonically with any examples, and one 70B model whose performance forms a U-shape—best at zero-shot, worst at two shots, partially recovering but never matching zero-shot. The paper also shows that a naive output parser can deflate measured F1 by more than the true effect, up to 206% for one model, making parser choice a first-order methodological concern.","feed_headline":"More examples can make LLMs worse: shot count is model-specific","feed_subtitle":"Five models show four distinct few-shot curves; one 70B model peaks at zero-shot, and parser choice swings F1 by up to 206%.","key_machinery":"The central object is the F1-versus-shot-count curve, measured across 30 model–configuration pairs and evaluated with bootstrap confidence intervals, permutation tests, and Cohen's d. The paper interprets the curves through a two-axis taxonomy: strength of the model's zero-shot classification prior versus sensitivity to format disruption from demonstrations. The mechanism proposed for the U-shaped curve is a competition between 'demonstration disruption' (which dominates at low shot counts and triggers deliberative reasoning that degrades label precision) and 'format re-anchoring' (which dominates at higher shot counts and partially restores concise outputs). The corrected regex parser—which","core_discovery":"The central claim is that the relationship between shot count and classification performance is not monotonic, not universal, and not predictable from model scale alone. Concretely, across five models and six shot counts (k=0,1,2,3,5,8), the authors observe four distinct regimes: Regime I (Gemini, GPT-4o-mini) shows modest, statistically insignificant gains; Regime II (Llama 3.1 8B) shows catastrophic zero-shot failure (F1=0.525) that recovers dramatically with one shot (F1=0.865, Cohen's d=10.98) before collapsing again at eight shots; Regime III (Llama 4 Scout) is optimal at zero-shot (F1=0.877) and degrades with examples; Regime IV (Llama 3.3 70B) traces a U-curve—0-shot F1=0.907, 2-shot","pith_inferences":["If the regime taxonomy holds beyond this single dataset and prompt realization, benchmarking claims about 'few-shot performance' of LLMs should be re-stated as conditional on demonstration selection, ordering, and parsing strategy.","The U-shaped curve in Llama 3.3 70B might be specific to four-class topic classification with short labels; testing on tasks with longer label sets or different label semantics would show whether the disruption mechanism generalizes.","An attention-based test of the proposed mechanism is feasible: measuring how much attention the model allocates to demonstrations versus the test article as shot count increases would directly confirm or reject the 'demonstration disruption and format re-anchoring' account.","The cost-latency implication is that zero-shot prompting is not only better but cheaper for Regime III and IV models, so efficient deployment can align with better accuracy."],"forward_implications":["Practitioners cannot assume that 3–5 demonstrations is universally best; the optimal shot count ranges from 0 to 8 across the five models tested, and choosing the wrong value can change F1 by as much as 0.34.","For models with strong zero-shot priors, adding any demonstrations can strictly hurt performance, so zero-shot evaluation should always be included as a baseline before paying for few-shot prompts.","Output parsing is not an implementation detail: a broken parser can flip the ranking of models, and parse failure rate should be reported alongside F1 as a standard transparency metric.","Large scale does not predict few-shot benefit; the 70B model gained nothing from demonstrations while the 8B model gained enormously, contradicting the intuition that more capable models benefit more from examples.","A practical decision protocol is suggested: profile zero-shot performance first; if F1 is high, skip few-shot; if low, add one or two examples; always use a robust parser."],"fun_headline_variants":["Few-shot curves: not universal, not monotonic","Parser artifact deflated 70B model by 206%","More examples can hurt: shot count is model-specific","Four regimes, five models: few-shot effects vary","Zero-shot isn't always baseline: parser matters"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"All five performance curves come from a single fixed demonstration pool with class-ordered presentation and one test seed, so the observed regimes may reflect the specific prompt realization rather than a general law of few-shot behavior.","fun_headline_variants_meta":{"raw":{"variants":["Few-shot curves: not universal, not monotonic","Parser artifact deflated 70B model by 206%","More examples can hurt: shot count is model-specific","Four regimes, five models: few-shot effects vary","Zero-shot isn't always baseline: parser matters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1407,"prompt_tokens":1012,"completion_tokens":395,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":756,"completion_tokens_details":{"reasoning_tokens":318}},"tokens_in":756,"tokens_out":395,"duration_ms":3806,"temperature":1.0,"reasoning_tokens":318,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T04:01:16.709211+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 30 model–shot configurations with demonstrations selected by similarity or with shuffled ordering instead of the fixed class-ordered pool; if the Llama 3.3 70B U-shape (0.907→0.635→0.785) flattens or vanishes under an equally valid demonstration set, the four-regime taxonomy loses its empirical footing.","supporting_citations":[],"review_version":1}