{"id":"9c9b90dd-d438-4f90-9ff8-a23a73f23283","arxiv_id":"2607.16202","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Sub-3B open models can already handle structured MCQ-style tasks, and cheap 4-bit adapter fine-tuning lifts the best ones by up to ~27 points on a held-out split.","lead":"This paper evaluates nine small open-weight language models (135M to 3B parameters) on a custom 1,085-question multiple-choice benchmark, then fine-tunes five of them with low-cost adapter methods to show how much local specialization can improve structured task accuracy. It is useful as a practical playbook for organizations that want to pick and adapt a small model they can run on their own hardware.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuning gains rest on a single 108-example split; base scores on this split diverge from full-benchmark results by up to 10 points, so the reported deltas are statistically fragile.","rationale":"The reader's weakest assumption — that the 108-example held-out split is a stable estimator of fine-tuning gains — is indeed the most load-bearing concern. It directly affects the quantitative claims that distinguish this paper from a pure base-model benchmark study. The full-benchmark base results (n=1,085) are solid, but the fine-tuning deltas are the evidence for the 'low-cost adaptation' half of the central claim, and they rest on a split with a tiny sample size. A different split could plausibly change the deltas, especially for the smaller models where the Wilson intervals already overlap. The paper is commendable for reporting Wilson intervals and acknowledging the limitation in Section 11, but the headline presentation (e.g., abstract) does not convey the fragility. This is an addressable weakness: multi-seed evaluation would either confirm or overturn the finding. Therefore the verdict remains CONDITIONAL, requiring the release of multiple seeds or a larger held-out split before the quantitative fine-tuning claim can be fully accepted. The reader's secondary concerns (missing data/code) and the paper's own scope caveats are noted but do not change the central assessment.","tokens_in":10829,"tokens_out":8252,"duration_ms":80362,"concrete_test":"Re-run the fine-tuning evaluation under 10 different 80/10/10 stratified splits (seeds 0–9) and report the distribution of deltas for each model, along with bootstrap CIs. Check whether the signs and ordering of deltas remain stable; also run a paired McNemar test on the current split to identify significant improvements.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative evidence for the claim that PEFT is decisive comes from Table 4, which reports deltas on a single 108-example held-out split (seed 42, Section 5.3). This split is small and evidently unrepresentative: Qwen Coder 3B's base accuracy on the split is 65.74% versus 75.67% on the full 1,085-example benchmark, while Qwen2.5 1.5B's split base is higher (70.37%) than its full-benchmark score (67.10%). Such deviations are within sampling error for n=108, but they mean the deltas are measured relative to a noisy baseline. Wilson 95% CIs for SmolLM2 360M ([26.8,44.6] vs [36.3,54.8]) and SmolLM2 135M ([18.6,34.9] vs [23.5,40.7]) overlap heavily, so the reported +10.18 and +5.55 improvements are not statistically significant. With a different seed, the ordering of deltas could change, and the claim that fine-tuning is 'frequently decisive' could be weakened. The paper acknowledges the small split in Section 11, but the abstract and headline deltas are presented without this caveat. The 108-example split is the load-bearing element for the adaptation component of the workflow.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a two-stage study: (i) construction of a controlled 1,085-item, 16-topic multiple-choice benchmark for structured local deployment, and (ii) a parameter-efficient fine-tuning (PEFT) pipeline using 4-bit NF4 quantization with DoRA/LoRA-style adapters applied to five models. Base evaluation of nine 135M–3B models shows Qwen Coder 3B leading at 75.67% strict accuracy, followed by Qwen2.5 1.5B (67.10%) and Qwen3.5 2B (64.98%). On a shared 108-example held-out split, fine-tuning yields positive observed gains for all five adapted models, with the largest gains for Qwen Coder 3B (+26.85 pts) and SmolLM2 1.7B (+25.92 pts). The paper argues that a reproducible workflow of benchmarking, cross-model evaluation, and low-cost specialization makes a subset of sub-3B models viable as local experts for structured niche workloads.","tokens_in":11114,"tokens_out":4676,"duration_ms":42268,"significance":"The base evaluation is a model of methodological discipline: deterministic decoding, strict accuracy with format validity, Wilson confidence intervals, McNemar tests, topic-conditioned and difficulty-stratified analyses, and a transparent split. The benchmark and scripts appear reproducible, and the paper correctly frames the results as benchmark-conditional. If the fine-tuning results are robust, the work would be a valuable demonstration of PEFT for democratized deployment. However, the PEFT evidence is currently weaker than the base evidence: the 108-item single-seed split yields wide intervals and overlapping CIs for two of the five models, and the selection of fine-tuned models/configurations is not fully specified. The central claim that PEFT is 'frequently decisive' therefore needs either additional experiments or more cautious language.","major_comments":[{"comment":"The fine-tuning deltas for SmolLM2 360M and 135M are not statistically significant: Wilson 95% CIs overlap heavily (360M: [26.8,44.6] vs [36.3,54.8]; 135M: [18.6,34.9] vs [23.5,40.7]). The §7.2 statement that 'PEFT is frequently decisive' is unsupported for these models. Moreover, the single seed-42 split (§5.3) yields baseline accuracy on the split that differs from the full benchmark by up to 10 points (e.g., Qwen Coder 3B 65.74 vs 75.67), so the deltas are measured against a noisy baseline. Please either provide multi-seed or larger held-out evaluation, or restrict the adaptation claims to the three models with non-overlapping intervals.","section":"§7.1, Table 4"},{"comment":"The criteria for choosing the five models for fine-tuning and the adapter rank (r=16 default) are not stated. If the choice was informed by base performance on the same 108-item split, the reported gains are vulnerable to selection bias. The paper should report the selection rule and ideally validate on a truly held-out split (e.g., a second split used only after model selection).","section":"§5.3, §6"}],"minor_comments":[{"comment":"The aggregate '+17.59 points mean improvement' mixes statistically significant and non-significant deltas; consider reporting it only for models with non-overlapping CIs or with a significance threshold.","section":"§7.2"},{"comment":"The paper refers to the 'v3 benchmark' without explaining earlier versions; define or remove the version label.","section":"§3.1"},{"comment":"The PEFT configuration uses RSLoRA, but no citation or explanation is given; please add a reference or a brief description.","section":"§5.3"},{"comment":"The McNemar discordant counts (203 vs. 199) are not self-explanatory; specify which model is the reference in each pairwise comparison.","section":"§6.1"},{"comment":"Section 11 correctly flags the small split, but the abstract and Section 7 present the fine-tuning results without the same caveat; the headline deltas should be qualified as single-split estimates.","section":"Abstract, §7"}],"recommendation":"major_revision","confidential_remarks":"The base evaluation is strong and likely publishable, but the fine-tuning claims need strengthening. I would request a revision that either adds multi-seed/larger split results or explicitly restricts the adaptation claims to the three models where the gains are statistically significant. The paper's own Section 11 acknowledges the need for this, so the revision should be straightforward in principle."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the base-evaluation work is the real contribution; the fine-tuning section is interesting but statistically fragile. The paper is honest about its scope, which earns it credit.\n\nWhat's new: a purpose-built 1,085-example, 16-topic MCQ benchmark with a strict one-letter output contract, plus a nine-model comparison under fixed decoding. That's not a big theoretical leap, but it's a usable instrument. The base eval is handled well: Wilson intervals, McNemar tests for adjacent ranks, difficulty-stratified analysis, failure composition, and cost frontiers. The authors explicitly say the findings are dataset-conditional, which is refreshingly disciplined.\n\nThe soft spot is the fine-tuning section. The headline deltas come from a single 108-example held-out split at seed 42. That split is not a stable baseline: Qwen Coder 3B's base accuracy drops from 75.67% on the full benchmark to 65.74% on the split, while Qwen2.5 1.5B goes in the opposite direction. The Wilson intervals for the two SmolLM2 models overlap heavily before and after tuning, so the reported +10.18 and +5.55 gains are not statistically meaningful. The large deltas for Qwen Coder 3B and SmolLM2 1.7B are credible even with n=108, but they should be treated as pilot results. The paper does acknowledge the small split in Section 11 and shows the intervals in Table 4, but the abstract presents the raw deltas without that caveat, which overstates what the evidence supports.\n\nAnother issue: the reproducibility statement talks about a released specification, but the code and data are not actually shipped. For a paper whose whole pitch is reproducible workflow, that gap matters. Releasing artifacts and adding even a few seeds would make the fine-tuning claims genuinely testable.\n\nCitation pattern is fine — LoRA, QLoRA, DoRA, SmolLM2, Qwen, all standard and correctly credited.\n\nBottom line: a useful, honest empirical study with one load-bearing weakness in the PEFT section. It deserves peer review, and a good referee will ask for artifact release and multi-seed evaluation, which are addressable. I'd cite it for the benchmark and workflow, and I'd bring it to a reading group to discuss how much a single small split can support a \"frequently decisive\" fine-tuning claim.","headline":"A solid, clearly-scoped small-model benchmark study whose PEFT claims rest on a single 108-item split; worth refereeing, but only after artifact release and multi-seed evidence.","tokens_in":11657,"tokens_out":2538,"would_cite":true,"duration_ms":25436,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A disciplined workflow of benchmark construction, cross-model evaluation, and low-cost adapter fine-tuning makes sub-3B open-weight language models viable as local experts for structured niche workloads.","keywords":["small language models","parameter-efficient fine-tuning","local deployment","structured benchmark","LoRA/DoRA","quantization","AI democratization","multiple-choice evaluation"],"falsifier":"An experiment that fine-tunes the same five models on, say, five different random splits with a 500-item held-out set; if the observed deltas shrink (e.g., the largest gain drops below +10 points) or if the gain for the 135M model reverses sign, the claim that low-cost PEFT is decisive for sub-3B models would not survive.","tokens_in":10640,"feed_emoji":"🤖","tokens_out":4444,"duration_ms":40333,"temperature":0.7,"pith_summary":"The paper argues that democratizing AI is not about matching frontier-scale generality; it is about giving ordinary institutions a reproducible workflow for selecting, auditing, and specializing small models under modest hardware. To test that, it builds a 1,085-item, 16-topic benchmark where every answer must be a single correct letter, evaluates nine open models from 135M to 3B parameters, and fine-tunes five of them with cheap quantized low-rank adapters on an L4-class GPU. On the base benchmark the best model scores 75.67% strict accuracy; after fine-tuning on a 108-example split it reaches 92.59%, and the mean gain across the five adapted models is +17.59 points. The takeaway is that a bounded, well-scored task plus low-cost specialization makes sub-3B models viable as local experts.","feed_headline":"3B model gains 27 points from cheap local fine-tuning","feed_subtitle":"Structured benchmarks plus low-rank adapters make sub-3B open models viable as on-site experts.","key_machinery":"The load-bearing machinery is the benchmark itself plus the fine-tuning recipe. The benchmark fixes a single output contract — exactly one option letter — and scores strict correctness, so formatting is part of correctness. The adaptation pipeline freezes a 4-bit quantized backbone, trains low-rank adapters (LoRA style, with DoRA's direction/magnitude decomposition), and optimizes only the answer span via a completion-only collator. That combination lets a small lab, on a single L4-class GPU, measure which models' decision boundaries improve under supervision and which remain capacity-bound.","core_discovery":"The central discovery is a conditional one: a meaningful subset of compact open-weight models already satisfies a strict, format-constrained answer contract on structured micro-tasks, and parameter-efficient fine-tuning can convert that base competence into substantially higher operating accuracy. The paper shows the largest gains appear in models that already have strong format control but retain semantic headroom, which is why it frames PEFT as decision-boundary specialization rather than output-shape repair. The strongest result: after adaptation, the leading 3B model reaches 92.59% strict accuracy on the shared held-out split, a +26.85 point jump, while all five adapted models improve by","pith_inferences":["The 108-example held-out split is small enough that the ordering of gains could be unstable; repeating the fine-tuning across multiple seeds and larger test sets would either confirm or overturn the claim that the top model's +26.85 point jump is robust.","The benchmark's template structure may understate the gap to real user traffic; testing the same workflow on more linguistically diverse, freely generated prompts would show whether the gains are durable.","If the local-expert story is right, routing each task to a specialist small model could beat a single large model on cost and accuracy per dollar, creating an economic incentive for SLM-based pipelines."],"forward_implications":["A small organization can now run, audit, and specialize useful open models on commodity hardware for bounded tasks.","PEFT is not a universal fix: it helps most when the base model already has strong format control, so the gains are concentrated in specific model-task combinations.","The preferred way to think about deployment is a portfolio of local experts matched to task families, not a single winner-take-all leaderboard.","Fine-tuning can sharpen some topic manifolds while slightly degrading others, so post-tuning validation must be topic-resolved."],"fun_headline_variants":["Low-cost fine-tuning lifts 3B model accuracy by 27 pts","Sub-3B models become viable experts with structured benchmarks","Compact LLMs get 27-point boost from parameter-efficient tuning","Local AI: 3B model reaches 92.6% after fine-tuning","Benchmark shows sub-3B models are local specialists"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's fine-tuning conclusions rest on the assumption that a single 108-example, one-seed held-out split gives a stable estimate of how much each model improves; the paper's own confidence intervals are wide enough that a different split could reorder the gains or erase them for smaller models.","fun_headline_variants_meta":{"raw":{"variants":["Low-cost fine-tuning lifts 3B model accuracy by 27 pts","Sub-3B models become viable experts with structured benchmarks","Compact LLMs get 27-point boost from parameter-efficient tuning","Local AI: 3B model reaches 92.6% after fine-tuning","Benchmark shows sub-3B models are local specialists"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000945,"raw_usage":{"total_tokens":3934,"prompt_tokens":868,"completion_tokens":3066,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":2976}},"tokens_in":612,"tokens_out":3066,"duration_ms":19865,"temperature":1.0,"reasoning_tokens":2976,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T14:51:17.108663+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An experiment that fine-tunes the same five models on, say, five different random splits with a 500-item held-out set; if the observed deltas shrink (e.g., the largest gain drops below +10 points) or if the gain for the 135M model reverses sign, the claim that low-cost PEFT is decisive for sub-3B models would not survive.","supporting_citations":[],"review_version":1}