{"id":"2191f2ed-a2d3-4f95-922e-3a6c9613fe80","arxiv_id":"2611.16414","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Prompt-optimized small language models show capability-dependent trade-offs between router size, classification performance, and downstream computational cost.","lead":"This paper tests whether small language models can decide which inference capabilities (reasoning, web search, calculator, retrieval) to activate before a downstream model answers a question. It finds that prompt optimization can make smaller routers competitive for some capabilities but not others, and that classification accuracy alone does not predict downstream efficiency.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"RQ3 prompt selection used the RQ2 test set, which shares questions with the RQ3 benchmark, creating a circular evaluation that inflates the reported efficiency gains.","rationale":"The reader correctly identified the lack of independent prompt validation as the weakest assumption. I agree this is the most load-bearing concern for the headline efficiency claim. My analysis adds the specific overlapping-data dimension, which makes the contamination more severe than a simple train/test split issue. The reader's CONDITIONAL verdict appropriately flags this; my concern does not change the verdict but sharpens the condition for acceptance: the paper must either rerun RQ3 with proper held-out prompt selection or, minimally, recompute savings on the non-overlapping subset. If the savings survive that test, the conditional can be lifted. If not, the central quantitative claim is unsupported. The paper's own acknowledgment that RQ3 is 'exploratory' aligns with this view.","tokens_in":19043,"tokens_out":4309,"duration_ms":33857,"concrete_test":"Identify the set of RQ3 questions that do NOT appear in the RQ2 test set (i.e., no overlap). Since Table 2 gives overlap per capability at the question-capability level, a conservative approach: remove all RQ3 questions that overlap ANY capability in the RQ2 test set. On the remaining held-out subset, recompute the 14B router's average downstream token consumption, median downstream latency, and BERTScore F1 relative to ALL_ON and ORACLE. If the token reduction drops below 50% or the latency reduction below 60% of the reported values (68.7% and 80.5%), the original efficiency claim is an artifact of test-set contamination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (68.7% token reduction, 80.5% latency reduction with minimal BERTScore drop) rests entirely on RQ3, where prompt configurations were selected based on the RQ2 test set (b166). Critically, the RQ2 test set and RQ3 benchmark overlap in questions: Table 2 shows 80+ overlapping question-capability pairs per capability. The paper harmonized labels but used original router outputs for RQ3. This means the prompt configurations were optimized (via F1 on RQ2 test) on data that includes the same questions later evaluated for downstream efficiency. Even if the labels were corrected post-hoc, the original router outputs were generated before correction, and the prompt selection process was influenced by those same questions' classifier performance. Hence the RQ3 efficiency numbers are not an independent estimate: they likely overstate gains because the router was effectively tuned to perform well on the evaluation set. The paper labels this 'exploratory' but the abstract and conclusion present the 68.7% number without qualification. This is the most load-bearing weakness: if the prompt configurations degrade on truly held-out questions, the headline resource savings may shrink or disappear.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper investigates whether prompt-driven small language models (1.5B–14B Qwen2.5) can serve as pre-inference routers for four capabilities (reasoning, RAG, web search, calculator) and whether classification gains translate into downstream efficiency. The authors examine RQ1 (effect of capacity under minimal prompting), RQ2 (effect of prompt optimization across strategies), and RQ3 (downstream answer quality, token consumption, and latency using a fixed GPT-OSS-120B model on 127 pharmaceutical regulatory questions). The central quantitative claim is that the 14B router achieves a mean BERTScore F1 of 0.8408 (versus 0.8418 with all capabilities enabled) while reducing downstream tokens by 68.7% and median latency by 80.5%.","tokens_in":19364,"tokens_out":2386,"duration_ms":19705,"significance":"If the findings are validated, the paper demonstrates a promising approach to capability-aware routing that can substantially reduce downstream computational cost with minimal degradation in semantic similarity. The decomposition into four independent binary classifiers and the systematic comparison of router capacity and prompt strategies are methodologically sensible. The within-family comparison reduces architectural confounds, and the authors are transparent about several limitations.","major_comments":[{"comment":"The RQ3 efficiency figures (68.7% token reduction, 80.5% latency reduction) rest on prompt configurations selected using the RQ2 test set, which overlaps substantially with the RQ3 benchmark (Table 2 shows 80+ overlapping question-capability pairs per capability). The authors label this 'exploratory' in §3.4, but the abstract and conclusion present the quantitative savings without this caveat. This circularity means the reported gains are not an independent estimate and likely overstate what would be observed on truly held-out data. To support the headline claims, the authors must either: (a) re-run RQ3 with prompt configurations selected on a held-out validation set that is disjoint from the RQ3 benchmark, or (b) explicitly qualify every appearance of the 68.7%/80.5% numbers with a statement that they are exploratory and may not generalize.","section":"§3.4, Table 2, b166","authors_acknowledge":"Because the prompting strategy for each capability was selected using the final RQ2 evaluation results rather than an independent validation set, the RQ3 compar- ison is interpreted as exploratory rather than as an independently validated estimate of generalization per- formance.","authors_acknowledge_p":0.91},{"comment":"The paper repeatedly acknowledges that BERTScore measures semantic similarity, not factual/numerical correctness, and that small differences do not establish statistical equivalence. However, the narrative in the abstract and conclusion relies on BERTScore to claim that the routed systems 'match' the ALL_ON condition. This tension is unresolved. The ALL_OFF condition also achieves a numerically similar BERTScore (§5.4), which suggests the metric may be insensitive to the very capabilities the router is designed to allocate. The authors should either provide task-specific correctness metrics (e.g., accuracy on arithmetic, retrieval precision) or tone down the quality-equivalence language throughout.","section":"§5.4, b258, b273, b274, b288","authors_acknowledge":"Because BERTScore measures semantic similar- ity rather than factual, numerical, or task correctness, these values should not be interpreted as demonstrating equivalent or preserved answer quality [13].","authors_acknowledge_p":0.92},{"comment":"The downstream benchmark contains only 127 questions from a single domain (pharmaceutical regulation), with highly imbalanced capability labels (e.g., only 10 web-search-required examples). The authors acknowledge this limitation (§6, b326). Nonetheless, the paper's significance claims are stated broadly. The narrow scope should be clearly stated in the abstract and conclusion, and the reported savings should be interpreted as specific to this benchmark, not as general evidence for the efficiency of the proposed routing approach.","section":"§3.2, Table 3, b326","authors_acknowledge":"The scope of the downstream benchmark further lim- its generalization.","authors_acknowledge_p":0.84},{"comment":"The headline efficiency numbers (68.7% token reduction, 80.5% latency reduction) refer only to downstream computation; router overhead is reported separately and increases with capacity (Fig. 8 shows 14B router median latency ~5.56s). Because router and downstream inference were measured in different environments (Colab vs. AWS Bedrock), no controlled end-to-end comparison is possible. The abstract should state explicitly that the reported savings exclude router overhead and that total system latency may be higher for larger routers.","section":"§3.4, §6, b323","authors_acknowledge":"Second, the reported downstream savings exclude router overhead, which increased with router capacity.","authors_acknowledge_p":0.75}],"minor_comments":[{"comment":"The abstract refers to 'GPT-OSS-120B' without a citation or explanation. Provide a reference or clarify the model provenance.","section":"Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The train/validation/test splits are highly imbalanced (e.g., reasoning has only 94 training examples vs. 752 test examples). Justify this split or discuss its implications for prompt optimization reliability.","section":"§3.2, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The reproducibility section mentions saving intermediate predictions to Excel checkpoints but does not state whether the code or prompts will be publicly released. State availability of code and data artifacts.","section":"§4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The confusion counts for 1.5B calculator show 65 false positives, but the downstream system may still avoid invoking the calculator. This nuance should be discussed in the main text rather than deferred to the caption.","section":"§5.4, Table 10","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 5 shows 'mean BERTScore F1' without error bars or confidence intervals. Provide per-question variability (e.g., standard deviation or box plots) to indicate spread.","section":"§5.4, Figure 5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The appendix provides representative prompts, but the full set of 80+ capability-model-prompt combinations is not included. At minimum, include all GEPA-optimized prompts in an ancillary file.","section":"§A.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The RQ1/RQ2 dataset originally had 3,573 records; after audit, 58 were excluded. The text states the audit 'identified 25 duplicated question-capability pairs with conflicting labels, corresponding to 58 records.' Clarify why 25 pairs yield 58 records (some may have more than 2 occurrences).","section":"§3.2","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Here is my take on this paper.\n\nWhat is actually new: a systematic within-family comparison (Qwen2.5 1.5B–14B) of router capacity and five prompting strategies for four independent capability decisions (reasoning, RAG, web search, calculator), plus a downstream evaluation that links classification error profiles to token and latency costs. The decomposition into independent binary decisions and the emphasis on False Positive/False Negative profiles matter for practical deployment. The paper is honest about its limitations; it flags the test-set contamination for prompt selection in §3.4 and the BERTScore insensitivity in §6. I appreciate that they actually report confusion counts and output reliability (unknown/error rates), not just F1.\n\nThe load-bearing soft spot is the RQ3 prompt selection: the authors used the RQ2 test set – which overlaps with the RQ3 benchmark questions – to pick the prompt for each capability. They call RQ3 'exploratory' in the text, but the abstract and conclusion present the 68.7% token reduction figure without that qualifier. That's a real discrepancy. The stress-test note is correct: the savings likely shrink on truly held-out questions. The paper's own Table 2 shows 80+ overlapping question-capability pairs per capability. Even with label harmonization, the prompt was optimized on data that includes those same questions. So the central quantitative claim lacks independent validation.\n\nOther issues: the benchmark is small (127 pharmaceutical questions) and domain-specific, with only 10 web-search-positive cases. BERTScore differences are tiny (0.8390 vs 0.8408) and the paper acknowledges BERTScore may not capture factual correctness. The ALL_OFF condition also achieves a numerically similar BERTScore, suggesting the benchmark may not require these capabilities at all—a point the authors raise but do not resolve.\n\nThat said, the RQ1/RQ2 results are cleaner and the overall design is sound. The paper is not claiming more than the evidence supports in those parts. The authors think clearly and are transparent about limitations.\n\nWho this is for: researchers building adaptive inference pipelines or studying SLMs for routing. A serious referee would want to see RQ3 repeated with proper train/valid/test splits before accepting the efficiency claims.\n\nRecommendation: send to peer review, but flag that the RQ3 results need independent validation before the efficiency numbers can be taken at face value. The rest of the paper is publishable after minor revision to tone down the abstract claims.","headline":"Solid empirical comparison of router capacity and prompting for capability routing, but the headline efficiency numbers in the abstract are not independently validated due to circular prompt selection—the paper acknowledges this, but the abstract overstates.","tokens_in":19786,"tokens_out":1237,"duration_ms":10765},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"A prompt-optimized 14B parameter router can match the answer quality of always-on capabilities while cutting downstream token use by 69% and latency by 81%.","keywords":["small language models","capability routing","prompt optimization","inference efficiency","Qwen2.5","BERTScore","retrieval-augmented generation","adaptive inference"],"falsifier":"If an independent evaluation with a held-out prompt validation set showed that the 14B router's downstream token savings drop below 50% while maintaining BERTScore F1 equivalence, or if a factual correctness metric (e.g., FActScore) reveals significantly lower accuracy compared to all-capabilities-on, the central efficiency claim would be undermined.","tokens_in":20557,"feed_emoji":"⚡","tokens_out":2864,"duration_ms":22781,"temperature":0.7,"pith_summary":"This paper investigates whether small language models, guided by carefully designed prompts, can decide which inference capabilities (reasoning, retrival, web search, calculator) to activate for each query before a large downstream model generates an answer. The authors find that prompt optimization can compensate for limited router capacity on reasoning and calculator decisions but not on retrival-augmented generation or web search, where larger models retain a clear advantage. In a downstream evaluation on 127 pharmaceutical questions, a 14B parameter router prompt-optimized per capability achieved a mean BERTScore F1 of 0.8408, essentially matching the 0.8418 of always-on capabilities, while reducing average downstream token consumption by 68.7% and median latency by 80.5%. However, the authors acknowledge that the prompt configurations were selected using the same test set on which they were evaluated, making the efficiency estimates exploratory rather than independently validated.","feed_headline":"Prompt-optimized router cuts LLM token use by 69% with near-equal quality","feed_subtitle":"A 14B-parameter router matches always-on capabilities while reducing latency by 80% on pharmaceutical QA.","key_machinery":"The central machinery is a set of four independent binary classifiers, each a Qwen2.5 small language model (1.5B, 3B, 7B, or 14B parametersution) that decides whether to enable a specific capability: reasoning effort, retrieval-augmented generation, web search, or calculator access. The classifiers are driven entirely by prompts — five strategies are compared (Minimal Baseline, Descriptive SME, DSPy Baseline, DSPy Few-Shot, and GEPA) — with no weight updates. The downstream model is a fixed GPT-OSS-120B, and the evaluation uses BERTScore for semantic similarity, token count, and latency to link routing decisions to downstream consequences.","core_discovery":"The paper establishes that the relationship between router capacity, prompt design, and downstream efficiency is capability-dependent. Under minimal prompting, larger router models do not consistently outperform smaller ones. Prompt optimization narrows the capacity gap for reasoning and calculator routing, but RAG and web search retain larger benefits from higher capacity. Crucially, similar classification F1 scores do not necessarily imply equivalent downstream behavior: false-positive activations increase unnecessary computation, while missed capabilities can reduce answer quality. The 14B router achieves a mean BERTScore F1 of 0.8408 compared with 0.8418 when all capabilities are enabled, while reducing average downstream token consumption by 68.7% (from 8,837 to 2,764 tokens) and median downstream latency by 80.5% (from 9.79 to 1.91 seconds). These savings exclude router overhead, which increases with router capacity and was measured in a separate inference environment.","pith_inferences":["The reported efficiency gains (68.7% token reduction, 80.5% latency reduction) may be overestimates because the prompt configurations for the downstream evaluation were selected using the same test set; an independent validation set would likely produce smaller savings.","Because BERTScore measures semantic similarity rather than factual correctness, a router that performs well on this metric might still degrade factual accuracy or numerical precision compared to always-on capabilities.","The cost-benefit comparison between router overhead and downstream savings is incomplete because the two were measured in different environments (local Google Colab vs. AWS Bedrock); a controlled end-to-end measurement could shift the optimal router size.","The approach could be extended to a wider range of capabilities, dynamic thresholds per capability, or router ensembles, but the capability-dependent results suggest that no single router size or prompting strategy will be optimal across all tasks."],"forward_implications":["Smaller prompt-optimized routers can substitute for larger ones on reasoning and calculator routing, potentially reducing the computational cost of the routing stage itself.","RAG and web search routing appear to require larger router capacity even after prompt optimization, suggesting that these tasks demand more nuanced understanding or broader context that smaller models cannot fully capture.","Router selection for production systems must consider not only classification F1 but also error profiles (false-positive and false-negative rates) because these directly determine downstream token consumption and latency.","Prompt optimization can substantially reduce inference cost in multi-capability LLM pipelines without fine-tuning, offering a lightweight deployment strategy.","The method suggests a path toward modular, capability-aware routing where each capability decision is made independently, allowing any subset of capabilities to be enabled per query."],"supporting_citations":[{"why":"Supplies the Qwen2.5 model family used for all router experiments, enabling a within-family capacity comparison.","marker":"[12]"},{"why":"Provides the DSPy framework used for structured prompting and few-shot demonstration optimization in router experiments.","marker":"[7]"},{"why":"Establishes the methodological baseline for prompt-driven small language model routing and demonstrates that prompt optimization can improve routing performance without weight updates.","marker":"[10]"},{"why":"Provides BERTScore, the semantic similarity metric used for downstream answer quality evaluation.","marker":"[13]"},{"why":"Provides the GEPA prompt optimization algorithm used to achieve the best results for RAG and web search routing.","marker":"[1]"},{"why":"Motivates adaptive retrieval by showing that not all queries require external evidence.","marker":"[6]"}],"fun_headline_variants":["Small router cuts LLM token use 69% with near-equal quality","14B router slashes latency 80% while matching full capability","Prompt-tuned small models route LLMs efficiently per task","Capacity and prompt design shape LLM router efficiency gains"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The prompt configurations used in the downstream evaluation were selected based on the same test set on which they were evaluated, rather than an independent validation set, so the reported token and latency savings may be overestimates that would shrink under proper held-out selection.","fun_headline_variants_meta":{"raw":{"variants":["Small router cuts LLM token use 69% with near-equal quality","14B router slashes latency 80% while matching full capability","Prompt-tuned small models route LLMs efficiently per task","Capacity and prompt design shape LLM router efficiency gains"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"cost":0.000225008,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000225008,"upstream_inference_prompt_cost":0.000118048,"upstream_inference_completions_cost":0.00010696},"total_tokens":1430,"prompt_tokens":1048,"completion_tokens":382,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":309}},"tokens_in":1048,"tokens_out":382,"cost_source":"openrouter_usage","duration_ms":3879,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":309,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T15:11:00.048062+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If an independent evaluation with a held-out prompt validation set showed that the 14B router's downstream token savings drop below 50% while maintaining BERTScore F1 equivalence, or if a factual correctness metric (e.g., FActScore) reveals significantly lower accuracy compared to all-capabilities-on, the central efficiency claim would be undermined.","supporting_citations":[{"cited_title":"Qwen2.5 Technical Report","cited_arxiv_id":"2412.15115","evidence_quote":"Supplies the Qwen2.5 model family used for all router experiments, enabling a within-family capacity comparison."},{"cited_title":"Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts","cited_arxiv_id":null,"evidence_quote":"Provides the DSPy framework used for structured prompting and few-shot demonstration optimization in router experiments."},{"cited_title":"Influence of Prompt Engineering on Small Language Models for Guarded Query Routing","cited_arxiv_id":"2607.24801","evidence_quote":"Establishes the methodological baseline for prompt-driven small language model routing and demonstrates that prompt optimization can improve routing performance without weight updates."},{"cited_title":"Wein- berger, and Yoav Artzi","cited_arxiv_id":null,"evidence_quote":"Provides BERTScore, the semantic similarity metric used for downstream answer quality evaluation."},{"cited_title":"GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning","cited_arxiv_id":"2507.19457","evidence_quote":"Provides the GEPA prompt optimization algorithm used to achieve the best results for RAG and web search routing."},{"cited_title":"Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity","cited_arxiv_id":null,"evidence_quote":"Motivates adaptive retrieval by showing that not all queries require external evidence."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}