{"id":"c4db69b5-dd4a-44f3-b92e-dc1816c7bb2a","arxiv_id":"2607.22621","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Per-question database-style plan search over multi-LLM DAGs improves QA quality under budgets by ~58% (MMLU-Pro) and ~41% (SimpleQA) versus reimplemented baselines.","lead":"OPTI-Q is a planner that picks which LLMs to call, and in what order, to answer each question under cost, latency, and energy limits. On two QA benchmarks it reports about 58% and 41% higher answer quality than reimplemented baselines at similar budget levels.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SimpleQA is scored by embedding cosine similarity, not the benchmark's factual-correctness metric; the headline 41% margin and the commercial-model comparison may be artifacts of that proxy.","rationale":"I read the paper in good faith. The system design is well-specified: the plan encoding, cost estimators, PERFDB coverage experiment, and planner comparisons are concrete and internally consistent. The PERFDB robustness study is a genuine contribution. However, the headline claim rests on two empirical pillars, and the SimpleQA pillar is weakened by the choice of evaluation metric. The paper uses cosine similarity for SimpleQA, a benchmark designed for factual correctness. This metric is used both in PERFDB (so the optimizer maximizes it) and in the headline evaluation. Under this metric, OPTI-Q reportedly surpasses frontier commercial models on SimpleQA, which is implausible if factual accuracy were the measure. The reader's weakest assumption was baseline fidelity; I partially agree, but I judge the metric-validity issue to be more fundamental because it affects not just the relative margin but the interpretation of what was optimized and measured. The proposed test—re-scoring with the official SimpleQA protocol—would settle whether the claimed 41% improvement reflects factual QA gains or an artifact of embedding similarity. My recommended verdict remains CONDITIONAL (unchanged): the paper's central idea is promising and well-argued, but the SimpleQA empirical claim should not be accepted at face value until re-scored. This is not a rejection of the framework; it is a demand for metric-validity evidence before the headline is trusted.","tokens_in":22747,"tokens_out":5676,"duration_ms":55262,"concrete_test":"Re-score all SimpleQA predictions from the released code/checkpoints using the official SimpleQA correctness protocol (exact match / contains-answer after normalization, per the SimpleQA repository). Compare OPTI-Q against FrugalGPT* and against the commercial models under the same budgets. If the relative improvement drops below about 10% or reverses, the headline should be revised to 'embedding-similarity QoA' or the SimpleQA claim should be removed. Additionally, compute the Spearman correlation between embedding-similarity QoA and official correctness on the 100-question sample; if the correlation is below 0.5, the proxy is not a valid surrogate for factual quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that OPTI-Q improves answer quality under budget—requires that the measured 'QoA' reflects genuine answer quality. For SimpleQA, the paper explicitly computes QoA as cosine similarity between all-MiniLM-L6-v2 embeddings of the predicted and reference answers (Section V, 'QoA Computation'; Section VI-A). SimpleQA is a short-fact factuality benchmark whose official protocol is exact-match/contains correctness. Embedding similarity can reward verbose, topically related, or stylistically similar but factually wrong answers. This is not merely a reporting nuance: PERFDB stores this cosine-similarity QoA, so OPTI-Q is planning to maximize embedding similarity, not factual correctness. The implausibility of the absolute results under a factual metric—OPTI-Q at 0.650 vs. Claude Opus 4.6 at 0.597 and GPT 5.4 at 0.605 on SimpleQA, with local open models—suggests the proxy is inflating local-model scores. Because baselines are scored with the same proxy, the 41% relative improvement could reflect shifts in output style or length rather than factual quality. The reader's baseline-fidelity concern is real but secondary: even with perfectly faithful baselines, the SimpleQA result would not establish improved factual QA unless re-scored on the official metric. This is the load-bearing weakness in the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents OPTI-Q, a plan-before-execute optimizer for multi-LLM question answering. It encodes candidate execution workflows as DAGs of LLM invocations with sequential, parallel, and blending operators; estimates QoA, financial cost, latency, and energy from a PERFDB statistics catalog; and searches the plan space using NSGA-II, dynamic programming, or hill climbing under user-specified budgets. The authors claim roughly 58% and 41% average QoA improvements over budget-aware baselines on MMLU-Pro and SimpleQA, and argue that database-style planning yields better quality-resource trade-offs. The main text includes the formal problem statement, the DAG encoding, cost/QoA estimators, and an evaluation on 200 sampled questions with five locally hosted models.","tokens_in":23119,"tokens_out":7382,"duration_ms":70468,"significance":"The framework is well structured and the cost/benefit formulation is a useful contribution. The strengths include a precise DAG encoding with canonicalization, token-aware cost estimators, a clean separation between PERFDB statistics, planners, and execution, and the release of code/results. The held-out configuration set is good practice. However, the headline empirical claims are not yet established: the SimpleQA quality metric is a cosine-similarity proxy rather than the benchmark's official factual-correctness metric; the evaluation is based on 200 questions with no reported confidence intervals; and all comparative numbers are against reimplemented baselines that are not validated. If the SimpleQA results were re-scored on the official metric and the MMLU-Pro results hold, this would be a solid systems contribution.","major_comments":[{"comment":"For SimpleQA, QoA is defined as cosine similarity between all-MiniLM-L6-v2 embeddings of the predicted and reference answers, not the benchmark's official exact-match/contains correctness. Because PERFDB stores and OPTI-Q optimizes this embedding-similarity score, the headline 41% SimpleQA gain and Table III's comparisons against Claude/GPT do not establish improved factual QA. Re-score all SimpleQA results on the official metric, or explicitly relabel the claim as 'semantic-similarity QoA' and remove SimpleQA from the factual-accuracy headline.","section":"§V, §VI-A(b), §VI-D"},{"comment":"The entire comparative evaluation uses 100 questions per benchmark (10 questions each from 10 sampled types) executed five times. The text says 95% confidence intervals are reported, but Figures 3–5 and Tables I–III report only point estimates, with no intervals or significance tests. Given the small sample, the '≈58%' and '≈41%' margins cannot be assessed statistically. Report CIs and ideally bootstrap or a larger testbed.","section":"§VI"},{"comment":"Baselines are author re-implementations: FrugalGPT*, ThriftLLM*, LLM-Ensemble*, LLM-Blender*, with FrugalGPT* adapted to a per-query budget and LLM-Ensemble* given a greedy budgeted variant. No evidence is provided that these reimplementations reproduce the published methods' behavior. The central comparisons in Figure 3 are therefore against a possibly weaker reference point. Provide validation against published numbers or a component-level ablation, and state explicitly which baseline is the 'strongest budget-aware baseline' used for the 58%/41% figures.","section":"§VI, Baselines and scope"},{"comment":"The claimed improvement magnitudes are not consistent with the body's numbers. For SimpleQA at b=3, the text reports OPTI-Q QoA ≈0.56–0.61 versus FrugalGPT* ≈0.46, which is a ~22–33% relative gain, not ~41%; the 41% figure appears to correspond to b=5 (0.65 vs 0.46). For MMLU-Pro, the b=3 gain (≈77% vs ≈45–47% baselines) is ~64%, not ~58%. The abstract and introduction must be reconciled with the actual reported operating point.","section":"Abstract, §VI-A"}],"minor_comments":[{"comment":"The canonicalization procedure (lexicographically maximal bit vector over all topological orderings) needs a precise algorithm and complexity analysis; the paper only states O(k^2) space. The extended version should contain this and the full enumeration method.","section":"§IV-A"},{"comment":"The sequential and blend update rules are heuristic; their free parameters and fallback choices (e.g., neutral factor 1, average-relative change) deserve a sensitivity analysis, especially since the robustness experiment only reports overall MAE, not per-component calibration.","section":"§IV-B, QoA Estimation"},{"comment":"The cumulative PERFDB trace counts read as '135,985,1,735,2,360'; likely missing separators. Also, the figure captions reference panels (4a/4b) but the text sometimes cites 'Figures 4a and 4b' without consistent panel labels.","section":"§VI-B"},{"comment":"For a journal version, the main text should not delegate so much essential material to an external 'extended version' ([39]) — including the full planner comparison, complete QoA lookup procedure, and additional sensitivity analyses. These are central to reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The SimpleQA metric issue is the most serious concern. If the authors cannot re-score on the official SimpleQA metric, the SimpleQA portion of the headline claim should be withdrawn. The 200-question sample and unvalidated reimplemented baselines further weaken the quantitative claims. The framework itself is promising, and the MMLU-Pro results, if they survive re-analysis, could support acceptance after the empirical corrections are made."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core of this paper is solid. The plan encoding is clean, the PERFDB statistics catalog is a practical contribution, and the coverage experiment—showing how planning quality degrades as historical data is removed—is the kind of honest evaluation more systems papers should include. The idea of treating per-question multi-LLM orchestration as a cost-based query-planning problem, with sequential/parallel/blend DAGs estimated from stored statistics, is new enough in this combination and worth building on.\n\nThe soft spots are real, though addressable. The biggest is the SimpleQA quality metric: QoA is cosine similarity between all-MiniLM-L6-v2 embeddings of predicted and reference answers, not SimpleQA's official factual-correctness scoring. That means the headline 41% improvement over baselines on SimpleQA is an improvement on a stylistic/semantic proxy, not on factual accuracy. The comparison against Claude Opus 4.6 and GPT 5.4, where OPTI-Q appears to beat both on SimpleQA, is likely an artifact of this proxy—embedding similarity rewards verbose, topically related answers. If the authors re-score on exact-match/contains, that section may look very different. This is the load-bearing weakness, and it deserves a direct answer before the empirical claims are taken at face value.\n\nThe other concerns are minor by comparison but real: the evaluation is on 200 questions, baselines are author re-implementations (even if adapted fairly), and the 58%/41% improvement is relative to a budget cap rather than realized spend. Confidence intervals are claimed but not reported. These are fixable with a fuller evaluation or more cautious wording.\n\nFor an internal letter: this is a serious, coherent paper by people who know what they're doing. It is not a field-shaping result, and the SimpleQA numbers as currently presented overstate the case. But the planner architecture and PERFDB design are worth engaging with, and the paper deserves a serious referee who will push the authors to re-run SimpleQA on the official metric and to clarify what exactly is being measured.","headline":"A well-specified planning system with a genuinely useful statistics catalog, but the SimpleQA results rest on an embedding-similarity proxy that undermines the factual-quality claim.","tokens_in":23580,"tokens_out":1525,"would_cite":true,"duration_ms":18791,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OPTI-Q proposes that answering a question with several LLMs should be a database-style planning problem: pick a per-question execution DAG from measured statistics before any model runs, then execute it under the user's budget.","keywords":["multi-LLM orchestration","query optimization","cost-based planning","Pareto frontier","quality-resource trade-off","question answering","statistics catalog"],"falsifier":"Re-run the comparison using the original, unmodified published baseline systems under the same per-question budgets and identical measurement of QoA, cost, latency, and energy on the 200 sampled MMLU-Pro and SimpleQA questions; if the roughly 58% and 41% QoA advantages at budget level 3 do not persist, the central claim fails.","tokens_in":22657,"feed_emoji":"🤖","tokens_out":4440,"duration_ms":44494,"temperature":0.7,"pith_summary":"OPTI-Q argues that orchestration of multiple LLMs for a single question should be treated as query optimization: the system constructs a DAG of model calls—sequential chains, parallel runs, and output blending—and estimates each candidate plan's answer quality, dollar cost, latency, and energy from a statistics catalog before executing anything. It then searches for Pareto-optimal plans under the user's budget and selects one by user weights. On sampled questions from MMLU-Pro and SimpleQA, the paper reports roughly 58% and 41% higher average QoA than budget-aware routing and ensemble baselines at comparable per-question cost. The central bet is that the best model combination and execution structure are question-dependent, and that historical statistics predict them better than greedy routing rules.","feed_headline":"Budgeted multi-LLM QA gets ~58% QoA boost from plan-before-execute","feed_subtitle":"A database-style optimizer that picks a per-question plan of parallel and sequential LLM calls beats routing and ensemble baselines under bu","key_machinery":"PERFDB, a topic-conditioned statistics catalog keyed by execution context, plus a compact canonical DAG encoding of plans (an upper-triangular adjacency bit vector with model assignments). PERFDB stores QoA and resource statistics for single models and composed subplans; the planner composes them into pre-execution estimates using a multiplicative relative-effect rule for sequential steps and an averaging blend rule for parallel-plus-blend nodes, with fallbacks for missing entries. The cost model is tokenizer-aware, applying each model's own tokenizer, expected output lengths from PERFDB, and fixed-plus-variable pricing, with parallel latency taken as the slowest branch. This machinery lets","core_discovery":"The discovery OPTI-Q tries to establish is that the cost-based optimization discipline used for SQL queries transfers to composing LLM calls. Model invocations are treated as physical operators with measured quality and resource profiles, and a planner synthesizes a question-specific execution DAG before execution. Using a statistics catalog (PERFDB) built from benchmarks and execution traces, OPTI-Q estimates token volumes, quality, and resources for candidate plans, then runs Pareto-frontier search—exact dynamic programming for small spaces, evolutionary search for larger ones—to pick a plan under user budgets. The empirical claim is that this beats routing, cascading, and ensembling basel","pith_inferences":["If the headline margins are real rather than an artifact of the baseline re-implementations, the plan-before-execute design likely transfers beyond QA to RAG and agentic workflows, where retrieval, reranking, verification, and tool calls are also operators with measurable cost and quality profiles.","The planning overhead (about 21 seconds at k=5 with the evolutionary backend) suggests per-question planning is best suited to offline or asynchronous use; caching plans keyed by topic or by similar questions, or moving to cheaper heuristic planners for interactive settings, would make the approach practical at low latency.","Even with full PERFDB coverage, QoA estimates remain imperfect (MAE around 0.27–0.34), so the planner's advantage likely comes from preserving relative ranking among candidate plans rather than from accurate absolute quality prediction—a property that should be tested explicitly as the catalog grows."],"forward_implications":["Overruns stay rare under cost and latency budgets: 88–97% of executed plans adhere to the budget, with overruns driven mainly by financial cost.","Plan complexity pays off most on classification-style reasoning: increasing the operation limit from k=1 to k=5 yields +54.7% QoA on MMLU-Pro versus +23.6% on SimpleQA, and at k=5 parallel and hybrid plans account for roughly 82% of selected plans.","Diversity saturates: the largest marginal QoA gain appears when moving from 2 to 3 distinct models (ΔQoA=0.038), while moving from 4 to 5 adds only 0.008.","OPTI-Q beats strong commercial API models on SimpleQA on cost-adjusted quality, with one leading API model 95.2x more expensive for modestly better QoA.","Richer PERFDB coverage improves planning: MMLU-Pro QoA rises from 0.40 at cold start to 0.67 at Level 4 coverage, and estimation error drops sharply across cost, energy, latency, and QoA."],"fun_headline_variants":["Plan-first multi-LLM QA: 58% better QoA under budget","DB-style planning lifts multi-LLM QA by 58% under budgets","Optimize LLM calls like SQL: 58% better QA on budget","Plan-before-execute lifts QA 58% across LLM blends","DB optimizer for LLMs: 58% QoA gain under budget"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The headline margins are measured against author re-implementations of the published baselines, adapted to OPTI-Q's per-question budget scheme; if those re-implementations are weaker than the original systems, the advantage reflects the comparison setup rather than the planner.","fun_headline_variants_meta":{"raw":{"variants":["Plan-first multi-LLM QA: 58% better QoA under budget","DB-style planning lifts multi-LLM QA by 58% under budgets","Optimize LLM calls like SQL: 58% better QA on budget","Plan-before-execute lifts QA 58% across LLM blends","DB optimizer for LLMs: 58% QoA gain under budget"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000618,"raw_usage":{"total_tokens":2723,"prompt_tokens":779,"completion_tokens":1944,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":1852}},"tokens_in":523,"tokens_out":1944,"duration_ms":13557,"temperature":1.0,"reasoning_tokens":1852,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T10:57:19.399115+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison using the original, unmodified published baseline systems under the same per-question budgets and identical measurement of QoA, cost, latency, and energy on the 200 sampled MMLU-Pro and SimpleQA questions; if the roughly 58% and 41% QoA advantages at budget level 3 do not persist, the central claim fails.","supporting_citations":[],"review_version":1}