{"id":"5bcf90da-81fe-4f78-ab34-ecbfb2fd2bf1","arxiv_id":"2504.18884","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Ensembling five Llama-3-8B inferences via the median reduces RMSE by 18.6% versus a single Llama-3-70B run on Yelp sentiment, at less than half the runtime.","lead":"This paper tests whether running a medium-sized language model several times and taking the median rating produces more stable sentiment predictions than one run of a large model. On 1,000 Yelp reviews, the ensemble cut the average rating error by 18.6% while using less than half the compute time, though accuracy was essentially unchanged.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 18.6% RMSE gain rests on a single 70B run; stochastic variance of the large-model baseline is unreported, so the comparison may not be stable.","rationale":"The reader's weakest-assumption analysis identifies the single-run 70B baseline as the key vulnerability, and I agree. The central empirical claim is a comparison between a stochastic process (one 70B inference) and an aggregated statistic (median of five 8B draws). Without repeated seeds, confidence intervals, or a significance test, the 18.6% RMSE gap could be within sampling noise. The accuracy component of the claim is also overstated, since Table 3 shows 0.778 vs 0.779, but the more load-bearing issue is the unquantified baseline variance because it threatens the quantitative headline. The paper has independent support: code is available, the pipeline is simple and re-implementable, and the ensemble-vs-single-8B comparison (0.424 vs 0.562 RMSE; 0.778 vs 0.749 Acc) plus the clear runtime advantage demonstrate that the strategy has real value. A conditional verdict requesting repeated-seed baselines is the right level; no rejection is warranted because the core direction of the finding is plausible and testable.","tokens_in":8049,"tokens_out":4800,"duration_ms":51154,"concrete_test":"Using the released code, run Llama-3-70B-Instruct on the same 1,000-review test set with at least five seeds, matching the seed values used for the 8B workers, and record RMSE/Acc for each single run plus the median-of-five ensemble. If the 70B single-run RMSE range overlaps 0.424, or the ensemble's RMSE advantage over the best single 70B run is not consistent across seeds, the headline 18.6% RMSE reduction is not established. Also report temperature and top-p to confirm that changing seeds actually produces diverse outputs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Table 3 pits a five-seed median ensemble of Llama-3-8B (RMSE 0.424) against one Llama-3-70B run (RMSE 0.521) and converts the gap into the headline 18.6% improvement. The paper does not report the seed, sampling temperature, or run-to-run distribution of the 70B baseline. Since LLM outputs are stochastic, a single 70B inference is a point estimate with unknown variance; another seed could plausibly put RMSE at 0.45-0.55 or lower, changing or erasing the claimed gain. The same missing uncertainty applies to the accuracy comparison (0.778 vs 0.779), which is already a tie. The paper's conclusion calls the difference 'significant' without a significance test. The available code makes this checkable, but the headline numerical claim is not yet supported at the reported precision.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a simple ensemble strategy for LLM-based sentiment classification: run Llama-3-8B-Instruct five times with different random seeds (as 'virtual workers'), aggregate the ordinal 1–5 predictions by the median, and compare the resulting ensemble against a single Llama-3-70B-Instruct run. On a 1,000-review subsample of the Yelp Open Dataset, the paper reports that the 8B median ensemble achieves RMSE 0.424 and accuracy 0.778, while the single 70B run achieves RMSE 0.521 and accuracy 0.779, with the 8B ensemble also being faster (27.2 vs 64.9 seconds per review). The paper frames this as evidence that medium-sized LLMs with multiple inference attempts are more robust and efficient than a single large-model attempt, and it claims an 18.6% RMSE reduction over the 70B baseline.","tokens_in":8207,"tokens_out":2294,"duration_ms":23792,"significance":"If the headline comparison were statistically supported, the result would be practically useful: it would show that cheap repeated inference with a small model can match or beat a much larger model at a fraction of the compute cost, without tuning, fine-tuning, or prompt engineering. The manuscript is commendably concrete: the strategy is simple, the code is released, the evaluation dataset is public, and the number of workers and the median aggregation rule are fixed a priori rather than tuned on the test set. These properties make the result easily checkable and reusable. However, the central quantitative claim currently rests on a single 70B run, and the claimed accuracy improvement is actually a tie, so the magnitude of the headline effect is not yet established.","major_comments":[{"comment":"The headline 18.6% RMSE reduction (0.521 vs 0.424) compares a five-run median ensemble against a single Llama-3-70B run. The paper provides no repeated-seed distribution, confidence interval, or significance test for the 70B baseline, even though LLM outputs are stochastic. A different seed or sampling temperature could plausibly move the 70B RMSE into the 0.45–0.55 range, which would shrink or erase the claimed gain. Please report the variance across repeated runs (for both the 70B single model and the 8B ensemble), state the exact seed values and sampling parameters, and run a paired significance test. This is load-bearing for the abstract's central claim.","section":"Section 4.3, Table 3"},{"comment":"The abstract says the ensemble produces 'more robust and accurate results' than the large model, but the reported accuracy is 0.778 for the ensemble and 0.779 for the 70B single run: statistically and practically a tie. The evidence for improved accuracy is therefore absent; the supportable claim is about RMSE (and even that currently lacks uncertainty quantification). Please revise the wording to distinguish the supported RMSE claim from the unsupported accuracy claim.","section":"Abstract and Section 4.3, Table 3"},{"comment":"The paper says 'the model's reproducibility parameter is utilized' and that five seed values create five workers, but it does not report the sampling temperature, top-p, or other decoding parameters used during inference. Since seed alone may not fully determine the sampling distribution across different inference stacks, please state the exact decoding configuration in the methods so that the reported numbers are reproducible and the ensemble diversity mechanism is precisely defined.","section":"Section 3.2, Eq. (1)"}],"minor_comments":[{"comment":"The text reads 'Llama 2 required more inference time than 80B model (Model 3)'; this should be '8B model'.","section":"Section 4.3"},{"comment":"The row 'Lift from the LLM baseline (8B)' reports -500.0% for time, which is confusing because a negative percentage could be read as a speedup rather than an increase; please clarify the sign convention in the table note.","section":"Table 3"},{"comment":"The evaluation uses only accuracy and RMSE; since the task is ordinal, reporting a confusion matrix or per-class error breakdown would help readers see where the median aggregation actually helps.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the practical question is timely, but the central quantitative claim is currently under-supported because the 70B baseline is a single stochastic run. The authors appear capable of fixing this with additional experiments and re-analysis; I would not recommend rejection unless the required repeated-run experiments are infeasible. The citation to the author's own prior work [27] is used appropriately as background and does not create a circularity problem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this paper ships code, a clean one-shot prompting setup, and a concrete cost-performance claim — Llama-3-8B with five-seed median aggregation beats a single Llama-3-70B run on RMSE (0.424 vs 0.521) in less than half the time. That is worth knowing. But the headline number rests on a single 70B run, with no seed, temperature, or repeated-run distribution, so the reported 18.6% RMSE improvement is a point estimate with unknown variance. The accuracy difference is a tie (0.778 vs 0.779), so 'more robust and accurate' overstates it.\n\nWhat's actually new: not the ensemble idea — sampling an LLM multiple times and aggregating is standard (self-consistency, majority voting, and the author's own prior work). The specific twist is using the median for ordinal 1-5 ratings, which is sensible for outlier robustness. The paper also provides a practical comparison with runtime, which is useful for applied settings.\n\nThe good: code is on GitHub, the prompt is simple, and the Yelp setup is described well enough to reproduce. The inclusion of non-LLM baselines like BERT and SVM helps contextualize. The paper is clear about its limitations (prompt sensitivity, same prompt for all workers, no different-scale combinations).\n\nThe soft spots, in proportion: the missing uncertainty quantification is the main one. A single-run baseline means we can't tell if 70B just had a bad seed. Also, the ensemble advantage is not compared against repeated runs of the 8B single model, so we can't attribute the gain to ensembling versus just averaging over seeds. The paper says 'significant' without a test. None of these are fatal to the practical suggestion — the runtime argument is compelling on its own — but they do mean the headline claim isn't supported at the reported precision.\n\nWho this is for: someone building LLM-based review classification in a business setting who wants a cheap, reproducible way to get more stable scores. A methods-oriented reader won't find much new, but the paper is a good reminder that stochasticity matters and that a medium model plus ensembling can be competitive with a big model.\n\nMy recommendation: send it to peer review. The finding is plausible and re-implementable; a good referee can ask for repeated-seed baselines and confidence intervals, which the author seems capable of providing. It's not a desk reject, but it needs a revision before the accuracy/RMSE claim can be taken at face value.","headline":"A practical, re-implementable cost-performance comparison that is undermined by a single-run 70B baseline and missing error bars; still worth a careful referee.","tokens_in":8728,"tokens_out":2506,"would_cite":false,"duration_ms":21781,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A five-run ensemble of an 8-billion-parameter language model produced lower average error than a single 70-billion-parameter run on restaurant review sentiment, at nearly identical accuracy and less than half the processing time.","keywords":["large language models","ensemble inference","sentiment analysis","median aggregation","label stability","RMSE","restaurant reviews","text classification"],"falsifier":"Run the 70-billion-parameter model on the same 1,000 reviews across, say, twenty random seeds, and compare the RMSE of every single run with the five-run 8B ensemble's 0.424. If most 70B seeds beat 0.424, the claimed 18.6 percent improvement is an artifact of one unlucky run.","tokens_in":7831,"feed_emoji":"🗳️","tokens_out":5913,"duration_ms":58055,"temperature":0.7,"pith_summary":"This paper claims that running a mid-sized language model several times and combining the answers is a better use of compute than one pass with a much larger model. On a 1,000-review restaurant sentiment task, the five-run ensemble of an 8-billion-parameter model reached a root mean square error of 0.424 and accuracy of 0.778, while a single run of a 70-billion-parameter model reached RMSE 0.521 and accuracy 0.779—an 18.6 percent smaller average error at nearly identical accuracy, in less than half the processing time. The proposed method requires no fine-tuning, and it borrows the common human-annotation practice of aggregating several judgments.","feed_headline":"Run a mid-size LLM five times, cut error 18.6 percent","feed_subtitle":"Five 8B inferences matched a 70B run on accuracy, with 18.6% less error and 58% less time.","key_machinery":"The load-bearing mechanism is a five-worker median ensemble. A single 8-billion-parameter instruction-tuned model is called five times with different random seeds, each call producing one star rating through a one-shot prompt, and the final prediction is the median of the five ratings. This turns the model's natural output variability into five virtual annotators and cancels single-run outliers. A 4-bit quantized implementation keeps the eight-billion-parameter runs cheap enough that five runs still cost less than one 70-billion-parameter run.","core_discovery":"On the paper's own terms, the central discovery is that stochastic variation in LLM inference can be exploited as a cheap ensemble: five independent generations by the same 8-billion-parameter model, aggregated with the median of the predicted 1–5 star scores, produce more stable and on average less wrong predictions than a single attempt with a 70-billion-parameter model. The paper reports RMSE 0.424 versus 0.521, an 18.6 percent reduction, and accuracy 0.778 versus 0.779, with runtime 27.2 versus 64.9 seconds per review. The median aggregation is robust to the occasional invalid or extreme token, so the ensemble gains do not depend on a scoring scheme that rewards only exactly correct labels.","pith_inferences":["A testable extension the paper leaves implicit: repeat the 70B run across several seeds to obtain a distribution, since the claimed 18.6 percent RMSE advantage is currently measured against one sample of the 70B output distribution.","The five-seed choice is arbitrary; plotting RMSE as a function of the number of workers would show whether gains saturate quickly and whether a 70B median ensemble would eventually pull ahead.","Because cost scales with parameters and with number of runs, the result suggests a compute-budget view of model choice: for a fixed wall-clock or dollar budget, repeated mid-size runs may dominate a single large run on ordinal classification, and checking this on other model families and tasks would test the generality."],"forward_implications":["If the central comparison holds, practitioners can replace one expensive large-model inference with several cheap mid-model inferences and get smaller average error in less wall-clock time.","Because the ensemble treats the model as an off-the-shelf scorer, it can be added to existing LLM annotation pipelines without training or extensive prompt engineering.","The method turns a reproducibility nuisance—random seed sensitivity—into an accuracy lever, so it should transfer to other stochastic models and other ordinal or categorical labeling tasks.","The RMSE gain with equal accuracy means the ensemble is not merely moving correct labels around; it shrinks the severity of misclassifications, which matters for star ratings and other graded scales.","The strategy's benefit should grow when single-run outputs include out-of-scope tokens, because the median discards extreme votes."],"supporting_citations":[{"why":"Supplies the two parameter scales (8B and 70B) whose single-run and ensembled behavior is compared.","marker":"[30]"},{"why":"Supplies the 1,000 restaurant reviews and star ratings that form the test set.","marker":"[34]"},{"why":"The 4-bit quantization implementation that makes five mid-size runs cheaper than one 70B run.","marker":"[33]"},{"why":"Documents the instability of LLM outputs, the problem the ensemble is designed to absorb.","marker":"[13]"}],"fun_headline_variants":["Five 8B runs cut error 18.6% vs one 70B","Aggregate five mid-size LLM runs: RMSE drops 18.6%","Small-model ensemble slashes RMSE by 18.6% vs single big run","Five 8B runs: 18.6% less error, 58% faster than one 70B"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison treats one 70-billion-parameter run as a fair representative of that model; since LLM outputs vary with seed, a single run could be atypical, and the paper gives no distribution of repeated 70B runs.","fun_headline_variants_meta":{"raw":{"variants":["Five 8B runs cut error 18.6% vs one 70B","Aggregate five mid-size LLM runs: RMSE drops 18.6%","Small-model ensemble slashes RMSE by 18.6% vs single big run","Five 8B runs: 18.6% less error, 58% faster than one 70B"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001113,"raw_usage":{"total_tokens":4561,"prompt_tokens":798,"completion_tokens":3763,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":414,"completion_tokens_details":{"reasoning_tokens":3666}},"tokens_in":414,"tokens_out":3763,"duration_ms":28631,"temperature":1.0,"reasoning_tokens":3666,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:06:33.709587+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the 70-billion-parameter model on the same 1,000 reviews across, say, twenty random seeds, and compare the RMSE of every single run with the five-run 8B ensemble's 0.424. If most 70B seeds beat 0.424, the claimed 18.6 percent improvement is an artifact of one unlucky run.","supporting_citations":[{"cited_title":"Ensemble based systems in deci- sion making","cited_arxiv_id":null,"evidence_quote":"Supplies the 1,000 restaurant reviews and star ratings that form the test set."},{"cited_title":"Ensemble methods in ma- chine learning","cited_arxiv_id":null,"evidence_quote":"The 4-bit quantization implementation that makes five mid-size runs cheaper than one 70B run."},{"cited_title":"Bold metric indicates that the model performs better than all references while cell shading represents the highest performance","cited_arxiv_id":null,"evidence_quote":"Documents the instability of LLM outputs, the problem the ensemble is designed to absorb."}],"review_version":1}