{"id":"7b7f5b3a-6752-4f10-9eb9-ac3a900a3603","arxiv_id":"2610.17205","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":7,"one_line_summary":"Open instruction-tuned language models, using only a letter-logit readout with temperature scaling, match Jev's decision capabilities zero-shot; a simple LoRA fine-tuning step exceeds them on the OpenDecisions benchmark and the JevBench leaderboard.","lead":"This paper shows that open-source instruction-tuned language models can match or exceed a proprietary decision-making system (Jev) by simply reading out probabilities from the model's logits for answer letters (A, B, C, etc.) in a single forward pass, with no text generation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Leaderboard draws are not shared: the 4.4-point margin over Jev on JevBench v1.7 compares separate question sets without a common reference, and the paper's own estimate of per-draw variance (0.2-point noise for a similar gap) undermines the precision of the claim.","rationale":"The reader correctly identified OpenDecisions as the weakest assumption but treated the JevBench leaderboard as a clean independent validation. The leaderboard comparison, however, shares a structural weakness: different draws without a common reference. The paper's own statement that a 0.2-point gap is 'within noise' due to separate draws implies that the 4.4-point gap carries similar, unquantified uncertainty—yet the paper does not apply that caveat to its headline comparison. The held-out datasets (Table 4) show a much smaller advantage (89.0 vs 88.0 macro accuracy), and the knowledge-heavy benchmarks show a clear deficit (88.8 vs 92.9). The 4.4-point JevBench margin is the strongest quantitative claim; if it is partially confounded, the paper's main conclusion softens from 'solidly exceeds Jev' to 'comparable on most metrics, leads on a carefully constructed development set, and shows a noisy advantage on a leaderboard with uncontrolled draws.' A single shared-draw evaluation would resolve this and strengthen the paper. Hence CONDITIONAL: the verdict should be finalized after this check.","tokens_in":33151,"tokens_out":8882,"duration_ms":94402,"concrete_test":"Submit Jev and Jeff-1.0-Large to the same JevBench v1.7 draw (i.e., the same revision, with the same 1,500 items). If the Capability gap is 4.4 ± 1.96 SE (approximately 2.5 points or more), the draw-confound concern is resolved. If the gap shrinks to ≤2 points or flips, the headline claim about the leaderboard is overstated and the paper's central result rests more heavily on OpenDecisions, whose acknowledged limitations (development set, Qwen-based labels) would then become more consequential.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central evidence that a short LoRA fine-tune 'exceeds Jev by 4.4 Capability points' on the official JevBench leaderboard (v1.7) rests on comparing two different leaderboard draws: Jeff-1.0-Large was measured on revision v1.6.2, Jev on v1.6.1 (Table 14). The paper itself notes that 'each system is measured on its own draw, so a difference of 0.2 points is within noise' when comparing Jeff-1.0-Large to Quyet-1.0-Large (81.5 vs 81.7). This same design—different draws, no shared baseline—applies to the 4.4-point gap against Jev. The paper provides no evidence that the v1.6.1 and v1.6.2 draws are of comparable difficulty. In fact, Jev's own Intelligence score varies by 8.3 points between OpenDecisions (71.9, Table 3) and the leaderboard (63.6, Table 14), hinting that question-set difficulty can shift Capability substantially. Without a system evaluated on both draws, part or all of the reported 4.4-point advantage could reflect draw difficulty rather than model quality. The paper acknowledges this issue for small gaps but does not extend the same caution to the headline comparison, making the leaderboard-based outperformance claim less secure than it appears.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"This paper shows that open instruction-tuned language models, using a simple letter-logit readout and temperature scaling, achieve performance comparable to the proprietary Jev model on decision tasks without any additional training. After a short LoRA fine-tuning procedure (400 steps, ~5 GPU-hours), the best model (Gemma 4 31B) scores 84.4 on the authors' OpenDecisions validation set and 81.5 on the JevBench v1.7 leaderboard, claiming a 4.4-point advantage over Jev. The paper also introduces OpenDecisions, a 2,352-question validation set, and releases models and code.","tokens_in":35322,"tokens_out":1550,"duration_ms":35406,"significance":"If the core claims hold, this work would be significant: it demonstrates that open-weight instruction-tuned LLMs already possess decision-making capabilities comparable to a proprietary system, and that a simple supervised fine-tuning recipe can further improve them—all without specialized architectures or reinforcement learning. The zero-shot finding (Figure 2) is especially valuable as it suggests that expensive proprietary training may be unnecessary for many decision tasks. The ablations (Table 5) cleanly identify LoRA adaptation of the backbone, rather than head training, as the source of gain. However, the headline result that fine-tuning 'exceeds Jev by 4.4 Capability points' on the official leaderboard is less secure than it appears.","major_comments":[{"comment":"The claim that Jeff-1.0-Large outperforms Jev by 4.4 Capability points on the JevBench leaderboard (v1.7) is based on comparing two different draws: Jev was measured on revision v1.6.1, Jeff-1.0-Large on v1.6.2 (Table14). The paper itself notes that 'each system is measured on its own draw, so a difference of 0.2 points is within noise' when comparing Jeff-1.0-Large to Quyet-1.0-Large. The same logic applies to the Jev comparison, yet the paper does not caution the reader. Moreover,Jev's own Intelligence score varies by8.3 points between OpenDecisions (71.9) and the leaderboard (63.6), indicating that question-set difficulty can substantially shift absolute scores. Without a system evaluated on both draws, part or all of the reported4.4-point advantage could reflect draw difficulty rather than model quality. This is a load-bearing issue for the most impactful claim in the paper.","section":"Abstract and Section 5.3 (Table14)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper repeatedly states that the trained model 'outperforms Jev by4.3 and4.4 Capability points respectively' (Abstract, Figure2 caption, Section5.2), but the4.4-point leaderboard gap is not accompanied by the cross-draw caveat. The manuscript should either (a) provide an indirect comparison via a system evaluated on both draws (e.g., the zero-shot model on both draws) or (b) qualify the claim as 'exceeds Jev's published score on a different draw' and state the uncertainty explicitly. As written, the headline number is misleading.","section":"Section5.3 (Table3 and leaderboard comparison)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"OpenDecisions is acknowledged as a development set (Section C.1), and its construction involves two models related to the training supervision: Qwen3.6-27B provides teacher labels, and Qwen3.8-27B participates in the label-panel verification. The paper notes this potential bias (Section6) and shows high Spearman correlation (0.90) with the leaderboard for14 public systems. However, the0.90 correlation does not guarantee that improvements on OpenDecisions transfer to truly independent evaluations; the residual error of3.0 Capability points (Section3) is non-negligible. The main training recipe was selected on OpenDecisions, so the OpenDecisions comparison to Jev (84.4 vs80.1) is not a valid generalization claim by itself. The held-out datasets (Table4) provide stronger evidence, but they show the trained31B model essentially ties Jev (Macro89.0 vs88.0) rather than exceeding it. The paper should temper the claim that the recipe 'exceeds Jev on a wide range of decision tasks' to better reflect the evidence.","section":"Section3 (OpenDecisions validity)","authors_acknowledge":"Finally, because OpenDecisions was used to select the training recipe, it is a development set rather than an independent test set. 17 Releasing the Inner Jev: A Simple Recipe for Open Decision Models Table 14.Official JevBench leaderboard(JevBench maintainers, 2026): the systems with the highest Capability, as published in October 2026 (revision v1.6.7).","authors_acknowledge_p":0.75}],"minor_comments":[{"comment":"The abstract states 'outperforming Jev by4.3 and4.4 Capability points respecively' without qualifying the draw difference for the leaderboard number. At the very least, a footnote should indicate that the leaderboard comparison uses different question draws.","section":"Abstract and Introduction","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The phrase 'leading Jev by4.3 ±2.5' on OpenDecisions is fine, but the next sentence should also note that OpenDecisions is a development set; this is acknowledged in Section C.1 but should be restated here.","section":"Section5.2 (first paragraph)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Table14 lists draws for each system. The text should explicitly state that Jev's draw (v1.6.1) and Jeff's draw (v1.6.2) are different and that direct comparison is confounded by draw difficulty. This is currently omitted.","section":"Table14 and surrounding text","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The limitations paragraph acknowledges the leaderboard comparison only indirectly (through the general caveat about OpenDecisions not representing all applications). It should also mention the cross-draw comparison issue.","section":"Section6 (Discussion and limitations)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper is otherwise thorough and well-written. The zero-shot results and ablations are particularly compelling. The training details and data checks are commendable.","section":"General","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"You should know two things about this paper. First, it is a competently executed study showing that existing open LLMs (Gemma 4, Qwen3.6) can match the proprietary Jev decision model zero-shot and exceed it after a cheap LoRA finetune. Second, the main caveats are honestly acknowledged but real: the leaderboard draw comparison is slightly weaker than advertised, abd OpenDecisions is a development set.\n\nWhat's new? Not the letter-logit readout — that is in prior work (SemIf). Not the LoRA finetuning — that is standard. The novelty is the systematic demonstration that these tools together are sufficient: temperature scaling helps calibration, and a 5 GPU–hour LoRA run on an instruction–tuned model pulls ahead of the proprietary alternative. That is useful for practitioners who want to avpid vendor lock–in.\n\nThe evaluation is thorough: three seeds, error bars, overlap checks, seven held–out public datasets, and the independent JevBench leaderboard. The ablations cleanly attribute improvement to LoRa rather than the learnable head. The code, models, and OpenDecisions are released. These are genuine strengths.\n\nSoft spots in proportion. The stress-test about leaderboard draws is osrt of right but overstated. Jeff–1.0–Large and Jev were measured on different draws (v1.6.2 vs v1.6.1), and the paper does not confirm equal difficulty. That matters for the headline 4.4–point claim. But the OpenDecisions comparison (84.4 vs 80.1 on the same 2,352 questions) gives a consistent 4.3–point advantage, so the robuster support comes from the authors' own development set, not the leaderboard. The paper acknowledges OpenDecisions is a development set (Section C.1), a point the reader and stress–test both repeat. The label quality concern (Qwen3.6–27B trains, Qwen3.8–27B eval labels) is also flagged by the authors. None of this sinks the paper, but a reasonable referee should ask for an independent held–out set for the final comparison, or at least a sensitivity analysis. \n\nSmaller models benefit less; the 12B and 4B gains are modest or zero. That limits the generality of the recipe but does not invalidate the conclusions for the 30B class.\n\nWh o is this for? Practitioners who want to build a decision system on open models and need a straightforward recipe. It confirms that expensive specialized architecture s and RL–based training are unnecessary for reaching current leaderboard parity.\n\nWould I accept for review? Yes. The paper is well–executed, the claims are (with the draw caveat) supported by mutiple lines of evidence, abd the realeased artifacts are useful. A smart referee can tighten the leaderboard discussion and ask for an independent validation set, but the core result will surviev moderate scrutiny.","headline":"Solid empirical paper showing open instruction-tuned LLMs can match a proprietary decision system with a simple letter-logit readout and cheap LoRA finetuning. The central claim holds up despite some caveats.","tokens_in":34129,"tokens_out":1504,"duration_ms":16281},"pith_extraction":null,"created_at":"2026-10-14T17:23:49.700836+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":null,"supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}