{"id":"3eadc323-ea98-4760-8d85-bd5f17c2e81c","arxiv_id":"2601.22025","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Adding generic prompt rules to task-specific LLM prompts is not monotonic: in 15-20 case local suites, Llama 3 and Qwen 2.5 sometimes pass fewer extraction and RAG checks, so prompt changes should be tested per task.","lead":"An evaluation framework called MVES is paired with small local experiments showing that adding generic 'helpful' prompt rules can lower extraction and RAG compliance while improving instruction-following on two open models. A generalist should read it for a reproducible reminder that prompt tweaks need regression testing rather than assumed improvements.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ablation does not support the central mechanism: for RAG, generic rules alone (C) show no degradation; only the full wrapper+rules prompt (D) drops, contradicting the claim that rules, not wrapper, cause regressions.","rationale":"The reader's weakest assumption focuses on small sample size and prompt-selection artifacts. My concern is narrower and more damaging: the paper's own four-condition ablation, as presented in Table 13, does not support the causal story attached to the central claim. For the RAG suite, condition C (generic rules alone) shows no degradation, while condition D (wrapper + rules) shows the full 13.3pp drop. Therefore the statement 'the degradation comes from generic rules conflicting with task-specific constraints rather than from the system wrapper itself' is contradicted by the data for the headline RAG result. This is not a generalization or power issue; it is an internal inconsistency between the claimed mechanism and the reported numbers. The extraction degradation does support a rules effect, but the RAG result is one of the two main quantitative findings and the paper explicitly uses it to conclude that generic rules, not wrappers, are the problem. Because the mechanism is central to the paper's contribution and is not supported by the evidence provided, the verdict should move from CONDITIONAL to REJECT unless the ablation can be re-run with properly isolated conditions and shows the rule effect on RAG. I agree partially with the reader: small samples are a concern, but the more load-bearing problem is that the ablation's own structure fails to isolate the claimed cause.","tokens_in":24751,"tokens_out":6999,"duration_ms":74039,"concrete_test":"Run a crossed 2×2 ablation on Llama 3 8B with the exact prompt files from the repository: system-wrapper absent/present × generic-rules absent/present, keeping the rule text identical between C and D. Record per-case pass/fail and apply McNemar's test to the A/B/C/D pairs. If C (rules only, no wrapper) shows no significant RAG drop while D does, the paper's mechanism claim is falsified and the RAG regression must be attributed to the wrapper or to wording/placement unique to D. Also verify by direct diff that the C and D rule texts are identical; if they differ, the C-versus-D comparison is confounded.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's headline claim includes a mechanism: degradation comes from generic rules conflicting with task-specific constraints, not from the system wrapper itself. The paper's own Table 13 contradicts this for the RAG suite, which is one of the two headline results. For Llama 3 8B RAG: A (baseline)=93.3%, B (+wrapper)=93.3%, C (+rules)=93.3%, D (full improved)=80.0%. If generic rules alone caused the RAG regression, condition C should drop; it does not. The drop appears only when the wrapper and rules are combined (D). Thus the RAG regression cannot be attributed to generic rules per se; it depends on the full prompt or on content/placement unique to D. The extraction result does support a rules effect (C drops 100→90), but the RAG result—the larger headline decline—does not. The conclusion in §12.9 and §14.4 that 'generic rules, not system wrappers, cause regressions' is therefore unsupported for the RAG condition. Additionally, the equality A=B is interpreted as 'wrapper has no effect,' but with n=15 a two-case shift is within binomial noise; the same low power undermines the causal attribution. The prompt texts for C and D are not included in the paper, so the reader cannot verify whether the rule wording is identical across conditions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This technical report proposes the Minimum Viable Evaluation Suite (MVES), a tiered framework for application-level LLM evaluation, and combines it with a substantial synthesis of test-set design, metrics, RAG evaluation, and LLM-as-judge methods. The empirical core (Section 12) compares a task-specific baseline prompt with a generic “improved” prompt across three small suites (extraction 20, RAG 15, instruction 15) using Llama 3 8B and Qwen 2.5 7B under Ollama. The paper reports non-monotonic effects: the generic prompt degrades extraction and RAG pass rates in several conditions while improving instruction-following, and a four-condition ablation is used to attribute the degradation to generic rules rather than the system wrapper. All artifacts, scripts, and raw logs are said to be in a public repository.","tokens_in":25124,"tokens_out":4472,"duration_ms":54629,"significance":"The practical message — that generic prompt changes can trade off one task against another and should be regression-tested — is a valuable and broadly concordant existence claim. The paper's strengths are its reproducible local harness, explicit failure taxonomy, detailed LLM-as-judge bias treatment, and an unusually honest set of threats-to-validity statements, including an admission that the suites are small and model coverage narrow. If the reported effects are corrected and re-statisticized, the paper would be a useful practitioner-oriented demonstration rather than a decisive benchmark study. At present, however, the quantitative headline, the mechanistic conclusion, and the failure analysis contain internal contradictions that prevent the results from being accepted as stated.","major_comments":[{"comment":"The abstract claims the largest observed decline is Qwen 2.5 on RAG from 26/30 to 9/30 when generic rules are appended to the user prompt. No result in the body supports this. Table 10 reports Qwen RAG at 93.3% (baseline) and 86.7% (improved), and Table 13 reports Qwen RAG A/B/C at 93.3% and D at 86.7%. There is no condition with 26/30 or 9/30. This is a direct contradiction in the paper's central quantitative claim and must be corrected before any further review.","section":"Abstract vs. §12.4 and Table 10"},{"comment":"The mechanism attribution is not supported by the RAG ablation. Table 13 shows Llama RAG A=93.3%, B=93.3%, C=93.3%, D=80.0%, and Qwen RAG A/B/C=93.3%, D=86.7%. If generic rules alone caused the RAG regression, condition C should drop; it does not. The regression appears only when the wrapper and rules are combined in D. Thus the conclusion that “generic rules, not system wrappers, cause regressions” is unsupported for the RAG suite, which is one of the two headline results. The extraction result does show a rules effect (C drops 100→90 for Llama), but the RAG result does not. The prompt texts for C and D are not provided in the paper, so the reader cannot verify whether the rule wording is identical across conditions. The authors should either revise the mechanistic claim to “the combined wrapper+rules prompt caused RAG degradation” or supply the actual prompts and add an ablation that i","section":"§12.9, §14.4, Table 13"},{"comment":"The effect sizes are stated as precise percentage-point declines despite being based on 15–20 cases per suite, and the paper itself states in §5.7 that detecting a 5% absolute difference requires 400–600 cases. For example, Llama RAG at 93.3%→80% is 14/15→12/15, and Qwen RAG at 93.3%→86.7% is 14/15→13/15. These are two- or three-case shifts and are well within binomial sampling variability. The N=5 repetitions in the ablation do not help because the runs are deterministic and identical. The authors should report raw counts, confidence intervals, or an exact paired test (e.g., McNemar) and, more generally, frame the results as qualitative existence demonstrations rather than calibrated effect sizes.","section":"§12.4, §12.9, §5.7, §12.10"},{"comment":"The failure categorization for Llama RAG is inconsistent with the reported pass rates. Table 11 lists 2/15 RAG failures under the improved prompt, which would yield a pass rate of 13/15 ≈ 86.7%, but Tables 10 and 13 report 80% (12/15). The table either omits a third failure category or the pass-rate numbers are incorrect. Since the failure table is used to argue that generic helpfulness pressure causes unsupported claims, this discrepancy undermines the diagnostic narrative and should be reconciled.","section":"Table 11 vs. Tables 10 and 13"}],"minor_comments":[{"comment":"The abstract says “expanded 30-case-per-suite ablations,” but the suites contain 20, 15, and 15 cases. Clarify whether this refers to 30 total conditions or something else.","section":"Abstract, §12.4"},{"comment":"The statement that the system wrapper has “no effect” (A vs. B) is an absence-of-evidence claim given the sample size. Rephrase as “no effect was observed in this suite” and avoid asserting a true null.","section":"§12.9"},{"comment":"Reference [13] lists the author as “Chin-Yun Lin”; the correct name is Chin-Yew Lin. Also check the ROUGE citation details.","section":"References"},{"comment":"The helpfulness rubric contains the phrase “hallmarks hallucinations”; this should be “hallucinates” or “contains hallucinations.”","section":"Appendix A.2"},{"comment":"Add raw counts (e.g., 14/15) to the percentage cells. Given the small suite sizes, percentages alone overstate precision and make paired comparisons harder to verify.","section":"Table 10"}],"recommendation":"major_revision","confidential_remarks":"The paper reads partly as a practitioner-oriented survey and partly as a small experimental study. The survey material is competent but not novel in a research sense. The experimental contribution is what would justify publication here, and at present it has internal contradictions that must be fixed. I recommend major revision rather than rejection because the central existence claim is plausibly salvageable with corrected reporting, explicit prompt texts, appropriate statistical caveats, and a revised mechanism statement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a skim for anyone shipping LLM apps, but it needs serious revision before it can stand as a report. The practical message — generic prompt improvements can regress some tasks while helping others, so run task-specific suites before deploying — is sound and consistent with the prompt-sensitivity literature. What the paper adds is a small, reproducible, local demonstration: two models, three tasks, and a four-condition ablation with code, datasets, and raw logs in the repo. That kind of artifact is genuinely useful, and the author is honest about the small suite sizes and limited model coverage in §12.10. Credit where due: the harness is simple and documented, and the ablation design is the right idea even if the interpretation is shaky.\n\nThe soft spots are real, and one is load-bearing. The abstract claims Qwen RAG drops from 26/30 to 9/30 when generic rules are appended; the body tables show Qwen RAG stays at 93.3% under rules-only and only drops to 86.7% under the full improved prompt. That is not a minor typo — it is a contradiction in the paper's headline result. Second, the stated mechanism — that generic rules, not the system wrapper, cause regressions — is not supported by the ablation for RAG. Table 13 shows condition C (rules only) holds RAG at 93.3% for both models; the drop appears only in condition D (wrapper plus rules). The rules-effect is visible for Llama extraction, but the RAG regression cannot be attributed to rules alone. The author even phrases §12.9 carefully, but then the abstract and §12.10 overclaim the mechanism. Third, with 15–20 cases per suite and no confidence intervals, the quantitative magnitudes (13.3 pp, etc.) are probably noise-level; the paper's own §5.7 says 400–600 cases are needed for 5% differences, which undercuts its own point estimates.\n\nWho is this for? LLM application engineers and evaluation practitioners who want a concrete template for regression-testing prompts. It is not a scientific advance, and the MVES framework is mostly a repackaging of existing RAGAS/harness/judge material. Still, the reproducible artifacts and the honest limitations section make it a legitimate candidate for peer review — not a desk reject. A referee should require the abstract and body numbers to be reconciled, the causal claim about rules vs. wrapper to be restricted to the extraction task, and either confidence intervals or explicit acknowledgment that the observed differences are within sampling noise.\n\nBring it to a reading group if you want a case study in how small-sample eval results get overinterpreted; otherwise cite it as a cautionary example.","headline":"Useful practical warning with reproducible code; the regression phenomenon is real but the paper overclaims the mechanism and has an internal data contradiction.","tokens_in":25530,"tokens_out":2888,"would_cite":false,"duration_ms":30030,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that generic 'improved' prompt templates can hurt LLM task accuracy, so prompt edits should be treated as regression risks and tested on task-specific suites before deployment.","keywords":["LLM evaluation","prompt engineering","regression testing","retrieval-augmented generation","prompt sensitivity","evaluation-driven iteration","structured extraction","instruction-following"],"falsifier":"Run the same four-condition ablation on a suite of 400-600 cases per condition, as the paper's own power guideline recommends for detecting 5% differences; if the generic-rules condition stops degrading extraction and RAG compliance at scale, the central non-monotonicity claim would not generalize beyond the paper's selected cases. The raw logs and scripts are provided so this can be checked directly.","tokens_in":24662,"feed_emoji":"🧪","tokens_out":4584,"duration_ms":46788,"temperature":0.7,"pith_summary":"This paper sets out to show that prompt 'improvements' are not monotonic: what helps one LLM task can hurt another. Using a small, reproducible local evaluation harness with two open-weight models, it finds that adding generic helpfulness rules degraded structured extraction pass rate from 100% to 90% and RAG citation/content compliance from 93.3% to 80% for Llama 3, while improving instruction-following. A four-condition ablation attributes the degradation to generic rules conflicting with task-specific constraints, not to the system wrapper itself. The broader point is that teams should validate every prompt change against a task-specific golden suite rather than assume best-practice templates help. The paper therefore proposes the Minimum Viable Evaluation Suite (MVES) as a tiered standard for doing that validation.","feed_headline":"Generic 'better' prompts can cut LLM accuracy by 10 points","feed_subtitle":"Task-specific prompts beat generic rules in small tests; treat every prompt edit as a regression risk.","key_machinery":"The carrying mechanism is a four-condition ablation (baseline, baseline plus system wrapper, baseline plus generic rules, full generic prompt) run over three small curated test suites: extraction, RAG citation compliance, and instruction-following. The comparison isolates the source of the regression: the system wrapper is inert, and the degradation comes from generic helpfulness/verbosity rules colliding with task-specific constraints. Around this, the paper builds the Minimum Viable Evaluation Suite (MVES), a tiered structure linking application type to failure modes, metrics, artifacts, and validation evidence, organized in a Define-Test-Diagnose-Fix loop.","core_discovery":"On the paper's own terms, the central discovery is that a generic 'improved' prompt template trades off behaviors: it can help open-ended instruction-following while degrading structured tasks. In local experiments with deterministic decoding, replacing task-specific extraction prompts with a generic system wrapper plus explicit rules lowered Llama 3's extraction all-pass rate by 10 percentage points and its RAG compliance by 13.3 points. The ablation's key result is that adding the system wrapper alone changed nothing — the regressions appeared only when generic rules were appended to the user prompt, where they conflicted with task-specific grounding and JSON-format constraints. This is of","pith_inferences":["If the conflict mechanism generalizes, the same non-monotonicity should appear for other 'helpfulness' additions such as role-priming sentences or few-shot examples that imply verbosity, not just system wrappers.","The all-pass-rate framing likely understates production risk: a prompt that degrades RAG compliance by 13 points on 15 cases could produce a much higher rate of uncited or unsupported claims at scale, where edge cases dominate.","A testable extension is to run the identical ablation on domain-stratified suites of 400-600 cases per condition; the paper's own power guideline implies that would determine whether the observed trade-off holds beyond these hand-picked examples.","One could also instrument each suite per quality dimension (grounding vs. helpfulness vs. format) to predict which template edits will trade off against which dimensions, turning the finding into a diagnostic rather than a warning."],"forward_implications":["Prompt changes should be treated like code changes: run the task-specific suite before deployment, because the same edit can improve one behavior and break another.","Small golden sets (tens of cases) can surface regressions that informal spot checks miss, provided they cover the task's main failure modes.","Generic system wrappers are not automatically harmful; the harm in these experiments came from generic rules appended to the user prompt conflicting with task-specific constraints.","Explicit output-contract prompts (JSON-only, cite-only-provided-sources) can improve strict extraction and grounding even when they feel restrictive.","Instruction-following tasks can benefit from generic rule scaffolding, so the same template may be right for one module and wrong for another."],"fun_headline_variants":["Generic prompt rules tank RAG score 26→9","When 'better' prompts actually hurt LLMs","Generic add-ons slash RAG compliance 17 pts","LLM prompt tweaks can backfire: test first","One-size-fits-all prompt rules cut accuracy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The small hand-built suites (20 extraction, 15 RAG, 15 instruction cases) and the hand-chosen baseline and improved prompts are representative enough that the measured pass-rate shifts reflect a general phenomenon rather than small-sample noise or prompt-selection artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Generic prompt rules tank RAG score 26→9","When 'better' prompts actually hurt LLMs","Generic add-ons slash RAG compliance 17 pts","LLM prompt tweaks can backfire: test first","One-size-fits-all prompt rules cut accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1118,"prompt_tokens":785,"completion_tokens":333,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":256}},"tokens_in":529,"tokens_out":333,"duration_ms":4017,"temperature":1.0,"reasoning_tokens":256,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T06:42:44.636648+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four-condition ablation on a suite of 400-600 cases per condition, as the paper's own power guideline recommends for detecting 5% differences; if the generic-rules condition stops degrading extraction and RAG compliance at scale, the central non-monotonicity claim would not generalize beyond the paper's selected cases. The raw logs and scripts are provided so this can be checked directly.","supporting_citations":[],"review_version":1}