{"id":"889ebbc5-d05b-41a7-b466-d5a10cb50e6c","arxiv_id":"2607.13934","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A policy-grounded, cache-routed LLM architecture with human-in-the-loop verification reports PAR 0.94 and 55% lower P95 latency on simulated Ghanaian farm queries.","lead":"The paper describes Pezego-HITL, a human-in-the-loop AI advisory system for Ghanaian farmers that combines policy-checked retrieval with a cache of expert-approved recommendations, and claims higher policy compliance (PAR 0.94) with 55% lower tail latency than baselines. The wider interest is as a template for making LLM advice in high-stakes public services safer and faster, though the evidence rests on a simulated benchmark and self-report surveys.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Simulated 1,240-query benchmark is unreleased and likely in-sample: VCM templates and routing thresholds are derived from the same data that produce the headline PAR/AUR/latency, and AUR has no human calibration.","rationale":"The central contribution is an architecture and an evaluation protocol; the paper's acceptance hinges on whether P-EVAL's numbers are trustworthy. I looked for the condition that would have to be true for the headline to hold: the 1,240 simulated queries must be a fair test, the judge must be valid for both PAR and AUR, and the reported operating point must not be cherry-picked. None of these is currently checkable. The manuscript never describes how the 1,240 cases were created or whether they overlap with VCM templates; the routing threshold is justified by an ablation on the same benchmark; and the judge's human calibration covers only binary modification decisions, not the multi-criteria utility scores. The reported accuracy arithmetic inconsistency is a red flag that the evaluation pipeline needs auditing. These are not accusations of bad faith—they are concrete missing pieces. The ESO/farmer surveys and the 135-report empirical verification are real evidence for usability and HITL acceptance, but they do not validate the simulated PAR/AUR/latency claims. Therefore the reader's CONDITIONAL verdict is appropriate; my concern reinforces, rather than moves, that verdict.","tokens_in":20147,"tokens_out":7518,"duration_ms":73666,"concrete_test":"After freezing α and τ, collect 200 new real farmer/ESO queries not used to build VCM or the judge; have ESOs gold-label PAR and AUR on the outputs; run Pezego-HITL and B4 under the same protocol and compare. If held-out PAR/AUR are materially below 0.94/0.95, or the latency advantage shrinks, the in-sample concern lands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Headline numbers—PAR=0.94, AUR=0.95, P95=12.9s, 59.6% VCM reuse—all come from a 1,240-case simulated workload whose construction is described only as 'representative' (Section 5.2). No sampling procedure, query log, or independence from the VCM template database is given. At the same time, the routing parameters α=0.6 and τ=0.83 (Eq. 1) are selected/ablated on this same benchmark (Fig. 4), so the reported numbers are in-sample selection results, not out-of-sample estimates. The LLM judge was calibrated on 135 ESO-reviewed reports with only 19 modification positives (Section 4.3), and calibration covers only binary compliance decisions, not the multi-criteria AUR scores in Eq. 4; AUR=0.95 therefore rests on an unvalidated judge dimension. A concrete tip that the evaluation numbers need auditing: the stated confusion matrix (TP=16, FP=2, TN=114) yields accuracy 96.3%, not the reported 91.9%. If the simulated cases were generated from the same policy/template database that populates VCM, then the 59.6% reuse ratio and the 0.94/0.95 scores are partly predetermined, and the claimed improvement over B4 is not a property of the architecture alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Pezego-HITL, a human-in-the-loop LLM decision-support architecture for agricultural extension in Ghana. It combines schema-aware SQL retrieval against registered pesticide/policy databases, an automated constraint auditing layer, critique agents, and an expert-verified case memory (VCM) cache that serves previously approved templates when a composite structural/semantic similarity exceeds a threshold. The paper also introduces P-EVAL, a multi-objective evaluation protocol measuring Policy Alignment Rate (PAR), Agronomic Utility Rate (AUR), P95 latency, and supervision overhead. The main empirical claims are based on a simulated 1,240-query workload: PAR 0.94, AUR 0.95, P95 latency 12.9 s versus 28.6 s for a critique-only baseline, and a 59.6% cache reuse ratio, with a parallel open-source model evaluation (PAR 0.86). The socio-technical section reports survey responses from 30 Ghanaian ESOs and 36 smallholder farmers.","tokens_in":20490,"tokens_out":4451,"duration_ms":45651,"significance":"The architectural idea of routing recurring, structurally similar queries to an expert-verified cache is practical and relevant for low-bandwidth agricultural settings, and the attempt to formalize safety, utility, latency, and supervision in one evaluation framework is useful. The two-cohort survey adds credible qualitative evidence about trust and usability. However, the paper's central quantitative claims currently rest on an unreleased, vaguely described simulated benchmark and on an LLM judge that was calibrated on only 135 reports with 19 positive cases. If the benchmark and judge validation were made rigorous and transparent, the contribution could be a valuable template for policy-constrained deployment; as submitted, the headline numbers are not yet established.","major_comments":[{"comment":"The simulated 1,240-query benchmark is described only as 'representative' of typical crop cycles in Eastern and Ashanti regions. No sampling procedure, source query log, crop/pest distribution, or independence from the VCM template database is given. If the queries were generated from the same policy/template database that populates the VCM, the 59.6% reuse ratio and the resulting PAR/AUR are partly predetermined. Please release the dataset or provide a detailed generation protocol, and evaluate on a held-out set constructed independently of the VCM templates.","section":"§5.2, Table 1"},{"comment":"The LLM judge calibration is both inconsistent and insufficiently validated. The confusion matrix in §5.1 (TP=16, FP=2, TN=114, with 3 FN implied) gives accuracy (16+114)/135 = 96.3%, not the reported 91.9%. Additionally, calibration covers only the binary compliance decision, whereas AUR in Eq. (4) is a multi-criteria score, so AUR = 0.95 has no human calibration. Report the corrected accuracy, per-criterion agreement for AUR, and confidence intervals, ideally with a larger human-labeled sample.","section":"§5.1, Table 1"},{"comment":"The routing parameters alpha = 0.6 and tau = 0.83 are selected or ablated on the same 1,240-case benchmark that produces the headline results. Figure 4 identifies tau = 0.83 as a 'Pareto-optimal' point on this benchmark, so the reported reuse ratio and PAR/latency numbers are in-sample selection results, not unbiased out-of-sample estimates. Please provide a nested or cross-validated evaluation, or report the variance of the metrics across resamples of the benchmark.","section":"§3.1, Eq. (1), Fig. 4"},{"comment":"The reported PAR/AUR for Pezego-HITL conflate two different processes: 59.6% of queries are served from the VCM cache and inherit prior expert approval, while only 40.4% go through the generation pipeline. Comparing the aggregate 0.94/0.95 to B4's 0.89/0.90 therefore overstates the architecture's generation quality. Report PAR/AUR separately for cache-routed and generation-routed cases, and compare B4 against Pezego-HITL on the cache-miss subset alone. The latency advantage of cache reuse is real but should be presented as a caching benefit, not as a generation-quality improvement.","section":"§5.2, Table 1"},{"comment":"No confidence intervals, standard errors, or significance tests are reported for any headline metric (PAR, AUR, P95 latency, reuse ratio). With N=1,240, bootstrapped intervals would be straightforward and are necessary to assess whether differences such as 0.94 vs 0.89 are meaningful, especially given the judge's calibration error rate.","section":"§5.2, Table 1"}],"minor_comments":[{"comment":"Correct the reported accuracy from 91.9% to 96.3% based on the given confusion matrix, or explain the discrepancy if a different denominator was used.","section":"§5.1"},{"comment":"The phrase 'high student-to-teacher ratio' appears to be a typo; in context it should be 'officer-to-farmer ratio'.","section":"§5.5"},{"comment":"The shaded operating region tau in [0.80, 0.85] is described qualitatively. Adding the actual PAR and latency values at the endpoints and at tau = 0.83 would make the sensitivity analysis more informative.","section":"Fig. 4"},{"comment":"The open-source model 'Qwen3.5-9B-DeepSeek-V4-Flash' is cited only via a HuggingFace URL with no version or release date; verify the model name and provide a stable citation.","section":"§4.1, Ref. [33]"},{"comment":"The introduction contains two overlapping paragraphs beginning 'To address these limitations...' and 'To address these issues...' that cover nearly identical ground; consolidate to reduce redundancy.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The core architecture is plausible and the survey component is a useful complement, but the quantitative evaluation as submitted is not defensible. The unreleased simulated benchmark, the in-sample selection of tau, the missing AUR calibration, and the confusion-matrix arithmetic error all need to be addressed before the paper can be considered. I do not see this as a reject-and-refile: the fixes are within the scope of a revision, provided the authors can release or fully specify the benchmark and re-run the evaluation with proper held-out data and uncertainty estimates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this as a systems paper, not an evaluation paper. The architecture is real and the human-in-the-loop motivation is well argued. What is actually new is the integration: schema-aware SQL-grounded RAG, an automated constraint auditing layer, expert-verified case-memory routing, and a multi-objective protocol (P-EVAL) that makes safety, utility, latency, and supervision workload explicit. The field deployment with a near-census ESO cohort and a farmer cohort is a genuine plus, and the prompt templates in the appendix are a good reproducibility gesture. The open-source model check is also a useful generalization test.\n\nThe soft spots are where the numbers come from. The 1,240-case simulated benchmark is not described in enough detail: no sampling procedure, no query log, no statement of independence from the VCM template database. The routing parameters alpha and tau are selected on that same benchmark, so the reported PAR/AUR/latency are in-sample selection results, not out-of-sample estimates. The 59.6% reuse ratio is partly by construction: a large share of the improvement over B4 comes from serving cached templates, not from better generation. That is fine as a design claim, but it should be labeled as such.\n\nThe LLM judge calibration is also thin. It is based on 135 reports with only 19 positive modification cases, and it validates binary compliance decisions only. The AUR metric — with its multi-criteria utility scoring — has no human calibration at all, so AUR=0.95 should not carry the weight the abstract gives it. There is also a small arithmetic inconsistency: the stated confusion matrix (TP=16, FP=2, TN=114) gives 96.3% accuracy, not the reported 91.9%. Minor, but the kind of thing that makes a referee want the raw data.\n\nThe surveys are small and self-reported, but the ESO census framing is reasonable and these results are not the load-bearing part of the paper. The central architecture is coherent and the direction is useful; the problem is that the evaluation protocol is not yet transparent enough to support the headline claims. All of this is fixable: release the benchmark and judge rubrics, report error bars or at least the selection procedure, and soften the language about guarantees.\n\nVerdict: worth sending to peer review, but with the explicit condition that the benchmark, judge rubric, and parameter selection be released or independently audited. If that condition is not met, desk rejection is defensible on reproducibility grounds. I would not cite the headline numbers in my own work until then.","headline":"A genuine systems contribution with a coherent architecture, but the headline numbers rest on an unreleased, likely in-sample simulated benchmark and an unvalidated judge dimension.","tokens_in":21034,"tokens_out":1430,"would_cite":false,"duration_ms":16797,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The authors claim that policy-constrained LLM decision support is best framed as an adaptive compute-allocation problem, and that a memory-routed architecture with an expert-verified cache can deliver policy-compliant, actionable advice at","keywords":["agricultural extension","large language models","retrieval-augmented generation","decision support systems","policy compliance","human-in-the-loop AI","smallholder farming","pest management"],"falsifier":"Run the same architecture on an independent set of, say, 300 real or freshly elicited farmer queries that were not drawn from the architecture's own policy templates or VCM entries, and have a panel of agronomists blind to the system label each output. If PAR/AUR drop substantially below 0.94/0.95, or if judge–expert agreement on the new set is much lower than κ=0.77, the headline numbers are partly an artifact of the evaluation setup.","tokens_in":20034,"feed_emoji":"🌾","tokens_out":4602,"duration_ms":40329,"temperature":0.7,"pith_summary":"The authors try to establish that safe, useful AI advice for smallholder crop protection is not mainly a matter of a better language model; it is a compute-allocation problem. They propose an architecture that answers recurring queries from a cache of expert-verified recommendations, and only runs expensive generation with policy checking for new cases. On 1,240 simulated field queries, they report that this routing lifts policy alignment to 0.94 and agronomic utility to 0.95 while cutting P95 latency from 28.6s to 12.9s, with about 60% of queries served from cache. They also introduce P-EVAL, a protocol that measures safety, usefulness, latency, and expert editing burden together. A sympathetic reader would care because the claimed trade-off—safety without sacrificing speed—is exactly what makes AI decision support viable in low-connectivity rural settings.","feed_headline":"Verified-case cache lifts policy-safe AI advice to 94%, halves latency","feed_subtitle":"A memory-routed architecture serves recurring farmer queries from expert-approved templates, cutting P95 latency from 28.6s to 12.9s.","key_machinery":"The load-bearing object is the Verified-Case Memory (VCM), a cache of expert-approved recommendation templates indexed by a composite similarity score s = α s_struct + (1−α)s_semantic, with α=0.6. Structural similarity is gated by exact matches on crop, pest, and growth stage; semantic similarity is cosine distance of dense embeddings. Queries with s ≥ τ=0.83 are served from cache, which the paper argues is what converts a safety-verified but slow pipeline (28.6s tail) into a usable one (12.9s) while preserving compliance. The threshold τ is the control knob on the safety-latency frontier: lower thresholds increase reuse but risk policy drift; higher thresholds preserve compliance at the cos","core_discovery":"The central claim is that policy-constrained LLM assessment should be treated as an adaptive test-time compute allocation problem, instantiated by Pezego-HITL. The architecture combines schema-aware retrieval that translates natural-language field queries into SQL constraints over a pesticide registry, an automated constraint-auditing layer that verifies active ingredients and growth-stage compatibility, and a Verified-Case Memory (VCM) that stores expert-approved recommendations. When an incoming query matches a cached case above a structural-semantic threshold (0.83), the system serves the verified recommendation directly, bypassing generation; otherwise it runs a multi-agent pipeline with","pith_inferences":["Editorial inference: if the simulated 1,240-case benchmark were partly generated from the same policy templates or VCM entries that the architecture uses, the reported PAR/AUR and reuse ratio would be optimistic; an independent field sample or hold-out set of queries would be needed to confirm.","Editorial inference: the cache-reuse mechanism suggests a testable general principle for other high-stakes advisory domains (e.g., clinical or legal triage): route by verified precedent first, generate only on novelty, and use expert edits to grow the precedent base.","Editorial inference: the paper leaves the generation of the simulated workload opaque; varying those cases to measure whether VCM reuse genuinely reflects recurring farmer questions could distinguish a caching effect from an in-sample artifact."],"forward_implications":["If the results hold, safety compliance and operational latency need not be in opposition: an expert-verified cache can absorb the recurring fraction of queries and reserve expensive generation for genuinely new cases.","The framework makes the supervision burden measurable: the paper reports a 14.1% expert modification rate and a mean edit ratio of 38.3% on 135 real reports, and folds these into the P-EVAL objective.","The architecture is model-agnostic: the same interventions raise the open-source model's PAR from 0.44 to 0.86 and cut P95 latency by 54.5%, suggesting local deployment is feasible without sacrificing compliance.","The VCM threshold τ acts as an explicit Pareto control; the paper claims τ=0.83 is a balanced operating point with 59.6% cache reuse, and that this generalizes across model classes."],"fun_headline_variants":["Expert-cached AI advice: 94% policy-safe, 55% faster for farmers","Verified-case memory boosts policy-safe farm AI to 94%, cuts latency 55%","Farm AI hits 94% policy alignment, halves response time","Ghana farm AI: 94% safe, 55% faster with expert-verified cache","Policy-safe AI for smallholders: 94% alignment, 55% lower latency"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the 1,240 simulated field queries faithfully represent real farmer questions, and that the LLM judge—calibrated on only 135 expert-reviewed reports—labels the remaining cases correctly; the paper does not describe how the cases were generated or how judge error propagates.","fun_headline_variants_meta":{"raw":{"variants":["Expert-cached AI advice: 94% policy-safe, 55% faster for farmers","Verified-case memory boosts policy-safe farm AI to 94%, cuts latency 55%","Farm AI hits 94% policy alignment, halves response time","Ghana farm AI: 94% safe, 55% faster with expert-verified cache","Policy-safe AI for smallholders: 94% alignment, 55% lower latency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1478,"prompt_tokens":881,"completion_tokens":597,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":488}},"tokens_in":625,"tokens_out":597,"duration_ms":6644,"temperature":1.0,"reasoning_tokens":488,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T03:16:21.265353+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same architecture on an independent set of, say, 300 real or freshly elicited farmer queries that were not drawn from the architecture's own policy templates or VCM entries, and have a panel of agronomists blind to the system label each output. If PAR/AUR drop substantially below 0.94/0.95, or if judge–expert agreement on the new set is much lower than κ=0.77, the headline numbers are partly an artifact of the evaluation setup.","supporting_citations":[],"review_version":1}