{"id":"5fc527fb-30c2-4e8a-97fe-89b4bcde536d","arxiv_id":"2608.06027","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A hybrid voice agent with LLM extraction plus rule-based validation can complete Hindi enrollment forms robustly, and end-to-end form completion, not component accuracy, should guide model choice.","lead":"The authors built FormBharo, a phone-based voice agent that fills a maternal health enrollment form in Hindi for low-literacy users, mixing LLMs with rule-based validation. They release a benchmark of simulated calls and show that component-level accuracy does not predict how well the whole system completes forms.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark STT runs on 16 kHz PCM, but deployed pipeline receives 8 kHz µ-law telephony audio; form-completion gaps and model rankings may not transfer to deployment.","rationale":"The reader's verdict is CONDITIONAL with the weakest assumption being benchmark realism. I agree with that broad category, but I want to sharpen it to a concrete, testable pipeline mismatch. The paper's central empirical claims—form-completion drops of up to ~41 points under real-speech transcripts and the ability of rule-based recovery to let cheaper models match frontier models—are computed on STT outputs from benchmark audio stored as 16 kHz PCM (Appendix E). Production audio is 8 kHz µ-law from Exotel (Section E). Section 1 lists telephone-channel distortion as a core difficulty, yet the benchmark never applies it. This is not a generic 'real conversations are messier' objection; it is a specific engineering discrepancy between the evaluation channel and the deployment channel. ASR error rates on 8 kHz telephony speech are generally higher and structurally different (loss of high-frequency energy affects fricatives and aspiration, which matter in Hindi), so the error-propagation and error-cancellation patterns the paper documents could change, potentially shifting model rankings. The test I propose is straightforward: resample/encode the existing clips to 8 kHz µ-law, rerun the three STT models, and recompute form completion. That directly tests whether the deployment recommendation survives. I also weighed the LLM-judge concern: the open-field judge is gpt-5.4-mini, itself an evaluated EXTRACT model, and calibration is on only 50 examples (Appendix B.3). This is a real measurement risk, but it affects only 3 of 12 fields and is partially mitigated by exact-match scoring for closed fields; the headline ~41-point drop is driven largely by closed fields under severe STT errors, so judge bias is less likely to overturn the central claim. The telephony mismatch affects every field and every model ranking, making it more load-bearing. That said, the paper has genuine strengths: human-recorded Hindi audio, a large simulated call set, a sensible hybrid architecture, and honest Section 7 limitations. The concern does not invalidate the benchmark as a research artifact; it means the deployment-oriented conclusions are conditional on a channel match that is currently untested. The reader's CONDITIONAL verdict already captures this; I am not proposing a change.","tokens_in":21346,"tokens_out":5487,"duration_ms":54511,"concrete_test":"Resample the 380 benchmark clips to 8 kHz and encode as µ-law (or downsampled to 8 kHz and back, if a codec is unavailable), re-run the three STT models used in integration tests (Scribe v2, Saaras v3, Nova-3), and recompute Tables 5 and 6 (form-completion accuracy and median drops vs. reference). If the median form-completion drop increases by more than ~2 points or the best EXTRACT model under Scribe v2 changes, the deployment model-selection result is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's deployable-configuration claims rest on STT outputs computed from benchmark audio stored as 16 kHz mono 16-bit PCM (Appendix E), while the production path uses Exotel, which delivers 8 kHz µ-law audio over a WebSocket. Section 1 explicitly lists telephone-channel distortion as a core challenge, and Section 7 acknowledges untested acoustic combinations but does not mention the telephony codec. Because 8 kHz µ-law bandwidth-limiting and companding change ASR error rates and error types (especially for Hindi retroflex/aspirated contrasts and fricatives), the measured ~41-point form-completion drop (Section 5.4, Table 5) and the ranking of cheaper EXTRACT models under Scribe v2 transcripts are conditional on the more forgiving 16 kHz channel. Error propagation and cancellation patterns could differ under higher or differently structured STT error, so the conclusion that rule-based recovery lets smaller models match or surpass frontier models may not hold in the real pilot. This is a concrete pipeline mismatch, not a general realism objection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents FormBharo, a hybrid voice agent for conversational form filling over phone calls, combining LLMs for extraction and reply generation with deterministic rule-based validation and flow control. It introduces FormVoiceAgentBench, a Hindi benchmark with 960 simulated calls assembled from 380 human recordings, covering 3,760 unit tests across transcription, extraction, reply generation, and end-to-end form completion. Experiments across five STT models and 11 LLMs show that component-level accuracy does not predict end-to-end form completion, that a rule-based layer recovers many extraction errors so cheaper models can match or surpass frontier models, and that no single model dominates accuracy, latency, and cost simultaneously; a weighted-sum scalarization is used to select a deployable configuration.","tokens_in":21601,"tokens_out":8209,"duration_ms":78718,"significance":"The benchmark and the hybrid architecture are a useful contribution to voice-agent evaluation in low-resource settings, and the release of human-recorded Hindi audio with multi-turn tests fills a gap. The paper's central finding - that per-turn extraction accuracy can mislead about end-to-end form completion - is important and well illustrated (e.g., GPT-5.5 leading extraction but not form completion). The authors are transparent about limitations in Section 7 and provide confidence intervals and judge calibration details. However, the transfer of the reported numbers to the actual telephony deployment is not yet established, and the integration-test design does not exercise branch-level error propagation.","major_comments":[{"comment":"The benchmark audio is stored and processed as 16 kHz mono 16-bit PCM (Appendix E), while the deployed system receives 8 kHz µ-law audio from Exotel over a WebSocket (Appendix E). Section 1 explicitly lists telephone-channel distortion as a core challenge, and the paper's headline result is the ~41-point form-completion drop when models receive error-prone real-speech transcripts (Section 5.4, Table 5). Band-limiting to 8 kHz and µ-law companding change ASR error rates and error types, particularly for Hindi retroflex/aspirated contrasts and fricatives, so the measured gaps and the ranking of EXTRACT models under Scribe v2 transcripts are conditional on a more forgiving channel than the one in the pilot. The authors should re-run the STT and downstream evaluations on audio downsampled and companded to the 8 kHz µ-law codec, or explicitly qualify all deployment claims as conditional on 16 kHz.","section":"Appendix E / Section 1"},{"comment":"The integration tests chain extractions by writing each turn's extracted values into the form state, but the conversation history and the next user turn are fixed by the simulated user's scripted path (Section 3.4). Consequently, an extraction error that would change a branching decision (e.g., misclassifying the pregnancy answer) does not alter the subsequent questions in the evaluation, even though Section 2 motivates the design by the risk that 'an incorrectly captured field can send the agent down the wrong path.' The end-to-end form-completion numbers therefore do not capture branch-level error propagation and may overestimate accuracy in real calls. The authors should either make the next question depend on the predicted form state in the integration loop, or clearly state in Section 7 that branch misrouting is not evaluated.","section":"Section 4.2 / Section 2"},{"comment":"The final EXTRACT selection depends on a 0.51-percentage-point difference in form-completion accuracy between Gemini 3.5 Flash (92.50±0.53) and Claude Sonnet 4.6 (93.01±0.50) on Scribe v2 transcripts. These 95% confidence intervals overlap, and since each configuration was run only once (Appendix E), item-level intervals do not account for run-level nondeterminism. The weighted-sum scalarization then selects different models for w_a in [0.50,0.61] versus [0.62,0.90], and the paper does not state the actual w_a used for deployment. The selection is therefore not statistically grounded. Please report repeated-run variability or a significance test, and specify the chosen weights.","section":"Section 5.5, Tables 11 and 13"}],"minor_comments":[{"comment":"The term 'piloted' is used to claim novelty ('first conversational voice agent piloted'), but no live pilot results are reported; the evaluation is entirely offline simulation. Please either temper the claim or clarify that the pilot is ongoing and this paper reports the pre-pilot benchmark.","section":"Section 1 / Abstract"},{"comment":"The deployment-constraint sentence ('discard models that fail our deployment constraints: p95 latency below 5s and form-completion accuracy above 90%') could be rephrased to specify which direction passes; as written it is ambiguous whether latency must be below or above 5s.","section":"Section 5.5"},{"comment":"The statement that each form field has 'three representations: the value, the desired form entry and the ground truth' is confusing because 'desired form entry' and 'ground truth' appear to be the same thing; clarify the intended distinction.","section":"Section 3.2"},{"comment":"The LLM judges were calibrated against human labels on only 50 test cases, and one judge (GPT-5.4-mini) is also an evaluated model; the small calibration set is acknowledged, but the report should also discuss the potential for judge bias toward outputs of the same model family.","section":"Appendix B.3"},{"comment":"LLM-WER is referenced to an external blog post; the manuscript would benefit from a self-contained definition of the metric, including how the LLM's semantic-equivalence classification is obtained and how inconsistent LLM judgments are handled.","section":"Section 4.1"},{"comment":"Costs are given as totals for transcribing the benchmark; it would be helpful to also report cost per audio hour for comparability.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is solid and the benchmark release is valuable, but the codec mismatch between the benchmark audio (16 kHz PCM) and the deployed telephony channel (8 kHz µ-law) is a concrete pipeline problem that directly affects the paper's central empirical claims. The fixed-script integration design also sidesteps the branch-misrouting failure mode that motivates the work. Both issues are fixable within the manuscript's scope: re-running on 8 kHz µ-law audio and either extending the integration loop or explicitly delimiting the claims. The statistical robustness of the final model selection should also be tightened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading and worth refereeing. The genuinely new piece is FormVoiceAgentBench, and the empirical lesson — component-level accuracy does not predict end-to-end form completion, and a cheap model plus rule-based validation can match a frontier model — is real and useful. The architecture (LLM extraction + rule-based guards) is not new, but the evaluation is careful enough that the finding lands.\n\nCredit where due: real Hindi audio from five speakers matched to the target demographic, four acoustic variations per utterance, 3,760 unit tests over 960 simulated calls, confidence intervals on all main tables, and an unusually honest limitations section. The rule-based layer is simple and sensible. They measure cost and latency, not just accuracy. Ranking STT by LLM-WER rather than raw WER is a sound choice. The GLM-5.1 collapse under Scribe v2 transcripts is a concrete illustration of how fragile some models are.\n\nSoft spots, in proportion. The benchmark is simulated: scripted, well-formed answers, one acoustic variation per call, five users. Section 7 concedes this, so credit. The bigger issue is in Appendix E: benchmark audio is 16 kHz mono PCM, but the production pipeline receives 8 kHz µ-law telephony from Exotel. The paper lists telephone-channel distortion as a core challenge in Section 1 but never tests it. The form-completion gaps and model rankings may shift under the codec, especially for Hindi retroflex/aspirated contrasts and fricatives. They should add an 8 kHz condition or temper the deployability claims. Second, the reference transcripts were generated by GPT-5.5, which is also an evaluated model, and GPT-5.4-mini serves both as an EXTRACT candidate and as the LLM judge. The calibration procedure helps, but 50 + 50 tests is small; the overlap is a mild circularity worth flagging. Third, code and data are promised but not yet released — for a benchmark paper, that is a real gap. Finally, each configuration was run once, so the confidence intervals reflect item-level variation only, not run-level variance; minor, but worth stating.\n\nWho this is for: people building voice agents for low-literacy LMIC settings, and anyone evaluating cascaded LLM pipelines with speech input. It deserves a serious referee. I would accept it conditional on releasing the benchmark, addressing the 8 kHz mismatch, and acknowledging the judge/transcript overlaps more explicitly.","headline":"A careful, honest evaluation with a useful new benchmark; the core findings hold, but the simulation and 8 kHz mismatch mean the numbers should be read as a floor, not a guarantee.","tokens_in":22065,"tokens_out":3647,"would_cite":true,"duration_ms":35363,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FormBharo pairs large language models with a deterministic rule-based validation layer to fill health enrollment forms over phone calls in Hindi, and its benchmark shows end-to-end form completion—not component accuracy—decides which…","keywords":["voice agents","conversational form filling","Hindi speech recognition","LLM extraction","rule-based validation","end-to-end form completion","low-literacy users","Pareto model selection"],"falsifier":"Collect a corpus of live or role-played full clinic calls containing wrong, partial, and self-corrected answers with combined acoustic degradations (noise plus distant microphone), run the same STT-extract-reply pipeline, and compare form-completion accuracy and model rankings against FormVoiceAgentBench; a large drop or a ranking flip would falsify the claim that the measured behavior transfers.","tokens_in":21178,"feed_emoji":"📞","tokens_out":9903,"duration_ms":88896,"temperature":0.7,"pith_summary":"FormBharo is a phone-call voice agent that fills a structured health-enrollment form by asking one question at a time in Hindi, for women who may not be able to read or write. The paper argues that a hybrid design—large language models for understanding and phrasing, plus deterministic rule-based validation and flow control for everything else—can complete these forms accurately under the tight latency and cost limits of a real NGO pilot. Its central empirical claim is that end-to-end form completion, not per-turn extraction accuracy, is the right measure: real-speech transcription errors can cut form completion by up to about 41 percentage points, while the rule layer recovers many extraction errors and lets smaller, cheaper models match or beat frontier models. If right, this means automated spoken enrollment for social-benefit programs is within reach for low-resource settings, and it gives the field a benchmark, FormVoiceAgentBench, to test such agents on noisy Hindi audio.","feed_headline":"Cheap LLMs match frontier models on spoken form-filling","feed_subtitle":"A hybrid voice agent with a rule-based checker completes over 90 percent of Hindi enrollment forms at a fraction of the cost.","key_machinery":"The load-bearing mechanism is the deterministic rule-based layer that sits between the EXTRACT and REPLY LLMs. It validates each extracted value against field-specific guards (ten-digit phone numbers, dates in the past, option indices), normalizes values such as strings returned for numeric fields, enforces retry limits, skips optional fields the caller declines, evaluates conditional branches, and selects the next question. This layer is what lets smaller, cheaper models reach the same form-completion accuracy as frontier models, because it absorbs type and format errors that would otherwise corrupt the form state. The benchmark itself—FormVoiceAgentBench, pairing human-recorded Hindi audio with 3,760 multi-turn conversation tests across 960 simulated calls—is the measuring instrument, and the key metric is form-completion accuracy: the fraction of final form fields captured correctly at the end of the call.","core_discovery":"The paper's central claim is that conversational form-filling over a phone call for low-literacy users is feasible with a pipeline that uses LLMs only where they add value: EXTRACT turns each transcribed answer into structured form values, and REPLY phrases the next question, while a deterministic rule-based layer validates values, handles retries and branching, and decides the next question. On FormVoiceAgentBench—3,760 unit tests and 960 simulated calls built from 380 human recordings of Hindi speech under noise, microphone-distance, and pace variations—the authors find that feeding LLMs real speech transcripts instead of clean reference transcripts drops form completion by up to about 41 points, a much larger drop than per-turn extraction accuracy suggests. The rule-based layer recovers many of those errors: Gemini 3 Flash scores 95.96 percent extraction accuracy but 100 percent form completion on reference transcripts because a type mismatch is normalized before storage. Consequently component performance does not predict end-to-end performance; GPT-5.5 leads extraction at 99.8 percent but ranks lower on form completion. No single model wins on accuracy, cost, and latency, so the authors select a deployable configuration by Pareto filtering and weighted-sum scalarization, landing on Scribe v2 for transcription, Gemini 3.5 Flash for extraction, and GPT-5.4-mini for reply generation. To the authors' knowledge, this is the first conversational voice agent piloted to fill an enrollment form for this population.","pith_inferences":["Live-call speech is the obvious stress test: the paper's own limitations section concedes real callers give wrong, partial, or self-corrected values, and distant-microphone-plus-noise combinations were never recorded, so a natural extension is to measure whether the ~41-point gap and the model rankings survive a corpus of full, naturalistic calls.","The 'errors both propagate and cancel' finding likely generalizes beyond speech: any multi-stage LLM pipeline may reward choosing models by end-to-end task success rather than by component benchmarks.","Adding cheap deterministic cross-field consistency checks (e.g., date-of-birth versus reported gestational age) could recover additional real-speech errors the current guards miss, since the rule layer already absorbs format and type errors.","Because the pilot form has only 12 fields with mostly fixed order, the rule layer's rescue effect may shrink on longer or more open-ended documents; testing the same hybrid on a larger form would bound the approach's generality."],"forward_implications":["Deployers can choose a cheap configuration that clears the completion bar: with the rule layer in place, Gemini 3.5 Flash as EXTRACT and GPT-5.4-mini as REPLY reach 92.50 percent and 96.70 percent accuracy respectively on Scribe v2 transcripts.","Real-speech robustness, not reference-transcript accuracy, determines deployed quality, since transcription errors can cut form completion by up to about 41 points; agents should be tuned and tested on error-prone transcripts from the chosen STT model.","Per-turn leaderboards can mislead model selection: GPT-5.5's 99.8 percent extraction accuracy does not translate into top form completion, so end-to-end evaluation should be the standard for voice-agent pipelines.","The hybrid architecture makes the system tolerant of one class of small-model errors: Gemini 3 Flash's type mismatches (numeric fields returned as strings) are normalized by the rule layer, yielding 100 percent form completion despite 95.96 percent extraction accuracy.","Deployment is an explicit multi-objective choice: Pareto filtering plus weighted-sum scalarization turns accuracy, latency, and cost priorities into a concrete model pick, giving a template for constrained deployments."],"supporting_citations":[{"why":"Supplies the LLM-WER metric used to rank STT models, which changes which transcripts feed the end-to-end tests.","marker":"(Sarvam AI 2026)"},{"why":"GPT-5.5 generated the reference transcripts and expected values that define the benchmark's ground truth and its clean-input condition.","marker":"(OpenAI 2026)"},{"why":"Provides the weighted-sum scalarization method used to select the deployed EXTRACT and REPLY models on the Pareto frontier.","marker":"(Marler and Arora 2010)"},{"why":"Empirical basis that speech interfaces can beat touch-tone for low-literate users, motivating the voice-agent approach.","marker":"(Sherwani et al. 2009)"},{"why":"VoiceBench is the closest prior voice-agent benchmark; this work extends evaluation to a constrained structured form-filling task on noisy Hindi audio.","marker":"(Chen et al. 2026)"},{"why":"Establishes the LLM-as-a-judge paradigm used to calibrate the judges scoring open-ended fields and replies.","marker":"(Zheng et al. 2023)"},{"why":"Documents code-mixing and spontaneous-speech challenges for Hindi ASR that the benchmark's acoustic variations are designed to stress.","marker":"(Diwan et al. 2021)"}],"fun_headline_variants":["Voice agent fills Hindi forms cheaply with hybrid LLM rules","Rule-based layer rescues form completion from noisy transcripts","Hybrid voice agent matches frontier models on enrollment forms","Cost-effective voice agent for low-literacy form filling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that scripted, well-formed answers stitched into calls with one acoustic variation per call faithfully represent real clinic conversations, and that reference transcripts generated by GPT-5.5—itself one of the evaluated models—are a neutral test input.","fun_headline_variants_meta":{"raw":{"variants":["Voice agent fills Hindi forms cheaply with hybrid LLM rules","Rule-based layer rescues form completion from noisy transcripts","Hybrid voice agent matches frontier models on enrollment forms","Cost-effective voice agent for low-literacy form filling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000397,"raw_usage":{"total_tokens":2204,"prompt_tokens":1194,"completion_tokens":1010,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":810,"completion_tokens_details":{"reasoning_tokens":945}},"tokens_in":810,"tokens_out":1010,"duration_ms":10104,"temperature":1.0,"reasoning_tokens":945,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T19:07:09.191422+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a corpus of live or role-played full clinic calls containing wrong, partial, and self-corrected answers with combined acoustic degradations (noise plus distant microphone), run the same STT-extract-reply pipeline, and compare form-completion accuracy and model rankings against FormVoiceAgentBench; a large drop or a ranking flip would falsify the claim that the measured behavior transfers.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LLM-WER metric used to rank STT models, which changes which transcripts feed the end-to-end tests."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-5.5 generated the reference transcripts and expected values that define the benchmark's ground truth and its clean-input condition."},{"cited_title":"T.; and Arora, J","cited_arxiv_id":null,"evidence_quote":"Provides the weighted-sum scalarization method used to select the deployed EXTRACT and REPLY models on the Pareto frontier."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Empirical basis that speech interfaces can beat touch-tone for low-literate users, motivating the voice-agent approach."},{"cited_title":"T.; and Li, H","cited_arxiv_id":null,"evidence_quote":"VoiceBench is the closest prior voice-agent benchmark; this work extends evaluation to a constrained structured form-filling task on noisy Hindi audio."},{"cited_title":"P.; Zhang, H.; Gonzalez, J","cited_arxiv_id":null,"evidence_quote":"Establishes the LLM-as-a-judge paradigm used to calibrate the judges scoring open-ended fields and replies."},{"cited_title":"M.; Khare, S.; Unni, V.; Vyas, S.; Rajpuria, A.; Yarra, C.; Mittal, A","cited_arxiv_id":null,"evidence_quote":"Documents code-mixing and spontaneous-speech challenges for Hindi ASR that the benchmark's acoustic variations are designed to stress."}],"review_version":1}