{"id":"0e79db4f-1b3d-4fd1-bd55-4e5b72b39106","arxiv_id":"2608.12779","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM generator-verifier refinement loop improves temporal ordering of symptoms in single-report vaccine narratives, evaluated on a new 5,347-report benchmark.","lead":"This paper introduces CRAFT, an LLM-based generator-verifier loop that builds symptom timelines from vaccine adverse event reports with few explicit time anchors, plus MedTempo, a benchmark of 5,347 expert-annotated narratives. It reports small but consistent accuracy gains over its own baselines on four LLMs, though the benchmark itself is the larger contribution.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 'consistently improves' rests on 0.1–1.6 EM-point gains with no significance testing, and the verifier hurts Claude; paired tests are needed before the central claim is established.","rationale":"Good-faith reading: the paper contributes both a benchmark (MedTempo) and a method (CRAFT), and the central claim is that CRAFT-Full empirically outperforms baselines. The benchmark construction is transparent, and the 93% inter-annotator agreement is real evidence for label reliability; the gold-standard anchoring concern is legitimate but not settled, and a blind re-annotation test would be valuable. However, the single point where the central claim is least secure is the quantitative evidence for 'consistently improves.' Section 6.1 reports margins of 0.1 to 1.6 EM points over baselines with no significance testing. Deterministic decoding removes sampling noise, but the relevant uncertainty is the instance-level sample, and a paired test is straightforward and should have been reported. The paper's own Table 6.1 shows CRAFT w/o V beating CRAFT-Full for Claude (37.83 vs. 37.14), and Appendix A.2 shows theta=3 was chosen on 100 GPT-4.1 development examples, so the verifier benefit is not shown to transfer across backbones. These points do not disprove the method: CRAFT-Full does beat both baselines on every backbone. But they move the conclusion from 'established' to 'plausible, pending paired inference.' CONDITIONAL therefore remains the right verdict, and I do not propose changing the reader's decision.","tokens_in":19691,"tokens_out":6006,"duration_ms":59511,"concrete_test":"Re-run all five configurations while logging per-report EM, then apply a paired McNemar exact test comparing CRAFT-Full against PIVOT, GUIDE, and CRAFT w/o V for each of the four backbones on the 3,166 MedTempo-T reports, along with a bootstrap 95% confidence interval for each EM difference. Report discordant-pair counts and p-values with a multiple-comparison correction (e.g., Bonferroni across the 12 comparisons). If the CRAFT-Full vs. PIVOT/GUIDE differences are not significant for MedGemma and possibly Llama or Claude, or if CRAFT-Full is significantly worse than CRAFT w/o V on Claude, the conclusion should be revised from 'consistently improves' to 'model-dependent and not statistically established.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the statistical support for the headline claim (abstract, Section 6.1) that CRAFT-Full 'consistently improves temporal ordering accuracy' and is 'the strongest configuration.' The reported EM margins over PIVOT are +1.0 for GPT-4.1, +0.8 for Llama-3.3-70B, +0.7 for Claude Sonnet 4.5, and +0.1 for MedGemma-27B on 3,166 reports; margins over GUIDE are +1.6, +2.0, +1.5, and +0.7. These correspond to roughly 3 to 63 report-level flips. No confidence interval, bootstrap, McNemar test, permutation test, or any paired-significance procedure appears in Section 6 or the appendix. Because decoding is deterministic, run-to-run variance is zero, but the finite sample of reports still leaves the differences potentially unremarkable; for the smallest gain, only about 3 changed reports separate the methods. The paper's own data also weaken 'consistently': in Table 6.1, Claude Sonnet 4.5 CRAFT-Full (37.14) is below CRAFT w/o V (37.83), and Section 6.4 explains this as threshold miscalibration from theta=3 selected only on GPT-4.1 (Appendix A.2). Thus, even granting MedTempo's gold labels, the central empirical claim currently reduces to a small, untested advantage over two internal baselines.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces CRAFT, a generator-verifier LLM framework for reconstructing stage-wise symptom timelines from single clinical narratives with weak temporal anchoring, and MedTempo, a benchmark of 5,347 VAERS narratives of which 3,166 have expert-validated temporal stage annotations. CRAFT iterates between a generator that proposes or edits a bucket-based timeline and a verifier that scores it with a 0-5 rubric, returning targeted feedback until an acceptance threshold or iteration cap is reached. The authors evaluate five configurations (CRAFT-Full, PIVOT, GUIDE, CRAFT-G, and CRAFT without verifier) on four LLM backbones using exact match, LCCS, and Kendall's tau-b, and report that CRAFT-Full attains the highest EM in every model block, with ablations attributing gains to the verifier and to full regeneration. They also report vaccine-stratified results and two case studies showing how verifier feedback corrects grouping errors and how a miscalibrated verifier can cause oscillation.","tokens_in":19959,"tokens_out":4879,"duration_ms":44429,"significance":"If the empirical claims are supported, the paper makes three useful contributions: it defines a clean, under-addressed task (structured trajectory reconstruction from anchor-sparse single reports), it releases a relatively large expert-validated benchmark with clear exclusion rules and 93% inter-annotator agreement, and it provides a systematic comparison of generator and verifier design choices across four model tiers. The ablation design is thoughtful, the three evaluation metrics are appropriate for bucket-sequence prediction, and the dataset construction is generally careful, including adjudication and reporting of acceptance, correction, and exclusion rates. The main reservations concern the statistical support for the headline claim, the uniform hyperparameter choice applied to all backbones, and the possibility that the gold standard inherits biases from the LLM that drafted the initial timelines; these issues are addressable but currently prevent the central conclusion from being fully established.","major_comments":[{"comment":"The headline claim that CRAFT-Full 'consistently improves temporal ordering accuracy' and is 'the strongest configuration' rests on EM margins of +1.0, +0.8, +0.7, and +0.1 over PIVOT and +1.6, +2.0, +1.5, and +0.7 over GUIDE, with the smallest margin corresponding to roughly three changed reports among 3,166. No confidence intervals or paired significance tests (McNemar, bootstrap, permutation) are reported anywhere in Section 6 or the appendix, so these differences are currently indistinguishable from sampling noise. Please add paired significance testing or, failing that, restrict the conclusion to a directional observation for the particular models and configurations tested.","section":"Section 6.1, Table 5.1"},{"comment":"The loop hyperparameters T_max=4 and theta=3 were selected on a 100-instance dev set using GPT-4.1 alone and applied uniformly to all backbones. The paper's own results show this choice is miscalibrated for Claude Sonnet 4.5, where CRAFT-Full (EM 37.14) falls below CRAFT without verifier (EM 37.83), and Appendix A.5 documents an oscillation caused by the same threshold behavior under GUIDE. Because the central claim is that CRAFT-Full is strongest across model tiers, the paper must either calibrate per model with a non-test selection protocol or report a sensitivity analysis over theta and T_max showing that the conclusion is robust; otherwise the 'strongest configuration' claim is only supported for GPT-4.1.","section":"Appendix A.2 with Section 6.4 and Table 6.1"},{"comment":"Gold timelines were initialized by GPT-4o mini and then reviewed by two annotators, with 80% of drafts accepted unchanged; Figure 4.1(a) instead labels the pre-annotation model as GPT-4.1. If the human reviewers anchored on the model draft, the benchmark inherits GPT-4o mini's temporal ordering biases, which would favor LLM-based systems of the same family and make the benchmark less reliable as an independent gold standard. The authors should report a human-only reannotation of a random subset (with agreement against the current gold) or otherwise quantify anchoring, and they should correct the model-name inconsistency.","section":"Section 3.3 and Figure 4.1(a)"}],"minor_comments":[{"comment":"The statement that '80% of LLM annotations were accepted without change' is useful, but the paper does not report the distribution of the 9% corrected cases by error type; a small breakdown (e.g., grouping versus ordering corrections) would help assess the anchoring risk.","section":"Section 3.3, Table 3.1"},{"comment":"The row label 'CRAFT G' should be 'CRAFT-G' to match Section 5.2 and the rest of the paper.","section":"Table 5.1"},{"comment":"The paper says deterministic decoding is used; please state the temperature and sampling settings explicitly for the proprietary APIs and note whether API-side nondeterminism was checked.","section":"Section 5.3"},{"comment":"The heatmaps would be easier to read with a defined color scale and explicit zero handling; the current captions describe counts but the color mapping is not explained.","section":"Figures 6.1 and A.1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and code release are likely to be useful to the NLP-for-clinical-text community, and the framework is clearly described. The main risk is that the central empirical claim is built on very small EM differences without any paired statistical testing, and the uniform hyperparameter choice demonstrably fails for one of the four tested models. I would ask the authors to add paired tests and a sensitivity analysis, and to provide evidence on the independence of the human annotation from the LLM draft, before accepting the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good benchmark, unproven headline.\n\nThe genuinely new thing here is MedTempo: 5,347 VAERS narratives, 3,166 with expert-validated stage-wise temporal annotations, for a task—bucket sequences over single anchor-sparse reports—that isn't covered by the existing pairwise-relation or multi-visit benchmarks. The curation looks careful: clear exclusion rules, stratified sampling, 93% inter-annotator agreement, and 11% excluded for ambiguity. That part of the paper deserves to be taken seriously.\n\nThe CRAFT framework itself is a reasonable structured-prediction spin on Self-Refine, and the authors do the right thing by reporting several ablations and iteration-level traces. The case study showing the anchor-based verifier sending Claude into a merge-split oscillation is exactly the kind of negative result that should be in a paper.\n\nThe soft spots are in the empirical claims. The abstract says CRAFT 'consistently improves temporal ordering accuracy,' but the EM margins over PIVOT are 0.1 to 1.0 points, and over GUIDE 0.7 to 2.0, with no confidence intervals or paired tests. On 3,166 reports that's between 3 and 63 flips. For the smallest difference, three changed reports separate the methods. The authors' own ablation shows Claude CRAFT-Full (37.14) below CRAFT w/o V (37.83), and they attribute this to threshold miscalibration from picking theta=3 on GPT-4.1 alone. So the 'consistent' claim doesn't hold up as stated.\n\nThere are also two secondary issues. Hyperparameters were chosen on a 100-instance dev set with one model; that's disclosed in Appendix A.2, but it means the headline configuration is partially tuned to that model. And the gold timelines were drafted by GPT-4o mini and then human-validated; the high IAA doesn't fully rule out anchoring on the draft. Minor internal inconsistency: Section 3.3 says GPT-4o mini, Figure 4.1 says GPT-4.1.\n\nNone of this kills the paper. The benchmark is real, and the negative results are informative. But the central method claim needs statistical support, ideally paired significance tests, and an external baseline or two (mention order, simple classifier) to calibrate what these gains mean. I'd send it to review, and push the authors to add that evidence. The benchmark alone is worth citing once it's released.","headline":"MedTempo is a solid new benchmark, but the paper's 'consistently improves' claim rests on small, untested EM gaps and is undercut by the authors' own Claude ablation.","tokens_in":20548,"tokens_out":2748,"would_cite":true,"duration_ms":25420,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A generator-verifier loop with a five-point rubric improves exact-match symptom-timeline reconstruction on 3,166 vaccine adverse-event reports.","keywords":["temporal reasoning","clinical narratives","LLM iterative refinement","generator-verifier framework","symptom timelines","MedTempo benchmark","vaccine adverse events","temporal ordering"],"falsifier":"Take a random sample of MedTempo-T reports and have annotators build gold timelines directly from the narratives without ever seeing the GPT-4o mini draft; compare the blind gold labels to the released labels on bucket composition and pairwise order. If agreement is far below the reported 93% inter-annotator rate, or if CRAFT's gains shrink when evaluated against blind labels, the central claim is not independent of the drafting model. A sharper test is to restrict evaluation to the 9% of reports whose draft labels were corrected: if CRAFT does not improve there, the loop mostly echoes the drafting model's biases rather than reasoning about time.","tokens_in":19421,"feed_emoji":"🩺","tokens_out":12259,"duration_ms":109374,"temperature":0.7,"pith_summary":"Clinical narratives about vaccine adverse events rarely state dates, so reconstructing the order in which symptoms appeared is hard. The paper proposes CRAFT: an iterative loop in which a generator asks an LLM to produce a stage-by-stage symptom timeline as an ordered list of time buckets, and a verifier scores the candidate against five explicit criteria, returning targeted feedback when the score is below a threshold. On MedTempo, a new benchmark built from 5,347 VAERS reports with expert-reviewed gold timelines, the paper claims this loop raises exact-match temporal ordering accuracy across four LLM backbones, and the full configuration posts the highest exact-match score in every model block against the two baselines. The largest gains appear in the first few refinement rounds for a capable model, while weaker models accept early and gain little from further iterations. If the claim holds, automatically built symptom timelines become a realistic tool for vaccine-safety monitoring and disease-progression tracking from ordinary text.","feed_headline":"Rubric-scored LLM loop sharpens symptom timelines on vaccine reports","feed_subtitle":"CRAFT improves exact-match temporal ordering on 3,166 VAERS narratives across four LLMs and a new benchmark.","key_machinery":"The load-bearing mechanism is the time-bucket trajectory representation together with the generator-verifier loop. Each report $r$ is mapped to a prediction $B(r) = (B_1,\\dots,B_K)$: a JSON list of non-empty buckets in which every finding from the provided symptom list appears exactly once and buckets run from earliest to latest. The generator $g$ maps the symptom list, free-text narrative, and any prior feedback to a candidate timeline; the verifier $v$ first runs FormatTool, a deterministic normalizer that repairs raw output into the bucket schema without touching temporal content, then scores the candidate on a five-point additive rubric and emits either ACCEPT or REVISE with feedback. The full-regeneration variant rewrites the timeline from the complete task prompt each round, which the ablation shows is safer than an edit-conditioned variant under sustained feedback. The loop converts temporal ordering from a single-shot generation problem into a constrained iterative structured-prediction problem with a fixed acceptance threshold $\\theta = 3$ and budget $T_{\\max} = 4$.","core_discovery":"The central discovery is that a fully automated generator-verifier refinement loop beats two alternative baselines at reconstructing ordered symptom timelines from anchor-sparse narratives, and it beats a single unverified pass for three of the four models tested. CRAFT-Full pairs a generator that regenerates the entire timeline from full task context at each iteration with a verifier that awards one point for each of five criteria: valid JSON in earliest-to-latest order, unmentioned symptoms placed as \"none\" in the final group, each symptom appearing exactly once, symptoms grouped only when they occur around the same time, and group order following the temporal cues in the text. A score of at least 3 accepts the candidate; otherwise the verifier returns specific corrective feedback and the loop repeats, up to four iterations. Across Claude Sonnet 4.5, GPT-4.1, Llama-3.3-70B, and MedGemma-27B, CRAFT-Full achieves the highest exact-match score in every model block relative to the baselines, and the ablations attribute most of the gain to the verifier, with full-context regeneration outperforming edit-conditioned refinement. The paper also reports that for Claude, whose first pass is already near gold, the fixed threshold can reject a correct output and make the loop slightly underperform a single unverified pass.","pith_inferences":["A per-backbone acceptance threshold is the obvious untested lever: the paper's own case study shows a fixed $\\theta = 3$ can reject or under-reward a near-correct Claude output and drive oscillation, so adapting the threshold to the model would likely recover the lost points.","Because the gold labels were drafted by GPT-4o mini and then human-reviewed, a blind re-annotation of a random sample, with annotators seeing only the narrative, would test whether MedTempo's target is independent of the drafting model family; the paper does not report such a check.","The same bucket-list representation and rubric loop could port to other anchor-sparse timeline tasks, such as inferring symptom progression from electronic-health-record narratives, with the verifier rubric re-specified for each new setting.","If the verifier's rubric is the main driver of gains, investing in better verifier criteria may matter more than scaling the generator."],"forward_implications":["Exact-match timeline reconstruction becomes achievable from a single report plus a symptom list, without requiring timestamps or multi-visit records.","For capable models, a longer refinement budget pays off: GPT-4.1's exact match grows from 26.90 at iteration 1 to 35.61 at iteration 4, while weaker models accept within one or two iterations and gain little.","The ablations show most of the gain comes from the verifier and from regenerating with full context, so improvements in feedback quality are likely more valuable than switching to local-edit refinement.","The stable model ranking across configurations and vaccine types makes MedTempo a usable standardized benchmark for comparing future temporal-reasoning methods.","The reserved non-temporal reports set up a separable next task, deciding whether a report contains temporal progression, that would let a future pipeline apply CRAFT selectively."],"supporting_citations":[{"why":"Supplies the iterative refinement paradigm that CRAFT adapts into a generator-verifier loop with a structured trajectory representation.","marker":"[17]"},{"why":"Model that drafted the initial stage-ordered timelines later reviewed and adjudicated by human annotators, so it sets the starting point of the gold labels.","marker":"[21]"},{"why":"VAERS source database from which MedTempo reports are sampled, filtered, and stratified.","marker":"[7]"},{"why":"Grounds the anchor-based verifier baseline that CRAFT-Full is compared against.","marker":"[33]"},{"why":"Provides the group-aware LCCS metric used to reward contiguous spans of correctly ordered stages.","marker":"[12]"},{"why":"Provides Kendall's tau-b with tie handling, the rank-correlation metric for predicted versus gold stage order.","marker":"[15]"},{"why":"Supplies the exact-match definition used as the headline evaluation metric for timeline reconstruction.","marker":"[24]"}],"fun_headline_variants":["Generator-verifier loop beats single-pass for temporal symptom mapping","LLM loop refines symptom timelines on vaccine adverse event reports","Iterative LLM refinement improves temporal ordering in clinical text","CRAFT: verifier-guided LLM loop boosts temporal reasoning in narratives"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premises are that MedTempo's gold timelines genuinely reflect the order symptoms appeared and that the provided symptom lists are complete and correctly coded; the riskier one is the gold timelines, because they were drafted by GPT-4o mini and then human-reviewed, so if the reviewers anchored on the draft the benchmark target inherits the drafting model's temporal biases and the measured gains are partly circular.","fun_headline_variants_meta":{"raw":{"variants":["Generator-verifier loop beats single-pass for temporal symptom mapping","LLM loop refines symptom timelines on vaccine adverse event reports","Iterative LLM refinement improves temporal ordering in clinical text","CRAFT: verifier-guided LLM loop boosts temporal reasoning in narratives"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000294,"raw_usage":{"total_tokens":1725,"prompt_tokens":977,"completion_tokens":748,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":676}},"tokens_in":593,"tokens_out":748,"duration_ms":7091,"temperature":1.0,"reasoning_tokens":676,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:25:53.025209+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of MedTempo-T reports and have annotators build gold timelines directly from the narratives without ever seeing the GPT-4o mini draft; compare the blind gold labels to the released labels on bucket composition and pairwise order. If agreement is far below the reported 93% inter-annotator rate, or if CRAFT's gains shrink when evaluated against blind labels, the central claim is not independent of the drafting model. A sharper test is to restrict evaluation to the 9% of reports whose draft labels were corrected: if CRAFT does not improve there, the loop mostly echoes the drafting model's biases rather than reasoning about time.","supporting_citations":[{"cited_title":"Journal of Biomedical Informatics , volume =","cited_arxiv_id":null,"evidence_quote":"Model that drafted the initial stage-ordered timelines later reviewed and adjudicated by human annotators, so it sets the starting point of the gold labels."},{"cited_title":"Evaluating Temporal Relations in Clinical Text: 2012","cited_arxiv_id":null,"evidence_quote":"VAERS source database from which MedTempo reports are sampled, filtered, and stratified."},{"cited_title":"2018 , month = nov, url =","cited_arxiv_id":null,"evidence_quote":"Grounds the anchor-based verifier baseline that CRAFT-Full is compared against."},{"cited_title":"Biometrika , volume=","cited_arxiv_id":null,"evidence_quote":"Provides Kendall's tau-b with tie handling, the rank-correlation metric for predicted versus gold stage order."},{"cited_title":"AMIA Annual Symposium Proceedings , year =","cited_arxiv_id":null,"evidence_quote":"Supplies the exact-match definition used as the headline evaluation metric for timeline reconstruction."}],"review_version":1}