{"id":"63eba5ce-a7b3-4212-a5fc-c3758dae553d","arxiv_id":"2607.16212","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Typed quantity verification exposes a canonical-equivalence blind spot in neural fact-checkers; Symbolic Augmentation fixes it (36.5%→98.2%) and transfers to SciFact-Open (+0.037 binary macro-F1).","lead":"Uses a five-way error taxonomy and a 1,500-item AI-labeled benchmark to show that fine-tuned ModernBERT mistakes physically equivalent rewrites (95 °C vs 368.15 K) for errors 63.5% of the time. Training on deterministic symbolic rewrites recovers that robustness to ~98%, and the paper argues training-time augmentation is the best place to inject symbolic knowledge.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-validation is stratified by gold label but not by the 741 source evidence sentences, so most test items likely share an evidence sentence with training items; the 36.5→98.2 probe gain and 0.899 macro-F1 may be inflated by evidence-level leakage.","rationale":"The paper is a good-faith attempt at a real problem. The typed taxonomy, rule-based verifier, per-family probes, negative results (features, silver labels), and SciFact-Open transfer are all useful and reduce the chance that the augmentation effect is entirely spurious. However, the strongest claim in the paper is quantitative: a 62-point jump on the F1 probe with no loss of in-distribution accuracy. That figure is only meaningful if test items are truly unseen at the item level. The reported CV does not guarantee this. Because each evidence sentence produced ~2 summaries on average, label-stratified folds will place siblings from the same evidence on both sides of the train/test boundary. The encoder can memorize evidence-specific patterns, turning the task from 'verify an unseen claim' into 'retrieve the training rewrites of this evidence.' This is not an internal inconsistency in the method, but it is a correctness risk in the reported evaluation. The data-split issue is the same one the reader identified, so I agree with that assessment. The probe/augmentation code-sharing contradiction (abstract says disjoint, Appendix C says shared) strengthens the concern because it removes one would-be defense. I do not think the paper should be rejected: the idea is sound, the negative results are informative, and the external SciFact result provides some independent signal. But the headline benchmark numbers should be treated as conditional until the grouped-split check is run and the data/code are available.","tokens_in":20296,"tokens_out":9766,"duration_ms":113489,"concrete_test":"Group the 5-fold cross-validation by the 741 unique evidence-sentence IDs (all items from the same evidence stay in the same fold), while preserving label balance as far as possible, and remove any exact (evidence, summary) duplicate across folds. Retrain the canonical and F1-augmented ModernBERT models under this grouped split; report 5-fold pooled macro-F1 and F1-probe accuracy. Also compute the proportion of test items whose evidence appears in the training folds and report probe accuracy conditioned on overlap vs. no overlap. If the probe gain remains ~61.7 points and macro-F1 is unchanged, the leakage concern is resolved; if the gain or macro-F1 drops materially, the headline numbers need revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (F1 probe 36.5→98.2, macro-F1 0.899, 'closes essentially all of the blind spot') is computed on a 5-fold split that is stratified by gold label only (Appendix G). The benchmark's 1500 items are rewrites of only 741 unique evidence sentences (Appendix D), so the evidence sentence — not the item — is the natural unit of independence. Under a label-stratified 80/20 split, a large fraction of test items will have one or more sibling items from the same evidence in the training folds. A ModernBERT encoder can then exploit evidence-level lexical memory: for a test summary whose evidence appeared in training with a different rewrite, the label can be inferred by comparing the summary against the training summaries of that evidence, rather than by generalizing to unseen evidence. This inflates macro-F1 and, more importantly, the F1 probe: probe items are F1-rewrites of test CORRECT items, while the augmented training set contains F1-rewrites of training CORRECT items; when those CORRECT items share an evidence sentence, the augmented model has effectively seen near-identical canonical-equivalent surfaces at train time. The manuscript never states a group-split guarantee, and the Limitations section does not list this leakage risk. The abstract's claim of a 'disjoint perturbation generator' is also not established: Appendix C states that 'Rule definitions and probe generation share code' with the augmentation, so the probe is not independent of the training transformation. The 36.5% baseline may itself be inflated by leakage, and the 98.2% post-augmentation number may further benefit from evidence-level overlap; the 62-point difference is therefore not a trustworthy estimate of the method's effect until the split is grouped by evidence and duplicate (evidence, summary) pairs are removed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a five-class typed-quantity verification task, a 1500-item benchmark built from PMC and arXiv evidence sentences and labeled by two LLM annotators with adjudication, and a fine-tuned ModernBERT encoder. It reports that the encoder reaches macro-F1 = 0.899 on the benchmark but collapses to 36.5% accuracy on canonical-equivalence probe rewrites (e.g., 95°C versus 368.15 K). The central proposal is Symbolic Augmentation: deterministic, label-preserving rewrite rules derived from the modules of a symbolic verifier (UGV) are used to generate augmented training data for the encoder. The paper claims this lifts F1-probe accuracy to 98.2%, slightly improves macro-F1 to 0.902, transfers to SciFact-Open (binary macro-F1 0.791 to 0.828), and matches a closed-frontier LLM at no inference cost. It further reports negative results for feature-level integration, silver-label teaching, and inference-time ensembling, concluding that training-time augmentation is the only effective integration point.","tokens_in":20699,"tokens_out":5021,"duration_ms":59680,"significance":"If the empirical claims hold, the paper makes a useful contribution: it identifies a concrete and plausible failure mode in neural fact-checkers, proposes a clean mechanism (inverting symbolic verifier rules into label-preserving augmentation), and provides an external transfer check. The deterministic nature of the rewrite rules and the explicit per-family probes are strengths, as is the attempt to compare five integration points. However, the headline numbers rest on a cross-validation protocol that does not separate items by source evidence sentence, and the benchmark/probe use LLMs in ways that create circularity risks. These issues are load-bearing for the central claims and require correction before the quantitative conclusions can be accepted.","major_comments":[{"comment":"The 1500 benchmark items are derived from only 741 unique evidence sentences (App. D), yet the five-fold cross-validation is stratified only by gold label (App. G). Nothing prevents summaries from the same evidence sentence from appearing in both training and test folds. A test summary whose evidence appeared in training can be classified by evidence-level lexical memory rather than by generalization to unseen evidence. This inflates the headline macro-F1 (Table 1, 0.899), the F1 probe gain (Table 2, 36.5% to 98.2%), and the SciFact-Open comparison is the only result not affected. The F1 probe is especially sensitive: probe items are rewrites of test CORRECT items, while augmentation adds rewrites of training CORRECT items; when those items share an evidence sentence, the augmented model has effectively seen near-identical canonical-equivalent surfaces at train time. Please re-run the ev","section":"§5.1, App. D, App. G"},{"comment":"The annotation pipeline is not as described. The rewriter is claude-haiku-4-5, and annotator A is also claude-haiku-4-5; only annotator B is from a different model family. The manuscript claims 'two independent LLM annotators' and 'drawn from different model families,' but annotator A is the same checkpoint family as the generator, so the inter-annotator agreement (Krippendorff's alpha = 0.882) partially reflects a model labeling its own outputs. The human spot-check is only 50 items (94% agreement), which is too small to rule out systematic bias. Please either replace annotator A with a model family not used for rewriting or provide a substantially larger human validation set, and revise the circularity discussion accordingly.","section":"App. D"},{"comment":"The abstract claims the probe set uses a 'disjoint perturbation generator' to address the self-labeling circularity concern, but Appendix C states that 'Rule definitions and probe generation share code' with the augmentation families. The probes are therefore not disjoint from the augmentation distribution; they are generated by the same transformations the model is trained on. The 98.2% probe result can be read as interpolation to the training augmentation distribution rather than as evidence of robustness to an independent perturbation family. Please either remove the 'disjoint' claim or design a held-out probe that uses surface transformations not present in the augmentation families.","section":"Abstract vs. App. C"}],"minor_comments":[{"comment":"The text says the all-four augmentation result is 'statistically indistinguishable' from baseline, but no variance estimate, confidence interval, or significance test is reported. With only one 5-fold pooled number, this claim is unsupported.","section":"§5.3"},{"comment":"The fine-tuned encoder is evaluated with 5-fold pooled out-of-fold predictions, while the closed-frontier LLMs are evaluated once on the full 1500-item set. The comparison in Table 1 is therefore not under identical protocols; this should be stated explicitly when comparing macro-F1 values.","section":"Table 1"},{"comment":"Table 3 reports a single-fold probe evaluation, while Table 2 reports 5-fold pooled results. The relationship between the two is not explained clearly; state why the single fold is representative, or report both with confidence intervals.","section":"Table 3"},{"comment":"The logit ensemble uses a single fixed alpha = 0.25. Since the conclusion is that ensembling fails, at least a small alpha sweep or a statement that the result is robust to alpha is needed to rule out a tuning artifact.","section":"§5.4 / App. H"}],"recommendation":"major_revision","confidential_remarks":"The central idea is interesting and the paper is clearly written, but the main quantitative claims are currently not supported by the reported evaluation protocol. The evidence-sentence leakage in the cross-validation is a correctness risk that cannot be waived by reinterpreting the text; the authors should re-run the evaluation with grouped folds. The self-labeling and probe-generation circularity issues also need to be addressed directly. If the grouped-CV results preserve the qualitative pattern, the paper could be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper names a real blind spot—neural fact-checkers treat canonical-equivalent quantity rewrites (95 °C vs 368.15 K) as errors—and the proposed fix, generating label-preserving training data by inverting symbolic verifier modules, is a good idea. But don't quote the numbers yet. The 5-fold CV is stratified by gold label, not by the 741 unique evidence sentences the 1500 items derive from. With roughly two rewrites per evidence sentence, most test items have a sibling in training, so the encoder can memorize evidence-level wording. That inflates macro-F1 0.899 and the F1 probe 36.5→98.2; the probe is also built by the same code as the augmentation, contradicting the abstract's 'disjoint perturbation generator' claim. These are fixable: group by evidence sentence, drop duplicate (evidence, summary) pairs, release code/data, report confidence intervals. Until then, treat the quantitative claims as provisional.\n\nWhat's actually new: the five-class typed-quantity taxonomy organized by symbolic axis (dimension/scale/relation/unsupported) rather than topic, and the augmentation mechanism—running deterministic verifier rules backwards for label-preserving rewrites rather than LLM paraphrases. The paper is honest about negatives: symbolic features as auxiliary input don't help, silver-label teaching hurts and scales negatively, logit ensembling is noise. Those are useful empirical findings. The SciFact-Open transfer (+0.037 macro-F1) is promising but small (n=206) and also computed from the same augmented model.\n\nThe LLM-generated gold labels with Krippendorff's alpha 0.882 and a 50-item human spot-check are a moderate concern, not fatal. The absence of code and data is more annoying than suspicious, but for a benchmark paper it's a real impediment. The split issue, though, is load-bearing: the central claim is that training-time augmentation closes the blind spot, and the current evaluation design cannot cleanly show that.\n\nWho this is for: people working on faithful summarization, fact verification, and symbolic-neural integration. It deserves serious peer review, but I'd send it out with the split and probe-independence issues as required revisions. The core idea is strong enough that the paper should get a chance to fix the evidence.","headline":"Typed-quantity verification and training-time symbolic augmentation are genuinely worth knowing about, but the headline numbers sit on a cross-validation split that ignores the 741 source evidence sentences; treat the gains as provisional until the split is grouped.","tokens_in":21189,"tokens_out":2504,"would_cite":false,"duration_ms":29274,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two spellings of the same physical quantity—95°C or 368.15 K—are routinely misread as conflicting claims by fine-tuned neural fact-checkers, and training on symbolically generated label-preserving rewrites closes this blind spot, lifting pr","keywords":["typed-quantity verification","unit hallucination","canonical equivalence","data augmentation","symbolic verification","fact-checking","scientific claim verification","five-class taxonomy"],"falsifier":"Split the data by the 741 evidence sentences so no source's rewrites straddle the train/test boundary, retrain, and re-run the probes; if the F1-probe gain of 36.5→98.2 collapses, the blind-spot fix was partly a memorization artifact of the split scheme.","tokens_in":20176,"feed_emoji":"🧪","tokens_out":5324,"duration_ms":48982,"temperature":0.7,"pith_summary":"The paper argues that neural fact-checkers fail not on paraphrases but on a narrow structural case: two spellings of the same physical quantity, such as 95°C and 368.15 K, are treated as conflicting claims. It proposes generating training data by running a symbolic unit verifier in reverse, so that every rewrite is guaranteed to keep its label. Trained on these rewrite families, the encoder's accuracy on canonical-equivalence probes rises from 36.5% to 98.2% while its in-distribution macro-F1 improves slightly. The authors further compare five places where symbolic information can enter the pipeline and find that only training-time augmentation helps; auxiliary features, logit ensembling, and silver-label teaching do not. The broader claim is that the right way to combine symbolic and learned verification is at the data layer, not the model layer.","feed_headline":"Training rewrites lift fact-checker accuracy from 36% to 98%","feed_subtitle":"Equivalent quantities like 95°C and 368.15 K were misread as errors; one family of symbolically generated pairs fixes the blind spot.","key_machinery":"Symbolic Augmentation: the modules of a deterministic symbolic verifier (quantity extraction, unit canonicalization, comparator and trigger detection) are run in reverse to define rewrite families (canonical-equivalence, comparator synonyms, unit-name paraphrase, scientific notation) that, applied to gold CORRECT items, generate new training pairs whose labels are correct by construction. The canonical-equivalence family, derived from the unit registry, is the one that closes the blind spot.","core_discovery":"A fine-tuned encoder that scores macro-F1 = 0.899 on five-class typed-quantity verification collapses to 36.5% accuracy when a gold-correct summary restates its quantity in canonical-equivalent form (95°C as 368.15 K, 0.5 mol/L as 500 mmol/L). The paper shows the failure is structural: 99 of 115 probe errors land in SCALEERROR, meaning the model reads a physically identical surface as an altered magnitude. The proposed fix, Symbolic Augmentation, inverts the modules of a symbolic verifier into deterministic, label-preserving rewrite rules; training on the resulting data raises the probe to 98.2%, slightly improves macro-F1, matches a closed-frontier LLM at no inference cost, and transfers to","pith_inferences":["The reverse-verifier recipe is general: any symbolic verifier whose stages expose deterministic rewrites (unit conversion, arithmetic, temporal normalization) could generate label-preserving training data for the same blind spot in other modalities.","If the evidence-sentence grouping were honored in cross-validation, the headline gains might shrink; the paper's tables stratify by label only, so a regrouping could change the headline numbers.","The probe suite is a reusable diagnostic: before deployment, run canonical-equivalence probes on any fact-checker to find whether numeric surface variation is misread as contradiction."],"forward_implications":["Fine-tuned encoders can reach closed-frontier-level robustness on units at negligible inference cost, if trained on label-preserving symbolic rewrites.","The four family-wide probes give a transferable diagnostic: any verifier can be probed for canonicalization blind spots before deployment.","The five integration-point comparison suggests that rule-based and learned components should be combined at training time, not inference time.","The gain transfers out of distribution: binary macro-F1 on an external scientific fact-checking benchmark rises from 0.791 to 0.828 with zero target-domain training data.","Scientific-notation rewrites also improve via cross-family transfer: F1 augmentation alone lifts the notation probe from 68% to 96%."],"fun_headline_variants":["Blind to unit equivalences: fact-checker fixed by reverse symbolics","Symbolic augmentation lifts fact-checker from 36% to 98% on unit rewrites","Fact-checker sees 95°C vs 368.15 K as different? New fix erases that","Reverse symbolic rewrites close canonical-equivalence gap in fact-checkers","Teaching fact-checkers that 95°C is 368.15 K: a training-time fix"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The 1500 benchmark items derive from 741 unique evidence sentences, but the cross-validation is stratified by gold label, not by evidence sentence, so rewrites of the same evidence can appear in both train and test folds and inflate the reported gains.","fun_headline_variants_meta":{"raw":{"variants":["Blind to unit equivalences: fact-checker fixed by reverse symbolics","Symbolic augmentation lifts fact-checker from 36% to 98% on unit rewrites","Fact-checker sees 95°C vs 368.15 K as different? New fix erases that","Reverse symbolic rewrites close canonical-equivalence gap in fact-checkers","Teaching fact-checkers that 95°C is 368.15 K: a training-time fix"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1249,"prompt_tokens":856,"completion_tokens":393,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":278}},"tokens_in":600,"tokens_out":393,"duration_ms":4746,"temperature":1.0,"reasoning_tokens":278,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T13:34:29.428752+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Split the data by the 741 evidence sentences so no source's rewrites straddle the train/test boundary, retrain, and re-run the probes; if the F1-probe gain of 36.5→98.2 collapses, the blind-spot fix was partly a memorization artifact of the split scheme.","supporting_citations":[],"review_version":1}