Pith.

Computation and Language · arXiv:2610.08675 v2 · submitted 6 Oct 2026

Same-Number Citation Swaps: Stress-Testing Jev as a Financial Evidence Judge

Chuhong Xu (Sofia University), Bo Su (Indiana University), Ziyao Chen (University of California, San Diego), Ruiyang Xu (Northeastern University), Shimeng Dai (Michigan State University), Xinyu Qiu (Northeastern University)

The paper says

“Financial reports repeat values across periods, metrics and accounting lines, allowing an LLM-generated calculation to be numerically correct while citing the wrong financial role.”

more of the abstract

We evaluate what probabilistic evidence verification adds beyond number matching using Jev as a source-support verifier for GPT-4.1-mini calculation traces. A signed-number-at-pointer baseline explains most recovery over exact quotation checks. To isolate the remaining role-recognition problem, we hold operands and arithmetic fixed, move citations between same-number cells, and retain controls that express equivalent facts. These contrasts reveal both wrong-role citations that pass and valid alternative citations that are withheld. Explicit column labels improve selected wrong-role decisions while also lowering support for some equivalent evidence. A constructed follow-up on 36 new source pages, labeled by a non-author reviewer, extends this evaluation and exposes the same tradeoff between detecting role errors and retaining valid citations. The contribution is a controlled evaluation that identifies what a probabilistic financial verifier distinguishes when numerical matching is held fixed. For LLM-based financial assistants, it makes numerical correctness, cited-role support and acceptance outcomes separately assessable.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 7 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. The same-number swap axis is a genuinely useful evaluation idea and the coarse finding survives, but the explicit-column benefit leg collapses under the contested labels the paper itself reports, leaving magnitude unpinned. This line is wrong

4 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at III-D, Fig. 3; Sec. VI

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.84):

“The labels reflect one reviewer per case; inter-human agreement remains unmeasured.”

Pith says: Objection 1 of 4 · major

The fresh-source headline that explicit columns improve role separation is not robust to the label uncertainty the paper itself documents. The paired-separation readout at 0.8 is supplied by one non-author reviewer whose inter-human agreement is unmeasured (Sec. VI: 'inter-human agreement remains unmeasured'), and the sensitivity analyses in III-D show that excluding case 444 under its contextual-equivalence reading makes paired separation 8/14 in both renderings, while excluding all eight author/reviewer disagreements also gives 8/14 in both renderings.

more of objection 1

Because the abstract presents explicit column labels as improving selected wrong-role decisions, this claim requires either independent confirmatory labels with agreement statistics, a pre-specified adjudication rule, or a reformulation reporting outcomes under each label set rather than a single net gain.

The paper says, at III-C, Figs. 2–3; Sec. VI

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.72):

“The case- level results expose sensitivity to role interpretation and near- threshold means; they describe this constructed bank rather than population rates.”

Pith says: Objection 2 of 4 · major

Several acceptance-level conclusions sit within repeat noise of the 0.8 cutoff. In the rendering study both equivalent-fact B means are 0.785, with one of four individual calls crossing 0.8; in the fresh bank, 16 and 13 withheld supported arms still have means at least 0.5, and the largest identical-input repeat range is 0.18.

more of objection 2

A shift of a few hundredths in the cutoff would therefore change most of the headline pass/withhold counts. The manuscript acknowledges that these means are close to the cutoff, but it still presents fixed-cutoff counts as the primary readout; please add cutoff-sweep or repeat-interval versions of the key counts, or explicitly demote acceptance outcomes to secondary status relative to the continuous support probabilities.

The paper says, at III-A, Table II

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

The claimed increment of probabilistic verification over the signed-number-at-pointer baseline rests on a single joint pass. Jev accepts 146 records with 142 joint passes versus 154 accepted and 141 joint passes for the relaxed rule, and the paired comparison gives a net gain of one joint pass: two recoveries on question 218 for one withheld valid trace on question 583.

more of objection 3

Removing question 218 leaves Jev at 140 versus 141 joint passes. Since the relaxed rule is post hoc and the one-pass difference receives no statistical inference, the abstract's statement that the baseline 'explains most recovery' is supported, but any stronger reading that Jev adds meaningful incremental value on the natural bank is not. Please present this leg explicitly as a single-case demonstration or add a question-level bootstrap test for the paired difference.

The paper says, at III-A, II-B

The review did not quote this passage.

Pith says: Objection 4 of 4 · major

The natural-record discrimination numbers (Brier 0.03378, AUROC 0.98587 on 190 binary labels) are quoted as evidence that the verifier is strong on natural records, but the Noul predicate was written after inspecting those same retained labels and tested in only four development checks before the saved runs.

more of objection 4

These aggregate figures are therefore in-sample, retrospective diagnostics rather than out-of-sample accuracy estimates. Please either evaluate a frozen prompt on a held-out label set or explicitly label these numbers as retrospective diagnostics, and soften the corresponding 'strong aggregate discrimination' statement in III-B.

5 minor objections

The paper says, at II-C

Pith says: minor objection 1 of 5

The model identifier 'returntypesafe/jev-1.13-20260917throughOpenRouter' appears with missing spaces; it should be formatted as code or inline text so the provider and model identity are readable.

The paper says, at Fig. 2

Pith says: minor objection 2 of 5

The figure would benefit from a small table of per-arm call counts, since question 930 shares a technical A input and the number of repeated calls per contrast is not readily recoverable from the plotted means.

show 3 more

The paper says, at Table II

Pith says: minor objection 3 of 5

The 'Accepted / 192' column mixes supported, unsupported, and unresolved traces; although the table already has an Unresolved column, a footnote stating explicitly that unresolved records are never counted as supported would prevent misreading.

The paper says, at III-C

Pith says: minor objection 4 of 5

The inline expression 'column_header=source.table[0][column_index]' should be set off as code, and the sentence should state explicitly that the column index is the same zero-based index used in the original pointer, to avoid ambiguity.

The paper says, at V-B

Pith says: minor objection 5 of 5

The discussion of reference [8] would be clearer if one sentence stated that the JEV-as-a-Judge escalation policy is distinct from the Noul source-support probability target defined in Section II-C; currently the reader must infer the relationship.

What would settle it

Run independent double-labeling of all 15 natural contrasts and all 36 fresh-source contrasts: recruit two additional reviewers, each blinded to model outputs and to the other reviewer, applying the returned reviewer's declared protocol (unaided reading, no probability/author-label access). Compute pairwise Cohen's kappa on the wrong-role/equivalent-fact/boundary and arm-level support labels, then recompute the headline acceptance counts under majority or consensus labels: accept-B among wrong roles (2/15 fresh), withhold-B among equivalents (11/19), withhold-A among A-only-supported (5/15 vs 6/15), and explicit-column paired separation (9/14 vs 8/14).

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 7 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 1 min 25 s · 14,916 tokens read, 10,187 written
  3. Editor wrote the verdict above 21 s · 16,641 tokens read, 866 written
  4. Referee major revision: 4 major objections, 5 minor 1 min 41 s · 15,528 tokens read, 8,230 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

The genuinely useful thing here is the evaluation axis: same-number citation swaps with equivalent-fact controls, holding operands and arithmetic fixed while moving the pointer to a different same-valued cell. That isolates financial-role recognition from numerical matching in a way I have not seen done for a probabilistic verifier. The signed-number-at-pointer baseline is the right control too, and it does real work—it explains most of the recovery over literal quotation checks, which keeps the paper honest about what Jev actually adds.

The paper is also honest in the right places. It reports that the net gain over the relaxed baseline is one joint pass in a single question (question 218, a contextual-sign case), and it reports its own fragility analyses in Section III-D: excluding the contested case 444 removes the explicit-column net gain (paired separation 8/14 in both renderings), and excluding all eight author/reviewer disagreements does the same. That is the load-bearing weak point, and the authors did not hide it. The stress-test note is right that this fragility is evidenced inside the manuscript, not hypothetical.

The central qualitative result survives the caveats. At the case level, with numerical agreement held fixed, Jev accepts some wrong-role citations (B means 0.865–0.950 on 906, 930 twice, 746) and withholds some equivalent facts (912 at 0.33, 9 at 0.695). That coarse observation is enough to support the paper's main claim that number matching and role support are separable dimensions for a probabilistic verifier.

What is not pinned down is the magnitude and the explicit-column asymmetry. The two equivalent-fact B means in the column-rendering study sit at 0.785 against an 0.8 cutoff—one technical repeat from crossing. In the fresh set, the 16 and 13 withheld supported arms still have means at least 0.5, and the largest identical-input repeat range is 0.18. The benefit leg of the tradeoff is movable under contested labels; the paper's own sensitivity says so.

The weaker premise is ground truth. Natural-record labels are unblinded assistant judgments with no independent confirmation; the fresh set is one non-author reviewer with no measured inter-human agreement. The paper acknowledges both (Sec. VI). For a referee, the headline counts are conditional on label stability, which the paper itself demonstrates is lacking at the margins, and no public artifacts are released.

Who this is for: people evaluating probabilistic evidence verifiers for financial LLM assistants, and anyone building adversarial evaluation axes for LLM-as-judge components. It deserves a serious referee, with the reasonable demand that the authors release the inspection packet and either add label-reliability evidence or pull the headline claims back to the coarse, surviving level.

The strongest objection

The central claim requires the wrong-role / equivalent-fact / boundary trichotomy to be correct and stable, because every headline count (accept-B among wrong roles, withhold-B among equivalents, explicit-column paired separation) is defined against it. The natural 15 contrasts use unblinded assistant labels with no independent confirmation (acknowledged, Sec. VI: 'Independent human confirmation of these natural-record labels remains incomplete'), and the fresh 36 use one returned reviewer with inter-human agreement unmeasured (acknowledged, Sec. III-D). The manuscript itself demonstrates the fragility: case 444 'admits a contextual-equivalence reading,' and post hoc exclusion of that single case removes the explicit-column net gain (paired separation 8/14 in both renderings); excluding all eight author/reviewer disagreements gives the same 8/14 in both renderings. Since the abstract presents explicit-column improvement as a headline result, that benefit leg collapses under a contested label. The cost leg is no sturdier: the two equivalent B means in the rendering study are 0.785 against an inclusive 0.8 cutoff, and in the fresh set '16 and 13' withheld supported arms still have means at least 0.5, while the largest identical-input repeat range is 0.18. Serving noise and label interpretation therefore both cross the decision boundary. What survives is the coarse case-level observation that some wrong-role B means (906, 930x2, 746; 0.865-0.950) pass and some equivalent facts (912 at 0.330, 9 at 0.695) are withheld; its magnitude and the explicit-column asymmetry are not pinned down. This is the load-bearing lever the reader identified, and it is empirically evidenced inside the manuscript rather than merely hypothetical.

The reader's weakest_assumption — that assistant and single-reviewer role labels are correct and stable enough to classify swaps — is the same load-bearing concern I reach independently, and the manuscript's own sensitivity analyses supply the decisive evidence: excluding contested case 444, or all eight reviewer/assistant disagreements, removes the explicit-column net gain (paired separation 8/14 in both renderings), while near-threshold means (two equivalent B means at 0.785 against an inclusive 0.8 cutoff; supported withholdings spread across 0.5-0.8; largest identical-input repeat range 0.18) place both the benefit and cost legs within measured serving noise and label-interpretation uncertainty. I do not escalate because the paper is unusually transparent: it reports these sensitivities and near-threshold weaknesses itself, labels are at least pre-inference and saved, the fresh bank avoids selection by verifier outcomes, and the coarse tradeoff (4/6 wrong-role passes, 2/6 equivalent withholdings in natural contrasts; accept-B 2/15 and withhold-A 5/15 fresh) survives the exclusions as a case-level claim. Other candidate concerns are weaker. The signed-number baseline result is internally consistent and the incremental joint-pass being a single contextual-sign question (218) is explicitly reported and leave-one-out tested. Selection bias in the natural contrasts is acknowledged and mitigated by the source-only fresh bank. The claim that Jev 'does not reliably solve role recognition' is a negative/qualitative finding that any verified counterexample would support; the risk is that specific counterexamples are label-dependent. Because the fix is a concrete reproducibility step (independent labels and artifact release) rather than a new experiment invalidating the design, CONDITIONAL remains the right verdict. My read therefore keeps the reader's verdict unchanged.

The referee's report

If the label-dependent findings hold, this is a useful controlled evaluation: it separates numerical matching from financial-role verification and provides a reusable perturbation design (same-number swaps, equivalent-fact controls, rendering sensitivity) for studying citation-support models. The saved inputs, exact probabilities, and unusually candid sensitivity analyses are strengths, as is the explicit comparison to a deterministic signed-number baseline. The main limitation is that the headline counts—wrong-role passes, equivalent-fact withholdings, and the explicit-column improvement—rest on role labels without independent confirmation and on probabilities very close to the acceptance cutoff. With added annotation reliability, cutoff-robust analyses, and a clear in-sample qualification for the natural-record discrimination numbers, the central tradeoff claim would be supported. As submitted, the contribution is best read as a carefully documented case study whose quantitative claims are more fragile than the abstract suggests.

Pith's reading of the paper (a machine's summary, not the authors')

The paper asks what a probabilistic evidence verifier contributes beyond numerical matching when checking an LLM-generated financial calculation and its cited source cells. To isolate the question, the authors hold operands and arithmetic fixed and move a citation to a different cell containing the same number, so any difference in the verifier's probability must come from recognizing the financial role (period, metric, cohort, status) of the cited cell rather than from number agreement. They find that at the inclusive 0.8 cutoff the verifier accepts some unsupported-role citations and withholds some equivalent-fact citations, while a deterministic rule that merely checks the signed number at the pointer explains most of the recovery gained over exact quotation checks. Explicitly repeating the column header fixes selected wrong-period and wrong-status errors but lowers support for some valid equivalent evidence, and a fresh-source follow-up on 36 new pages with an independent reviewer reproduces the same tradeoff. The contribution is a controlled evaluation that makes numerical correctness, cited-role support, and acceptance outcomes separately measurable.

What carries the argument. The same-number citation swap: an A/B contrast in which arm A retains the original citation and arm B points one operand at a different cell containing the identical numerical value, with question, source, operands, operation, and executed result held fixed, so both arms satisfy the same signed-number-at-pointer criterion. The changed pointer and its resolved context are the only intended difference, making any probability shift a test of financial-role recognition rather than numerical matching. Equivalent-fact controls (for example, an ending balance and the next period's opening balance representing the same carried-forward amount) separate harmless citation changes from genuine role errors, and the signed-number-at-pointer rule itself is the deterministic baseline against which incremental probabilistic value is measured.

Core claim. On the paper's own terms: a probabilistic source-support verifier that shows strong aggregate discrimination on natural records (AUROC 0.98587, Brier 0.03378) nevertheless fails to reliably recognize the financial role of a cited number once numerical agreement is held fixed. Under same-number citation swaps, two of six clear wrong-role contrasts are still accepted at the inclusive 0.8 cutoff and two of six equivalent-fact controls are withheld; a deterministic signed-number-at-pointer baseline recovers 31 of 33 supported failures that exact quotation checks reject, locating most of the probabilistic verifier's apparent value in number matching rather than role judgment. Explicitly repeating the column header changes the tested decisions in both directions: all eight swapped wrong-period and wrong-status calls are withheld while equivalent-fact support falls, and the 36-question fresh-source evaluation with a returned non-author reviewer shows the same tradeoff between detecting role errors and retaining valid citations. The natural-bank comparison also found no release benefit for the originally specified numerical-selection gate, leaving the source-verifier evaluation as the measured evidence-verification component.

The premise it rests on. The ground-truth labels classifying each citation swap as wrong-role, equivalent-fact, or boundary come from one unblinded assistant review of the natural records and one non-author reviewer on the fresh-source follow-up, with no independent confirmation or inter-human agreement measurement; if those role labels are wrong or unstable, the pass and withhold counts and the stated tradeoff collapse.

If the paper is right (Pith's inferences, not the authors')

A cheap deterministic check of the signed operand value at the cited location captures most of what probabilistic verification recovers over exact quotation checks, so rule-based matching should be the baseline any probabilistic verifier must beat for financial citations.

Financial-role recognition, not numerical matching, is the open problem: same-value citations to the wrong period, cohort, or status pass even at the 0.8 cutoff despite near-perfect aggregate discrimination on natural records.

Explicit column headers can recover specific wrong-period and wrong-status errors but simultaneously lower support for some valid equivalent facts, so the input representation shifts the acceptance tradeoff rather than removing it.

Numerical correctness, cited-role support, and answer release must be assessed as separate dimensions, since 27 of 192 key-matching traces cited unsupported roles while their derivations were recoverable elsewhere.

The observed row-versus-column asymmetry in swap effects suggests the verifier attends more to the neighboring numbers and text around a cited cell than to the semantic header that names the column; a direct test is to perturb the header text alone while keeping the cell neighborhood fixed.

The tradeoff between role-error detection and equivalent-fact retention is likely to recur in real financial reports, where carried-forward balances, repurchase-program footnotes, and reused values across periods are common; reliable verification may need explicit accounting-rule knowledge rather than better probability calibration.

If the result holds at scale, deploying LLM financial assistants should pair probabilistic verification with a deterministic citation-role check rather than trusting the support probability alone.

The follow-up's finding that 25 of 53 reviewer-supported arms are withheld at the 0.8 cutoff implies the default cutoff is conservative for valid evidence; a cost-sensitive choice of threshold could trade role-error recall against equivalent-fact acceptance.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Financial reports repeat values across periods, metrics and accounting lines, allowing an LLM-generated calculation to be numerically correct while citing the wrong financial role. We evaluate what probabilistic evidence verification adds beyond number matching using Jev as a source-support verifier for GPT-4.1-mini calculation traces. A signed-number-at-pointer baseline explains most recovery over exact quotation checks. To isolate the remaining role-recognition problem, we hold operands and arithmetic fixed, move citations between same-number cells, and retain controls that express equivalent facts. These contrasts reveal both wrong-role citations that pass and valid alternative citations that are withheld. Explicit column labels improve selected wrong-role decisions while also lowering support for some equivalent evidence. A constructed follow-up on 36 new source pages, labeled by a non-author reviewer, extends this evaluation and exposes the same tradeoff between detecting role errors and retaining valid citations. The contribution is a controlled evaluation that identifies what a probabilistic financial verifier distinguishes when numerical matching is held fixed. For LLM-based financial assistants, it makes numerical correctness, cited-role support and acceptance outcomes separately assessable.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-07

The authors' main claim, as the classifier picked it out 0.51

“The contribution is a controlled evaluation that identifies what a probabilistic financial verifier distinguishes when numerical matching is held fixed.”

How the main claim is stated: asserted 0.99

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.79 · human participants 0.49 · controlled comparison 0.93 · randomized assignment 0.03 · states quantitative result 0.52 · compares with prior results 0.91 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Financial reports repeat values across periods, metrics and accounting lines, allowing an LLM-generated calculation to be numerically correct while citing the wrong financial role. background 0.97 0.14 0.06 asserted 0.91
We evaluate what probabilistic evidence verification adds beyond number matching using Jev as a source-support verifier for GPT-4.1-mini calculation traces. contribution 0.83 0.91 0.06 asserted 0.75
A signed-number-at-pointer baseline explains most recovery over exact quotation checks. result 0.97 0.76 0.17 asserted 1.00
To isolate the remaining role-recognition problem, we hold operands and arithmetic fixed, move citations between same-number cells, and retain controls that express equivalent facts. method 0.99 0.93 0.05 asserted 0.78
These contrasts reveal both wrong-role citations that pass and valid alternative citations that are withheld. result 0.87 0.91 0.23 asserted 1.00
Explicit column labels improve selected wrong-role decisions while also lowering support for some equivalent evidence. result 1.00 0.90 0.14 asserted 1.00
A constructed follow-up on 36 new source pages, labeled by a non-author reviewer, extends this evaluation and exposes the same tradeoff between detecting role errors and retaining valid citations. method 0.75 0.87 0.12 asserted 1.00
The contribution is a controlled evaluation that identifies what a probabilistic financial verifier distinguishes when numerical matching is held fixed. contribution 1.00 0.89 0.45 asserted 0.99
For LLM-based financial assistants, it makes numerical correctness, cited-role support and acceptance outcomes separately assessable. interpretation 0.58 0.49 0.17 asserted 0.96

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.90

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-07: 1 request, 6,787 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (3)
Figure 1 of the paper
Figure 1. Same-number citation contrasts under the original rendering. A retains the original citation; B cites a different same-number cell. Shading separates
Figure 2 of the paper
Figure 2. Explicit-column intervention: all two-call A/B means and interactions. Interaction is explicit (B minus A) minus original (B minus A). Negative values
Figure 3 of the paper
Figure 3. Fresh-source outcomes under one reviewer’s declared blinded labels. Decisions use two-call means and inclusive cutoffs. (a) All six outcomes at 0.8;

Figures are the authors', reproduced from the paper.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment