Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Technical Challenges in Maintaining Tax Prep Software with Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Hybrid ranking beats single metrics for LLM-written tax code.

desk verdict A modest, honest workshop study of LLM-based tax-code updates; the ranking claim is plausible but under-evidenced and there is a concrete table error to fix. read the letter →

arxiv 2504.18693 v1 pith:2WDZQKOH submitted 2025-04-25 cs.SE cs.AI

Sina Gogani-Khiabani , Varsha Dewangan , Nina Olson , Ashutosh Trivedi , Saeid Tizpaz-Niari This is my paper · ORCID
classification cs.SEcs.AI
keywords taxpreparationsoftwarelargelanguagemodelscodegenerationBertScoreMajorityVotemetamorphictestingmaintenancelawupdates
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Tax preparation software must be updated every year as U.S. tax law changes, and doing this by hand is slow and error-prone. This paper asks whether large language models can translate official tax amendments into working code updates, and argues that the hard part is not generating code but choosing which generated candidate is correct. To solve that, it proposes ranking candidates by a weighted combination of CodeBertScore, a measure of semantic similarity to the reference material, and MajorityVoteScore, the fraction of test inputs for which a candidate agrees with the most common output. In experiments on tax brackets, deductions, and the Earned Income Tax Credit, the combined score ranked candidates better than either metric alone, and giving the model the previous year's code improved accuracy. The payoff, if the approach holds, is that developers could focus their review on a few promising candidates instead of inspecting every generated version.

What carries the argument

The machinery is the two-signal ranking pipeline. CodeBertScore is a similarity score obtained from a pretrained code model (CodeBERT): it compares each generated candidate against the reference code and the tax policy text, returning a cosine similarity. MajorityVoteScore is computed by running every candidate on a shared random set of tax inputs, taking the most frequent output for each input as the presumed-correct answer, and scoring each candidate by how often it matches that consensus. The two are combined into a weighted score (0.6 and 0.4) that is used to order the candidates before any human inspection or further validation. This ranking is the load-bearing component of the proposed framework, because the framework's later refinement loop only makes sense if the candidates it feeds forward are the ones most likely to be fixable.

What would settle it

A reader could test the central claim by constructing a scenario where most generated candidates are seeded with the same erroneous tax bracket or deduction value; if the combined ranking still places that wrong consensus at the top while a correct minority candidate scores lower, and this happens systematically across scenarios, the claim that the combination identifies promising candidates would be refuted. The paper's own tables already show isolated instances of this pattern, such as Brackets+Deductions with GPT-3.5, where candidates with perfect MajorityVoteScore match ground truth on only 1 of 100 inputs.

Watch

Extended reading notes

Core claim

The central claim is that the correctness of LLM-generated tax code can be effectively ranked by combining two weak signals: semantic similarity to the task description and previous code (CodeBertScore), and agreement among candidate outputs on random tax inputs (MajorityVoteScore). The paper reports that a weighted average—0.6 CodeBertScore and 0.4 MajorityVoteScore—outperformed each metric used alone in ranking candidates against ground-truth implementations. The experiments also show that providing the previous year's code as context substantially raises the accuracy of top-ranked candidates, and that even a weaker model like GPT-3.5 can match GPT-4 for error-tolerant scenarios when given that context. At the same time, the paper observes that a perfect MajorityVoteScore does not guarantee correctness: in several cases all top candidates agreed perfectly yet matched ground truth on only about one percent or twenty-one percent of test inputs, which motivates pairing the ranking with metamorphic testing as validation.

Load-bearing premise

The ranking method assumes that the output most generated candidates agree on is the correct tax result, so if the candidates share the same bug, the majority vote can be confidently wrong.

Editorial extensions

If this is right

  • Developers maintaining compliance software can use the combined score to cut a pool of ten LLM-generated candidates down to a short list worth reviewing.
  • Providing the previous year's code as context is a reliable way to boost accuracy: top candidates reached 100 percent ground-truth match on simple bracket updates and stayed close on harder scenarios.
  • When exact correctness is not required, the ranking still surfaces candidates whose outputs lie within a tolerance band of ground truth, making them candidates for automated repair.
  • A high majority-vote score alone is not enough; the paper's own tables show consensus candidates that are entirely wrong, so the ranking must be paired with metamorphic or other correctness testing before deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the ranking generalizes, the same two-signal idea could be applied to other annually updated compliance software, such as payroll withholding or benefits eligibility, where the governing text is formal and the prior year's code exists.
  • The majority-vote assumption is the fragile point: when several generated candidates inherit the same mistake, consensus becomes a confident wrong answer. A natural extension would be to weight votes by semantic similarity or to cluster candidates to detect a single dominant failure mode.
  • The paper's tolerance-band results suggest a cheaper proxy for correctness: checking whether a candidate lands within a few percent of the majority on held-out inputs could serve as a screening test before expensive metamorphic validation.
  • The next testable step is whether the feedback loop can actually repair a top-ranked candidate; the ranking alone identifies promising code, but the framework's success depends on targeted prompts turning near-misses into correct implementations.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper addresses the problem of maintaining tax-preparation software when tax laws change, proposing to use LLMs to generate updated code from IRS publications. The main experimental contribution is a ranking study: for three tax-calculation scenarios (brackets, brackets plus deductions, and brackets plus deductions plus EITC) and two LLMs (GPT-3.5 and GPT-4), with and without the previous year's code as context, the authors generate ten candidate implementations, score them with CodeBertScore and a proposed MajorityVoteScore, and combine them with a weighted score. The central claim, stated in the abstract, is that the combination outperforms each metric in isolation. The paper also outlines a broader framework including metamorphic testing and a feedback loop, although those components are not evaluated in the reported experiments.

Significance. If the ranking claim were rigorously established, it would be practically useful: developers could prioritize LLM-generated tax-code candidates that are likely to be correct and reduce manual review effort. The paper also makes a useful observation that LLM consensus can be confidently wrong when most generated candidates share the same bug, and it is partially transparent about that limitation. The paper's concrete strengths are its explicit ground-truth scoring of generated candidates, its prompt templates, and its articulation of the technical challenges. However, the empirical evidence for the central claim is currently anecdotal: there is one run of ten candidates per scenario, no confidence intervals, no statistical significance test, and the combination weight was selected on the same data used for evaluation. As presented, the value of the paper is in describing the challenges and illustrating them with examples rather than in validating the proposed ranking method.

major comments (5)
  1. [Section 5, Tables 1 and 2; Section 4 (Ranking and Selection)] The central claim that the combined WeightedScore outperforms CodeBertScore and MajorityVoteScore individually is not supported by the presented evaluation. There is one run of ten candidates per scenario/model, no confidence intervals, no significance test, and no held-out validation. The 0.6/0.4 weights were chosen after 'various experiments' on the same scenarios used for evaluation, so the reported superiority of the combination may be an artifact of fitting the weight to the test data. Please add a validation split or multiple independent generation runs with a paired comparison of ranking quality (e.g., precision@k or nDCG), or explicitly limit the claim to a qualitative observation.
  2. [Tables 1 and 2] Many WeightedScore entries are inconsistent with the stated formula, WeightedScore = 0.6 * CodeBertScore + 0.4 * MajorityVoteScore. For example, Table 2, Brackets, GPT4, Version9 lists scores 0.911, 1.0, and 0.592, but the stated formula gives 0.947; Table 2, Brackets+Ded+EITC, GPT3.5, Version7 lists 0.977, 0.56, and 0.727, while the formula gives 0.810; and Table 1, Brackets+Ded+EITC, GPT3.5, Version6 lists 0.852, 1.0, and 0.941, while the formula gives 0.911. Because the headline evidence depends on these tables, the discrepancies must be resolved and every row checked before the ranking comparison can be assessed.
  3. [Section 4, MajorityVoteScore] The method assumes that the majority output among candidates is correct. The paper's own results show that this assumption fails badly in several scenarios: in Table 2, Brackets+Deductions, GPT-3.5, all four top-ranked candidates have MajorityVoteScore=1.0 but Ground Truth Score=21/100; in Table 1, Brackets+Deductions, GPT-3.5, the four top-ranked candidates have MajorityVoteScore=1.0 and Ground Truth Score=1/100. The combined WeightedScore still ranks these candidates at the top (0.99 and about 0.916, respectively), so the reported examples do not demonstrate that the combination fixes the core failure. The paper needs an explicit analysis of when the majority is trustworthy, or the ranking claim must be restricted to settings where the consensus assumption is verified.
  4. [Section 5, Tables 1 and 2] The 'Ground Truth Score' column is not defined. The paper does not state how many test cases are used, how outputs are compared (exact match vs. tolerance), how floating-point amounts are judged, or whether the fraction represents the percentage of test inputs with matching output. Without this definition, the ranking comparisons cannot be reproduced or interpreted, and the subsequent error-margin analysis in Figures 5 and 6 is also missing the precise threshold mechanism.
  5. [Section 5.2 and Figure 6] The text claims that in the Brackets+Ded+EITC scenario with prior code, GPT-3.5 'showed great performance' and has 'better top ranked codes than GPT-4', but Table 2 shows only a small difference in the best candidate's Ground Truth Score (56/100 versus 48/100) and no repeated trials. Similarly, Section 5.1 states that 'the ranking part of our framework works well' without a quantitative comparison against the individual metrics. These comparative statements are not justified by the single-run data and should either be supported statistically or softened to describe the specific examples.
minor comments (5)
  1. [Section 1] The text refers to 'GPT 3.0's candidates' in the results paragraph; the experiments use GPT-3.5 and GPT-4.0, so this appears to be a typo.
  2. [Figure 2 caption] The caption says prior code is listed 'only for clarity' and 'does not impact the code generation process,' but the figure is in a section about generation without prior code; please clarify that the prior code is shown only for the reader's reference and is not part of the prompt.
  3. [Section 3] The phrase 'ground truth score (or validation score) is 0.45' introduces a scale without explanation; a definition of the ground truth score should appear before its first use.
  4. [Section 6] The discussion mentions prior work on ranking code 'primarily in C programming language,' but all experiments in this paper are reported in Python; please clarify the relationship or remove the remark.
  5. [Section 4, Step 4 (Metamorphic Testing)] The framework describes metamorphic testing as the next stage, and Section 5.1 says top-ranked candidates undergo it, but no metamorphic testing outcomes are reported; the text should state explicitly that this paper evaluates only the ranking component.

Circularity Check

1 steps flagged · score 6.0 of 10

WeightedScore weights are tuned on the same evaluation scenarios, so the abstract's claim that the combination outperforms each metric in isolation is partially a restatement of that tuning.

  1. fitted input called prediction [Section 4 (Ranking and Selection, WeightedScore) and Abstract]
    "We perform various experiments and found that assigning a weight of 0.6 to CodeBertScore and 0.4 to the MajorityVoteScore works well in practice. ... Our experiments show that a combination of CodeBertScore and MajorityVote outperformed each metric in isolation."

    WeightedScore is defined as 0.6*CodeBertScore + 0.4*MajorityVoteScore, and the 0.6/0.4 split is selected after 'various experiments' on the same scenarios and tables later used to demonstrate the combination's superiority. No held-out validation set or significance test is reported. Consequently, the abstract's 'combination outperformed each metric in isolation' is, at least in part, a restatement of the tuning criterion: a weight chosen to make the weighted average look good on the evaluation data. The MajorityVoteScore assumption that consensus equals correctness is explicitly acknowledged as imperfect in the paper (e.g., Table 2 rows with MajorityVoteScore 1.0 but Ground Truth 21/100); that is a correctness risk rather than a circularity step.

full rationale

The paper's central ranking claim rests on a WeightedScore whose weights were chosen after experiments on the same data used for evaluation, matching the 'fitted input called prediction' pattern. The text explicitly says the 0.6/0.4 weight 'works well in practice' based on 'various experiments', and then the abstract reports that the combination outperforms each metric in isolation without a held-out validation or statistical test. This is a partial circularity: the reported superiority of the combination is partly a consequence of tuning the weight to the evaluation set. The MajorityVoteScore's consensus-as-truth assumption is a real limitation but is not circular, because the paper evaluates against ground truth and even highlights cases where majority vote diverges from ground truth. The self-citations to prior metamorphic-testing work are contextual and not load-bearing for the ranking claim, since the paper explicitly states that integrating metamorphic testing remains future work. I also note a concrete arithmetic inconsistency in Table 2 (Version9 for Brackets/GPT4 reports WeightedScore 0.592 despite CodeBertScore 0.911 and MajorityVoteScore 1.0, which would give 0.947 under the stated formula); this is a correctness/quality risk, not circularity. Overall, the central claim is partially circular due to weight fitting on the evaluation data, but the underlying experiments include ground-truth measurements and are not fully reducible to the metric definitions.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central ranking result rests on the assumptions that majority vote proxies correctness, that CodeBERTScore correlates with functional quality, and that OpenTaxSolver 2021 is a valid ground truth. The only fitted free parameters are the weights 0.6/0.4 in the combined ranking score, tuned on the same data used for evaluation.

free parameters (2)
  • weight_w_CodeBertScore = 0.6
    Weight assigned to CodeBertScore in WeightedScore; chosen after 'various experiments' on the same data, not from a held-out set (Section 4).
  • weight_w_MajorityVote = 0.4
    Complement weight for MajorityVoteScore in the same tuned combination (Section 4).
assumptions (3)
  • domain assumption Majority vote of LLM-generated candidates approximates the correct tax computation
    The MajorityVoteScore assumes most candidates share correct behavior; the paper itself observes cases where high majority vote co-occurs with low ground-truth match (Section 5.1), weakening this assumption.
  • domain assumption OpenTaxSolver 2021 implementation is a valid ground truth
    Ground-truth scores compare generated code outputs to a reference implementation of the 2021 tax law (Section 5.1), but that implementation is not independently verified in this paper.
  • domain assumption CodeBERTScore semantic similarity correlates with functional correctness
    The ranking uses CodeBERTScore as a proxy for quality; Section 3 shows a faulty snippet with CodeBertScore 0.972 and the paper admits semantic similarity alone is insufficient.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Technical Challenges in Maintaining Tax Prep Software with Large Language Models." pith.science (2026). https://pith.science/paper/2WDZQKOH

@misc{pith2026250418693,
  author       = {Pith},
  title        = {Pith review of: Technical Challenges in Maintaining Tax Prep Software with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2WDZQKOH}},
  note         = {Machine review of arXiv:2504.18693}
}
read the original abstract

As the US tax law evolves to adapt to ever-changing politico-economic realities, tax preparation software plays a significant role in helping taxpayers navigate these complexities. The dynamic nature of tax regulations poses a significant challenge to accurately and timely maintaining tax software artifacts. The state-of-the-art in maintaining tax prep software is time-consuming and error-prone as it involves manual code analysis combined with an expert interpretation of tax law amendments. We posit that the rigor and formality of tax amendment language, as expressed in IRS publications, makes it amenable to automatic translation to executable specifications (code). Our research efforts focus on identifying, understanding, and tackling technical challenges in leveraging Large Language Models (LLMs), such as ChatGPT and Llama, to faithfully extract code differentials from IRS publications and automatically integrate them with the prior version of the code to automate tax prep software maintenance.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Technical Survey of Reinforcement Learning Techniques for Large Language Models

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A survey of RL methods for LLMs that organizes the field by reward modeling, feedback source, and optimization strategy, with benchmark tables favoring a scalar-regression UNA variant over DPO and KTO in offline alignment.

Reference graph

Works this paper leans on

24 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    I. 524. Credit for the elderly or disabled. https://www.irs.gov/publications/p524, 2021. Online

    work page 2021

  2. [2]

    I. 596. Earned income credit (eic). https://www.irs.gov/pub/irs-pdf/p596.pdf, 2021. Online

    work page 2021

  3. [3]

    I. 8812. Credits for qualifying children and other dependents. https://www.irs.gov/ forms-pubs/about-schedule-8812-form-1040 , 2021. Online

    work page 2021

  4. [4]

    I. 8863. Education credits. https://www.irs.gov/pub/irs-pdf/f8863.pdf, 2021. Online

    work page 2021

  5. [5]

    E. T. Barr, M. Harman, P . McMinn, M. Shahbaz, and S. Yoo. The oracle problem in software testing: A survey. IEEE Transactions on Software Engineering, 41(5):507–525, 2015

    work page 2015

  6. [6]

    Breiman, J

    L. Breiman, J. Friedman, R. Olshen, and C. Stone. Classification and regression trees . Wadsworth: Belmont, CA, 1984

    work page 1984

  7. [7]

    J. Cherry. Use opentaxsolver as an open source alternative to turbotax. https:// opensource.com/article/20/2/open-source-taxes, 2020. Online

    work page 2020

  8. [8]

    R. L. Community. Open source alternative to turbotax called open-source tax solver. https://www.reddit.com/r/linux/comments/bhp3cq/open_source_alternative_ to_turbotax_called/, 2019. Online

    work page 2019

Show all 24 references
  1. [9]

    Escher and N

    N. Escher and N. Banovic. Exposing error in poverty management technology: A method for auditing government benefits screening tools. Proc. ACM Hum. Comput. Interact. , 4(CSCW):064:1–064:20, 2020

    2020

  2. [10]

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang. Large language models for software engineering: Survey and open problems. arXiv preprint arXiv:2310.03533, 2023

    2023 arXiv

  3. [11]

    Hindle, E

    A. Hindle, E. T. Barr, M. Gabel, Z. Su, and P . Devanbu. On the naturalness of software. Communications of the ACM, 59(5):122–131, 2016

    2016

  4. [12]

    U. IRS. Filing taxes 101: Common errors taxpayers should avoid. https://www.irs.gov/ newsroom/filing-taxes-101-common-errors-taxpayers-should-avoid , 2020. Online

    2020

  5. [13]

    R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023

    2023 arXiv

  6. [14]

    Mehta, S

    P . Mehta, S. Kumar, R. Kumar, C. Babu, et al. Enhancement to training of bidirectional gan: An approach to demystify tax fraud. arXiv preprint arXiv:2208.07675, 2022

    2022 arXiv

  7. [15]

    A. Roberts. Open tax solver. https://sourceforge.net/projects/opentaxsolver/, 2021. Online

    2021

  8. [16]

    Schedule-A

    I. Schedule-A. Itemized deductions. https://www.irs.gov/pub/irs-pdf/f1040sa.pdf,

  9. [17]

    Srinivas, R

    D. Srinivas, R. Das, S. Tizpaz-Niari, A. Trivedi, and M. L. Pacheco. On the potential and limitations of few-shot in-context learning to generate metamorphic specifications for tax preparation software, 2023. The Proceedings of the Natural Legal Language Processing Workshop, E...

    2023

  10. [18]

    Tizpaz-Niari, V

    S. Tizpaz-Niari, V . Monjezi, M. Wagner, S. Darian, K. Reed, and A. Trivedi. Metamorphic testing and debugging of tax preparation software. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) , pages 138– 149, 2023

    2023

  11. [19]

    Six reasons 90 percent of people will e-file their tax returns

    US-IRS. Six reasons 90 percent of people will e-file their tax returns. https://www.irs.gov/ pub/irs-utl/oc-e-file.pdf/ , 2020. online

    2020

  12. [20]

    Direct file, a free online tool for tax filing by the irs

    US-IRS. Direct file, a free online tool for tax filing by the irs. https://directfile.irs.gov/,

  13. [21]

    Filing statistics for week ending december 11 2020

    US.IRS. Filing statistics for week ending december 11 2020. filing-season-statistics-for-week- ending-december-11-2020. online

    2020

  14. [22]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  15. [23]

    I. World. Tax preparation services in the us — market size, industry analysis, trends and forecasts. https://www.ibisworld.com/united-states/market-research-reports/ tax-preparation-services-industry/ , 2023. Online

    2023

  16. [24]

    S. Zhou, U. Alon, S. Agarwal, and G. Neubig. CodeBERTScore: Evaluating code generation with pretrained models of code. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 13921–13937, Sing...

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.