{"id":"1e47916f-2707-498b-9eca-451c6cdd0960","arxiv_id":"2504.18693","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A combination of CodeBERTScore and majority voting ranks LLM-generated tax code updates better than either metric alone, but results are preliminary and limited by small samples.","lead":"The authors test whether ChatGPT and GPT-4 can turn IRS tax law updates into code patches for tax preparation software, and they propose a voting-based way to pick the best candidate patch. The paper reports that combining a similarity score with majority voting helps, but the evidence is preliminary and the method is not yet reliable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Concern: Table 2 contains a likely label/entry error (Version9 for Brackets/GPT4 shows WeightedScore 0.592 despite CodeBertScore 0.911 and MajorityVoteScore 1.0), and the central ranking claim is supported only by qualitative top-4 comparisons rather than a statistically rigorous evaluation.","rationale":"The reader's verdict is CONDITIONAL, which matches my assessment. The reader's weakest_assumption about MajorityVoteScore is well-founded and is indeed load-bearing: the paper itself documents cases where high majority vote coincides with low ground-truth accuracy, so the majority-vote assumption is not reliable in the very scenarios the paper reports. However, the single most concrete and decisive issue I identified is different: the internal score inconsistency in Table 2. The row 'Brackets, GPT4, Version9' reports a WeightedScore of 0.592 despite component scores that should yield 0.947 under the stated formula. This is not an interpretive matter; it is a direct contradiction in the reported evidence, and it suggests either a typo in the table or that the WeightedScore is not computed as described. The reader's emphasis on the majority-vote weakness is correct, but this specific data error is more easily verified and more damaging to the reliability of the reported claim. A simple re-derivation of the WeightedScore for that row settles whether my concern lands. Even if the row is corrected, the paper's primary claim remains weakly supported because the evaluation is qualitative and top-4 oriented, with no held-out validation of the weight and no statistical significance test. Therefore, CONDITIONAL remains the appropriate verdict: the research direction is plausible and the paper is transparent, but the empirical support needs correction and statistical strengthening.","tokens_in":11967,"tokens_out":1847,"duration_ms":16627,"concrete_test":"Recompute the full ranking tables from raw candidate scores (or ask authors to release the per-candidate CodeBertScore, MajorityVoteScore, WeightedScore, and ground-truth values). In particular, recompute WeightedScore for the row 'Brackets, GPT4, Version9' in Table 2 with the stated weights 0.6/0.4; if it does not equal 0.947, the table contains an error. Then run a permutation test or rank-correlation test (e.g., Kendall's tau) between each metric and ground truth across all 10 candidates per scenario, and validate the 0.6/0.4 weight on a held-out split, to determine whether the combined ranking significantly outperforms either metric alone.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that combining CodeBertScore and MajorityVoteScore outperforms each metric in isolation. The evaluation has two weaknesses: (1) In the reader's weakest_assumption is directly supported by the paper's own Table 2: e.g., Brackets+Deductions with GPT-3.5 shows all top candidates with MajorityVoteScore 1.0 but Ground Truth Score 21/100, so the majority vote rewards a confidently wrong consensus. The paper states this limitation explicitly and routes it into the combined WeightedScore, but the weight (0.6/0.4) is tuned on the same data and no validation set or statistical significance test is reported. (2) Beyond the weight, Table 2 has a concrete internal inconsistency: the row 'Brackets, GPT4, Version9' reports CodeBertScore=0.911, MajorityVoteScore=1.0, WeightedScore=0.592, while the stated formula WeightedScore=0.6*CodeBertScore+0.4*MajorityVoteScore would yield 0.947. The same row appears to be missing a ground-truth value and is labeled GPT-3.5, while neighboring rows are GPT4. Because the headline evidence relies on these tables and score values, an internal inconsistency in a top-4 row directly affects the credibility of the reported comparison and whether the stated weight was in fact used. This is a concrete, checkable correctness risk.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses the problem of maintaining tax-preparation software when tax laws change, proposing to use LLMs to generate updated code from IRS publications. The main experimental contribution is a ranking study: for three tax-calculation scenarios (brackets, brackets plus deductions, and brackets plus deductions plus EITC) and two LLMs (GPT-3.5 and GPT-4), with and without the previous year's code as context, the authors generate ten candidate implementations, score them with CodeBertScore and a proposed MajorityVoteScore, and combine them with a weighted score. The central claim, stated in the abstract, is that the combination outperforms each metric in isolation. The paper also outlines a broader framework including metamorphic testing and a feedback loop, although those components are not evaluated in the reported experiments.","tokens_in":12412,"tokens_out":11277,"duration_ms":93802,"significance":"If the ranking claim were rigorously established, it would be practically useful: developers could prioritize LLM-generated tax-code candidates that are likely to be correct and reduce manual review effort. The paper also makes a useful observation that LLM consensus can be confidently wrong when most generated candidates share the same bug, and it is partially transparent about that limitation. The paper's concrete strengths are its explicit ground-truth scoring of generated candidates, its prompt templates, and its articulation of the technical challenges. However, the empirical evidence for the central claim is currently anecdotal: there is one run of ten candidates per scenario, no confidence intervals, no statistical significance test, and the combination weight was selected on the same data used for evaluation. As presented, the value of the paper is in describing the challenges and illustrating them with examples rather than in validating the proposed ranking method.","major_comments":[{"comment":"The central claim that the combined WeightedScore outperforms CodeBertScore and MajorityVoteScore individually is not supported by the presented evaluation. There is one run of ten candidates per scenario/model, no confidence intervals, no significance test, and no held-out validation. The 0.6/0.4 weights were chosen after 'various experiments' on the same scenarios used for evaluation, so the reported superiority of the combination may be an artifact of fitting the weight to the test data. Please add a validation split or multiple independent generation runs with a paired comparison of ranking quality (e.g., precision@k or nDCG), or explicitly limit the claim to a qualitative observation.","section":"Section 5, Tables 1 and 2; Section 4 (Ranking and Selection)"},{"comment":"Many WeightedScore entries are inconsistent with the stated formula, WeightedScore = 0.6 * CodeBertScore + 0.4 * MajorityVoteScore. For example, Table 2, Brackets, GPT4, Version9 lists scores 0.911, 1.0, and 0.592, but the stated formula gives 0.947; Table 2, Brackets+Ded+EITC, GPT3.5, Version7 lists 0.977, 0.56, and 0.727, while the formula gives 0.810; and Table 1, Brackets+Ded+EITC, GPT3.5, Version6 lists 0.852, 1.0, and 0.941, while the formula gives 0.911. Because the headline evidence depends on these tables, the discrepancies must be resolved and every row checked before the ranking comparison can be assessed.","section":"Tables 1 and 2"},{"comment":"The method assumes that the majority output among candidates is correct. The paper's own results show that this assumption fails badly in several scenarios: in Table 2, Brackets+Deductions, GPT-3.5, all four top-ranked candidates have MajorityVoteScore=1.0 but Ground Truth Score=21/100; in Table 1, Brackets+Deductions, GPT-3.5, the four top-ranked candidates have MajorityVoteScore=1.0 and Ground Truth Score=1/100. The combined WeightedScore still ranks these candidates at the top (0.99 and about 0.916, respectively), so the reported examples do not demonstrate that the combination fixes the core failure. The paper needs an explicit analysis of when the majority is trustworthy, or the ranking claim must be restricted to settings where the consensus assumption is verified.","section":"Section 4, MajorityVoteScore"},{"comment":"The 'Ground Truth Score' column is not defined. The paper does not state how many test cases are used, how outputs are compared (exact match vs. tolerance), how floating-point amounts are judged, or whether the fraction represents the percentage of test inputs with matching output. Without this definition, the ranking comparisons cannot be reproduced or interpreted, and the subsequent error-margin analysis in Figures 5 and 6 is also missing the precise threshold mechanism.","section":"Section 5, Tables 1 and 2"},{"comment":"The text claims that in the Brackets+Ded+EITC scenario with prior code, GPT-3.5 'showed great performance' and has 'better top ranked codes than GPT-4', but Table 2 shows only a small difference in the best candidate's Ground Truth Score (56/100 versus 48/100) and no repeated trials. Similarly, Section 5.1 states that 'the ranking part of our framework works well' without a quantitative comparison against the individual metrics. These comparative statements are not justified by the single-run data and should either be supported statistically or softened to describe the specific examples.","section":"Section 5.2 and Figure 6"}],"minor_comments":[{"comment":"The text refers to 'GPT 3.0's candidates' in the results paragraph; the experiments use GPT-3.5 and GPT-4.0, so this appears to be a typo.","section":"Section 1"},{"comment":"The caption says prior code is listed 'only for clarity' and 'does not impact the code generation process,' but the figure is in a section about generation without prior code; please clarify that the prior code is shown only for the reader's reference and is not part of the prompt.","section":"Figure 2 caption"},{"comment":"The phrase 'ground truth score (or validation score) is 0.45' introduces a scale without explanation; a definition of the ground truth score should appear before its first use.","section":"Section 3"},{"comment":"The discussion mentions prior work on ranking code 'primarily in C programming language,' but all experiments in this paper are reported in Python; please clarify the relationship or remove the remark.","section":"Section 6"},{"comment":"The framework describes metamorphic testing as the next stage, and Section 5.1 says top-ranked candidates undergo it, but no metamorphic testing outcomes are reported; the text should state explicitly that this paper evaluates only the ranking component.","section":"Section 4, Step 4 (Metamorphic Testing)"}],"recommendation":"major_revision","confidential_remarks":"The paper is best read as an experience/position report rather than a rigorous empirical study. The concrete table inconsistencies and the weight-tuning issue are fixable, but the central claim in the abstract needs to be either supported with a proper statistical evaluation or explicitly downgraded to a hypothesis. If the authors are willing to make that change, the paper could become suitable after major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Paper is a preliminary workshop study on using LLMs to update tax-prep code. What's actually new: applying LLM-generated candidate ranking to IRS publication-to-code updates, and specifically combining CodeBertScore with majority voting for this domain. The framework is clearly laid out, the experiments cover three tax scenarios and two LLMs, and the authors are upfront that they are only addressing the ranking stage, not validation. That transparency is genuine.\n\nThe central empirical claim—that the weighted combination beats either metric alone—is plausible but not demonstrated. You have one run of 10 candidates per scenario, no confidence intervals, and the 0.6/0.4 weight was chosen after 'various experiments' on the same data. The majority vote also assumes consensus equals correctness, which the paper itself shows can fail (Table 2, Brackets+Deductions GPT-3.5: high majority vote, ground truth 21/100). Those honestly describe the failure mode, but it's exactly the load-bearing assumption of the ranking.\n\nThere is also a concrete arithmetic inconsistency in Table 2: the GPT-4 Brackets row for Version9 lists CodeBertScore 0.911 and MajorityVoteScore 1.0 with WeightedScore 0.592, but 0.6*0.911+0.4*1.0 = 0.947. Either the score or the formula is wrong, and since the headline evidence is the ranking comparison, this has to be fixed before the results can be trusted. The stress-test note about a missing ground-truth value and mismatched label doesn't hold up on reading: that row has 100/100 and is under GPT-4. So the main concrete problem is the arithmetic typo, not a label error.\n\nThat said, the paper is honest about its scope and the authors' prior TenForty work is a real, reproducible foundation. This is not a model-fitting exercise presented as prediction—it's a clearly framed first pass. The right fix is to add a validation set or at least report per-candidate variability, and to change the abstract's wording from 'outperformed' to something like 'showed promise in our preliminary runs.'\n\nWho is this for? Researchers working on LLMs for domain-specific code updates, especially in regulated domains. It's not a broad contribution, but within the tax-software niche it's a useful baseline. I'd send it to a serious referee for the workshop; it shouldn't be desk-rejected. But it needs a revision, not acceptance as-is.","headline":"A modest, honest workshop study of LLM-based tax-code updates; the ranking claim is plausible but under-evidenced and there is a concrete table error to fix.","tokens_in":12815,"tokens_out":2549,"would_cite":false,"duration_ms":22790,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hybrid ranking beats single metrics for LLM-written tax code.","keywords":["tax preparation software","large language models","code generation","CodeBertScore","MajorityVoteScore","metamorphic testing","software maintenance","tax law updates"],"falsifier":"A reader could test the central claim by constructing a scenario where most generated candidates are seeded with the same erroneous tax bracket or deduction value; if the combined ranking still places that wrong consensus at the top while a correct minority candidate scores lower, and this happens systematically across scenarios, the claim that the combination identifies promising candidates would be refuted. The paper's own tables already show isolated instances of this pattern, such as Brackets+Deductions with GPT-3.5, where candidates with perfect MajorityVoteScore match ground truth on only 1 of 100 inputs.","tokens_in":11790,"feed_emoji":"🧾","tokens_out":7040,"duration_ms":65873,"temperature":0.7,"pith_summary":"Tax preparation software must be updated every year as U.S. tax law changes, and doing this by hand is slow and error-prone. This paper asks whether large language models can translate official tax amendments into working code updates, and argues that the hard part is not generating code but choosing which generated candidate is correct. To solve that, it proposes ranking candidates by a weighted combination of CodeBertScore, a measure of semantic similarity to the reference material, and MajorityVoteScore, the fraction of test inputs for which a candidate agrees with the most common output. In experiments on tax brackets, deductions, and the Earned Income Tax Credit, the combined score ranked candidates better than either metric alone, and giving the model the previous year's code improved accuracy. The payoff, if the approach holds, is that developers could focus their review on a few promising candidates instead of inspecting every generated version.","feed_headline":"Hybrid ranking beats single metrics for LLM-written tax code","feed_subtitle":"Combining semantic similarity with candidate consensus surfaces promising updates before human review.","key_machinery":"The machinery is the two-signal ranking pipeline. CodeBertScore is a similarity score obtained from a pretrained code model (CodeBERT): it compares each generated candidate against the reference code and the tax policy text, returning a cosine similarity. MajorityVoteScore is computed by running every candidate on a shared random set of tax inputs, taking the most frequent output for each input as the presumed-correct answer, and scoring each candidate by how often it matches that consensus. The two are combined into a weighted score (0.6 and 0.4) that is used to order the candidates before any human inspection or further validation. This ranking is the load-bearing component of the proposed framework, because the framework's later refinement loop only makes sense if the candidates it feeds forward are the ones most likely to be fixable.","core_discovery":"The central claim is that the correctness of LLM-generated tax code can be effectively ranked by combining two weak signals: semantic similarity to the task description and previous code (CodeBertScore), and agreement among candidate outputs on random tax inputs (MajorityVoteScore). The paper reports that a weighted average—0.6 CodeBertScore and 0.4 MajorityVoteScore—outperformed each metric used alone in ranking candidates against ground-truth implementations. The experiments also show that providing the previous year's code as context substantially raises the accuracy of top-ranked candidates, and that even a weaker model like GPT-3.5 can match GPT-4 for error-tolerant scenarios when given that context. At the same time, the paper observes that a perfect MajorityVoteScore does not guarantee correctness: in several cases all top candidates agreed perfectly yet matched ground truth on only about one percent or twenty-one percent of test inputs, which motivates pairing the ranking with metamorphic testing as validation.","pith_inferences":["If the ranking generalizes, the same two-signal idea could be applied to other annually updated compliance software, such as payroll withholding or benefits eligibility, where the governing text is formal and the prior year's code exists.","The majority-vote assumption is the fragile point: when several generated candidates inherit the same mistake, consensus becomes a confident wrong answer. A natural extension would be to weight votes by semantic similarity or to cluster candidates to detect a single dominant failure mode.","The paper's tolerance-band results suggest a cheaper proxy for correctness: checking whether a candidate lands within a few percent of the majority on held-out inputs could serve as a screening test before expensive metamorphic validation.","The next testable step is whether the feedback loop can actually repair a top-ranked candidate; the ranking alone identifies promising code, but the framework's success depends on targeted prompts turning near-misses into correct implementations."],"forward_implications":["Developers maintaining compliance software can use the combined score to cut a pool of ten LLM-generated candidates down to a short list worth reviewing.","Providing the previous year's code as context is a reliable way to boost accuracy: top candidates reached 100 percent ground-truth match on simple bracket updates and stayed close on harder scenarios.","When exact correctness is not required, the ranking still surfaces candidates whose outputs lie within a tolerance band of ground truth, making them candidates for automated repair.","A high majority-vote score alone is not enough; the paper's own tables show consensus candidates that are entirely wrong, so the ranking must be paired with metamorphic or other correctness testing before deployment."],"supporting_citations":[{"why":"Supplies the CodeBertScore metric used as the semantic-similarity half of the ranking.","marker":"[24]"},{"why":"Provides the metamorphic testing and debugging framework that motivates validation of ranked candidates, and the case study of tax software failures.","marker":"[18]"},{"why":"Contributes the 33 metamorphic specifications from official tax publications that define correctness properties the ranking ultimately feeds into.","marker":"[17]"},{"why":"Supplies the open-source tax preparation software whose prior-year code is used as reference and update context in the experiments.","marker":"[15]"},{"why":"Establishes the oracle problem that justifies why ranking generated candidates is necessary instead of simply checking outputs.","marker":"[5]"},{"why":"Motivates the ranking problem by describing chain-of-thought prompting, which the paper plans to use for iterative candidate refinement.","marker":"[22]"}],"fun_headline_variants":["Hybrid scoring tops single metrics for LLM tax code","Weighted combo ranks LLM tax code better","Combined signals improve LLM tax code ranking","Previous code context boosts LLM tax updates","LLM tax code ranking: hybrid metric wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking method assumes that the output most generated candidates agree on is the correct tax result, so if the candidates share the same bug, the majority vote can be confidently wrong.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid scoring tops single metrics for LLM tax code","Weighted combo ranks LLM tax code better","Combined signals improve LLM tax code ranking","Previous code context boosts LLM tax updates","LLM tax code ranking: hybrid metric wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1385,"prompt_tokens":867,"completion_tokens":518,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":446}},"tokens_in":483,"tokens_out":518,"duration_ms":5243,"temperature":1.0,"reasoning_tokens":446,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:11:25.378575+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could test the central claim by constructing a scenario where most generated candidates are seeded with the same erroneous tax bracket or deduction value; if the combined ranking still places that wrong consensus at the top while a correct minority candidate scores lower, and this happens systematically across scenarios, the claim that the combination identifies promising candidates would be refuted. The paper's own tables already show isolated instances of this pattern, such as Brackets+Deductions with GPT-3.5, where candidates with perfect MajorityVoteScore match ground truth on only 1 of 100 inputs.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CodeBertScore metric used as the semantic-similarity half of the ranking."},{"cited_title":"Tizpaz-Niari, V","cited_arxiv_id":null,"evidence_quote":"Provides the metamorphic testing and debugging framework that motivates validation of ranked candidates, and the case study of tax software failures."},{"cited_title":"Srinivas, R","cited_arxiv_id":null,"evidence_quote":"Contributes the 33 metamorphic specifications from official tax publications that define correctness properties the ranking ultimately feeds into."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the open-source tax preparation software whose prior-year code is used as reference and update context in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the oracle problem that justifies why ranking generated candidates is necessary instead of simply checking outputs."}],"review_version":1}