{"id":"d40263f7-99d5-4fcd-938e-1e541c9c2e0a","arxiv_id":"2411.10145","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A judge-extract-code-conclude workflow beats plain prompting on numeric long-context tasks and cuts API cost, but underperforms CoT on one of two benchmarks.","lead":"This paper proposes a four-step workflow for numeric long-context questions: judge what data is needed, filter and extract that data from chunks using small models, then let a large model write and execute code for the arithmetic. On a dense numeric benchmark it reports 99% accuracy at low API cost, but on a sparse benchmark chain-of-thought prompting still wins.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"On the sparse Loong benchmark, the proposed method is less accurate than CoT, contradicting the abstract's claim of general accuracy improvement.","rationale":"I focused on the sparse benchmark contradiction because it is the most direct, falsifiable piece of evidence against the paper's central claim. The extraction concern is speculative; the Table 1 numbers are concrete. The reader did mention the Loong underperformance in their rationale, so there is partial agreement, but their primary flagged assumption (relevance filter/extraction reliability) is different. My test is a direct reproduction/verification of the published numbers; if they hold, the abstract and conclusion need revision. I keep the verdict at CONDITIONAL because the dense results are promising and the paper could be fixed by reframing the claim, not because the concern is trivial.","tokens_in":127,"tokens_out":4319,"duration_ms":49793,"concrete_test":"Reproduce the Loong subset experiments with the released pipeline (or, if code is absent, ask the authors to provide exact per-sample results) and verify that the workflow indeed scores 36.6% and 64.2% versus CoT's 41.2% and 84.4%. If the numbers hold, the paper must be revised to state that the method improves accuracy only on dense numeric tasks, or to frame the sparse results as an accuracy-cost trade-off rather than an accuracy improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 shows that on the numerical-sparse Loong subset, the workflow achieves 36.6% (comparison) and 64.2% (cluster), while the CoT baseline achieves 41.2% and 84.4%. The abstract states that the workflow 'can not only improve accuracy, but also significantly reduce the cost' and the conclusion repeats the claim without qualification. Since the sparse benchmark is a numeric-involved long-context task, the central claim of general accuracy improvement is directly contradicted by the paper's own data. The paper only notes that the method 'can still improve the accuracy by near 10%' compared to normal prompting, but the comparison to CoT in the same table shows a clear accuracy regression. This makes the core claim as written misleading; at best the method improves accuracy on dense numeric tasks while sacrificing accuracy on sparse ones for a large cost reduction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a workflow for numeric-involved long-context QA that decomposes the task into four subtasks: analyzing the question, removing irrelevant context chunks via a 1.5B Qwen filter, extracting data into a dataframe via a 7B Qwen model, and then having a large LLM (Gemini-1.5-flash) generate and execute code to compute the answer. The method is evaluated on a dense numeric subset (100 samples from the author's difficult-retrieval benchmark) and a sparse numeric subset (180 samples from Loong financial report analysis). The paper reports 99% accuracy on the dense subset versus 6% for normal prompting and 28% for a CoT-like baseline, at a fraction of the API cost; on the sparse subset the method improves over normal prompting but is less accurate than the CoT baseline (e.g., 36.6% vs 41.2% on comparison, 64.2% vs 84.4% on cluster). The abstract and conclusion claim the workflow improves accuracy generally, without qualification.","tokens_in":5605,"tokens_out":3842,"duration_ms":35740,"significance":"If the results are reliable, the framework is a practical, low-cost engineering solution for dense numeric long-context tasks; the 99% versus 28% gap on the dense benchmark is striking and the cost reduction is clearly demonstrated. The use of small open-weights models for filtering and extraction is a sensible division of labor and could be useful to practitioners. However, the evidence base is thin (no error bars, no significance tests, no validation of the GPT-4o judge), and the sparse-benchmark results directly contradict the unqualified accuracy claim. The contribution is therefore a promising idea whose generality remains unestablished.","major_comments":[{"comment":"The abstract and Section 4 state without qualification that the workflow 'can not only improve accuracy' on numeric-involved long-context tasks. Table 1 shows that on the numerical-sparse subset, the proposed method achieves 36.6% (comparison) and 64.2% (cluster) accuracy, while the CoT baseline achieves 41.2% and 84.4%, respectively. Thus the method is less accurate than CoT on one of the two evaluation benchmarks. The central claim must be revised to specify that accuracy improvement is demonstrated only on the dense benchmark, or the method must be improved/analyzed further so that the sparse regression is addressed rather than mentioned only as a 'near 10%' gain over normal prompting.","section":"Abstract and Section 3.3, Table 1"},{"comment":"The evaluation is too minimal to support the headline numeric claims. Only 100 dense and 180 sparse samples are used, with temperature set to 0 but no repeated runs, no confidence intervals, no bootstrap, and no significance tests. Accuracy is judged by GPT-4o against reference answers, but no human validation or agreement measure is reported. The dense result (99% vs 28%) is large enough to likely be robust, but the sparse comparison (e.g., 36.6% vs 41.2%) is within a range where sampling noise could change the conclusion. The paper should add variance estimates or multiple runs, and should validate the GPT-4o judge on a sample of human-annotated outputs.","section":"Section 3.1, 3.2, and Table 1"},{"comment":"The workflow's accuracy depends on the assumptions that the 1.5B relevance filter never discards a chunk containing needed data and that the 7B extractor captures every required value without loss or duplication. If either step drops a relevant value, the code-execution stage cannot recover it. The Limitations section correctly states that extraction accuracy varies with auxiliary-model choice and that this influence has not been studied. The paper should report precision/recall of the intermediate filtering and extraction steps, or at least ablate the auxiliary-model choice, because the final accuracy claims are conditional on these unmeasured steps.","section":"Section 2, 'Remove irrelevant parts' and 'Extract data'; Section 5"},{"comment":"The code-generation prompt shows the main LLM only the 'first a few rows' of the extracted dataframe, with the implicit assumption that those rows suffice for writing correct processing code. This is not justified in the paper. For example, if a column contains null or non-numeric values only in later rows, the generated code could fail or produce wrong results. The paper should demonstrate that the first-row display is sufficient across the tested tasks, or modify the prompt to include column statistics or a data sample that is robust to schema heterogeneity.","section":"Section 2, 'Process data' and Appendix A"}],"minor_comments":[{"comment":"There are several typos and grammar errors, including 'seperated' instead of 'separated', 'Qwen2.5-7b-instruct and is used' (duplicate 'and'), and the prompt text containing 'Y es' and 'Y ou' instead of 'Yes' and 'You'. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The sentence 'there is currently no research targeting at solving this type of problems' is ungrammatical and also overstates the novelty, since code-execution prompting has been widely explored; the paper's contribution is the specific decomposition, not the use of code execution.","section":"Section 1"},{"comment":"The claim that parallel chunk processing will 'increase the speed exponentially' is overstated; the speedup is at most linear in the number of parallel workers, and communication/merge overhead is ignored.","section":"Section 2"},{"comment":"The cost calculation uses a fixed price of $5 per million input tokens and $15 per million output tokens, but the paper does not specify whether this corresponds to Gemini-1.5-flash or a mix of models; since the workflow uses Qwen models locally, the API cost should be broken down by component to make the comparison transparent.","section":"Section 3.2 and Table 1"},{"comment":"The reference entry for 'gkamradt' contains repeated 'titleTranslation:' text, which is a citation-formatting artifact and should be cleaned up.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's evaluation resources overlap heavily with the author's prior work (Yu et al., 2024): the dense benchmark and the CoT baseline both come from that paper. This is not circular, but an independent external benchmark would substantially strengthen the claims. The manuscript reads more like a workshop or industry report than a full research paper; the evaluation depth, particularly the absence of error bars and judge validation, would need to be substantially increased for this venue. The sparse-benchmark contradiction is the most serious issue and must be addressed head-on in any revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The workflow itself is sensible: split the task into judging, filtering, extracting, and code-executing, and use a small model to throw away irrelevant chunks before the medium model builds a table. That is a smart engineering move, and the cost savings are real—$0.01 per sample versus $0.09–$0.15 for the baselines. The 99% accuracy on the dense dataset is eye-catching, and if it survives scrutiny, the recipe is genuinely useful for finance-style numeric QA over long documents. The paper also gets credit for an honest limitations section: it explicitly says the impact of auxiliary model choice on extraction accuracy has not been studied.\n\nThe soft spots are significant. The abstract and conclusion claim the method \"improves accuracy\" without qualification, but Table 1 shows the opposite on the sparse Loong benchmark: the workflow gets 36.6% vs 41.2% (CoT) on comparison and 64.2% vs 84.4% on cluster. That internal contradiction is the kind of thing a referee would flag immediately. The evaluation is also thinner than the headline warrants: 100 dense and 180 sparse samples, no error bars, no repeated runs, and a GPT-4o judge without human validation. One of the two benchmarks comes from the author's own prior work, and the CoT baseline is also taken from there—a mild self-citation concern, though the other benchmark (Loong) is external. The method recombines known ideas (chunking, relevance filtering, table extraction, code execution) and never cites or compares with prior code-generation prompting methods like Program of Thoughts or PAL. The novelty is in the composition, not the components.\n\nStill, I would not dismiss it. The dense result is striking enough to justify a serious referee. The paper needs a qualified accuracy claim, a more rigorous evaluation (error bars, multiple runs, human-checked judgments), code and data release, and a comparison against existing code-prompting baselines. The limitations paragraph already concedes the main unknown; a revision that addresses these points could turn a plausible engineering recipe into a useful, citable result.\n\nThis is a paper for people who care about practical cost-accuracy tradeoffs in long-context numeric QA, not for someone looking for a new method. I would not cite it in its current form, but I would bring it to a reading group to discuss the sparse-regression contradiction and the evaluation standards. Send it to peer review, but expect major revisions.","headline":"A practical cost-cutting pipeline that shines on dense-numbered QA but whose own data contradicts the abstract's unqualified accuracy claim; worth a rigorous referee, not yet a convincing paper.","tokens_in":6233,"tokens_out":1888,"would_cite":false,"duration_ms":19368,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A four-step workflow lets LLMs handle numeric long-context tasks by moving arithmetic into code, reaching 99% accuracy on a dense benchmark.","keywords":["numeric-involved long-context","task decomposition","LLM prompting","code interpreter","data extraction","chain-of-thought","API cost reduction","long-context QA"],"falsifier":"Build a synthetic long-context document with hundreds of records and place one required number only inside a chunk that otherwise discusses unrelated topics but happens to contain a header keyword; if the lightweight filter rejects that chunk, the pipeline's final answer should be wrong while a direct prompt reading the whole context could find the correct one. A companion test is to split a company's name and its profit across two separate 1000-token chunks so the extractor sees an incomplete row and discards it, and then check whether the final answer changes.","tokens_in":5233,"feed_emoji":"🧮","tokens_out":9697,"duration_ms":83322,"temperature":0.7,"pith_summary":"Large language models can retrieve a sentence from a 100k-token document but stumble badly when the question requires comparing, sorting, or counting numbers scattered through that document. This paper proposes to stop asking the model to read everything and calculate at once: instead, the task is decomposed into judging which data fields matter, filtering the context to relevant chunks, extracting the needed values into a table, and having generated code perform the arithmetic. The claim is that this decomposition makes numeric long-context tasks tractable and cheap, because small models do the voluminous reading while the large model only sees the question, a table head, and code. On a dense benchmark with hundreds of resumes the workflow reaches 99% accuracy at $0.01 per sample, versus 6% at $0.09 for plain prompting and 28% at $0.15 for a chain-of-thought-style baseline; on a sparse financial-report benchmark it improves over plain prompting but trails the stronger chain-of-thought baseline on accuracy while spending far less.","feed_headline":"Judge-extract-code pipeline hits 99% on numeric long-context tasks","feed_subtitle":"Splitting the work into judge, extract, code, and conclude beats plain prompts on accuracy while slashing API cost.","key_machinery":"The load-bearing mechanism is the judge–extract–process–conclude pipeline. 'Judge' is a short prompt that makes the model state the table header and primary key implied by the question, so the later stages know exactly which data fields matter. 'Extract' is a two-pass retrieval step: a lightweight binary classifier removes chunks judged irrelevant, the remaining chunks are merged and re-segmented at roughly 8,000 tokens, and a mid-size model transcribes each chunk into a markdown table, with incomplete or uncertain rows discarded and tables concatenated and deduplicated on the primary key. 'Process' gives the main model the question and the head of the resulting dataframe, and the model writes pandas code that is executed externally to load the data, transform types, and compute the answer. 'Conclude' is a final prompt that turns the code output into a natural-language answer. The work of the argument is carried by the fact that the large model never reads the long context and never performs arithmetic by token prediction.","core_discovery":"The paper's central claim is that LLM failures on numeric-involved long-context questions stem from conflating retrieval with arithmetic, and that a four-stage pipeline—analyze the question to define a data-table header and primary key, filter chunks with a lightweight model, extract values into markdown tables with a mid-size model, and then delegate computation to code written by a large model—fixes both problems at once. After filtering and extraction, the individual tables are concatenated and deduplicated on the primary key, and the main model writes a pandas program that the paper runs through an external interpreter; the final conclusion is then phrased from the program's output. The reported evidence is 99% accuracy on the dense student-resume benchmark at $0.01 per sample, against 6% and 28% for the plain and chain-of-thought baselines, and a large cost cut on the financial-report benchmark where the chain-of-thought baseline remains somewhat more accurate.","pith_inferences":["An implicit consequence is that on these tasks the binding constraint is faithful extraction rather than arithmetic; if extraction recall improves, the pipeline's accuracy should rise without any change to the code step.","The same judge–extract–code–conclude pattern is a candidate for non-numeric structured questions involving dates, locations, or names where exact values must be recovered before any reasoning; the paper does not test this.","The reported cost advantage assumes proprietary API pricing for the large model; if all stages run on open-weights models locally, the economics change but the filter's recall becomes the main risk.","A natural stress test is to hide a required number in a chunk the lightweight filter would likely classify as irrelevant; the paper's own limitation note implies the pipeline would lose that value permanently."],"forward_implications":["On numeric-dense long contexts (hundreds of records), the workflow reaches 99% accuracy versus 6% for plain prompting and 28% for a chain-of-thought-style baseline.","API cost per dense sample falls to $0.01 from $0.09 and $0.15, because the expensive model processes only the question, the table head, and the code it writes.","On sparse financial-report contexts of 40k–200k tokens, the workflow improves accuracy over plain prompting on both comparison and cluster subtasks, although the chain-of-thought baseline still has the higher accuracy.","Because each chunk is processed independently, the extraction stages can be parallelized across GPUs; the paper argues this makes the workflow scale to even longer contexts with a large speed-up.","The workflow needs no fine-tuning; it is a combination of prompting and external code execution and can be switched on manually for suitable long-context QA tasks."],"supporting_citations":[{"why":"Supplies the Loong financial-report subset used as the sparse numerical long-context benchmark.","marker":"Wang et al., 2024"},{"why":"Supplies the difficult-retrieval dense student-resume benchmark and the chain-of-thought-style baseline that the paper compares against.","marker":"Yu et al., 2024"},{"why":"Provides chain-of-thought prompting, the baseline that motivates the decomposition and that the paper improves on in cost.","marker":"Wei et al., 2022"},{"why":"Provides the Qwen2.5 auxiliary models used for classifying irrelevant chunks and extracting data tables.","marker":"Yang et al., 2024"},{"why":"Provides the Gemini model used as the main LLM for question analysis, code writing, and final conclusion.","marker":"Team et al., 2023"},{"why":"Provides GPT-4o, the judge model used to score whether the workflow's answers match the reference answers.","marker":"OpenAI, 2023"}],"fun_headline_variants":["Four-step pipeline boosts LLM arithmetic in long docs","99% accuracy on numeric long-context at a fraction of cost","Let code do the math: new LLM workflow for numeric tasks","New framework fixes LLM long-context calculation failures","Divide and conquer: judge-extract-code improves LLM numeracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The workflow assumes that the lightweight relevance filter never discards a chunk containing a needed value and that the extractor never drops or corrupts a needed value, because the code step can only use data that survived those two stages.","fun_headline_variants_meta":{"raw":{"variants":["Four-step pipeline boosts LLM arithmetic in long docs","99% accuracy on numeric long-context at a fraction of cost","Let code do the math: new LLM workflow for numeric tasks","New framework fixes LLM long-context calculation failures","Divide and conquer: judge-extract-code improves LLM numeracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00086,"raw_usage":{"total_tokens":3726,"prompt_tokens":935,"completion_tokens":2791,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":2706}},"tokens_in":551,"tokens_out":2791,"duration_ms":20765,"temperature":1.0,"reasoning_tokens":2706,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:54:50.619330+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a synthetic long-context document with hundreds of records and place one required number only inside a chunk that otherwise discusses unrelated topics but happens to contain a header keyword; if the lightweight filter rejects that chunk, the pipeline's final answer should be wrong while a direct prompt reading the whole context could find the correct one. A companion test is to split a company's name and its profit across two separate 1000-token chunks so the extractor sees an incomplete row and discards it, and then check whether the final answer changes.","supporting_citations":[],"review_version":1}