REVIEW 4 major objections 5 minor 4 references
Efficacy of AI RAG Tools for Complex Information Extraction and Data Annotation Tasks: A Case Study Using Banks Public Disclosures
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A retrieval-augmented generation tool lets analysts extract data from bank disclosures up to 10 times faster and, when used interactively, with a statistically significant accuracy gain.
desk verdict Credible time-savings result from RAG in a real annotation task, but the interactive-condition accuracy gain is confounded by question reuse and needs a cleaner test. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a retrieval-augmented generation (RAG) pipeline: a chat interface (OpenWebUI) backed by Claude 3.5 Sonnet, with documents chunked into 2000-character pieces with 200-character overlap and the 20 most relevant chunks retrieved to answer each prompt. The experimental machinery is a within-subjects design in which each of 30 questions per bank is randomly assigned to a control or AI condition, followed by a delayed interactive AI+ phase on the same AI-assigned questions. Accuracy is scored by agreement with the final vetted answers of the original human research team, treated as the benchmark.
What would settle it
Re-run the AI+ phase on a fresh set of questions that participants have never seen, or include a no-treatment re-answer control group matched for delay; if the AI+ advantage over control disappears or shrinks to the naive-AI advantage, the interactive-use effect is not real.
Extended reading notes
Core claim
The paper claims that a simple, off-the-shelf RAG configuration can replace most manual document searching in a demanding real-world data extraction task without sacrificing accuracy, and that letting annotators interact with the tool improves accuracy beyond both human-only and naive-AI performance. Agreement rates were 52.9% in the control condition, 58.9% in the naive AI condition, and 65.9% in the interactive condition; the interactive gain is statistically significant in a logistic regression controlling for annotator and question complexity, while the naive gain is not. The paper also claims that annotator skill with AI tools, not just domain expertise, drives both speed and accuracy: the annotator with the highest AI-literacy score improved most under AI conditions, while the annotator with the highest domain score but lowest AI-literacy performed slightly worse with AI help.
Load-bearing premise
The load-bearing premise is that the two-week-to-three-month gap between the AI and AI+ phases fully removed participants' memory of the answers they had already produced, so the AI+ accuracy gain can be attributed to interactive tool use rather than recall.
Editorial extensions
If this is right
- Forced to rely only on the AI's first answer, annotators finished questions in about 0.3 minutes versus 4.0 minutes by hand, a 90% time reduction, with accuracy that was not statistically worse.
- Interactive AI use raised agreement with the expert benchmark to 65.9% from 52.9% in the control, a statistically significant improvement driven mainly by simpler yes/no and low-complexity questions.
- Extrapolated to the full 150-question, 29-bank task, the naive AI condition could save about 268 hours relative to human-only annotation.
- High-complexity questions remain the weak point: agreement on those stayed near 40% even in the interactive condition, and complexity was the strongest negative predictor of accuracy.
- Annotator AI skill appears to matter: the participant with the highest AI-literacy score more than doubled control accuracy under AI help, while the participant with the lowest AI-literacy did slightly worse with AI than without.
Reading between the lines
- If the interactive accuracy gain partly reflects recall of answers seen two weeks to three months earlier, the true benefit of interactive RAG use may be smaller than the headline 13 percentage-point improvement over control.
- The 268-hour savings extrapolation assumes the same speedup applies to all 150 questions and all 29 banks; the strong negative effect of high complexity suggests real savings would be lower for harder questions.
- The non-significant coefficients for document-checking and number of follow-up messages in the AI+ condition hint that annotators use their discretion mostly on hard questions; a design that randomizes whether these checks are allowed would test that directly.
- Because the tool itself is a simple, non-state-of-the-art RAG setup, the results may understate what more advanced retrieval or agentic systems could achieve; they also set a floor for what a typical researcher with commodity tools should expect.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a within-subjects experiment in which three financial analysts annotated a subset of a real-world dataset (30 questions per bank for 9 European GSIBs) under three conditions: a human-only Control condition, a naive AI RAG condition (accept the first answer), and an interactive AI+ condition (follow-up questions and/or PDF consultation). The benchmark is the vetted dataset of Beltran et al. (2023). The paper finds that the naive AI condition reduces per-question time by roughly 90% relative to control, the interactive AI+ condition reduces time by roughly 70%, and the AI+ condition shows a statistically significant agreement improvement over control in a logistic regression. The authors extrapolate potential savings of up to 268 hours for the full 150-question, 29-bank task. They also report that annotator AI skill and subject-matter skill appear to affect performance.
Significance. If the main results held, the paper would provide a practically valuable measurement of RAG-tool productivity effects in a realistic financial-analysis workflow, with a credible external benchmark that avoids circularity. The time-savings result in the naive AI condition is large, cleanly identified by the randomized within-subjects comparison, and robust to many concerns; this is a real strength. However, the headline accuracy claim for the interactive condition is not identified by the current design, and the statistical analysis contains several issues that need correction before the significance claims can be accepted. As a case study, the paper is informative but requires substantive revision.
major comments (4)
- [5.1, Table 14] The AI+ accuracy comparison is confounded with question re-administration. The AI+ condition is applied only to the exact questions that participants had already answered under the naive AI condition in the earlier phase, while control questions were not re-administered. The two-week to three-month delay is asserted to 'mitigate potential recall effects,' but the paper provides no evidence that recall or repeated-exposure effects have dissipated. Under this design, the significant AI+ coefficient in Table 14 (p=0.014) could reflect memory of the earlier AI answer, practice with the same items, or increased familiarity with the tool, rather than the value of interactive follow-up. Because re-answering is constitutive of the AI+ condition, treatment assignment and question familiarity are collinear. The time-savings result and the naive-AI accuracy finding are less affected, but the abstract's claim that accuracy is enhanced 'particularly in the interactive condition' is not identifiable from these data. A fresh-question control condition, or at minimum a clearly reported analysis restricted to 'among previously answered questions,' is needed before this claim can be sustained.
- [Tables 15 and 16] The logistic regression treats all 494 question-answer observations as independent, but the data are clustered by participant (3 participants) and by bank (9 banks). With only three participants, the participant-level effects are estimates of fixed convenience-sample differences, not population variability, and the standard errors almost certainly understate the uncertainty. The p=0.014 for the AI+ coefficient should be re-estimated with cluster-robust standard errors (clustered by participant, or by participant-bank) or with a multilevel model that includes random intercepts for participant and bank. Without this, the statistical significance claim is not reliable.
- [5.2, Eq. (5)] The reported standard errors in Tables 15 and 16 are implausible and appear to contain transcription errors. The intercept standard error is reported as 0.05400 in both Table 15 (n=173, within AI+ only) and Table 16 (n=494, all conditions), which is not credible: for a logistic regression with n=173 and an agreement rate near 66%, the expected intercept standard error is approximately 0.16, not 0.054. Similarly, the coefficient standard errors of roughly 0.25 in Table 16 seem too small for a GLMM with only three participants. These values should be independently recomputed; as printed, the implied z-statistics and p-values cannot be trusted. This issue directly affects the claim that time savings in the GLMM are highly significant.
- [5.2, Eq. (5)] The model specification in Equation (5) includes Complexity_Low, Complexity_High, and Complexity_Med as separate regressors, which with the intercept creates perfect collinearity. The accompanying Table 16 omits the medium-complexity dummy, suggesting the equation is mis-specified. Please clarify whether the equation should include only two of the three complexity dummies (with one baseline), and reconcile the equation with the reported table.
minor comments (5)
- [3.3] The section number 3.3 is used twice, for 'Participants' and for 'Materials'; renumber the subsequent sections.
- [Table 4] The count for 'External Organizations' is 26 out of 494 total questions, which is 5.3%, not the reported 7.3%.
- [References] The reference to Financial Times should be 'Gillian Tett,' not 'Gilian Tett.'
- [Table 14] The variable name 'Condtion_AI' is a typo for 'Condition_AI.'
- [Abstract, Table 5] The abstract says 'accelerated task execution by up to a factor of 10,' but Table 5 reports a mean control time of 4.0 minutes versus 0.3 minutes for the naive AI condition, which is a roughly 13-fold reduction; the phrasing 'up to a factor of 10' is imprecise and could be read as an upper bound that the data exceed.
Circularity Check
No significant circularity; the benchmark is external and the claims rest on direct measurement.
full rationale
The paper's central claims are empirical comparisons against an external benchmark: the annotated dataset from Beltran et al. (2023), compiled by a separate research team and provided to the present authors. Accuracy is measured by having a member of that original team code agreement with the benchmark answers, so the AI tool's outputs are not fitted to the benchmark and no equation or fitted parameter is renamed as a prediction. Time savings are a direct arithmetic extrapolation of self-reported per-question times, and the accuracy regressions use standard logistic and GLMM models on observed outcomes. The AI+ condition does re-answer questions previously handled under the naive AI condition, and the paper relies on a two-week-to-three-month delay to control for recall effects; this is an internal validity concern and a plausible confound, but it is not circularity because no claim reduces by construction to its own input or to a self-citation. The authors make no appeal to their own uniqueness theorems or prior work, and the few citations to prior literature are not load-bearing for the empirical result. The paper is appropriately cautious about its small sample and first-pass accuracy, further supporting that the reported findings are not manufactured by definition.
Assumptions & free parameters
assumptions (5)
- domain assumption The Beltran et al. (2023) vetted answers are an acceptable proxy for true accuracy in a task with no known ground truth.
- domain assumption A two-week minimum delay between the AI and AI+ phases adequately controls for recall of the same questions.
- domain assumption Unstratified random assignment produced balanced question characteristics across conditions.
- domain assumption Self-reported wall-clock times to the nearest minute are accurate enough for treatment comparisons.
- domain assumption The statistical models' independence assumption holds for repeated measures.
Cite this review
Pith. "Pith review of Efficacy of AI RAG Tools for Complex Information Extraction and Data Annotation Tasks: A Case Study Using Banks Public Disclosures." pith.science (2026). https://pith.science/paper/LBLT7MMI
@misc{pith2026250721360,
author = {Pith},
title = {Pith review of: Efficacy of AI RAG Tools for Complex Information Extraction and Data Annotation Tasks: A Case Study Using Banks Public Disclosures},
year = {2026},
howpublished = {\url{https://pith.science/paper/LBLT7MMI}},
note = {Machine review of arXiv:2507.21360}
}
read the original abstract
We utilize a within-subjects design with randomized task assignments to understand the effectiveness of using an AI retrieval augmented generation (RAG) tool to assist analysts with an information extraction and data annotation task. We replicate an existing, challenging real-world annotation task with complex multi-part criteria on a set of thousands of pages of public disclosure documents from global systemically important banks (GSIBs) with heterogeneous and incomplete information content. We test two treatment conditions. First, a "naive" AI use condition in which annotators use only the tool and must accept the first answer they are given. And second, an "interactive" AI treatment condition where annotators use the tool interactively, and use their judgement to follow-up with additional information if necessary. Compared to the human-only baseline, the use of the AI tool accelerated task execution by up to a factor of 10 and enhanced task accuracy, particularly in the interactive condition. We find that when extrapolated to the full task, these methods could save up to 268 hours compared to the human-only approach. Additionally, our findings suggest that annotator skill, not just with the subject matter domain, but also with AI tools, is a factor in both the accuracy and speed of task performance.
Reference graph
Works this paper leans on
-
[1]
Generative AI and labour productivity: a field experiment on coding
Bank for International Settlements (BIS) (2024): “Generative AI and labour productivity: a field experiment on coding”, BIS Working Papers, no 1208, September. Beltran, D O, H Bensen, A Kvien, E McDevitt, M V Sanz, and P Uysal (2023): “What are large global banks doing about climate change?”, International Finance Discussion Papers, no 1368, Board of Gove...
work page 2024
-
[2]
The rapid adoption of generative AI: productivity effects and time savings across occupations
System, January. Bick, A, A Blandin, and D Deming (2025): “The rapid adoption of generative AI: productivity effects and time savings across occupations”, NBER Working Paper Series, no 32966, February Chollet, F (2019): “On the measure of intelligence”, arXiv preprint, arXiv:1911.01547. arxiv Dua, D, Y Wang, P Dasigi, G Stanovsky, S Singh, and M Gardner (...
arXiv 2025
-
[3]
Artificial Intelligence Index Report 2025
Research, July. Maslej, N, L Fattorini, R Perrault, Y Gil, V Parli, J Manyika, J Etchemendy, K Ligett, T Lyons, and Y Shoham (2025): “Artificial Intelligence Index Report 2025”, Stanford Institute for Human-Centered Artificial
work page 2025
-
[4]
Wake up to the dangers of digital bank runs
Intelligence, April. Tett, Gilian (2023): “Wake up to the dangers of digital bank runs” Financial Times 4/20/23 Zhu, F, W Lei, Y Huang, C Wang, S Zhang, J Lv, F Feng, and T-S Chua (2021): “TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance”, arXiv preprint, arXiv:2105.07624
arXiv 2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.