{"id":"5bd2448b-be39-49c0-9e85-38eb5131d321","arxiv_id":"2506.18942","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Four worked case studies show generative AI can improve claim cost models, automate annual report analysis, classify car damage, and migrate actuarial code, with notebooks on GitHub.","lead":"This paper reports four implemented case studies using generative AI in actuarial work: extracting claim features from text, pulling market data from annual reports, classifying car damage from images, and migrating legacy R code to Python. It is a practical, code-backed demonstration of where LLMs can help actuaries, with honest discussion of risks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Case Study 4's validation is strongest for deterministic outputs, but the stochastic bootstrap component is only sanity-checked; a resampling bug could pass all tests while producing materially wrong reserve distributions.","rationale":"The reader's weakest assumption (Case Study 4 test-suite comprehensiveness) is shared, but I specify a more concrete gap: the stochastic bootstrap outputs are not numerically validated against R at all, only shape/type/sanity checked. This is a load-bearing weakness because the paper's own conclusion highlights 'code migration with automated validation,' and the most complex example's stochastic component is exactly where a silent translation error is plausible. Other case studies are comparatively more robust: Case Study 1 has a corrected paired t-test and ablation study; Case Study 2 uses repeated runs and exact-match ground truth with public documents; Case Study 3, while lacking error bars, supports only a modest 'enabled' claim rather than a strong comparative one. I therefore agree with the CONDITIONAL verdict but do not escalate to REJECT, because the concern is a demonstrable limitation, not an internal inconsistency, and the proposed concrete test could confirm or refute it. The paper's honest discussion of validation limits (Section 6.4) actually supports treating this as a conditional-accept issue rather than a fatal flaw.","tokens_in":113,"tokens_out":3698,"duration_ms":45880,"concrete_test":"Independently re-run the R GLM bootstrap with a fixed random seed, then run the translated Python code with the same seed and documented resampling scheme. Compare the empirical distributions of total reserve (e.g., 75th and 95th percentiles, or full CDF) across repeated runs using a two-sample Kolmogorov–Smirnov test or a quantile-error tolerance (e.g., 1% of R's value). If the distributions differ significantly or percentiles are outside tolerance, the current test suite is insufficient to support the 'validated migration' claim; if they match, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim's fourth component, 'supported code migration with automated validation,' rests on the pre-written test suites in Section 6.3. For the GLM-based reserving example, 7 of 15 tests are content/numerical, but the paper states (Section 6.3) that only 'shape, type, and sanity tests' cover the stochastic bootstrap outputs, with numerical values 'varying naturally across runs.' No test compares the Python-generated bootstrap reserve distribution to the R-generated distribution (e.g., percentiles, CDF, or process variance). If the translation silently changed the resampling mechanism—sampling with replacement from residuals versus parametric bootstrap, incorrect process-variance formula, or wrong iteration count—the structural and deterministic tests could still pass while the reserve distribution materially differs. The paper itself acknowledges in Section 6.4 that LLMs 'may produce silently incorrect outputs that pass tests but introduce subtle errors in untested code paths' [28]. This gap directly weakens the claim of automated validation because the validation does not actually verify the stochastic output's numerical correctness, only its superficial structure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents four implemented case studies demonstrating generative AI applications in actuarial science: (1) LLM-extracted features from unstructured claim descriptions improve claim cost prediction, evaluated with a held-out test, stratified 4-fold CV, and a corrected paired t-test; (2) a Retrieval-Augmented Generation pipeline extracts structured financial data from three insurers' annual reports, benchmarked across five LLMs with exact-match ground truth and repeated runs; (3) a fine-tuned vision-enabled GPT-4o classifies car damage types and provides damage localization, compared against a CNN baseline; (4) a five-agent system migrates actuarial R code to Python with pre-written test suites combining structural and numerical checks. The paper also surveys additional GenAI applications and discusses regulatory, security, dual-use, reproducibility, privacy, and governance risks. All case studies are implemented in Jupyter notebooks available on GitHub.","tokens_in":27371,"tokens_out":6133,"duration_ms":67158,"significance":"If the results hold, the paper provides a valuable, reproducible resource for the actuarial community. Its strengths include open-source Jupyter notebooks with pinned dependencies, pinned LLM version identifiers, cached LLM outputs in Case Study 1, a proper holdout with stratified CV and a corrected significance test in Case Study 1, and repeated-run stability analysis in Case Study 2. The paper is notably transparent about its limitations: Section 5.4 explicitly acknowledges the single-CNN-baseline and single-split evaluation in Case Study 3, and Section 6.4 explicitly acknowledges that LLM-generated code may pass tests while containing subtle errors in untested code paths. These self-acknowledged limitations are weighed in the assessment below.","major_comments":[{"comment":"The 'automated validation' claim in the conclusion (Section 9) overstates what the test suites verify for the stochastic bootstrap outputs of the GLM-based reserving example. The paper states in Section 6.3 that only 'shape, type, and sanity tests' cover the bootstrap distributions, with numerical values 'varying naturally across runs'; no test compares the Python-generated bootstrap reserve distribution to the R-generated distribution (e.g., percentiles, CDF, or process variance). Since Section 6.4 itself acknowledges that LLMs 'may produce silently incorrect outputs that pass tests but introduce subtle errors in untested code paths' [28], the fourth component of the central claim is not fully supported. The authors should either add a distributional comparison test (e.g., comparing quantiles or a two-sample test between the Python and R bootstrap outputs) or narrow the claim to 'supported code migration with automated validation of deterministic outputs and structural checks for stochastic outputs.'","section":"§6.3, Table 12; §6.4; §9"},{"comment":"The comparison among the CNN, non-fine-tuned GPT-4o, and fine-tuned GPT-4o is based on a single train/validation/test split, with no confidence intervals, repeated runs, or error bars. The paper acknowledges this in Section 5.4, but the headline statement that the fine-tuned model achieves performance 'comparable to the convolutional neural network' should be softened or accompanied by uncertainty estimates, since the observed differences (0.837 vs. 0.880 accuracy) may not be stable across different random splits.","section":"§5.3, Table 9"}],"minor_comments":[{"comment":"The corrected paired t-test is computed on the 4-fold CV scores, but hyperparameters were selected using a separate 4-fold CV on the same training set; this selection can introduce optimistic bias in the CV performance estimates. The held-out test result is the more trustworthy evidence, and the paper should clarify that the reported p-values apply to the CV evaluation rather than to the held-out test.","section":"§3.3"},{"comment":"The ground-truth reference values in Case Study 2 were manually extracted by the authors. Although the annual reports are public and the values are listed in Tables 5-7, a second annotator or an explicit annotation protocol with inter-annotator agreement would strengthen the reliability of the reported pass rates.","section":"§4.3"},{"comment":"The exact-match pass criterion for discount rates requires all durations to be present and to match the ground truth exactly; this is strict and may penalize correct extractions with minor rounding differences. Consider reporting a partial-credit or tolerance-based analysis to complement the strict pass rates.","section":"§4.3"},{"comment":"The reference list is extensive and generally appropriate, but the paper would benefit from citing the original Kaggle competition and dataset pages in the reference list rather than only in footnotes, and from providing DOIs where available for the datasets used in Case Studies 1 and 3.","section":"§10, References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid practitioner-oriented contribution with open code and transparent limitations. The main concern is the overstated 'automated validation' claim in Case Study 4's stochastic bootstrap component, which is acknowledged in the text itself; this is fixable by either adding a distributional comparison test or qualifying the claim. I recommend major revision rather than rejection because the central contributions are sound if appropriately scoped and the paper has clear value for the actuarial community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is the package: four implemented case studies, Jupyter notebooks on GitHub, pinned model versions, and evaluations that mostly take holdout discipline seriously. Case Study 1 is the strongest: proper stratified split, corrected paired t-test, an ablation, and a deterministic codebook that avoids target leakage. Case Study 2 is a clean multi-model benchmark with exact-match ground truth and 900 runs, even if the ground truth is author-labeled and only three companies. Case Study 3 is honest about being an illustration rather than a rigorous comparison, and it explicitly flags the missing error bars and single CNN baseline. Case Study 4 is the most ambitious and the most fragile.\n\nThe stress-test note lands. The claim of 'automated validation' for the R-to-Python migration rests on pre-written test suites, and for the GLM bootstrap example only shape, type, and sanity tests cover the stochastic output. No test compares the Python-generated reserve distribution to the R-generated one at the percentile or process-variance level. The paper itself concedes in Section 6.4 that LLMs can produce silently incorrect outputs that pass tests. So the validation is real for deterministic outputs, but weaker for the bootstrap distribution than the conclusion implies. This is a fixable gap: add distributional checks (e.g., percentiles, mean and variance of the reserve distribution) against R ground truth, and rerun the 10 executions.\n\nThe other soft spots are minor and mostly acknowledged: Case Study 1 uses synthetic data, Case Study 2 has a small author-labeled reference set, Case Study 3 has no confidence intervals. None of these undercut the central claim that GenAI can assist these actuarial tasks on the demonstrated datasets. The paper doesn't overclaim; the final section is measured about the gap between prototype and production.\n\nI didn't execute the notebooks, so reproducibility rests on the artifacts and the paper's description. But the artifacts are real, the documentation is unusually thorough, and the citation pattern looks solid. This is a paper written by people who know the actuarial domain and want other actuaries to build on their work.\n\nWho is it for? Practicing actuaries and applied researchers who want concrete templates and honest caveats, not a novel algorithmic contribution. It doesn't reshape machine learning, but it doesn't need to. Deserves a serious referee, and the referee should push the authors to close the bootstrap-validation gap and add uncertainty estimates to Case Study 3.","headline":"Four worked actuarial GenAI case studies with honest limitations; the weakest link is the stochastic-output validation in the code-migration study, but the paper is worth refereeing.","tokens_in":27930,"tokens_out":1380,"would_cite":true,"duration_ms":18160,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Four implemented case studies show generative AI measurably supports actuarial work across text, reports, images, and code.","keywords":["generative AI","actuarial science","large language models","retrieval-augmented generation","fine-tuning","multi-agent systems","claim cost prediction","legacy code migration"],"falsifier":"Design a deliberate sabotage test: take the two translated Python programs, introduce a subtle but material change (for example, an off-by-one origin-year shift, a wrong volume-weight denominator, or a different dispersion link in the GLM) that leaves all declared test assertions satisfied, and check whether the pipeline's tests still pass. If they do, the claim that the system validates the translation against the original code's outputs is falsified for that example.","tokens_in":26953,"feed_emoji":"🤖","tokens_out":6751,"duration_ms":66569,"temperature":0.7,"pith_summary":"Generative AI can be put to work in actuarial practice today, not just speculated about: this paper reports four fully implemented case studies in which LLM-based systems improve real tasks. The paper claims that extracting structured injury features from free-text claim descriptions improves a gradient-boosting claim-cost model (18.2% lower RMSE on a held-out test set), that a retrieval-augmented pipeline reliably pulls solvency ratios, discount rates, and financial-strength ratings out of insurers' annual reports, that fine-tuning a vision-enabled LLM makes it outperform a convolutional baseline on car-damage classification, and that a five-agent system migrates R reserving code to Python with 100% pass rates against pre-written tests. What matters if true is that actuaries have a concrete, reproducible template for each of these four capabilities, plus a candid map of the governance and validation risks that come with them.","feed_headline":"GenAI cuts claim costs, extracts reports, reads damage, migrates code","feed_subtitle":"Four fully implemented case studies report double-digit accuracy gains, near-perfect extraction, and 100% test pass rates.","key_machinery":"Four named mechanisms carry the argument. First, LLM feature extraction: a commercial LLM is prompted with a strict schema to return injured body part, cause of injury, and injury count from claim descriptions, and a deterministic regular-expression codebook maps the free-form strings to 8 body regions and 13 cause classes so the features are stable and reusable. Second, Retrieval-Augmented Generation with Structured Outputs: annual-report text is chunked, embedded, and retrieved by cosine similarity, and the LLM is constrained by JSON/Pydantic schemas, with manually curated ground truth for exact-match evaluation. Third, fine-tuned vision-enabled LLM: a pre-trained multimodal model is fine-tuned on labelled car-damage images and then asked to output both class and optional location under a structured schema. Fourth, multi-agent migration system: five specialised agents (R analysis, translation, compilation, test runner, report) are wired in a hardcoded sequential graph with conditional retry loops, and validation rests on pre-written, R-verified test suites whose ground-truth values are hidden from the translation agent.","core_discovery":"The paper's central claim is that each of the four GenAI application patterns works on the reported datasets: LLM-derived features from claim text carry real predictive signal beyond tabular data (RMSE reduction from 1.345 to 1.101 on the held-out test set, with the gain statistically significant under a corrected paired t-test); a shared RAG pipeline with Structured Outputs achieves near-perfect or perfect extraction of the studied financial fields across five commercial LLMs; fine-tuning raises a vision-enabled LLM's accuracy from 0.823 to 0.880 on six-way car-damage classification, slightly above a CNN baseline; and a multi-agent R-to-Python migration pipeline passes all 14/15 tests in every one of ten runs per example, with deterministic outputs matching R to floating-point precision. The authors present these as evidence that generative AI can improve predictive modelling, automate document-driven comparison tasks, perform image-based damage assessment, and automate code migration with validation, while emphasizing that production deployment still requires human oversight and governance.","pith_inferences":["The feature-extraction pattern in Case Study 1 should transfer to underwriting and fraud-detection pipelines wherever free-text notes sit unused beside tabular data, but the paper demonstrates it on only one synthetic workers' compensation dataset.","The 100% pass rates in Case Study 4 are bounded by test coverage; a natural stress test would be to inject subtle, test-invisible numerical errors and check whether the pipeline's validation catches them, since the paper itself concedes LLMs can pass tests while hiding errors in untested paths.","The RAG benchmark's near-perfect scores on three narrow financial fields may not generalize to open-ended or ambiguous document queries, where the failure taxonomy the paper cites suggests harder cases will still need human review.","Model and API churn is the quiet vulnerability: all results depend on pinned commercial model versions, so the reported accuracies should be re-measured whenever those versions are retired."],"forward_implications":["Actuaries can treat unstructured claim text as a feature source: LLM-derived body-part and cause-of-injury categories improve prediction beyond tabular data, with body-part features contributing the largest single gain.","Annual-report-style extraction can be automated with high reliability for well-defined fields, and retrieval quality and prompt specificity matter as much as the choice of LLM.","Fine-tuning a vision-enabled LLM is a viable alternative to training a CNN for damage classification, and it adds location and context extraction the CNN cannot provide.","Legacy actuarial code migration can be automated with a test-gated multi-agent workflow, achieving 100% pass rates on the two reserving examples while retries absorb occasional translation errors.","All four implementations are released as runnable Jupyter notebooks, so the claimed workflows are reproducible and adaptable by other practitioners."],"supporting_citations":[{"why":"Supplies the corrected paired t-test used to declare the Case Study 1 improvement statistically significant.","marker":"[36]"},{"why":"Defines the retrieval-augmented generation method that is the core of Case Study 2's extraction pipeline.","marker":"[37]"},{"why":"Provides the BERT-based frequency and severity framework that the claim-text feature model extends.","marker":"[26]"},{"why":"Represents the earlier actuarial word-embedding approach that LLM feature extraction builds on.","marker":"[34]"},{"why":"Source of the caveat that LLM-generated code can pass tests while containing subtle errors, delimiting the migration validation claim.","marker":"[28]"},{"why":"The self-debugging paradigm behind the translation agent's targeted retry fixes.","marker":"[52]"},{"why":"The chain-ladder development-factor method that the first migration example reproduces.","marker":"[53]"},{"why":"The GLM with bootstrap reserving method that the second migration example reproduces.","marker":"[54]"},{"why":"Cited to show the broader applicability of vision LLMs to insurance visual tasks.","marker":"[47]"}],"fun_headline_variants":["GenAI improves claims, extracts reports, reads damage, migrates code","Four GenAI case studies: claims, reports, damage, code migration","GenAI in actuarial science: from claim text to code migration","Beyond ChatGPT: GenAI for claims, reports, damage, code","GenAI sharpens claims, reads damage, migrates legacy code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The code-migration claim rests on the assumption that the pre-written test suites (14 tests for chain-ladder, 15 for GLM) are comprehensive enough to catch any meaningful translation error; if untested code paths hide errors, a 100% pass rate would not prove the translation is correct.","fun_headline_variants_meta":{"raw":{"variants":["GenAI improves claims, extracts reports, reads damage, migrates code","Four GenAI case studies: claims, reports, damage, code migration","GenAI in actuarial science: from claim text to code migration","Beyond ChatGPT: GenAI for claims, reports, damage, code","GenAI sharpens claims, reads damage, migrates legacy code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000759,"raw_usage":{"total_tokens":3379,"prompt_tokens":962,"completion_tokens":2417,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":2324}},"tokens_in":578,"tokens_out":2417,"duration_ms":18509,"temperature":1.0,"reasoning_tokens":2324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:22:50.440958+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Design a deliberate sabotage test: take the two translated Python programs, introduce a subtle but material change (for example, an off-by-one origin-year shift, a wrong volume-weight denominator, or a different dispersion link in the GLM) that leaves all declared test assertions satisfied, and check whether the pipeline's tests still pass. If they do, the claim that the system validates the translation against the original code's outputs is falsified for that example.","supporting_citations":[{"cited_title":"Machine Learning 52(3), 239–281 (2003)","cited_arxiv_id":null,"evidence_quote":"Supplies the corrected paired t-test used to declare the Case Study 1 improvement statistically significant."},{"cited_title":"Variance16(2) (2023)","cited_arxiv_id":null,"evidence_quote":"Provides the BERT-based frequency and severity framework that the claim-text feature model extends."},{"cited_title":"In: Proceedings of the Twelfth International Conference on Learning Representations (ICLR 2024) (2024)","cited_arxiv_id":null,"evidence_quote":"The self-debugging paradigm behind the translation agent's targeted retry fixes."},{"cited_title":"ASTIN Bulletin23(2), 213–225 (1993)","cited_arxiv_id":null,"evidence_quote":"The chain-ladder development-factor method that the first migration example reproduces."},{"cited_title":"British Actuarial Journal8(3), 443–518 (2002) 41","cited_arxiv_id":null,"evidence_quote":"The GLM with bootstrap reserving method that the second migration example reproduces."}],"review_version":2}