{"id":"53ce3804-f076-44f6-b1d7-7224a1036f22","arxiv_id":"2502.05439","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A CrewAI-based multi-agent system with human oversight built financial models and carried out model risk management checks on three public credit datasets, with results comparable to AutoML and Kaggle baselines.","lead":"This paper describes two teams of AI agents, built on the CrewAI framework, that automate machine learning model building and model risk management for credit card fraud, credit approval, and portfolio credit risk tasks. It is a practical demonstration of using LLM multi-agent systems with a human in the loop for regulated financial workflows.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human-in-the-loop instructions carry the modeling decisions; the paper does not isolate agentic capability, so the central effectiveness claim is under-supported.","rationale":"The reader's weakest_assumption focused on code-generation reliability and audit effectiveness. I agree that reliability is necessary, but the more load-bearing gap is the conflation of agent execution with agent decision-making: the logs show the human specifying algorithm choices, preprocessing steps, and performance criteria, which means the observed outcomes do not establish that the crew 'effectively collaborate[s]' rather than that the crew faithfully codes human decisions. This is not an internal inconsistency—the system is honestly described as human-orchestrated—but it undermines the abstract's implication of autonomous agentic capability. The concern is addressable by an ablation or intervention audit, so a conditional verdict is appropriate rather than rejection. The paper does provide real evidence that LLM agents can generate and execute working code in a structured framework, and the comparisons to AutoML/Kaggle baselines are useful, which is why I am not recommending a harsher verdict. The robustness evidence is also weak due to unspecified perturbations, but the primary issue remains the attribution of effectiveness.","tokens_in":33349,"tokens_out":3370,"duration_ms":39372,"concrete_test":"Re-run the portfolio credit risk use case under two conditions: (A) high-level human prompts only (e.g., 'build and validate a default-risk model according to the organizational guide') and (B) the exact prompts and feedback used in the paper. Log every human intervention, code error, and correction. If condition A requires a comparable number and depth of human decisions to achieve the reported metrics, or if condition B's success depends on the detailed instructions, then the claimed agentic effectiveness is not demonstrated; if condition A succeeds with few interventions, the original claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that agentic crews with a human-in-the-loop module can effectively collaborate to perform complex modeling and MRM tasks. The evidence, however, does not separate the agents' contribution from the human expert's substantive direction. In §3.1 the authors explicitly propose that a human expert act as system orchestrator, and the provided logs show that the human supplies the domain-level modeling decisions, not just exception handling. In Log 1 the human instructs the agent to drop the Time variable and use an 80/20 split; in Log 2 the human specifies the preprocessing recipe (KNN imputation, standard scaler, ordinal encoding, KBest); in Log 3 the human dictates the tuning protocol (use preprocessor, 40% sample, CV=5, roc_auc); and in Log 4 the human corrects code-level errors (missing joblib import, missing predict call). These interventions are not residual 1–10% corrections (the paper's own §3.1 estimate); they constitute the core modeling methodology. The reported success is therefore compatible with the interpretation that the human, not the agent crew, is performing the complex tasks, with agents serving as code-generating and executing tools. Since the claim is about agentic effectiveness, the absence of any ablation or control condition separating human direction from agent autonomy is a load-bearing gap. The robustness claim is similarly vulnerable: the perturbation procedure in the Outcome Analyzer is described only as multiplication or addition of fixed/randomized values, and the observed shifts produce large metric drops (fraud F1 88.9→65; credit-risk accuracy 93.4→79.1), yet the paper concludes robustness without a standard benchmark or statistical comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two CrewAI-based agentic crews: a modeling crew (data extraction, EDA, feature engineering, meta-tuning, model training, model evaluation, documentation, and a judge agent) and a model risk management crew (documentation compliance, model replication, conceptual soundness, outcome analysis, documentation, and a judge agent). A human expert acts as system orchestrator, delegating tasks and supplying corrective feedback. The framework is demonstrated on three public financial datasets: credit card fraud detection, credit card approval, and portfolio credit risk. The reported model metrics are competitive with external Kaggle solutions and H2O AutoML baselines, and the MRM crew produces replication, feature-importance, and stress-test reports. The central claim is that such crews can effectively and robustly collaborate on modeling and MRM tasks with human oversight.","tokens_in":33660,"tokens_out":5459,"duration_ms":51345,"significance":"If the central claim were established with proper controls, this would be a useful demonstration of LLM multi-agent workflows in a regulated financial-services setting. The paper has real strengths: it benchmarks against external Kaggle and H2O AutoML solutions, includes detailed interaction logs (Logs 1-4), uses public datasets, and candidly reports manual verification (§3.5.5) and a 1-10% residual-error estimate (§3.1). These features make the workflow transparent at the narrative level and would serve as a reproducible case study if code and prompts were released. However, the evidence does not yet separate the human expert's substantive modeling decisions from agent contributions, and the robustness and compliance evidence is not fully specified. The contribution is best read as an architectural demonstration until those gaps are closed.","major_comments":[{"comment":"The human expert, not the agents, supplies the core modeling methodology. In Log 2 the human specifies the preprocessing recipe (KNN imputation, standard scaler, ordinal encoding, KBest), and in Log 3 the human specifies the tuning protocol (use the preprocessor, 40% sample, CV=5, roc_auc). These are substantive modeling decisions, not the residual 1-10% corrections estimated in §3.1. Because the paper reports no ablation or control condition that removes or randomizes such human direction, the results are compatible with the human performing the complex modeling while agents act as code-generating and executing tools. Please add an ablation (e.g., generic task prompts without human-specified recipes, a single-LLM control given the same instructions, or a quantitative ledger of human decisions versus agent decisions) to support the claimed agentic effectiveness.","section":"§3.1, Logs 1-4"},{"comment":"The Outcome Analyzer's perturbation procedure is not specified enough to support the robustness claim. §3.4 says shifted inputs are regenerated by multiplication or addition of fixed/randomized values, and Appendix B reports only the final code outcome; no seed, perturbation magnitudes, feature selection rule, or generation algorithm is given. All reported metrics in §3.5 are single-run point estimates with no confidence intervals or repeated trials, and the 1-10% failure estimate in §3.1 is not derived from observed error counts. Please provide the exact perturbation algorithm and parameters, report variability across runs or seeds, and give empirical failure counts so the robustness results are reproducible and statistically meaningful.","section":"§3.4, Appendix B"},{"comment":"The compliance check is performed against an AI-generated modeling guide created for this paper. Footnote 1 states the guide 'is AI-generated and intended for illustrative purposes only' and is not the actual internal guide of the employer. The MRM claim that the modeling crew 'adheres to the organization's guidelines' is therefore self-referential: it compares one LLM output against another LLM-generated standard. Please evaluate the documentation against an independent, authoritative standard (e.g., published regulatory guidance or an externally provided modeling checklist) or explicitly re-scope the claim as an internal consistency check.","section":"§3.4, footnote 1"},{"comment":"The class-imbalance statement appears internally inconsistent. The text says '78.18% of instances belonging to the majority class (‘loan_status’ = 1)' after defining loan_status=1 as default. In the cited Kaggle credit-risk dataset, the majority class is non-default (loan_status=0) at roughly 78%. If the paper's statement is correct, the dataset is pathological; if incorrect, the data description and conclusions about imbalance are wrong. Please verify the target encoding and correct the text, because the model's precision/recall tradeoff and the MRM stress-test interpretation depend on which class is the majority.","section":"§3.5.3"}],"minor_comments":[{"comment":"Equation (2.1) is typeset incorrectly: the traditional softmax denominator is not shown as a sum over j, and the temperature-adjusted formula lacks parentheses. Please rewrite it with proper LaTeX.","section":"Eq. (2.1)"},{"comment":"The abstract says the crews 'can effectively collaborate' but the paper never defines success criteria for effectiveness; please state quantitative acceptance criteria before the experiments.","section":"Abstract and §1"},{"comment":"The log transcripts contain many typographical artifacts (e.g., 'spliting', 'ana lys is', 'e xte rn al'), which should be cleaned; the Keywords line also has a double colon.","section":"Logs and general formatting"},{"comment":"No code repository or full prompt templates are provided; consider releasing them in a supplementary artifact to support reproducibility of the claimed workflow.","section":"Reproducibility"},{"comment":"The phrase 'record-high metrics' for the Kaggle benchmark is vague and could be misread as a state-of-the-art claim; please rephrase it as 'high on the leaderboard' or give the specific leaderboard context.","section":"§3.5.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a plausible industry case study, but the central claim of agentic effectiveness needs stronger separation between human and agent contributions. The AI-generated compliance guide footnote is a notable circularity risk and should be addressed openly in revision. I do not see grounds for rejection if the authors are willing to reframe the contribution as a workflow demonstration and add the requested controls and specifications."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a sincere, well-documented demo of CrewAI-based modeling and model risk management crews on three public credit datasets, with honest HITL orchestration and external baselines. It deserves a serious referee, but as a system demonstration, not as evidence that the agents are the ones doing the complex reasoning. The reader's take is about right; the stress-test note also lands, with one caveat: the paper's own framing makes HITL central, so the criticism bites hardest if the title/abstract are read as claiming autonomous agentic capability.\n\nWhat is new: applying the CrewAI multi-agent template to financial modeling plus a separate MRM crew (compliance, replication, conceptual soundness, outcome analysis), with logs of the interactions. The benchmark comparisons to Kaggle and H2O AutoML are fair; they flag leakage in the top fraud Kaggle solution and use a cleaned credit-card approval baseline. The model replication result is reproducible in spirit and a nice touch. They are transparent about the AI-generated modeling guide and about manual verification. The citation pattern looks fine.\n\nSoft spots, in proportion. The main one is attribution. The logs show the human expert specifying the split, dropping Time, the preprocessing recipe (KNN imputation, standard scaler, ordinal encoding, KBest), the tuning protocol (preprocessor, 40% sample, cv=5, roc_auc), and even code-level fixes (missing joblib import, missing predict call). Those are the core modeling decisions, not the 1-10% residual corrections the paper's Section 3.1 suggests. The result is compatible with \"human does ML, agents transcribe and run code.\" The stress-test note is right to call this load-bearing: no ablation or control isolates agent capability. Since the stated design is HITL, this is not fatal to the system, but it undercuts the \"agentic effectiveness\" claim.\n\nSecond, the robustness claim is weak. The perturbation procedure is described only as multiplication/addition of fixed or randomized values; the observed drops (fraud F1 88.9 to 65, credit accuracy 93.4 to 79.1) are presented as robustness without a baseline or statistical comparison. Common ML practice would call those failures, or at least sensitivity findings. This needs a precise spec and/or benchmarks.\n\nThird, the compliance check is circular: the organizational modeling guide was AI-generated for this paper, and the CAG tool compares the crew's documentation against it. The footnote says illustrative, which is honest, but then \"compliance\" is self-referential and cannot support the MRM claim. Missing code/data artifacts and single-run metrics without uncertainty are also there, but those are minor for a demo.\n\nFor whom: practitioners in regulated finance wanting a starting blueprint or a discussion piece on HITL guardrails; not for anyone needing evidence of autonomous agentic problem-solving. Recommendation: send to peer review, conditional, requiring at least: an ablation with the human giving only high-level goals, specified perturbation code/data, an external compliance standard or a clearly illustrative-only framing, and released artifacts.","headline":"A sincere, well-documented HITL system demo that deserves peer review, but the claimed agentic effectiveness is under-supported because the human expert supplies the core modeling decisions.","tokens_in":34192,"tokens_out":2821,"would_cite":false,"duration_ms":28808,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two multi-agent “crews” of LLM specialists, coordinated by a human expert and a judge agent, can carry out financial model development and independent model risk management on standard credit datasets.","keywords":["Large Language Models","Multi-Agent Systems","Agentic Systems","Multi-Agent Collaboration","Human-in-the-Loop","Model Risk Management","Financial Modeling","Credit Risk"],"falsifier":"Run the modeling and MRM crews on a new, unseen financial dataset without any human feedback or code corrections, and compare the autonomously produced metrics to those from a standard AutoML run; if the agents require frequent human fixes, silently produce wrong results, or cannot reproduce their own metrics, the claimed effectiveness and robustness is not supported.","tokens_in":33171,"feed_emoji":"🤖","tokens_out":6185,"duration_ms":57631,"temperature":0.7,"pith_summary":"This paper tries to show that a small team of specialized AI agents—an “agentic crew”—can perform two jobs that banks normally assign to separate groups: building a machine learning model and independently checking that model for risk. The modeling crew runs exploratory data analysis, feature engineering, model selection, training, and evaluation; the model risk management crew replicates the model, checks documentation against a modeling guide, examines conceptual soundness, and stress-tests the model on shifted and adversarial inputs. A human expert orchestrates the workflow, with a judge agent reviewing the other agents’ code and output. On credit card fraud, credit card approval, and portfolio credit risk datasets, the crews produce models whose metrics are comparable to AutoML and Kaggle baselines, while the risk-management tests reveal where the models degrade under distribution shift.","feed_headline":"Two LLM crews automate financial modeling and model-risk review","feed_subtitle":"A human expert orchestrates, a judge agent reviews, and the system matches AutoML baselines on three credit datasets.","key_machinery":"The crews themselves are the central mechanism: modular teams of role-playing LLM agents organized in a hierarchical human-agent-task-tool structure. Each agent carries a persona and backstory, uses a memory stream to pass task context and results to the next agent, and activates specialized tools (Python code execution, EDA analysis, cache-augmented generation for compliance checking); a judge agent audits the others’ code and output, and the human expert delegates tasks, adds instructions, and corrects errors. This combination of role specialization, memory chaining, and human oversight is what the paper credits for keeping outputs aligned with financial modeling standards.","core_discovery":"The central claim is that LLM-powered multi-agent crews, built on a role-playing collaboration framework with a human-in-the-loop as orchestrator, can effectively collaborate on complex financial modeling and model risk management tasks. Each agent has a persona (data analyst, data scientist, machine learning engineer, model validation analyst, or documentation writer), a shared memory stream that chains outputs between stages, and tools such as a Python code executor, an exploratory-data-analysis tool, and a cache-augmented-generation checker. The judge agent reviews the actions and code of the other agents and reports to the human, who can inject feedback and corrections at any step. The paper reports that on the three datasets the agentic pipeline selects competitive models (CatBoost and XGBoost), produces evaluation metrics on par with AutoML and popular Kaggle solutions, and generates model risk reports that expose significant performance drops under shifted inputs while remaining robust to outlier inputs.","pith_inferences":["The same crew architecture could be transplanted to other regulated modeling tasks—insurance pricing, anti-money-laundering screening, or climate-risk scoring—where documentation and independent validation are mandatory.","The compliance-check step is only as authoritative as the modeling guide it checks; because the paper’s organizational guide is AI-generated and illustrative, the demonstration establishes workflow feasibility, not regulatory approval.","As LLM code reliability improves and the judge agent gets better at spotting subtle errors, the human role could shift from checking every artifact to managing exceptions, which would change how much staff time these workflows actually save.","A cost-sensitive deployment test comparing total human audit hours and residual agent failure rates against manual modeling would quantify the practical benefit that this paper demonstrates qualitatively."],"forward_implications":["Financial model development can be compressed into a sequence of delegated agent tasks, with the human expert reviewing handoffs rather than writing every step by hand.","Independent model risk management can be run as a parallel crew, reproducing the model, checking documentation compliance, and stress-testing under shifted and adversarial inputs.","Agentic results are competitive with automated machine learning baselines, so model quality need not be sacrificed for automation.","MRM stress tests can surface real vulnerabilities—for example, accuracy holding while precision and F1 collapse under input shift—making model risk reporting a systematic byproduct of the workflow."],"supporting_citations":[{"why":"CrewAI framework; supplies the agent role, memory, and tool orchestration that the modeling and MRM crews are built on.","marker":"[12]"},{"why":"CrewAI course materials; defines cooperation, guardrails, and role-playing components adopted in the system design.","marker":"[11]"},{"why":"Taxonomy of LLM multi-agent collaboration structures; the crews use the hierarchical and nested collaboration patterns described here.","marker":"[14]"},{"why":"Defines the general components of an LLM agent (core, memory, tools, planning); the architecture section follows this breakdown.","marker":"[58]"},{"why":"Credit card fraud detection dataset; one of the three benchmarks used to demonstrate the modeling and MRM crews.","marker":"[22]"},{"why":"Credit card approval dataset; the second benchmark that the agentic pipeline is applied to.","marker":"[25]"},{"why":"Portfolio credit risk dataset; the third benchmark, with detailed end-to-end agent interaction logs in the appendices.","marker":"[24]"},{"why":"Most-upvoted Kaggle fraud solution that is examined and rejected for data leakage, motivating the paper's different preprocessing and benchmarking choices.","marker":"[23]"},{"why":"Credit card approval Kaggle solution; its ETL approach and results serve as the benchmark the agentic output is compared against.","marker":"[5]"},{"why":"Upvoted Kaggle solution for portfolio credit risk; provides the accuracy benchmark that the agentic and AutoML results are measured against.","marker":"[26]"}],"fun_headline_variants":["Agentic LLM crews handle modeling and risk review in finance","Multi-agent LLM system matches AutoML for financial modeling and MRM","Human-in-the-loop LLM crews automate financial model and risk work"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM agents generate correct-enough Python code and reasoning on their own most of the time, so a human expert can catch the residual failures; the paper itself concedes a 1-10% chance that outputs do not match expectations and notes that the authors manually verified every output, so this reliability assumption has not been tested outside the authors' controlled runs.","fun_headline_variants_meta":{"raw":{"variants":["Agentic LLM crews handle modeling and risk review in finance","Multi-agent LLM system matches AutoML for financial modeling and MRM","Human-in-the-loop LLM crews automate financial model and risk work"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000705,"raw_usage":{"total_tokens":3155,"prompt_tokens":895,"completion_tokens":2260,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":2211}},"tokens_in":511,"tokens_out":2260,"duration_ms":15089,"temperature":1.0,"reasoning_tokens":2211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T19:19:56.073899+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the modeling and MRM crews on a new, unseen financial dataset without any human feedback or code corrections, and compare the autonomously produced metrics to those from a standard AutoML run; if the agents require frequent human fixes, silently produce wrong results, or cannot reproduce their own metrics, the claimed effectiveness and robustness is not supported.","supporting_citations":[],"review_version":1}