{"id":"8b07351f-2cbd-43eb-af78-032d9db83504","arxiv_id":"2505.16229","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CT-Agent combines an LLM planner, region-specific LoRA adapters, and global/local token compression to improve 3D chest CT report generation and question answering on CT-RATE and RadGenome-ChestCT.","lead":"A new AI system called CT-Agent answers questions and writes reports from 3D chest CT scans by splitting the scan into ten anatomical regions, each handled by a specialized model, and compressing hundreds of slices into a compact set of visual tokens. In tests on public chest CT benchmarks it reports large gains over prior methods, though the evaluation compares question answering against only one baseline and does not release code.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central QA comparison is confounded: the test pairs are generated from the same RadGenome GPT template/annotation pool used to train CT-Agent, while the sole baseline LLaVA-CT is not trained on this QA task, so Table 2 may reflect template reproduction rather than clinical superiority.","rationale":"I reviewed the full manuscript. The proposed architecture is coherent, and the ablations in Section 5.3 provide internal evidence that token compression, exemplar retrieval, and region routing each contribute to the system's own performance. However, the central claim that CT-Agent 'consistently outperforms existing methods' rests on two empirical comparisons. The report-generation comparison is weakened by the inconsistent F1 values in Table 1 and the unvalidated 'improved EM' CE metric. The QA comparison is weakened more fundamentally by the distributional asymmetry described above: the test QA pairs are generated from the same RadGenome GPT annotation and template pool on which CT-Agent's anatomy tools and query-rewriting module are trained, while the sole baseline LLaVA-CT is not trained on that QA task. Under these conditions, high F1 could simply reflect successful template reproduction rather than clinically superior understanding. This is the single most load-bearing concern because if it holds, the headline QA gains in Table 2 (0.646 vs 0.589 presence; 0.532 vs 0.393 abnormality) do not establish a general method advantage. A concrete test is to train LLaVA-CT on the same QA pairs and evaluate both on clinician-written held-out questions. If the gap persists, the claim is strengthened; if it collapses, the current comparison is an artifact. The reader's weakest assumption identified the same test-set and metric issue, so I agree, and the conditional verdict remains appropriate pending this check.","tokens_in":16238,"tokens_out":6108,"duration_ms":46224,"concrete_test":"Fine-tune LLaVA-CT on the same 1.91M RadGenome template QA training split used for CT-Agent, then evaluate both systems on a held-out set of naturally phrased clinician questions that were not produced by the template fill-in procedure, with answers adjudicated by a radiologist. If LLaVA-CT reaches or exceeds CT-Agent's F1 once trained on the same QA pairs, or if CT-Agent's margin shrinks materially on non-template questions, the claimed superiority is an evaluation artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of consistent outperformance is not yet supported because the QA evaluation is biased toward CT-Agent's own training distribution. Section 5.1.1 states that the QA pair dataset is constructed by filling unified templates with RadGenome anatomical region names and deriving answers directly from RadGenome GPT annotations, and that this same dataset provides both the 1,914,448 training pairs and the 119,200 test pairs. CT-Agent's anatomy-aware tools are explicitly trained on those pairs (Section 4.3.1), and its inference pipeline rewrites any user question into 'a standardized format aligned with the model's training distribution' (Section 4.3.3) before generating an answer from a standardized template (Appendix A). The only QA baseline, LLaVA-CT, is instead described as fine-tuned directly on CT-RATE CT images and reports (Section 5.1.2), i.e., it was never trained on the RadGenome template QA task being tested. The comparison therefore conflates task-specific fine-tuning and query-template normalization with the agent architecture itself. Additionally, Table 1's CE-F1 values are internally inconsistent with its stated P/R columns (e.g., CT-Agent P=0.423, R=0.477 implies F1 approximately 0.448, not 0.420; several baselines also mismatch), so the report-generation advantage is also not internally reproducible. The overarching concern is that the reported gains may be artifacts of evaluating on the method's own template/annotation distribution rather than on clinically meaningful held-out questions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CT-Agent, a multimodal-LLM agent for 3D chest CT radiology report generation and region-guided question answering. The system combines a planning module built on Deepseek-v3, an action space of ten anatomy-specific LoRA-tuned reasoning tools, hierarchical global-local token compression, memory-guided few-shot exemplar retrieval, and query normalization. The authors evaluate on CT-RATE and RadGenome-ChestCT, reporting that CT-Agent outperforms existing methods on both report generation and question answering, with ablations attributed to planning, token compression, and retrieval components.","tokens_in":16579,"tokens_out":5083,"duration_ms":38550,"significance":"If the claims are validated, CT-Agent would be a useful contribution to 3D medical VQA: the anatomy-aware decomposition is a sensible response to the complexity of chest CT, the token compression strategy addresses a real computational bottleneck, and the use of public datasets with detailed implementation settings supports reproducibility. The ablations are also structured to test each component. However, the evaluation as reported cannot currently support the headline conclusions: the central report-generation table contains internally inconsistent precision/recall/F1 values, the QA benchmark is generated from the same template and GPT-label pool used for training CT-Agent while the sole baseline is not trained on that task, and the adapted Clinical Efficacy metric is not validated. These issues are load-bearing for the abstract's claim of consistent superiority.","major_comments":[{"comment":"The CE precision/recall/F1 triplets in Table 1 are internally inconsistent. For CT-Agent, P=0.423 and R=0.477 imply F1 = 2*0.423*0.477/(0.423+0.477) ≈ 0.448, not the reported 0.420. Similar mismatches appear for CT2Rep (0.355/0.132 implies 0.192, not 0.175) and M3D (0.407/0.009 implies 0.018, not 0.148). Since the paper's report-generation superiority claim rests on these numbers, the table must be corrected and the conclusions recomputed before the clinical-efficacy advantage can be assessed.","section":"Table 1"},{"comment":"The QA evaluation is confounded by training/test distribution overlap. Section 5.1.1 states that the QA pair dataset is constructed by filling unified templates with RadGenome anatomical region names and deriving answers directly from RadGenome GPT annotations, and that the same constructed dataset provides both the 1,914,448 training pairs and the 119,200 test pairs. CT-Agent is explicitly trained on these pairs (Section 4.3.1), and its query-rewriting tool reformulates user questions into 'a standardized format aligned with the model's training distribution' (Section 4.3.3) before final answers are generated from a standardized template (Appendix A). The only QA baseline, LLaVA-CT, is described as fine-tuned directly on CT-RATE CT images and reports (Section 5.1.2), not on this QA task. Table 2 therefore conflates task-specific fine-tuning and query-template normalization with the agent architecture itself. Please retrain LLaVA-CT (and ideally a second baseline) on the same QA pairs, or evaluate on independently written, human-annotated questions with varied phrasing, and additionally report what fraction of outputs exactly match the templates.","section":"Sections 5.1.1, 4.3.1, 4.3.3, Appendix A"},{"comment":"The 'improved Exact Match' adaptation of CT2Rep's Clinical Efficacy metric is introduced without validation. There is no comparison against the original CE metric, no clinician agreement study, and no analysis of how the lenient EM matching behaves on template-generated QA pairs. Given the very large QA gains in Table 2 (for example, Trachea & bronchi abnormality F1 rising from 0.018 for the baseline to 0.790 for CT-Agent), it is important to show that the metric is not merely rewarding template-consistent outputs. Please validate the metric on a human-annotated subset and report per-abnormality precision/recall rather than only a micro-averaged aggregate.","section":"Section 5.1.3"},{"comment":"No confidence intervals, standard deviations, or significance tests are reported anywhere in the experiments. Several reported differences are very small (e.g., BLEU-3 +0.004 and METEOR +0.004 over the best baseline in Table 1), and some Table 2 region-level comparisons go in the opposite direction (e.g., Lung presence precision 0.973 vs. 0.962, Thyroid abnormality F1 0.540 vs. 0.453). The abstract's 'consistently outperforms' claim is therefore not statistically supported. At minimum, report variance or bootstrap confidence intervals for the main metrics, and significance tests for the differences that drive the conclusions.","section":"Tables 1-4"}],"minor_comments":[{"comment":"The affiliation line contains a typo: 'Hanghou' should be 'Hangzhou'.","section":"Author affiliation"},{"comment":"The caption uses 'BLUE-4' instead of 'BLEU-4'; please correct this typo.","section":"Table 1 caption"},{"comment":"Notation is inconsistent: the global aggregated token matrix is called Z_f in Eq. (6) but Z_global in Eq. (11); please align the symbols.","section":"Equations (6) and (11)"},{"comment":"The query-rewriting prompt includes a size-estimation template ('What is the approximate size of the {abnormality} in the {region}?'), but Section 5.1.1 says the QA templates are only presence detection and abnormality identification; please clarify whether size questions are part of the evaluation set or only an auxiliary prompt template.","section":"Appendix A"},{"comment":"The table reports overall and per-region metrics but no sample sizes per region or per task; please add the number of test questions for each anatomical region and task type.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a medical-imaging or vision-language venue, and the architecture is plausible. The two blocking issues are the internally inconsistent CE table and the QA benchmark contamination; both are fixable with additional experiments and corrected reporting. I would not accept the current version, but I would be willing to review a revised version that addresses these points, including retraining the QA baseline on the same training distribution and validating the adapted metric."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper delivers a genuinely new system: an LLM agent with ten anatomy-specific LoRA tools, a global-local token compression scheme, and prediction-guided exemplar retrieval. The ablations are informative and show each component earns its place. The report-generation comparison against CT2Rep, MS-VLM, and others with standard NLG metrics is a fair and useful benchmark, and the gains on CE-F1, though based on an unvalidated metric, are at least computed against the same metric for all baselines. The soft spot is the QA evaluation, and it is substantial. The test QA pairs are built from the same RadGenome templates and GPT annotation pool used to train the anatomy tools, and the sole QA baseline, LLaVA-CT, was never trained on that QA task. CT-Agent also rewrites user questions into a format explicitly aligned with its training distribution. So Table 2 likely measures template reproduction and distribution overlap more than clinical superiority. That is a real confound, and the stress-test note is right. The fix is straightforward: evaluate on held-out questions that are not templated from the training pool, add a QA baseline that is fine-tuned on the test distribution (or at least on the same templates), and report per-region confidence intervals. A second, smaller issue: Table 1's CE-F1 values are internally inconsistent with the reported P and R. For CT-Agent, P=0.423, R=0.477 gives F1 about 0.448, not the stated 0.420. Some baselines match, some do not. That is sloppy and needs a check. No error bars, no code release, and the improved Exact-Match CE metric is not validated against the original CT2Rep CE, so the exact magnitudes should be treated as provisional. These are addressable problems, not a load-bearing flaw in the architecture. The report-generation evidence is credible enough that the paper deserves referee time, with the understanding that the QA claim needs reworking. I would not desk reject it. I also would not cite it yet; wait for a corrected version with an honest QA evaluation.","headline":"The architecture and ablations are solid and worth building on, but the QA evaluation is biased by train/test template overlap and a single untrained baseline, so the headline QA gains should not be taken at face value.","tokens_in":641,"tokens_out":1906,"would_cite":false,"duration_ms":24737,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"3D chest CT agent beats baselines on reports and questions","keywords":["CT radiology question answering","3D chest CT","multimodal large language model","LLM agent","visual question answering","token compression","LoRA fine-tuning","radiology report generation"],"falsifier":"Run CT-Agent and LLaVA-CT on a held-out set of QA pairs written by radiologists rather than generated from templates, and have radiologists score the answers blind; if CT-Agent's F1 advantage over LLaVA-CT shrinks to near zero, the gains are artifacts of template overlap rather than better clinical reasoning.","tokens_in":16055,"feed_emoji":"🩻","tokens_out":7336,"duration_ms":56773,"temperature":0.7,"pith_summary":"CT-Agent is a multimodal LLM agent designed to answer questions about 3D chest CT volumes and to generate radiology reports. The paper argues that the two obstacles to good CT question answering are anatomical complexity and the difficulty of capturing relationships across hundreds of slices, and that both can be addressed by giving the agent a set of anatomy-specific tools. Each of ten anatomical regions gets its own lightweight LoRA adapter on a shared vision-language backbone, while a global-local token compression scheme reduces roughly 240 slices to about 64 visual tokens per slice without dropping the information needed for cross-slice reasoning. On CT-RATE and RadGenome-ChestCT the system reports clinical-efficacy F1 of 0.420 for report generation versus 0.261 for the best baseline, and QA F1 of 0.646 and 0.532 for presence and abnormality tasks versus 0.589 and 0.393 for LLaVA-CT. A sympathetic reader would take the paper to establish that decomposing a 3D medical volume by anatomy, rather than feeding the whole volume to one model, is a workable path to clinically useful CT question answering.","feed_headline":"3D chest CT agent beats baselines on reports and questions","feed_subtitle":"Per-region LoRA tools and token compression lift clinical F1 to 0.420, far above the best baseline's 0.261.","key_machinery":"The load-bearing object is the agent's action space: ten region-specific LoRA reasoning tools mounted on one frozen multimodal LLM backbone, each built on a two-stage hierarchical token compression. Global Token Aggregation applies a token-wise mixture-of-experts and slice-wise averaging to fold 240 slices into a compact global representation; Local Token Selection keeps the top attention-dominant tokens and merges the rest by key similarity, yielding 54 dominant plus 10 contextual tokens per slice. A planning module driven by an LLM classifies the task, identifies the anatomical region, rewrites free-form questions into templates, and routes the compressed visual tokens to the right LoRA plugin, with a prediction-guided retrieval mechanism pulling similar exemplar reports from memory. Together these components reduce the token count by roughly 75% while preserving semantic integrity, and that reduction is what makes whole-volume 3D reasoning feasible inside an LLM context window.","core_discovery":"The paper's central claim is that CT-Agent consistently outperforms existing methods on both radiology report generation and region-guided question answering for 3D chest CT, on two public datasets. In report generation it reports a clinical-efficacy F1 of 0.420, with recall of 0.477, against a best baseline F1 of 0.261 (MS-VLM) and 0.221 for LLaVA-CT, while also improving most NLG metrics except BLEU-4. For question answering, over ten anatomical regions, it reports overall presence-detection F1 of 0.646 versus 0.589 for LLaVA-CT and abnormality-detection F1 of 0.532 versus 0.393. The authors attribute the gains to three mechanisms working together: the planning module that routes each query to the relevant anatomy, the ten region-specific LoRA plugins that specialize without retraining the backbone, and the dual-path token compression that keeps global context while trimming local redundancy. The paper frames the result as showing that anatomy-aware, agent-style reasoning is superior to treating the volume as a homogeneous input.","pith_inferences":["Because the test QA pairs are built from the same template-and-annotation scheme used to train the region tools, part of the measured gain likely reflects the query-rewriting step aligning user questions to the training distribution; a reader should not conclude that the agent reasons better on genuinely out-of-distribution questions.","The exact-match clinical-efficacy metric rewards the presence of predefined abnormality terms, so a system that produces template-consistent but clinically incomplete reports could score well, making a clinician-rated evaluation the natural next test.","The anatomy-first recipe is not tied to CT: the same planner-plus-LoRA-tools structure could be applied to MRI or whole-body PET by replacing the region taxonomy and the predefined query pool.","If the gains are real, the practical bottleneck for deploying such an agent shifts from model capacity to the quality and coverage of the region taxonomy and question templates."],"forward_implications":["CT-Agent handles both radiology report generation and region-guided question answering through one planning module, so a single system can serve both clinical workflows.","Routing each query to a region-specific LoRA plugin removes anatomical-confusion errors, such as mistaking mediastinum findings for lung findings.","Compressing each of 240 slices to 64 tokens cuts the visual token budget by roughly 75%, making whole-volume 3D input feasible for a frozen vision-language backbone.","Prediction-guided exemplar retrieval improves report BLEU, ROUGE-L, and METEOR over static few-shot and zero-shot variants, indicating that memory-grounded prompting is part of the gain.","The modular design means existing regions can be improved or new regions added by training only that region's LoRA adapter, without retraining the shared backbone."],"supporting_citations":[{"why":"Supplies the CT-RATE dataset used for training and testing, including the 50k chest CT volumes and reports that ground the report-generation experiments.","marker":"[27]"},{"why":"Supplies RadGenome-ChestCT, the source of the ten-region masks, GPT-generated pathological labels, and the QA pairs used to train and test the region tools.","marker":"[28]"},{"why":"CT2Rep is the main report-generation baseline and the origin of the Clinical Efficacy metric that the paper adapts into the exact-match CE scorer.","marker":"[13]"},{"why":"MS-VLM is the strongest report-generation baseline on CE-F1, the number the paper's 0.420 result is compared against.","marker":"[18]"},{"why":"LLaVA-Med is the frozen vision-language backbone on which all region-specific LoRA plugins are built and is also the basis of the LLaVA-CT baseline.","marker":"[6]"},{"why":"LoRA is the parameter-efficient fine-tuning method used to create the ten lightweight, region-specific adapters.","marker":"[32]"},{"why":"VisionZip motivates the dominant-token selection and contextual merging used in the local token compression pathway.","marker":"[34]"},{"why":"CLIP ViT-B/16 is the frozen vision encoder that produces the slice-level visual tokens fed into the compression module.","marker":"[31]"}],"fun_headline_variants":["Anatomy-savvy CT agent tops baselines in QA and reports","CT-Agent: A multimodal LLM that reads 3D CT scans better","Per-region tools boost CT question answering and reports","Agent with anatomy tools outdoes monolithic CT models","3D CT agent beats baselines on chest scans and QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported superiority rests on the assumption that the test questions and the exact-match clinical metric actually measure clinically meaningful performance, because the test QA pairs come from the same template-and-annotation scheme used for training and the adapted CE metric has not been validated against radiologist judgment.","fun_headline_variants_meta":{"raw":{"variants":["Anatomy-savvy CT agent tops baselines in QA and reports","CT-Agent: A multimodal LLM that reads 3D CT scans better","Per-region tools boost CT question answering and reports","Agent with anatomy tools outdoes monolithic CT models","3D CT agent beats baselines on chest scans and QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000619,"raw_usage":{"total_tokens":2891,"prompt_tokens":983,"completion_tokens":1908,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":1831}},"tokens_in":599,"tokens_out":1908,"duration_ms":11535,"temperature":1.0,"reasoning_tokens":1831,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:04:30.333250+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CT-Agent and LLaVA-CT on a held-out set of QA pairs written by radiologists rather than generated from templates, and have radiologists score the answers blind; if CT-Agent's F1 advantage over LLaVA-CT shrinks to near zero, the gains are artifacts of template overlap rather than better clinical reasoning.","supporting_citations":[{"cited_title":"Ct2rep: Automated radiology report generation for 3d medical imaging","cited_arxiv_id":null,"evidence_quote":"CT2Rep is the main report-generation baseline and the origin of the Clinical Efficacy metric that the paper adapts into the exact-match CE scorer."},{"cited_title":"Llava-med: Training a large language-and-vision assistant for biomedicine in one day","cited_arxiv_id":null,"evidence_quote":"LLaVA-Med is the frozen vision-language backbone on which all region-specific LoRA plugins are built and is also the basis of the LLaVA-CT baseline."}],"review_version":1}