{"id":"a9290d4f-6ffa-4781-92f1-f501e8522d0d","arxiv_id":"2507.00993","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A 3D ResNeSt50 model with slice removal and weighted cross-entropy achieves a Macro F1 of 0.80 on the Fair Disease Diagnosis Challenge validation set.","lead":"The paper reports a deep-learning system that processes whole 3D chest CT volumes and classifies them into four lung conditions, achieving a Macro F1 of 0.80 on the Fair Disease Diagnosis Challenge validation set. The method is a routine combination of existing components, and the report is a benchmark entry rather than a new scientific result.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.80 Macro F1 is a single uncontextualized validation number; with 25 rare-class cases and no comparator or error bars, it cannot support 'strong performance.'","rationale":"The reader's weakest assumption identifies the validation-set Macro F1 as the fragile link, and I agree: the paper's only quantitative evidence is a single 0.80 score on a small validation set with a rare class. My concern is not that the number is impossible or that the methodology is internally inconsistent—the pipeline is standard and the preprocessing is described at a reasonable level for a challenge report. The problem is that the conclusion is load-bearing on an unquantified estimate. A change of a few correct/incorrect predictions in the 25-case squamous-cell class would move the macro average by a non-negligible amount, and no confidence interval, per-class breakdown, or seed variation is reported. The 'strong performance' characterization is also relative, yet no baseline or leaderboard comparison is provided. This is an absence of evidence rather than evidence of error, so the appropriate response is to require the missing stability and comparison evidence, matching the reader's CONDITIONAL verdict. No new concern stronger than the reader's was found, so the verdict remains unchanged.","tokens_in":4000,"tokens_out":5041,"duration_ms":138736,"concrete_test":"Have the authors rerun the complete training pipeline with five different random seeds on the fixed validation split and report per-class and Macro F1 for each run, using the official challenge evaluation script. If the range of Macro F1 across seeds is wider than about 0.04, or if the squamous-cell carcinoma F1 varies by more than about 0.1, the single reported 0.80 is not a stable basis for the 'strong performance' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The entire central claim rests on one reported number: Macro F1 = 0.80 on the validation set (§4.3). The load-bearing assumption is that this number is stable and meaningful. The validation set has only 25 squamous-cell carcinoma cases (13 female, 12 male) out of 155 volumes, so the per-class F1 for that class is a small-sample estimate; the macro average over four classes is a direct function of that unstable component. The paper reports no per-class F1, no confidence interval, no number of runs or seeds, no ablation of the slice-removal or weighted-loss components, and no independent test set. The word 'strong' additionally presupposes a comparison—against baselines, other challenge entries, or a published benchmark—that is never supplied. As written, the conclusion cannot be separated from the particular validation split and one training run.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a three-stage pipeline for four-class lung disease classification from 3D chest CT volumes: removal of non-lung slices, a 3D ResNeSt50 backbone, and a weighted cross-entropy loss intended to counteract class imbalance. The reported result is a Macro F1 of 0.80 on the validation split of the Fair Disease Diagnosis Challenge, and the paper states that this demonstrates strong performance. The evaluation consists of a single aggregate number, with no held-out test set, no per-class results, no confidence intervals, no baselines, and no ablations.","tokens_in":4118,"tokens_out":3541,"duration_ms":42875,"significance":"The proposed combination of components is plausible and the paper provides useful implementation details (volume resizing, augmentation, optimizer settings, batch size, epoch count) that would help others reproduce the pipeline. If the 0.80 Macro F1 were shown to be stable and favorably compared against alternatives, the recipe would be of practical interest for challenge-style lung-disease classification. However, the current evidence is too thin to support the stated conclusion: a single validation-set number, computed on a small cohort with a rare class, cannot by itself demonstrate 'strong performance' without error bars, per-class breakdowns, or comparators.","major_comments":[{"comment":"The central claim rests entirely on a single Macro F1 of 0.80 computed on the validation set of 155 volumes. With only 25 squamous cell carcinoma cases (13 female, 12 male) in that split, the per-class F1 for the rare class has wide sampling uncertainty, and the macro average is directly sensitive to that component. The paper reports no confidence interval, no per-class F1 values, and no number of training runs or seeds, so the reader cannot judge whether 0.80 is a stable estimate or a fluctuation of a particular split and initialization.","section":"§4.3 and Table 1"},{"comment":"The word 'strong' presumes a comparison that is never supplied. No baseline architecture, no variant without slice removal, no variant without weighted loss, and no reference to other challenge entries or published benchmarks are provided. Without a comparator, 0.80 is an uncontextualized number; adding even a small set of ablations (e.g., plain 3D ResNet, unweighted cross-entropy, or no slice removal) would make the claim falsifiable and interpretable.","section":"§4.3"},{"comment":"The weighted cross-entropy loss is presented as a contribution, but the class weights w_c are never specified. The reader cannot tell how much weight was given to squamous cell carcinoma or whether the weights were derived from the class frequencies in Table 1. Moreover, there is no ablation isolating this component, so the paper does not provide evidence that the weighting, rather than the backbone or the preprocessing, contributes to the reported 0.80.","section":"§2, Eq. (1)"},{"comment":"The slice-removal preprocessing is described only qualitatively. There is no specification of how non-lung slices are identified, how many slices are removed on average, or whether the threshold is fixed or scan-dependent. This matters because the method assumes that removed slices never contain diagnostically relevant tissue; if the rule occasionally removes apical or basal slices containing lesions, the evaluation could be biased. At minimum, the manuscript should quantify the removed slice range and validate the assumption on the training set.","section":"§4.1 and §4.2"}],"minor_comments":[{"comment":"The figure labels 'ResNestBlock' and 'ResNest' use a different capitalization from the text's 'ResNeSt50'; please make the notation consistent throughout.","section":"Figure 2"},{"comment":"The notation y, ŷ, y_c, and ŷ_c is not defined in detail; please state explicitly that y_c is the one-hot ground-truth component and ŷ_c is the predicted probability for class c.","section":"Eq. (1)"},{"comment":"The caption uses abbreviations 'A', 'Covid', and 'G' without explanation; the table would be clearer if the caption defined that A stands for adenocarcinoma, G for squamous cell carcinoma, and Covid for COVID-19.","section":"Table 1"},{"comment":"The augmentation description 'random cropping on the vertical section to 64' is ambiguous given that volumes are already resized to depth 64 in §4.1; please clarify whether the crop is applied to the original depth before resizing or is a depth-preserving random crop.","section":"§4.2"},{"comment":"The paper does not state whether code or trained models will be made available; adding a code-availability statement would improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This is a very short challenge-style report whose evidence is a single validation number. The main weakness is not an internal inconsistency but a lack of experimental support for the headline claim. I believe the paper could be made acceptable after a major revision that adds per-class results, uncertainty estimates, baselines, and ablations, or alternatively by repositioning the paper as a brief challenge contribution without the strong comparative claim. Given the journal context, major revision seems more appropriate than outright rejection because the proposed pipeline is concrete and the reported result may be reproducible with the provided hyperparameters."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a challenge-participation write-up that reports one number — Macro F1 0.80 on the Fair Disease Diagnosis Challenge validation set — with no baseline, no ablation, no per-class breakdown, and no error bars. The method itself is a sensible composition of standard pieces: 3D volume input, removal of non-lung slices, ResNeSt50 backbone, class-weighted cross-entropy. That is fine for its apparent purpose as a workshop report, but the paper does not earn the 'strong performance' claim.\n\nWhat's actually new: a single validation-set measurement on a public challenge dataset. That is a legitimate data point, and the preprocessing choice (dropping neck/abdomen slices) is a reasonable and common trick to cut compute and focus the model. The use of weighted cross-entropy for the rare squamous cell carcinoma class is appropriate. The writing is clear and the methodology is reproducible in principle, though no code or data are provided.\n\nThe soft spots: the whole conclusion rests on 0.80 on 155 validation volumes, with only 25 squamous cell carcinoma cases. The macro F1 is directly sensitive to that small class, and no confidence interval or per-class F1 is given. There is no comparison to a trivial baseline (e.g., majority class, 2D slice-based model, or a plain ResNet), so 'strong' is unsubstantiated. The loss weighting is described as 'lesion-aware' in the intro but Equation (1) is plain class-weighted cross-entropy, which is a small mismatch. Also, tuning preprocessing and weights on the validation set without a separate test set means the reported number is at risk of being optimistic; the authors do not mention using a held-out split.\n\nThe citation pattern is fine — lots of self-citations but that's typical in this subfield, and the COVID-19 detection work they cite is relevant.\n\nWho this is for: someone tracking challenge entries or looking for a quick baseline reference on this dataset. It does not advance methodology. A serious referee could justify spending time on it if the venue accepts challenge-participation notes, but as a standalone research contribution it is thin. My recommendation: if this comes to a journal, treat it as a workshop note; require at least one baseline and per-class results before accepting, or accept as a short report with appropriate caveats. For a main-track conference, desk reject is reasonable.","headline":"Single uncontextualized validation score with no baselines or error bars; the method is a competent composition of standard components, but 'strong performance' is not supported.","tokens_in":4703,"tokens_out":2243,"would_cite":false,"duration_ms":24140,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Four-way lung CT classification reaches 0.80 Macro F1 with a 3D ResNeSt model.","keywords":["lung disease classification","3D CT scans","ResNeSt50","weighted cross-entropy loss","class imbalance","Macro F1","Fair Disease Diagnosis Challenge","computed tomography"],"falsifier":"Re-run the same pipeline on multiple random splits of the provided validation set, or obtain the challenge test labels and compute per-class F1; if the Macro F1 drops below roughly 0.75 on any split, or the squamous cell carcinoma F1 confidence interval overlaps the majority-class baseline, the paper's 'strong performance' conclusion fails.","tokens_in":3745,"feed_emoji":"🫁","tokens_out":4419,"duration_ms":45414,"temperature":0.7,"pith_summary":"The paper proposes a complete-3D approach to classifying chest CT scans into four categories: normal, adenocarcinoma, squamous cell carcinoma, and COVID-19. It claims that cutting away slices that contain no lung tissue, feeding the remaining volume through a 3D ResNeSt50 network, and weighting the cross-entropy loss toward rare classes yields a Macro F1 of 0.80 on the validation set of the Fair Disease Diagnosis Challenge. The authors present this as evidence that 3D volumes, rather than stacks of independently analyzed 2D slices, can support automated lung disease triage.","feed_headline":"Lung CT pipeline reaches 0.80 Macro F1 across four classes","feed_subtitle":"Removing non-lung slices and reweighting rare classes lets one 3D network separate cancer subtypes from COVID-19.","key_machinery":"The load-bearing machinery is the combination of three components: (1) a preprocessing step that detects and removes non-lung slices from each 3D volume, cutting computation and focusing the model on lesions; (2) a 3D ResNeSt50 backbone, a residual network whose blocks include split-attention that weights interactions across feature-map groups; and (3) a weighted cross-entropy loss $L_{\\text{wCE}} = -\\sum_{c=1}^{C} w_c \\, y_c \\log(\\hat{y}_c)$, where class weights $w_c$ are chosen to up-weight underrepresented classes such as squamous cell carcinoma. Together these allow the whole volume, not individual 2D slices, to drive the classification decision.","core_discovery":"On the paper's own terms, the discovery is a pipeline: after resizing each CT series to (64,256,256) and clipping intensities to [0,1], the model discards slices without lung tissue, extracts features with a 3D ResNeSt50 backbone whose split-attention blocks let feature-map groups interact, and trains with a lesion-aware weighted cross-entropy loss. The reported outcome is a Macro F1 score of 0.80 on the challenge validation set, where Macro F1 averages per-class F1 across normal, adenocarcinoma, squamous cell carcinoma, and COVID-19. The paper treats this number as showing the model's strong ability to distinguish lung conditions.","pith_inferences":["The 0.80 Macro F1 is reported without error bars or a per-class breakdown; with only 25 squamous cell carcinoma validation cases, that rare-class F1 likely has a wide confidence interval, so the aggregate score may not be stable across splits.","The non-lung slice removal is a fixed heuristic; a model that learns which slices matter from data, or one tested across scanners with different fields of view, would reveal whether the heuristic itself is essential.","A direct comparison pairing this pipeline with the same ResNeSt50 trained on raw full volumes would isolate how much of the gain comes from preprocessing versus the backbone or the loss weighting."],"forward_implications":["If the 0.80 Macro F1 holds on an independent test set, the same preprocessing and weighting recipe can be carried over to other volumetric medical classification tasks with rare classes.","Removing non-lung slices lowers the input dimensionality, so the model can train on fewer compute resources (the paper uses four RTX 3090 GPUs) than a full-volume model.","The weighted loss directly targets the squamous cell carcinoma class, so per-class F1 for that category should be the first number to inspect in any replication.","Treating CT as a single 3D volume instead of a bag of 2D slices is claimed to avoid the suboptimal performance of slice-based approaches."],"supporting_citations":[{"why":"Supplies the ResNeSt50 backbone architecture whose split-attention blocks serve as the feature extractor.","marker":"[21]"},{"why":"The Adam optimizer used for training, part of the implementation that produces the reported score.","marker":"[8]"},{"why":"Prior 3D CT COVID-19 classification work by the same group that this paper extends to a four-class setting.","marker":"[19]"},{"why":"Related 3D CT domain-adaptation method that provides comparison context for the proposed pipeline.","marker":"[20]"},{"why":"Earlier 3D chest CT COVID-19 detection baseline that motivates treating volumes as whole 3D inputs.","marker":"[12]"}],"fun_headline_variants":["Lung CT diagnosis hits 0.80 Macro F1 with ResNeSt50","3D CT lung pipeline reaches 0.80 Macro F1, four classes","ResNeSt50 and weighted loss improve lung CT to 0.80 F1","Lung disease CT: non-lung removal plus reweighting yields 0.80 F1","0.80 Macro F1 for lung CT: ResNeSt50 and slice filtering"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative claim rests on the validation-set Macro F1 being stable and meaningful, but with 155 volumes, only 25 in the rarest class, and no error bars, per-class breakdown, or separate test set, the 0.80 score could shift materially with the split or the official F1 aggregation rule.","fun_headline_variants_meta":{"raw":{"variants":["Lung CT diagnosis hits 0.80 Macro F1 with ResNeSt50","3D CT lung pipeline reaches 0.80 Macro F1, four classes","ResNeSt50 and weighted loss improve lung CT to 0.80 F1","Lung disease CT: non-lung removal plus reweighting yields 0.80 F1","0.80 Macro F1 for lung CT: ResNeSt50 and slice filtering"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1451,"prompt_tokens":781,"completion_tokens":670,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":397,"completion_tokens_details":{"reasoning_tokens":557}},"tokens_in":397,"tokens_out":670,"duration_ms":7256,"temperature":1.0,"reasoning_tokens":557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:00:54.735029+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same pipeline on multiple random splits of the provided validation set, or obtain the challenge test labels and compute per-class F1; if the Macro F1 drops below roughly 0.75 on any split, or the squamous cell carcinoma F1 confidence interval overlaps the majority-class baseline, the paper's 'strong performance' conclusion fails.","supporting_citations":[{"cited_title":"Resnest: Split-attention networks","cited_arxiv_id":null,"evidence_quote":"Supplies the ResNeSt50 backbone architecture whose split-attention blocks serve as the feature extractor."},{"cited_title":"Advancing covid-19 de- tection in 3d ct scans","cited_arxiv_id":null,"evidence_quote":"Prior 3D CT COVID-19 classification work by the same group that this paper extends to a four-class setting."},{"cited_title":"Domain adaptation us- ing pseudo labels for covid-19 detection","cited_arxiv_id":null,"evidence_quote":"Related 3D CT domain-adaptation method that provides comparison context for the proposed pipeline."},{"cited_title":"Mia-cov19d: Covid-19 detection through 3-d chest ct image analysis","cited_arxiv_id":null,"evidence_quote":"Earlier 3D chest CT COVID-19 detection baseline that motivates treating volumes as whole 3D inputs."}],"review_version":1}