{"id":"99f1fcb6-5a73-46ef-898a-a330ed20845a","arxiv_id":"2411.08992","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"IDCIA is a new public immunocytochemistry image dataset with dot annotations across seven antibody stains, and five existing counting models all fail to reach expert-level accuracy on it.","lead":"The authors release IDCIA, a new set of 262 fluorescence microscopy images of rat neural stem cells stained with seven different antibodies, with every cell marked and counted. The paper also runs five deep learning counting models on the set and finds that none meets biologists' 5 percent accuracy bar.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated manual annotations are the load-bearing weak point: no inter-annotator agreement or expert recount is reported, so the dataset's gold standard and the 'no model is acceptable' conclusion are not yet established.","rationale":"The reader and I identify the same weak point: manual annotations are the sole ground truth for both the dataset and the benchmark conclusion. I looked for other places where the argument might break. The novelty claim about covering seven antibodies is plausible in light of Table 1, which lists only single-stain counting datasets. The five baseline models and the training protocol are standard and the code and trained models are publicly released, which is real supporting evidence. The ACP metric is clearly defined but depends entirely on label accuracy. The absence of inter-annotator agreement is therefore the decisive gap. I would not move the verdict: CONDITIONAL is appropriate because the issue is specific and addressable by a small re-annotation study. If such a study shows expert counts agree within 5%, the dataset claims would be substantially strengthened. If it shows disagreement, the conclusions about model inadequacy would need to be re-derived with uncertainty intervals and possibly corrected labels.","tokens_in":11193,"tokens_out":3545,"duration_ms":38268,"concrete_test":"Have an expert in fluorescent cell image analysis independently recount a stratified random sample of at least 30 IDCIA images (covering all seven antibodies and the full cell-density range) using the same ImageJ Cell Counter protocol, blinded to the original dot files. Compute per-image absolute percent count differences and dot-level agreement. If the mean absolute percent difference exceeds 5%, or if re-evaluating the five models on expert-corrected test labels shifts any ACP value by more than 5 percentage points, then the unvalidated gold standard is a real threat to the central claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central value claim is that IDCIA is a reliable annotated dataset and that existing models cannot count accurately enough (ACP at most 17%, Section 5, Table 5). Everything hangs on the ground truth from Section 3.1: 'manual annotation was performed by a group of undergraduate students led by a graduate student with more than three years of experience in cellular image analysis' using the ImageJ Cell Counter tool. No inter-annotator agreement, no expert re-count, and no quality-control statistic is reported anywhere; Section 3.2 only describes extracting dot coordinates by color thresholding. The statement that imaging and counting were performed blind guards against bias about the stimulation condition, but it does not validate counting accuracy. The ACP metric (Eq. 3) is defined as predictions within 5% of the manual count, so if the manual counts are themselves off by more than 5% on a meaningful fraction of images, then the reported ACP values (0-17%) and the conclusion that manual counting cannot be replaced are not trustworthy. Table 6's per-antibody MAEs are also single-point estimates from five runs without confidence intervals, but the more fundamental issue is the unvalidated reference standard. This does not undermine the dataset's existence or its potential usefulness, but it means the benchmark claims rest on an assumption that the paper has not tested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces IDCIA, a new annotated fluorescence microscopy dataset of rat adult hippocampal progenitor cells, containing 262 images at 800x600 resolution, stained with DAPI and six primary antibodies (TuJ1, MAP2ab, RIP, GFAP, Nestin, Ki67), with per-image cell counts and dot-annotation coordinates. The authors describe the data collection and annotation pipeline, the dataset structure, suggested metrics (MAE, RMSE, and a new ACP metric based on a 5% acceptable-error threshold), and baseline experiments with five deep-learning counting models (CNN regression, CSRNet, MCNN, Count-ception, FCRN-A) using a stratified 60:20:20 train/validation/test split, grid search for hyperparameters, and five training runs per model. The main claims are that IDCIA covers more staining methods than existing cell-counting datasets and that none of the five evaluated models achieves acceptable counting accuracy, with the best ACP at 17% (CSRNet) and the best MAE at 15.47 (Count-ception). The dataset, code, and trained models are publicly available.","tokens_in":11578,"tokens_out":3953,"duration_ms":32715,"significance":"If the ground-truth annotations are valid, IDCIA is a useful new benchmark for cell counting under diverse immunolabeling conditions, and the paper provides a reproducible evaluation protocol with public data and code. The finding that existing counting models fail the proposed ACP criterion would be a valuable cautionary result for the community. However, the central benchmark conclusion currently rests on an unvalidated manual annotation process: no inter-annotator agreement or expert re-count is reported, and the reported performance metrics lack uncertainty estimates. These issues are addressable and do not negate the dataset's existence, but they must be fixed before the paper's main claims can be accepted.","major_comments":[{"comment":"The ground-truth cell counts are the output of manual dot annotation by a group of undergraduate students, and the manuscript reports no inter-annotator agreement, no expert re-count, and no quantitative quality-control statistic. Because ACP (Eq. 3) is defined as the fraction of predictions within 5% of the manual count, unvalidated manual counts directly undermine the headline result that the best model reaches only 17% ACP and the conclusion that no model can replace manual counting. The statement that imaging and counting were performed blind guards against bias about stimulation condition, but it does not validate counting accuracy. I recommend reporting inter-annotator agreement (e.g., count-level ICC or Cohen's kappa on a subset), an expert re-count of a random sample, or an explicit error-propagation analysis showing how annotation noise affects ACP.","section":"Section 3.1 / Eq. (3) / Table 5"},{"comment":"Tables 4 and 6 report only average MAE over five runs without standard deviations or confidence intervals, and Table 5 reports ACP as a single point per model with no uncertainty. With only five runs and small test partitions, the reader cannot assess whether the ranking among models is stable. For example, Table 6's per-antibody MAEs are averaged over at most a handful of test images (the GFAP antibody has 23 images total, so about 4-5 test images under the 60:20:20 split). Please report mean ± std or 95% confidence intervals for all metrics and explicitly state the number of test images per antibody subset.","section":"Tables 4, 5, and 6"},{"comment":"The per-antibody baseline results are based on very small test sets: antibodies with 23-25 images contribute only roughly 4-5 test images each. Consequently, the claims that CNN Regression is best on RIP and GFAP and that MCNN is best on Ki67 are not statistically supported; a single image can shift the MAE substantially. I recommend either reporting confidence intervals, aggregating antibodies with small sample sizes, or clearly labeling these results as preliminary observations rather than definitive rankings.","section":"Section 5 / Table 6"},{"comment":"The 5% acceptable-error threshold used in the ACP metric is described as 'the experts' acceptable error rate' without citation or derivation. Since ACP is the metric used to conclude that all models are unacceptable, this threshold is load-bearing. Please justify the 5% value with a reference or a domain-expert survey, and report sensitivity to the threshold (e.g., ACP at 10% and 15%) so readers can judge how robust the conclusion is.","section":"Section 4 / Eq. (3)"}],"minor_comments":[{"comment":"The phrase 'protein components of immune responses against invaders' is an imprecise definition of antibodies; consider simplifying to 'proteins used to detect specific markers' or similar.","section":"Abstract"},{"comment":"The sentence describing scaffolds contains a repetition: '...scaffolds, which are structures providing support for cells to grow within an interdigitated electrode region. Then the voltage is applied to the electrode pads of the scaffold, which are structures providing support for cells to grow.' Please remove the duplicated description.","section":"Section 1, paragraph 3"},{"comment":"References [4] and [26] both cite the same Count-ception paper (Cohen et al., ICCVW 2017); one entry should be removed or the citations merged.","section":"References"},{"comment":"The affiliation line for Surya K. Mallapragada contains an extraneous 'pilcrow' symbol, likely a LaTeX error, and the contact email 'abdu@iastate.com' should probably be 'abdu@iastate.edu'.","section":"Author affiliations"},{"comment":"The caption labels rows inconsistently: 'Row 1 (A-C)' but then describes DAPI in (B and C), while 'Row 2 (D-E)' is used for the dot-annotated images; please clarify the panel layout and the color assignments.","section":"Figure 2"},{"comment":"When the ground-truth count y_i is zero, the ACP condition becomes |prediction| ≤ 0, which counts only exact zero predictions; this edge case should be addressed explicitly since the dataset has images with zero cells.","section":"Eq. (3)"},{"comment":"The mean cell count for RIP is reported as '49.542' with three decimals while other entries use two; please format consistently.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The dataset and public code are genuine contributions, and the paper is close to publishable once the annotation validation is added. The main risk is that the headline 'no model is acceptable' conclusion may be an artifact of noisy manual ground truth; I would make the inter-annotator/expert validation a mandatory condition of acceptance. The novelty claim about covering multiple staining methods is appropriately hedged, but the comparison to prior datasets is somewhat selective; I would encourage the authors to cite LIVECell and other large microscopy datasets for context. Overall, the work is within scope and should be reconsidered after the revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper contributes a genuinely new dataset, IDCIA, of 262 fluorescence microscopy images of rat adult hippocampal progenitor cells stained with seven different antibodies, with dot annotations for cell locations and counts. It also ships the code, the trained models, and baseline experiments on five existing counting methods. No prior public cell counting dataset covers this antibody diversity, so this is a real addition to the field.\n\nWhat it does well: the data collection is described in enough detail to reproduce, the train/validation/test split is stratified by antibody, and the authors report a grid search over hyperparameters with five runs per model. The headline result—that none of the five models reaches the domain experts' 5% acceptable error criterion, with the best ACP at 17%—is a useful baseline for future work. The per-antibody breakdown in Table 6 is informative and suggests that different models suit different stains.\n\nThe soft spots are real but addressable. The biggest one is the ground truth. Annotations were done by undergraduate students using ImageJ, led by a graduate student, and the paper reports no inter-annotator agreement, no expert re-count, and no quality control statistic. The ACP metric is defined relative to those manual counts. If the manual counts are themselves off by more than 5% on a meaningful number of images, the conclusion that automated methods are unacceptable is not yet established. The blind counting helps with bias about stimulation condition, but it does not validate counting accuracy. This doesn't invalidate the dataset as a resource, but it does mean the benchmark conclusion rests on an untested assumption.\n\nA smaller issue: Tables 4-6 report average MAE and ACP across five runs without standard deviations or confidence intervals. Given the high variance in cell counts per image, the reader can't tell whether the differences between models are meaningful.\n\nThe citation pattern looks fine; the electrical stimulation context is the authors' own prior work and is legitimate. The claim about covering the most staining methods seems credible.\n\nAll in all, this paper deserves a serious referee. It's a useful dataset and a reasonable baseline study. I'd ask for two revisions before acceptance: validate the annotations (a small expert re-count or inter-annotator agreement on a subset) and report uncertainty on the metrics. With those, it's a solid contribution.","headline":"A useful new multi-antibody cell counting dataset and baseline study; the main caveat is unvalidated manual ground truth.","tokens_in":11982,"tokens_out":2438,"would_cite":true,"duration_ms":23828,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new 262-image immunolabeled cell dataset shows automated counters still fall short.","keywords":["immunocytochemistry","cell counting","fluorescence microscopy","deep learning","annotated dataset","stem cells","density map estimation","benchmark"],"falsifier":"Take a random sample of IDCIA images and have two independent annotators, plus one experienced cell biologist, re-count every cell without seeing the original labels; if inter-annotator agreement is low, or the original counts differ systematically from the expert recount, the conclusion that no automated model is acceptable cannot be separated from errors in the gold standard.","tokens_in":11017,"feed_emoji":"🔬","tokens_out":6806,"duration_ms":61399,"temperature":0.7,"pith_summary":"The paper introduces IDCIA, a public dataset of 262 fluorescence microscopy images of rat adult hippocampal progenitor cells, each annotated with the location of every cell and a total cell count. The images come from an electrical-stimulation study and cover seven antibody staining methods, which the authors say makes it the most stain-diverse annotated fluorescent cell dataset available. Evaluated on this dataset, five established deep-learning counting models all underperform: the best average count error is about 15 cells per image, and the highest share of images falling within a domain expert's 5 percent tolerance is 17 percent. The authors' claim is that accurate automated counting on this kind of immunolabeled data is still an open problem, and IDCIA provides a benchmark for solving it.","feed_headline":"Best automated cell counter scores just 17% on new benchmark","feed_subtitle":"IDCIA adds 262 fluorescence images across seven antibody stains; no tested model meets the 5% error experts demand.","key_machinery":"Two things carry the argument. First, the ground-truth format: dot annotations mark each cell, and the paper argues this avoids double counting, is fast to acquire, and supports both detection- and regression-based methods. Second, the evaluation protocol: dot annotations are converted to geometry-adaptive Gaussian density maps for training, and performance is judged by mean absolute error and by a new metric, ACP (Acceptable Error Count Percent), which counts the fraction of images whose predicted count is within 5 percent of the true count. ACP is what converts average error into acceptable to a biologist, and it is the metric on which every tested model fails.","core_discovery":"The central contribution is the dataset itself plus a baseline evaluation. IDCIA contains 262 grayscale 800x600 images of rat adult hippocampal progenitor cells labeled with DAPI and six neural markers (TuJ1, MAP2ab, RIP, GFAP, Nestin, Ki67), with manual dot annotations converted to cell coordinates. Five regression- or density-based deep models were trained and tested with a stratified split, and none met the acceptance standard of a 5 percent error: Count-ception had the lowest mean absolute error (15.47), while CSRNet had the highest acceptable-error percentage (17 percent), and MCNN scored 0 percent on that metric. The paper concludes that existing counting models are not yet accurate enough to replace manual cell counting and that performance varies strongly by antibody stain.","pith_inferences":["Beyond the paper: if ACP becomes a standard metric, automated counting systems will be judged by clinical tolerance rather than by mean error, which is a stricter and more decision-relevant bar.","Beyond the paper: the absence of reported inter-annotator agreement means the benchmark's ceiling could change under a re-annotation study; a second annotation pass on a sample would reveal how much of the 17 percent is label noise.","Beyond the paper: the strong per-antibody performance differences suggest antibody-aware or stain-conditional models, not a single generic counter, may be the fastest route to expert-acceptable accuracy.","Beyond the paper: because the images come from a specific electrical-stimulation protocol and scaffold, models trained on IDCIA may need domain adaptation before transferring to other cultures or imaging setups."],"forward_implications":["The dataset gives the community a public, multi-antibody benchmark for cell counting, with per-image antibody labels that can be used to diagnose where models fail.","The baseline results establish that density-map methods transfer only partially from crowd counting to immunolabeled stem cells, motivating architectures tuned for antibody-specific appearance.","Since no model reaches the 5 percent tolerance, the paper's own stated target is a new counting method that does; the dataset is positioned as the training and evaluation ground for that effort.","The dot annotations are reusable beyond counting, as weak supervision for segmentation or detection, and the antibody label supports an auxiliary classification task."],"supporting_citations":[{"why":"Supplies the widely used synthetic bacterial fluorescence dataset and the learning-to-count approach that later density-map methods build on.","marker":"[18]"},{"why":"One of the existing cell-counting datasets IDCIA is compared against, with bright-field bone marrow images.","marker":"[13]"},{"why":"Existing colorectal cancer histology cell dataset used in the comparison table and as a counting benchmark.","marker":"[31]"},{"why":"Existing human embryonic stem cell image dataset used to position IDCIA's novelty in staining coverage.","marker":"[22]"},{"why":"Count-ception, one of the five evaluated models and the best performer on mean absolute error.","marker":"[26]"},{"why":"Fully convolutional regression network, a cell-counting model evaluated on IDCIA and the worst MAE performer.","marker":"[35]"},{"why":"CSRNet, the crowd-density model that achieved the highest acceptable-error percentage on IDCIA.","marker":"[19]"},{"why":"MCNN, the multi-column crowd-counting model adapted to density maps for the baseline comparison.","marker":"[36]"},{"why":"The cell-counter tool used to produce the manual dot annotations that define ground truth.","marker":"[28]"}],"fun_headline_variants":["New cell dataset stumps all 5 tested AI counters","No AI model passes cell counting bar on new dataset","IDCIA dataset: 262 images, 7 stains, zero models pass","17% is best: AI cell counters fail on new dataset","New benchmark: 7 stains, 262 images, all AI counters fail"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole benchmark rests on the manual dot annotations being correct, but the paper reports no check of agreement between annotators or a re-count by an expert, so if those labels are noisy or biased the reported error rates are not a clean measure of model accuracy.","fun_headline_variants_meta":{"raw":{"variants":["New cell dataset stumps all 5 tested AI counters","No AI model passes cell counting bar on new dataset","IDCIA dataset: 262 images, 7 stains, zero models pass","17% is best: AI cell counters fail on new dataset","New benchmark: 7 stains, 262 images, all AI counters fail"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000628,"raw_usage":{"total_tokens":2872,"prompt_tokens":885,"completion_tokens":1987,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":1907}},"tokens_in":501,"tokens_out":1987,"duration_ms":14486,"temperature":1.0,"reasoning_tokens":1907,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:10:27.668379+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of IDCIA images and have two independent annotators, plus one experienced cell biologist, re-count every cell without seeing the original labels; if inter-annotator agreement is low, or the original counts differ systematically from the expert recount, the conclusion that no automated model is acceptable cannot be separated from errors in the gold standard.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the widely used synthetic bacterial fluorescence dataset and the learning-to-count approach that later density-map methods build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the existing cell-counting datasets IDCIA is compared against, with bright-field bone marrow images."},{"cited_title":"Fu, Shenghua He, Sabine Dietmann, Steven C","cited_arxiv_id":null,"evidence_quote":"Existing human embryonic stem cell image dataset used to position IDCIA's novelty in staining coverage."},{"cited_title":"Glastonbury, Henry Z","cited_arxiv_id":null,"evidence_quote":"Count-ception, one of the five evaluated models and the best performer on mean absolute error."}],"review_version":1}