{"id":"1b92bc8c-82cd-444a-88d6-d7abb93f91b6","arxiv_id":"1908.02919","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AxoNet, a U-Net neural network, counts retinal ganglion cell axons in rat and monkey optic nerve images with accuracy comparable to or better than existing tools.","lead":"This paper presents AxoNet, an automated neural network tool that counts retinal ganglion cell axons in microscope images of rat and monkey optic nerves. It matters because glaucoma researchers rely on axon counts to measure nerve damage, and AxoNet promises faster, more consistent counting than existing software.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rat test accuracy is likely inflated: train/test split is at sub-image level (Methods II.A.3-II.A.4), so images from the same optic nerve and same animal appear in both training and testing; the reported 4.4-axon MAE needs a nerve-level split before AxoNet's outperformance claim can be accepted.","rationale":"Reader's weakest assumption is exactly the one I find most load-bearing. The paper's strongest quantitative evidence is the rat testing subset; if those numbers are inflated by shared-nerve leakage, the abstract's 'outperformed' conclusion is unsupported. The authors did disclose the normal-axon limitation and the per-dataset bias-correction procedure, and the plugin/code/data availability is genuine evidence. However, the evaluation design does not match the deployment scenario of counting a new nerve, so the central claim should be conditional on a nerve-level validation. I recommend keeping the verdict CONDITIONAL (no change), with the condition being the split-level re-analysis.","tokens_in":13195,"tokens_out":5217,"duration_ms":57515,"concrete_test":"Re-run the rat evaluation with a nerve-level or animal-level split: assign all sub-images from a nerve to exactly one split, using the same proportions (or leave-one-nerve-out cross-validation), retrain AxoNet, recalculate the validation correction equations on held-out nerves, and report MAE, R2, and Bland-Altman limits on the held-out nerve test set. If the MAE rises materially (e.g., from 4.4 to more than 7) or R2 drops below about 0.85, the sub-image-level split is load-bearing and AxoNet's comparative claim should be qualified to 'within the same nerve distribution.' Also report how many test sub-images share a nerve with training sub-images.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is the independence assumption in the rat evaluation. The 1,474 sub-images come from only 27 optic nerves from 14 animals (Methods II.A.1, II.A.3), yet Methods II.A.4 randomly divides the sub-images themselves into training/validation/test subsets. Sub-images from the same nerve (and often from the same animal, since most animals contributed one glaucomatous and one control nerve) therefore fall on both sides of the split. Because AxoNet is a fully convolutional U-Net, it can exploit shared staining intensity, section thickness, tissue processing, and damage level within a nerve; a test image from a nerve already represented in training is much easier than an image from a never-seen nerve. This can inflate the rat test-set R2=0.901 and MAE=4.4 (Fig. 5) and the claim that AxoNet outperforms AxonMaster and AxonJ. The NHP dataset is also randomly split at image level, and AxoNet's NHP edge over AxonMaster is not statistically significant (MAE 17.8 vs. 18.2, p=0.976; R2 0.945 vs. 0.938). I do not see a comparable hidden flaw in the architecture, loss, or correction-equation procedure; the transparent 'normal-axon' limitation is disclosed. The decisive question is whether the rat-test advantage survives when whole nerves, not sub-images, are held out.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AxoNet, a U-Net-based convolutional neural network that predicts pixelwise axon count density in light-microscopy images of optic nerve cross sections; the predicted density is integrated and divided by a learned scaling factor to produce an axon count. The network is trained on 1,474 manually annotated rat optic nerve sub-images drawn from 27 nerves of 14 rats, and is then applied without retraining to a separate non-human primate (NHP) dataset previously used to validate AxonMaster. The authors compare AxoNet against AxonMaster and AxonJ on both datasets, using linear bias-correction equations fitted on validation subsets, and report R², mean absolute error, and Bland-Altman limits of agreement. The central claims are that AxoNet outperforms the existing tools on both the rat and NHP datasets, and that it is robust to tissue damage, image quality, and species.","tokens_in":13543,"tokens_out":4444,"duration_ms":50396,"significance":"If the performance claims hold, AxoNet would be a practically useful, openly available tool for a labor-intensive neuroscience assay, and the count-density approach is a sensible alternative to per-axon segmentation. The paper's strengths are real: the code and training data are released, the tool is packaged as an ImageJ/Fiji plugin, the training procedure is described in enough detail to reproduce, the scaling-factor procedure in Eqs. (2)-(3) is an internal optimization device that is divided out of the final count and does not force agreement, and the limitation to 'normal'-appearing axons is explicitly disclosed. The main risk to the rat claim is the image-level rather than nerve-level split of the training/validation/testing data; the main risk to the NHP claim is that the reported statistics do not support the word 'outperformed.' These are load-bearing because the abstract and conclusion rest on them.","major_comments":[{"comment":"The rat evaluation split is at the level of individual 12×12 µm sub-images, not at the level of optic nerves or animals, even though the 1,474 sub-images were sampled from only 27 nerves belonging to 14 rats. Because sub-images from the same nerve (and often the same animal) appear in both training and testing, the fully convolutional network can exploit nerve-specific staining intensity, section thickness, tissue processing, and damage level. This makes the reported rat test results (R²=0.901, MAE=4.4, and the claim that only AxoNet's limits of agreement fall within the manual-count threshold in Fig. 6) potentially inflated. The authors should re-evaluate with whole nerves (or whole animals) held out, for example via nested cross-validation over nerves, and report the range of performance across folds. The same image-level split is used for the NHP dataset, so the NHP comparisons may also be optimistic.","section":"II.A.3-II.A.4, II.B"},{"comment":"The abstract states that AxoNet 'outperformed the existing tools on both the rat and NHP optic nerve datasets,' but the NHP results in Section III.B do not support this. AxoNet's NHP MAE was 17.8 versus AxonMaster's 18.2 with p=0.976 (not significant), and R² was 0.945 versus 0.938, with 'comparable limits of agreement.' At best, AxoNet ties with AxonMaster on the NHP dataset. The abstract and any summary statements should be revised to say that AxoNet performed comparably to AxonMaster on NHP and better than AxonMaster and AxonJ on rat, pending the nerve-level split analysis requested above.","section":"Abstract and III.B"}],"minor_comments":[{"comment":"The caption for Figure 10 says 'Each data point is a single sub-image from the rat testing subset,' but the figure shows the NHP testing subset; this should be corrected.","section":"Fig. 10 caption"},{"comment":"The caption for Figure 4 says 'Each data point is obtained from a single sub-image from the rat testing subset,' but the figure and the text indicate that the data are from the validation subset; this discrepancy should be fixed.","section":"Fig. 4 caption"},{"comment":"There is a typo: 'Kuskal-Wallis' should be 'Kruskal-Wallis.'","section":"II.D.2"},{"comment":"The sentence 'This including image mirroring and rotation...' should read 'This included image mirroring and rotation...'","section":"II.C.2"},{"comment":"The NHP dataset is not quantified in this manuscript (number of images, number of animals, selection procedure); since the split-level concern also applies there, the authors should provide these details or a clear reference to where they are reported.","section":"II.B"}],"recommendation":"major_revision","confidential_remarks":"The image-level split is the central technical risk, and it directly affects the headline rat result. The authors should be asked to provide nerve-level or animal-level cross-validation. If the advantage over AxonMaster disappears under such a split, the rat outperformance claim should be downgraded. I do not see evidence of intentional circularity; the model is trained against manual counts on held-out images, and the density scaling factor is divided out before computing counts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful thing to know about AxoNet: it is a competent, clearly described application of U-Net density regression to counting retinal ganglion cell axons in light microscopy, and the authors ship code and training data. That combination is rarer than it should be. The tool likely works. But the paper's central comparative claim—that AxoNet beats AxonMaster and AxonJ on rat and NHP—rests on an evaluation split that is too optimistic. I would not take the rat outperformance at face value until the test set is split by nerve or animal.\n\nWhat is genuinely new: adapting density-map regression (Lempitsky-style) to optic nerve axon counting in light microscopy, and showing a rat-trained network transfers to NHP without retraining. The authors also package it as an ImageJ plugin, which is practical. They are transparent about the 'normal-appearing axon' limitation, and they make the training data public. The comparison against AxonMaster and AxonJ is thorough in spirit: correction equations, Bland-Altman limits, and inter-observer variability. That is careful work.\n\nThe soft spot is the split. The 1,474 rat sub-images come from 27 nerves (14 animals), and the split into train/validation/test is at the sub-image level. Sub-images from the same nerve almost certainly share staining intensity, tissue processing, and damage extent. A fully convolutional U-Net can latch onto those shared cues, making test images from seen nerves easier than images from a never-seen nerve. The reported MAE of 4.4 and R2=0.901 are therefore likely inflated. A nerve-level split should be the first thing a referee asks for. On the NHP side, the results show parity, not superiority: MAE 17.8 vs 18.2, p=0.976. The abstract's \"outperformed\" overstates that. The NHP ground truth is the same group's semi-automated counts, which is a limitation but a disclosed and reasonable one.\n\nNone of this suggests the method is broken. The architecture, loss, and correction procedure look sound. The normal-axon limitation is honestly stated. The main fix is methodological: redo the split at the biological unit level and tone down the comparative claims.\n\nThis paper is for vision/glaucoma researchers who need an automated axon counter. It deserves a serious referee—the application is real and the reproducibility is a plus—but it needs revision before acceptance. My recommendation: send it out, with the split issue front and center.","headline":"A solid, reproducible axon-counting tool whose rat evaluation split likely inflates the reported advantage; on NHP it is parity, not superiority.","tokens_in":14070,"tokens_out":2138,"would_cite":true,"duration_ms":22386,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A neural network trained on rat optic nerves counts axons in rat and monkey tissue, outperforming existing tools.","keywords":["retinal ganglion cell axons","optic nerve","axon counting","deep learning","U-Net","count density regression","glaucoma","automated image analysis"],"falsifier":"Hold out whole optic nerves from training and test AxoNet on sub-images from those nerves; if mean absolute error rises from the reported 4.4 axons per image to the 8.6 level of AxonMaster or higher, the claimed accuracy is an artifact of the random sub-image split.","tokens_in":13013,"feed_emoji":"🔬","tokens_out":6060,"duration_ms":59120,"temperature":0.7,"pith_summary":"This paper develops AxoNet, a deep learning tool that counts retinal ganglion cell axons in light microscopy images of optic nerve cross sections. Using a U-Net architecture modified to predict a pixelwise axon-count density rather than a segmentation, the network was trained on manually annotated rat optic nerve sub-images and then applied to non-human primate images without retraining. The paper reports that AxoNet outperforms the existing automated tools AxonMaster and AxonJ on both rat and NHP datasets, with mean absolute errors of 4.4 and 17.8 axons per image after bias correction. If these results hold, AxoNet offers a species-general, labor-saving way to quantify glaucomatous optic nerve damage.","feed_headline":"Neural network counts optic nerve axons in rats and monkeys","feed_subtitle":"Rat-trained AxoNet beats AxonMaster and AxonJ on both rat and monkey images with near-manual accuracy.","key_machinery":"The load-bearing mechanism is count-density regression by a U-Net convolutional network. Instead of segmenting individual axons, the network's final layer uses a ReLU activation to output a pixelwise density map; the axon count for an image is the sum of the density map divided by a learned scaling factor (m=1000). Training minimizes mean squared error between predicted and Gaussian-blurred manual annotation densities, with edge mirroring to avoid border artifacts. Because no hand-crafted features are used, the same network trained on rat data can be applied to monkey images.","core_discovery":"The paper claims that AxoNet, built by adapting the U-Net architecture to regress a pixelwise axon-count density, counts 'normal'-appearing retinal ganglion cell axons in light-microscopy optic nerve images with near-manual accuracy in rats (mean absolute error 4.4 axons per 12x12 µm sub-image after linear bias correction) and transfers to non-human primate images without retraining (R²=0.945 against semi-automated manual counts). On both datasets it outperforms the existing automated tools AxonMaster and AxonJ by mean absolute error, regression R², and Bland-Altman limits of agreement.","pith_inferences":["A nerve-level validation study would determine whether the R²=0.901 rat result reflects true generalization to unseen tissue or memorization of the 27 source nerves.","The same count-density approach could be retrained on transmission electron microscopy images to address the known ~30% underestimation of light microscopy counts.","AxoNet's counts describe 'normal-appearing' axons, so it should not be read as a measure of total surviving retinal ganglion cells without checking how many abnormal-looking axons still function."],"forward_implications":["AxoNet can replace fully manual axon counting in rat and NHP glaucoma studies, cutting analysis time to about 15 minutes per full nerve versus hours of manual counting.","A rat-trained model works on monkey images, so the method does not require species-specific retraining to move between mammalian models.","Because the network outputs a count density map rather than a segmentation, it can be extended to quantify other optic nerve features such as glial processes and nuclei.","The reported 84% success rate for AxoNet counts falling within the 95% confidence interval of manual counters means automated counts are typically indistinguishable from a human counter within inter-observer variability.","Full nerve counts around 80,000 'normal' axons in healthy rat nerves are consistent with published manual counts, validating the tool at the whole-nerve scale."],"supporting_citations":[{"why":"Supplies the NHP dataset and semi-automated manual ground truth, and the linear bias-correction method used for all tools.","marker":"[12]"},{"why":"AxonJ, one of the two existing automated counting tools used as a baseline comparator.","marker":"[13]"},{"why":"The U-Net encoder-decoder architecture that AxoNet adapts.","marker":"[15]"},{"why":"The count-density regression framework that replaces axon segmentation.","marker":"[16]"},{"why":"Defines the 'normal' axon appearance criterion and documents the ~30% underestimation of light microscopy counts, motivating the counting task.","marker":"[5]"},{"why":"Defines the Bland-Altman limits-of-agreement analysis used to judge whether each tool's errors fall within manual counter variability.","marker":"[28]"}],"fun_headline_variants":["AxoNet AI counts retina axons with near-manual accuracy","Deep learning tool beats existing axon counters","U-Net tool tallies optic nerve axons across species","Rat-trained AI counts monkey axons accurately","AI axon counter surpasses AxonMaster and AxonJ"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported rat accuracy assumes that randomly splitting sub-images into training and test sets keeps test images independent, even though all 1,474 sub-images come from only 27 optic nerves, so images from the same nerve can appear in both sets.","fun_headline_variants_meta":{"raw":{"variants":["AxoNet AI counts retina axons with near-manual accuracy","Deep learning tool beats existing axon counters","U-Net tool tallies optic nerve axons across species","Rat-trained AI counts monkey axons accurately","AI axon counter surpasses AxonMaster and AxonJ"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000475,"raw_usage":{"total_tokens":2366,"prompt_tokens":961,"completion_tokens":1405,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":1330}},"tokens_in":577,"tokens_out":1405,"duration_ms":10120,"temperature":1.0,"reasoning_tokens":1330,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:29:42.952054+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out whole optic nerves from training and test AxoNet on sub-images from those nerves; if mean absolute error rises from the reported 4.4 axons per image to the 8.6 level of AxonMaster or higher, the claimed accuracy is an artifact of the random sub-image split.","supporting_citations":[{"cited_title":"Automated quantification of optic nerve axons in primate glaucomatous and normal eyes --method and comparison to semi-automated manual quantification,","cited_arxiv_id":null,"evidence_quote":"Supplies the NHP dataset and semi-automated manual ground truth, and the linear bias-correction method used for all tools."},{"cited_title":"Au tomated Axon Counting in Rodent Optic Nerve Sections with AxonJ,","cited_arxiv_id":null,"evidence_quote":"AxonJ, one of the two existing automated counting tools used as a baseline comparator."},{"cited_title":"U -Net: Convolutional Networks for Biomedical Image Segmentation,","cited_arxiv_id":null,"evidence_quote":"The U-Net encoder-decoder architecture that AxoNet adapts."},{"cited_title":"Learni ng To Count Objects in Images,","cited_arxiv_id":null,"evidence_quote":"The count-density regression framework that replaces axon segmentation."},{"cited_title":"A semiautomated targeted sampling method to assess optic nerve axonal loss in a rat model of glaucoma,","cited_arxiv_id":null,"evidence_quote":"Defines the 'normal' axon appearance criterion and documents the ~30% underestimation of light microscopy counts, motivating the counting task."},{"cited_title":"Measuring agreement in method comparison studies,","cited_arxiv_id":null,"evidence_quote":"Defines the Bland-Altman limits-of-agreement analysis used to judge whether each tool's errors fall within manual counter variability."}],"review_version":1}