{"id":"5d477c4d-dbb1-4545-8a3c-10de6dcef0b1","arxiv_id":"1908.11203","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Diffusion-MRI Monte-Carlo simulations need at least 5e5 particles, 1e4 time steps, and roughly 10,000 cylinders per substrate to avoid variance and bias that compromise their use as ground truth.","lead":"Monte-Carlo simulations used to generate ground-truth diffusion MRI signals are sensitive to the number of simulated particles, the time-step count, the shape of axons, and the size of the simulated tissue sample. This paper quantifies these sensitivities, gives minimum parameter recommendations, and introduces a framework for building more realistic crossing-fiber substrates.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The threshold numbers (5e5 particles, 1e4 steps, 10,000 cylinders) are derived from plots without a pre-specified tolerance or error bars, so the exact values are not reproducible findings.","rationale":"I read the central claim as providing specific minimum parameter values (5e5 particles, 1e4 steps, 10,000 cylinders) that should be used for reproducible MCDS. For these numbers to be valid, the experiments must define 'significant variance' and 'important bias' with a reproducible criterion and must estimate the uncertainty of the thresholds. The paper does neither. Section 4.1 reports RMAE values and bootstrap repetitions, but the cutoffs are not tied to a tolerance or test; the reader cannot reconstruct why 5e5 rather than 2e5 or 1e6 was chosen. Section 4.3 reports a single realization per substrate size and no intermediate sizes between 1,000 and 10,000, so the 10,000 boundary is an unmeasured interpolation. I do not think this overturns the paper; the qualitative warnings about undersized simulations are probably correct and the simulator is a useful contribution. But the headline numbers should be treated as provisional, exactly as the CONDITIONAL verdict states. The reader's stated weakest assumption (the extra-axonal gold-standard) is related but less decisive: the Methods cite an independent FEM validation for the simulator, so the reference is not unvalidated, though the exact substrate's absolute accuracy remains unquantified. My concern is more specific: even with a perfect reference, the thresholds lack a defined criterion. If the proposed re-analysis with explicit tolerances and replicates confirms the same thresholds, the concern is resolved.","tokens_in":17227,"tokens_out":14011,"duration_ms":129012,"concrete_test":"Recompute the recommended thresholds using a pre-registered tolerance: for Section 4.1 choose the smallest particle and step counts such that the 95th percentile of the RMAE across the 50 bootstrap repetitions is at or below 1%; for Section 4.3, generate at least 10 independent substrates at N=2,000, 5,000 and 8,000 cylinders and choose the smallest size whose 95% confidence interval for the radial anisotropy metric (std/mean) lies below a pre-set bound (e.g., 0.5%). If the resulting minima differ materially from 5e5, 1e4 and 10,000, the headline thresholds are not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claims in Sections 4.1 and 4.3 are thresholds, but no threshold criterion is ever defined. In Section 4.1, the conclusion that simulations with fewer than 5e5 particles and 1e4 steps carry 'significant variance' is based on visual inspection of RMAE plots from 50 repetitions; there is no predetermined tolerance and no separation of sampling variance from mean offset. Moreover, when the number of steps is varied, the gold-standard uses the maximum value (2e4 steps), so the RMAE for smaller step counts conflates fixed-step discretization bias with Monte Carlo variance, yet the text labels the effect 'variance.' In Section 3.3/4.3, the 'important bias' for substrates with fewer than 10,000 cylinders is quantified by an anisotropy metric (std/mean over in-plane directions) with no stated threshold; the tested sizes are 100, 1,000, 10,000, 50,000 and 100,000, with no intermediate counts and one random substrate per size, so 'less than 10,000' is an interpolation without uncertainty. Because the abstract and conclusion present these numbers as the main deliverable, the lack of an explicit, reproducible criterion makes the thresholds as stated not independently checkable from the manuscript.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript studies three methodological pitfalls in Monte-Carlo diffusion simulations (MCDS) used as ground truth for diffusion MRI: the number of particles and time steps, the intra-axonal representation (straight vs. undulating cylinders), and the size of the extra-axonal substrate. It reports that simulations with fewer than 5e5 particles and 1e4 steps show significant variability, and that substrates with fewer than 10,000 sampled cylinders induce a bias in the radial symmetry of the extra-axonal signal. It then presents a framework for generating complex crossing substrates that preserves volume in the crossing region and achieves high packing density, and it evaluates axon diameter estimation in such substrates. The paper also provides a complexity analysis of the simulator and makes the code and substrate data available.","tokens_in":1532,"tokens_out":5301,"duration_ms":78179,"significance":"If the central quantitative claims hold, the paper is practically important because many published MCDS-based ground-truth studies use parameter values below the proposed minima, and the paper identifies a concrete reproducibility risk. The bootstrap design in Section 3.1, with 50 repetitions per parameter combination, is a clear strength, as is the effort to validate the simulator against analytical solutions and an independent FEM approach. The proposed crossing-substrate framework is a useful step toward more realistic numerical phantoms, and the volume-preservation property is a genuine improvement over naive crossings. However, the paper's headline numbers (5e5 particles, 1e4 steps, 10,000 cylinders) are currently derived from visual inspection without a pre-specified tolerance, and the extra-axonal gold standard is itself a Monte-Carlo simulation whose systematic errors have not been independently bounded. These issues make the main recommendations not yet fully reproducible or falsifiable as stated, so the manuscript needs substantial revision before it can serve as a reliable reference for parameter choice.","major_comments":[{"comment":"The central thresholds '5×10^5 particles' and '1×10^4 steps' are presented in the conclusions without a pre-specified acceptance criterion. In Figures 5 and 6 the RMAE is plotted against sample size, but no tolerance on the RMAE is defined and no confidence bounds are derived from the 50 repetitions; the reader cannot determine why 5×10^5 rather than 2×10^5 or 1×10^6 is the cut-off. In addition, when the number of steps is varied the reference uses the maximum value (2×10^4), so the RMAE for smaller step counts mixes the fixed-step discretization bias with Monte Carlo variance, and the text labels the combined effect 'variance'. Please define an explicit reproducibility criterion (e.g., a maximal acceptable RMAE with a confidence interval) and separate the discretization bias from the sampling variance, ideally by varying the step count in the gold-standard reference as well.","section":"§4.1, §6"},{"comment":"The extra-axonal gold standard is itself a Monte-Carlo simulation (20×10^6 particles, 2×10^4 steps), and the paper reports no comparison of this reference against an independent method beyond an assertion of convergence; the earlier FEM validation [38] is cited but not shown for this substrate and protocol. If the fixed-step integration or the cylinder packing carries a systematic bias, every RMAE in Section 4.1 inherits it, so the recommended minima could be miscalibrated. Please add either a convergence study for the extra-axonal gold standard with respect to step size (not just particle number) or a comparison with an independent numerical/analytical reference for the identical geometry.","section":"§3.1, §4.1"},{"comment":"The conclusion that substrates with fewer than 10,000 cylinders induce an 'important bias' is not supported by a quantitative threshold. The anisotropy metric (std/mean of the radial signal) is reported for one random substrate per size, with only sizes 100, 1,000, 10,000, 50,000, and 100,000 tested; the boundary 'less than 10,000' is an interpolation with no uncertainty. Please repeat the measurement over multiple random packing realizations, report the spread of the anisotropy metric, and state a pre-defined isotropy criterion (or at least a tolerance on the deviation from the large-substrate limit).","section":"§3.3, §4.3"},{"comment":"The diameter-fitting intervals in Figure 7 depend on the arbitrary threshold of '1% difference from the minimum fitting error'. Because the RMAE surface is strongly protocol-dependent (Figure 8), the reported intervals and the qualitative claim of 'considerable mis-estimation' are sensitive to this ad hoc choice. Please justify the 1% level (e.g., by linking it to the noise level or the known sensitivity of the protocol) or report how the intervals change when the threshold is varied.","section":"§3.2, §4.2"}],"minor_comments":[{"comment":"The list of tested particle numbers includes 2×10^6, but the preceding sentence says particles were varied from 1×10^3 to 1×10^6; please make the range consistent.","section":"§3.1"},{"comment":"The sentence 'choosing a the parameters that shows almost almost no variance' contains repeated words ('a the', 'almost almost') and a subject-verb agreement error; please correct it.","section":"§3.2"},{"comment":"'This parameters where chooseng from the previous results' should read 'These parameters were chosen from the previous results'.","section":"§3.3"},{"comment":"The caption says 'RMAE of all the repetition' but should read 'RMAE of all the repetitions'; please also indicate which panels correspond to the intra- and extra-axonal cases in the text, since the figure has six panels.","section":"§4.1, Figure 5"},{"comment":"The optimization time (about 42 hours) and simulation time (less than 24 hours) are reported for a single run; please state whether these are wall-clock times and on what specific hardware configuration they were obtained, to make the computational claims reproducible.","section":"§3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and addresses an important reproducibility issue for MCDS-based ground truth in diffusion MRI. The main concern is that the headline parameter thresholds are not derived from an explicit, reproducible criterion and rely on a self-referential Monte-Carlo gold standard; both issues are fixable with additional analyses. I recommend major revision rather than rejection, because the experimental design and the framework are solid despite these calibration gaps."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading if you do Monte-Carlo diffusion simulations or build numerical phantoms. The practical message: below about 5e5 particles and 1e4 time-steps the simulated signal shows noticeable variation, below about 10,000 cylinders the extra-axonal radial signal gets biased and anisotropic, and undulating axons badly break straight-cylinder diameter fitting. Those numbers will probably get cited as default parameters, and that is mostly fine: the direction of every finding holds up, and the bootstrap design in Study 1 (50 repetitions per parameter combination, RMAE against an analytical intra-axonal reference) is the strongest part of the paper.\n\nThe genuinely new pieces are the quantitative thresholds under modern ActiveAx-style protocols and the volume-preserving crossing-substrate framework. The interdigitating, gamma-distributed crossing reaching about 48% ICVF at the lowest resolution is a real step beyond the roughly 20% of earlier phantoms, and the authors are honest: Section 5.1 frames the results as a lower bound, and the simulator carries independent validation against analytical solutions and FEM.\n\nNow the soft spots, in proportion. The headline thresholds are stated as if measured, but no threshold criterion is ever defined. In Section 4.1 the 'significant variance' call is visual; in the step-count arm the gold standard uses the maximum step count, so RMAE at low step counts conflates discretization bias with Monte-Carlo variance, though the text calls it variance. In Section 4.3, 'less than 10,000 cylinders' is an interpolation from one random substrate at each of five sizes with no intermediate counts. So the exact numbers are provisional: well-informed calibration points, not measured limits. Ask the authors to define a tolerance or report confidence bands.\n\nSecond, the extra-axonal gold standard is itself a simulation (2e7 particles, 2e4 steps), so convergence to it is not convergence to the true signal. Given the prior FEM validation this is a moderate concern, not a fatal one. Third, Studies 2-4 are single realizations without error bars, the diameter fitting uses an arbitrary 1% threshold, and the optimization weights in Equation 8 are undisclosed. One minor internal wrinkle: the extra-axonal study uses 1e3 steps and the crossing demo uses 5e3 steps, both below the paper's own 1e4 recommendation; defensible (different diffusivity and particle counts), but it should be acknowledged. The code and meshes are promised via a GitHub URL without a pinned commit, so exact reproducibility is not yet verifiable.\n\nWho this is for: people generating ground-truth diffusion-MRI data and anyone testing microstructure models on simulated data. It deserves a serious referee; the right outcome is conditional acceptance with the thresholds made reproducible and the weights and meshes released.","headline":"A solid calibration paper with a useful new substrate-generation framework; the headline thresholds are provisional since no threshold criterion is defined, but the direction of each finding checks out and the bootstrapped first study is well done.","tokens_in":18043,"tokens_out":6667,"would_cite":true,"duration_ms":59086,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Monte-Carlo diffusion-MRI simulations require at least 500,000 particles and 10,000 cylinders before their ground-truth signals become reproducible.","keywords":["Monte-Carlo diffusion simulation","diffusion-weighted MRI","microstructure imaging","simulation reproducibility","axonal undulation","substrate generation","extra-axonal bias","ground truth validation"],"falsifier":"Recompute the extra-axonal signal for the same substrate with an independent finite-element or analytical effective-medium solver across the full acquisition protocol; if the reference signal disagrees by more than the reported 0.4–0.7% convergence margin, the recommended parameter minima are miscalibrated.","tokens_in":17002,"feed_emoji":"🧠","tokens_out":8423,"duration_ms":71190,"temperature":0.7,"pith_summary":"Monte-Carlo simulations are widely used to generate ground-truth diffusion-MRI signals for testing microstructure models, but the cheap settings many studies use may not be safe. This paper argues that below $5\\times10^5$ particles and $1\\times10^4$ time steps, the simulated signal carries significant variance in both the compartment inside the axons and the space outside them, and that substrates built from fewer than 10,000 cylinders produce an extra-axonal signal that is no longer directionally symmetric. It also shows that representing axons as straight cylinders hides a real diameter-estimation error once undulation is introduced, and it presents a substrate-generation framework that builds complex, non-overlapping crossing fibre configurations while preserving volume and packing density. If these claims hold, past and future Monte-Carlo validations should report particle counts, step counts, and substrate sizes, and rethink what a reliable ground truth requires.","feed_headline":"Diffusion-MRI simulations need 500,000 particles and 10,000 steps","feed_subtitle":"Study links undersized Monte-Carlo settings to biased diffusion-MRI ground-truth signals.","key_machinery":"The argument rests on two mechanisms. The first is a bootstrapped error analysis: repeated Monte-Carlo runs at each particle and step count are compared with a high-quality reference, namely the analytical Gaussian-phase-distribution solution for restricted diffusion in cylinders for the intra-axonal signal and a 20-million-particle Monte-Carlo run for the extra-axonal signal, using the Relative Mean Absolute Error (RMAE) to turn raw signal deviations into convergence curves that set the recommended minima. The second is an energy-optimization substrate generator: an objective function penalizes overlap, curvature, and length of parametrized strands, and a modified cylinder-cylinder collision term lets those strands be subdivided into gamma-distributed diameters so that crossing bundles interdigitate without intersecting, preserving the volume fraction in the crossing area.","core_discovery":"On its own terms, the paper establishes three quantitative claims. First, for the substrate and acquisition settings studied, simulations with fewer than $5\\times10^5$ particles or $1\\times10^4$ steps show a significant, avoidable variance between repeated runs in both the restricted intra-axonal signal and the hindered extra-axonal signal; the mean relative absolute error settles near 0.4–0.7% only above those values. Second, substrates built from fewer than 10,000 sampled cylinders produce extra-axonal radial signals whose directional symmetry is visibly broken and whose mean amplitude is biased, with the bias disappearing around a 200–230 µm voxel scale for the tested diameter distribution. Third, replacing straight cylinders with helically undulating axons shifts the apparent diameter recovered from a straight-cylinder model, severely for 1 µm axons, where the mis-estimation can exceed 300%. The paper also presents a framework for generating complex crossing substrates whose strands interdigitate without overlap, preserving volume in the crossing region and achieving an intra-axonal volume fraction above 48% even at low resolution.","pith_inferences":["A practical next step the paper does not take is to derive a scaling law for the particle and step minima as a function of diffusivity, diffusion time, and packing density, since the recommended values are tied to the specific protocol and substrate studied here.","Because the extra-axonal reference is itself a Monte-Carlo run, an independent finite-element or effective-medium calculation on the same substrate would give a stronger absolute calibration of the recommended thresholds.","The generator could be used to build controlled dose-response phantoms that vary undulation amplitude, crossing angle, and volume fraction independently, letting models be scored against known ground truth rather than inferred tissue properties.","The observed substrate-size bias in the mean extra-axonal signal implies that shrinking the simulated voxel is not a neutral computational shortcut, and time-dependent diffusion metrics extracted from such simulations may inherit a false time-dependence."],"forward_implications":["Validations run below $5\\times10^5$ particles and $1\\times10^4$ steps should be treated as carrying unexplained run-to-run variance; reporting a single run is not enough.","Extra-axonal signals from substrates with fewer than about 10,000 cylinders are not radially isotropic, so fitting procedures that assume axial symmetry inherit a systematic bias.","Axon-diameter estimates from straight-cylinder models can be off by more than 300% for 1 µm undulating axons, meaning undulation is a confounding factor in diameter mapping.","The proposed substrate generator can produce crossing-fibre ground truths with preserved volume and an intra-axonal volume fraction above 48% at coarse resolution, giving crossing-aware models a harder, more realistic test."],"supporting_citations":[{"why":"Supplies the packing algorithm and parameter-choice methodology the paper builds on; its convergence approach is extended to more complex substrates.","marker":"[22]"},{"why":"Introduced the helical undulation substrate representation that the intra-axonal geometry experiment generalizes.","marker":"[31]"},{"why":"Provides the ActiveAx acquisition protocol and the orientationally invariant diameter and density indices used in simulations and fitting.","marker":"[2]"},{"why":"Original tractography-phantom strand optimization framework that is modified into the complex substrate generator.","marker":"[12]"},{"why":"Gives the Gaussian phase distribution analytical cylinder signal used as the intra-axonal ground truth.","marker":"[48]"},{"why":"Independent finite-element validation of the particle dynamics, supporting the simulator's physical accuracy.","marker":"[38]"},{"why":"Established fixed step-size Monte-Carlo for non-homogeneous media and analysed particle effects in simple geometries.","marker":"[5]"},{"why":"Prior evidence on realistic voxel sizes and reduced signal variation that the substrate-size result is compared with.","marker":"[23]"}],"fun_headline_variants":["500k particles, 10k steps: robust diffusion-MRI Monte-Carlo","Oversimplified substrates bias diffusion-MRI simulated signals","Diffusion-MRI simulation pitfalls: particle count and substrate design","New simulator tackles Monte-Carlo bias in diffusion-MRI","Ground-truth diffusion-MRI needs proper Monte-Carlo settings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 20-million-particle, $2\\times10^4$-step extra-axonal run is itself an accurate reference; if that Monte-Carlo simulation carries a systematic bias from the fixed step size, substrate representation, or cylinder packing, then every error percentage and the recommended $5\\times10^5$-particle, $1\\times10^4$-step minimum inherit that bias.","fun_headline_variants_meta":{"raw":{"variants":["500k particles, 10k steps: robust diffusion-MRI Monte-Carlo","Oversimplified substrates bias diffusion-MRI simulated signals","Diffusion-MRI simulation pitfalls: particle count and substrate design","New simulator tackles Monte-Carlo bias in diffusion-MRI","Ground-truth diffusion-MRI needs proper Monte-Carlo settings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000638,"raw_usage":{"total_tokens":2972,"prompt_tokens":1014,"completion_tokens":1958,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":1870}},"tokens_in":630,"tokens_out":1958,"duration_ms":14812,"temperature":1.0,"reasoning_tokens":1870,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:21:35.182777+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the extra-axonal signal for the same substrate with an independent finite-element or analytical effective-medium solver across the full acquisition protocol; if the reference signal disagrees by more than the reported 0.4–0.7% convergence margin, the recommended parameter minima are miscalibrated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the packing algorithm and parameter-choice methodology the paper builds on; its convergence approach is extended to more complex substrates."},{"cited_title":"Nilsson, J","cited_arxiv_id":null,"evidence_quote":"Introduced the helical undulation substrate representation that the intra-axonal geometry experiment generalizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ActiveAx acquisition protocol and the orientationally invariant diameter and density indices used in simulations and fitting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Original tractography-phantom strand optimization framework that is modified into the complex substrate generator."},{"cited_title":"Van Gelderen, D","cited_arxiv_id":null,"evidence_quote":"Gives the Gaussian phase distribution analytical cylinder signal used as the intra-axonal ground truth."},{"cited_title":"Rafael-Patino, A","cited_arxiv_id":null,"evidence_quote":"Independent finite-element validation of the particle dynamics, supporting the simulator's physical accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Established fixed step-size Monte-Carlo for non-homogeneous media and analysed particle effects in simple geometries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior evidence on realistic voxel sizes and reduced signal variation that the substrate-size result is compared with."}],"review_version":1}