{"id":"99b5d082-845a-44df-b60f-6ac79d27e20e","arxiv_id":"2505.18303","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"OTMR is a new public dataset of 232,881 microscope images across 18 optical microrobots and 176 poses, benchmarked on pose classification and depth regression with eight deep learning models.","lead":"This paper releases OTMR, a public dataset of 232,881 microscope images of 18 optical microrobot designs with pose and depth labels. It benchmarks eight deep learning models, finding ViT best for pose classification and deeper networks better for depth regression.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark numbers may be inflated by frame-level splits: consecutive frames from the same z-trajectory can appear in both training and test folds.","rationale":"The reader's weakest assumption is exactly the concern I identify: temporal leakage from random frame-level splits. This is the most load-bearing issue because it directly affects every headline benchmark result, not just a secondary detail. The paper's own description of data collection (III.B) and evaluation (IV.A) makes the risk concrete and testable. Other issues, such as benchmark coverage being narrower than the abstract implies (pose classification shown only for two robots, depth only for six) and the absence of error bars, are real but secondary. The dataset's existence and public availability are not undermined by this concern, so a conditional verdict remains appropriate. If the sequence-aware split confirms the inflation, the benchmark numbers would need to be recomputed and the model-ranking and scaling conclusions would likely weaken; that is precisely what the conditional acceptance should require.","tokens_in":11473,"tokens_out":2141,"duration_ms":19503,"concrete_test":"Re-run the five-fold protocol with sequence-grouped folds: assign all frames from one z-trajectory (or one pose instance) to a single fold, then train the same eight models on the same folds and recompute Tables II and IV. If accuracy or R2 drops materially (e.g., classification accuracy falls by more than 3 points, or depth R2 falls below 0.95), the reported benchmarks are inflated and the ranking/scale conclusions need revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claims—that ViT is best for pose classification, that deeper models are best for depth regression, and that more data helps—all rest on the five-fold cross-validation described in Section IV.A. Section III.B states that images were acquired by moving the piezoelectric stage along z, generating depth-varying image sequences per pose. Figure 6 confirms that depth is recorded over time, so consecutive frames in a sequence are near-duplicates with only slightly different depth labels. The evaluation protocol says the dataset is 'evenly partitioned into five subsets' (IV.A) with no mention of grouping frames by sequence or pose. A random frame-level split therefore allows temporally adjacent frames from the same trajectory to appear in both training and test folds; a model can effectively memorize one appearance and interpolate to its near-duplicate. This inflates classification accuracy and depth R2, and makes the model ranking and data-scaling conclusions unreliable. The dataset may still be a valid resource, but Tables II and IV cannot be trusted as benchmark reference points without a sequence-aware split. The paper gives no sequence identifiers, no error bars, and no per-sequence analysis to rule this out.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OTMR, a dataset of 232,881 optical microscopy images of 18 microrobot designs across 176 poses, collected with an optical tweezer system and a piezo stage that generates depth-varying image sequences. The authors benchmark eight deep learning models on two tasks: pose classification (pitch/roll) and depth regression, reporting that ViT achieves the highest pose accuracy, deeper architectures generally improve depth regression, NAS-optimized CNNs outperform the baseline CNN, and increasing training data size improves depth performance. The dataset, CAD models, and benchmarking code are claimed to be publicly available.","tokens_in":11644,"tokens_out":3310,"duration_ms":25990,"significance":"If the evaluation concerns are addressed, OTMR fills a clear gap: it is, to the authors' knowledge, the first public dataset dedicated to microrobot pose and depth perception under optical microscopy, and its scale (232,881 images, 18 designs, 176 poses) is a substantial step beyond previous single-robot studies. The paper also contributes a reproducible benchmarking suite with eight architectures, computational cost comparisons, Grad-CAM analysis, and an open release of resources. The main significance is therefore as a community resource and reference point, provided the benchmark numbers are made trustworthy through a sequence-aware evaluation protocol.","major_comments":[{"comment":"The five-fold cross-validation described in Section IV.A ('the dataset is evenly partitioned into five subsets') does not group frames by acquisition sequence or by pose, yet Section III.B states that images were acquired by moving the piezoelectric stage along z, generating depth-varying image sequences, and Fig. 6 shows depth evolving continuously over time. Consecutive frames in such a sequence are near-duplicates with only slightly different depth labels, so a random frame-level split allows temporally correlated images to appear in both training and test folds. This can inflate the accuracy and R2 values in Tables II and IV and can bias the model ranking and the data-scaling conclusion in Fig. 9. Please implement a sequence-aware split that keeps each z-sweep sequence (or each pose) entirely within one fold, report per-sequence or per-pose performance, and quantify the difference between frame-level and sequence-level splits.","section":"IV.A, III.B"},{"comment":"The abstract claims that increasing the size of the training dataset leads to 'substantial improvements across both tasks', but Section IV.F only evaluates depth regression (Robot Type 8 with ResNet50, Fig. 9); no pose classification experiment with varying data size is reported. Please either add a pose-classification scaling experiment or restrict the claim to depth regression.","section":"IV.F, Abstract"},{"comment":"All benchmark results are reported as single averages over five folds without standard deviations, confidence intervals, or per-fold breakdowns. Several model differences are small relative to the granularity of the metrics (e.g., Robot 1 pitch accuracy: ViT 0.999 vs. VGG16 0.996; Robot 10 R2: ResNet18 0.994, ResNet50 0.994, EfficientNet 0.995). Without fold-wise variance or a significance test, the claimed rankings (ViT best for pose, deeper architectures better for depth) are not established. Please report mean ± std across folds or include per-fold results.","section":"Tables II and IV"},{"comment":"The transfer learning experiment tests a ViT trained on Robot Type 3 directly on Robots 1, 4, and 5, with no fine-tuning and no comparison to a model trained on each target robot or to a control model. The conclusion that structural similarity and horizontal orientation determine transfer performance (Fig. 7) is therefore not supported by the data as presented. Please add baseline comparisons (e.g., per-target training) or soften the interpretation to a descriptive observation.","section":"IV.D"}],"minor_comments":[{"comment":"The text says pose angles vary from 0° to 90°, but the Fig. 5 caption shows pitch and roll variations from 0° to 70°; please reconcile these numbers.","section":"III.C.1, Fig. 5"},{"comment":"Reference [17] (the JEFF-3.3 nuclear data library) appears unrelated to the sentence it supports about data scarcity limiting model development; this is likely a citation error and should be corrected.","section":"References"},{"comment":"The column header 'Params (MB)' is ambiguous: parameter counts are not normally measured in megabytes. Please clarify whether this is the model file size or the parameter count, and label accordingly.","section":"Table III"},{"comment":"The conclusion states that ViT achieved the highest pose classification accuracy 'exceeding 99%', but Table II shows Robot 3 ViT accuracy of 0.965 (pitch) and 0.971 (roll); please qualify this claim to avoid overstatement.","section":"Conclusion"},{"comment":"The dataset is referenced only via a project website in a footnote; a persistent identifier such as a DOI or a versioned repository link would make the 'publicly available' claim more robust for reproducibility.","section":"Data Availability"}],"recommendation":"major_revision","confidential_remarks":"The core resource is valuable and likely appropriate for a robotics or automation venue. The main risk is the frame-level evaluation protocol, which undermines the quantitative benchmark conclusions as currently presented. I would not reject the dataset paper on this basis, but the revision must address the sequence-aware split and the missing error bars before the benchmark numbers can be used as reference points by the community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing you should know: this paper is worth knowing for the dataset, not for its benchmark numbers. OTMR is a 232,881-image collection of optical microrobots under a microscope, covering 18 designs and 176 poses, and it looks genuinely to be the first public resource of its kind. If the website actually ships the images and code, it fills a real gap in microrobot perception.\n\nWhat it does well: the data collection is described concretely (Nanoscribe fabrication, optical tweezer setup, piezo-stage z-motion, Wiener filtering and Canny cropping), and the benchmarking is unusually broad for the area—eight models including a NAS search, transfer experiments, and Grad-CAM. Giving the field a standardized testbed is a solid contribution, and the qualitative finding that structural symmetry affects difficulty is useful.\n\nWhere it goes soft: the evaluation protocol. Section III.B says images were captured as depth-varying sequences while the piezo stage moves along z, and Figure 6 confirms that depth changes smoothly over time. Section IV.A then says the dataset was 'evenly partitioned into five subsets' with no mention of grouping by sequence. A random frame-level split puts near-duplicate frames from the same trajectory into train and test, so the high ViT accuracies and R2 values are likely inflated and the model ranking is unreliable. The paper also reports no error bars, tests the data-size effect only on depth regression for a single robot, and the abstract claims data helped 'both tasks' when only one was tested. Pose results are only shown for two of the six robot types. None of this kills the dataset resource, but it means Tables II and IV cannot serve as trustworthy reference points as written.\n\nOne caveat: I could not access the website to confirm the data release, and the absence of prior public datasets is asserted more than evidenced. Both are verifiable.\n\nBottom line: this is a genuine dataset contribution with a fixable methodological hole. It deserves a serious referee, but the benchmark claims should be relabeled as preliminary until the authors rerun with sequence-aware splits and report variance.\n\nI'd send it to review with a request for major revision.","headline":"A genuinely useful first public dataset for optical microrobot perception, but the benchmark numbers are likely inflated by temporal leakage in the frame-level split.","tokens_in":12163,"tokens_out":3432,"would_cite":true,"duration_ms":28460,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces OTMR, the first publicly available image dataset for optical-microrobot perception, with 232,881 microscope images of 18 printed microrobot designs in 176 out-of-plane poses.","keywords":["microrobot perception","optical tweezers","image dataset","pose classification","depth regression","deep learning benchmark","Vision Transformer"],"falsifier":"Re-run the five-fold benchmark using sequence-aware folds so that all frames from one z-sweep stay in the same split; if pose accuracy and depth R2 drop markedly, the original frame-level split is inflating the reported generalization.","tokens_in":11267,"feed_emoji":"🔬","tokens_out":4202,"duration_ms":37582,"temperature":0.7,"pith_summary":"The paper introduces OTMR, the first publicly available image dataset for optical-microrobot perception, containing 232,881 microscope images of 18 separately fabricated microrobot designs and 176 distinct out-of-plane poses. It argues that this resource lets deep-learning models be trained and compared reproducibly on two tasks: classifying a robot's pitch and roll angles, and regressing its depth from defocus patterns. Benchmarks across eight architectures show a Vision Transformer reaching over 99 percent pose accuracy, deeper architectures helping depth regression, and more training data improving both tasks. A sympathetic reading is that the dataset is large and diverse enough to become a standard evaluation surface for microscope-robot perception.","feed_headline":"First open microrobot vision dataset holds 232,881 images","feed_subtitle":"Pose and depth benchmarks across 18 designs give deep learning a common starting line.","key_machinery":"The dataset itself is the load-bearing object, and the two controlled generation mechanisms are what make it usable as a benchmark. Pose labels come from fabricating microrobots in defined pitch and roll orientations, while depth labels come from translating the robot along the z-axis with a piezo stage and recording the true depth at each frame. These mechanisms convert inherently ambiguous microscope images into supervised targets, and the five-fold cross-validation protocol turns the collection into a shared evaluation surface for comparing models.","core_discovery":"The central discovery is OTMR itself: a public collection of 232,881 cropped grayscale images of 18 two-photon-polymerized microrobots, with ground-truth pitch and roll labels for 176 poses and depth labels obtained by moving a piezoelectric stage along the optical axis. On this dataset, the paper finds that the Vision Transformer achieves the highest pose classification accuracy, exceeding 99 percent on a simple robot, while deeper residual networks generally perform best for depth regression. Neural architecture search tuned to the two tasks finds a deeper model for depth regression than for pose classification, and the paper reports that increasing the training-data fraction consistently lowers depth MSE and raises R2.","pith_inferences":["A sequence-aware re-split that keeps all frames from one z-sweep in the same fold would test whether the reported accuracies and R2 values are inflated by temporal near-duplicates in the training and test sets.","The public CAD models allow synthetic images with known pose and depth to be generated, which could test whether models trained only on OTMR transfer to unseen microrobot designs.","The observed transfer pattern, where structurally similar robots transfer better than dissimilar ones, suggests that a quantitative measure of design similarity could predict cross-robot generalization performance."],"forward_implications":["If OTMR becomes the standard benchmark, new perception algorithms for optical microrobots can be compared on identical data and metrics instead of single-robot demonstrations.","The Vision Transformer's high pose accuracy suggests transformer-based models are strong candidates for real-time closed-loop microrobot control.","Depth regression improves with deeper architectures and with more training data, so scaling both model depth and dataset size is a promising path.","The NAS-found depth model has more convolutional layers than the pose model, indicating that the two tasks have different optimal architectures.","Because CAD models and benchmark code are released with the dataset, future work can build sim-to-real transfer directly on OTMR."],"supporting_citations":[{"why":"Supplies the planar multi-spot optical tweezer actuation design that enables controlled out-of-plane rotation and depth, generating the pose and depth variations in OTMR.","marker":"[6]"},{"why":"Represents a prior micro-object pose estimation method validated on a single robot model, which OTMR extends to 18 designs and a public benchmark.","marker":"[9]"},{"why":"Provides an earlier CNN-LSTM approach for 3D pose and depth estimation of transparent microrobots, a key context for the tasks OTMR benchmarks.","marker":"[25]"},{"why":"Describes a data-driven microscopic pose and depth estimation method using deep residuals and Gaussian processes, one of the single-robot baselines OTMR generalizes beyond.","marker":"[26]"},{"why":"Provides the two-photon polymerization fabrication technique used to print the 18 microrobot designs in the dataset.","marker":"[27]"}],"fun_headline_variants":["ViT tops pose, deep nets win depth on new microrobot dataset","First open microrobot dataset benchmarks pose and depth AI","232k microrobot images enable pose and depth perception tests","Microrobot vision benchmark: 232k images, 18 designs, 176 poses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported benchmark numbers assume the 232,881 images are independent samples; because depth images are captured as time sequences from a moving stage, frames recorded close together are near-duplicates, so a random frame split can let training and test sets share almost identical images and make accuracy and R2 look higher than they are.","fun_headline_variants_meta":{"raw":{"variants":["ViT tops pose, deep nets win depth on new microrobot dataset","First open microrobot dataset benchmarks pose and depth AI","232k microrobot images enable pose and depth perception tests","Microrobot vision benchmark: 232k images, 18 designs, 176 poses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1687,"prompt_tokens":912,"completion_tokens":775,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":696}},"tokens_in":528,"tokens_out":775,"duration_ms":6236,"temperature":1.0,"reasoning_tokens":696,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:33:11.991204+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the five-fold benchmark using sequence-aware folds so that all frames from one z-sweep stay in the same split; if pose accuracy and depth R2 drop markedly, the original frame-level split is inflating the reported generalization.","supporting_citations":[{"cited_title":"Data-driven microscopic pose and depth estimation for optical microrobot manipulation,","cited_arxiv_id":null,"evidence_quote":"Describes a data-driven microscopic pose and depth estimation method using deep residuals and Gaussian processes, one of the single-robot baselines OTMR generalizes beyond."},{"cited_title":"Finer features for functional microdevices,","cited_arxiv_id":null,"evidence_quote":"Provides the two-photon polymerization fabrication technique used to print the 18 microrobot designs in the dataset."}],"review_version":1}