Towards Active Robotic Vision in Agriculture: A Deep Learning Approach to Visual Servoing in Occluded and Unstructured Protected Cropping Environments

Chris Lehnert; Paul Zapotezny-Anderson

REVIEW 4 major objections 5 minor 2 references

Towards Active Robotic Vision in Agriculture: A Deep Learning Approach to Visual Servoing in Occluded and Unstructured Protected Cropping Environments

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a single-camera CNN can learn the direction a robot arm should move to reveal an occluded fruit, matching the nine-camera 3DMTS baseline in simulation.

desk verdict A solid, honest step toward single-camera 3DMTS, but the equivalence claim rests on 24 trials with no statistics. read the letter →

arxiv 1908.01885 v1 pith:UA6XJYCF submitted 2019-08-05 cs.RO

Paul Zapotezny-Anderson , Chris Lehnert This is my paper · ORCID

classification cs.RO

keywords agriculturalroboticsvisualservoingdeeplearningconvolutionalneuralnetworksmove-to-seeocclusionhandlingprotectedcroppingsweetpepperharvesting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the expensive multi-camera stage of 3D Move To See (3DMTS) can be replaced by a single camera plus a convolutional neural network without losing guidance quality. 3DMTS finds a better view of occluded fruit by comparing nine images and computing a direction gradient; Deep-3DMTS regresses that same gradient from one reference-camera image. In the simulation comparison, the learned controller guided the arm to an occlusion-free view of a sweet pepper, ending within 11.4 mm of the baseline's final position and enlarging the fruit's image area by a factor of 17.8, against 16.8 for the baseline. The paper argues this matters because the nine-camera setup limited 3DMTS to about 1 Hz, and a single-camera CNN removes most of the data-processing burden that caused the bottleneck. The validation is entirely in simulation, so the practical claim depends on how well that simulator matches real greenhouses.

What carries the argument

The central object is a learned map $\Phi: I_{\text{ref}} \mapsto \nabla\hat{f}$, a CNN with a three-output regression head that replaces the standard method's nine-camera array and its one-shot optimisation of $f(\mathbf{x}) = w_1 p(\mathbf{x}) + w_2 m(\mathbf{x})$. The learned controller uses $w_2 = 0$, so the direction comes purely from image content rather than from a mobility measure. The CNN does the load-bearing work: it is trained on reference-camera images paired with gradient targets from the baseline, and at run time its output is used directly to update the end-effector position. Image segmentation is retained only to keep the camera pointed at the fruit and to measure the fruit-pixel proportion for comparison.

What would settle it

Run the trained CNN on a physical arm in a greenhouse with sweet peppers occluded at the same leaf-pose ranges used in simulation, ending guidance by the same criteria (gradient magnitude below 1.5 or fruit above 40% of the image); if the average end position differs from the nine-camera baseline by more than about 11.4 mm, or the average fruit-size enlargement factor is below about 16.8, the equivalence claim is refuted. A cheaper first check is to feed real greenhouse images through the CNN and compare its predicted gradients with gradients computed from the 3DMTS objective; a large jump from the simulated validation MSE of 1.896 would expose the gap between simulation and reality.

Watch

Extended reading notes

Core claim

The central claim is that the local direction gradient $\nabla f(\mathbf{x})$—the vector the nine-camera 3DMTS obtains by segmenting images from a 3D camera array—can be regressed directly from a single reference-camera image. In Deep-3DMTS, the CNN's three outputs are the components $\nabla f_x, \nabla f_y, \nabla f_z$, and the learned objective omits the mobility term so that only image content sets the direction. The network was fine-tuned on 1155 simulated image–gradient pairs (495 validation pairs) produced by running the baseline under random occlusion poses. In the main comparison series, the single-camera controller ended on average 11.4 mm from the baseline end position and enlarged the fruit from 1.69% of the image to 30.19%, a factor of 17.8, versus the baseline's factor of 16.8; every trial reached an occlusion-free view under the termination criteria.

Load-bearing premise

The load-bearing premise is that the simulator's occluded sweet-pepper scenes reproduce real greenhouse occlusion geometry and image appearance closely enough that a CNN trained and tested there will guide a physical robot the same way.

Editorial extensions

If this is right

A single-camera 3DMTS is feasible: Deep-3DMTS ended within 11.4 mm of the nine-camera baseline and produced an occlusion-free view in every trial of both series.
Removing the nine-camera array and its image-processing and optimisation stages addresses the bottleneck that limited standard 3DMTS to roughly 1 Hz, so the learned controller is the concretely motivated route to a faster servoing loop.
The controller tolerated random starting-position offsets up to 50 mm in each coordinate with almost no performance change (end position difference -0.3 mm, final fruit size -0.40 percentage points).
In heavily occluded cases Deep-3DMTS used up to 19 fewer steps than the baseline, because it moves laterally and toward the fruit at the same time rather than first circling the occlusion.
The method still depends on the segmentation stage for pointing the camera at the fruit and for evaluation, so the reduction is from nine cameras to one camera plus a CNN, not a fully self-contained vision front end.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the training scheme only needs paired images and gradients from the teacher 3DMTS, the same recipe could be applied to other crops or other target objects, provided the training data covers their appearance; nothing in the method is sweet-pepper-specific except the data.
The paper does not report closed-loop latency, so the claimed practical motivation—a faster visual servoing rate—is a plausible corollary but not yet measured; timing the CNN forward pass and control update on the actual robot would settle it.
Since the learned objective drops the mobility term and the CNN is trained on the same simulator it is tested on, the most informative next experiment is physical deployment: if performance there falls well short of the simulated equivalence, the limitation is transfer from simulation to reality rather than the learned gradient itself.
One implicit consequence is that the CNN's notion of a good view is tied to the fruit-pixel proportion used in the baseline objective; a view with more fruit area may not always be the view that makes the peduncle graspable, so harvesting success should eventually be the evaluation metric.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is exactly the kind of paper that should be reviewed if the venue wants incremental but useful robotics work. The authors take the stated future work from Lehnert et al. (2018) — replace the nine-camera array with a CNN that regresses the direction gradient from one image — and they actually build it. The method is clearly described, the simulation setup matches the baseline, and the results are plausible. It is not a breakthrough, but it is a legit contribution to the agricultural visual servoing niche.

What is good: the CNN training data is generated from the 3DMTS simulator, the architecture is sensible (fine-tuned ResNet18), and the integration into the control loop is straightforward. The comparison against the baseline in the same simulator is fair. The paper is honest about what is not solved: it still uses segmentation for the performance metric and orientation control, it ignores mobility in the objective, and it does not measure the servoing rate that motivated the work. The citations are appropriate; the prior art is properly credited.

Where it gets soft: the empirical evidence for 'equivalent performance' is thin. Two series of 12 trials, with no standard deviations, confidence intervals, or significance tests. The 11.4 mm mean difference could easily be compatible with a practically meaningful difference. The 70:30 random per-image split from 55 trajectories is a real leak: consecutive frames from the same run appear in both training and validation, so the reported validation loss of 1.896 is optimistically biased. The actual evaluation trials are separate, which helps, but n=12 is still small. No timing data, so the promised servoing rate improvement is unmeasured. No code or data, which would make the result much easier to trust.

None of this is fatal. The paper makes no grand claims; it says 'towards' and flags the sim-to-real gap itself. The right response is revision, not rejection: report error bars or individual trial values, redo the split by trajectory, and add a table of wall-clock servoing rates. Even a quick physical test on a single plant would help.

I would send this to peer review. It deserves a serious referee, mostly to push the authors to tighten the statistics and add the timing numbers. For a reader in ag robotics, it is a useful incremental data point.

Referee Report

4 major / 5 minor

Summary. The paper presents Deep-3DMTS, a deep learning variant of the 3DMTS visual servoing method. The key idea is to train a convolutional neural network (ResNet18) to regress the direction gradient that standard 3DMTS computes from a nine-camera array, using only the central reference camera image. Training data are generated in a V-REP simulation used previously to develop 3DMTS, and the CNN is evaluated by guiding a simulated Harvey robot end effector around an occluding leaf to obtain an unobstructed view of a sweet pepper. The reported results are two series of 12 trials, stating that Deep-3DMTS reaches final positions within 11.4 mm of the baseline and increases fruit image size by a factor of 17.8 versus 16.8 for the baseline. The paper concludes that single-camera Deep-3DMTS performs equivalently to the nine-camera baseline in the tested simulation.

Significance. If the equivalence claim held, this would be a useful practical step: replacing a nine-camera array with a single camera and a CNN could reduce hardware complexity and increase the servoing rate of the 3DMTS approach, whose 1 Hz rate is a known limitation. The teacher-student formulation is sensible and the authors are appropriately careful to state that validation is simulation-only. The work is also honest about the retained image-processing stages, which are not part of the gradient regression. The main weakness is that the evidence for equivalence is statistically thin: 12 trials per condition with no confidence intervals, significance tests, or equivalence margins, and a training/validation split that is likely leakage-prone. These issues are load-bearing because the central claim is precisely that the new method performs 'equivalently' to the baseline.

major comments (4)

[Section 3.1, Table 3] The central claim of equivalent performance is not statistically supported. Table 3 reports only means, maxima, and minima for two series of 12 trials; no standard deviations, confidence intervals, or tests for equivalence are given. With n=12, the observed mean difference of 11.4 mm could easily be consistent with a practically meaningful difference, especially since the maximum difference in Series 1 is 28.7 mm and in Series 2 is 19.5 mm. The authors should report per-trial data or at least confidence intervals and define an equivalence margin before claiming equivalence.
[Section 3.1, validation splitting] The 70:30 random split of 1650 images from 55 trajectories is performed per image, not per trajectory. Because consecutive frames from the same simulation run are highly correlated and likely appear in both training and validation, the reported validation loss of 1.896 is optimistically biased. The evaluation should use a per-trajectory split or a separate set of held-out trajectories to avoid temporal leakage.
[Section 4, experimental protocol] The equivalence claim rests on only 24 new trials, and all trials are generated from a single fruit pose and a single type of occlusion (one leaf) with randomized offsets. This is a narrow basis for the statement that Deep-3DMTS is 'equivalent' to the baseline. The paper would be stronger if the evaluation included variations in fruit position, leaf shape, or occlusion geometry, or at least a clear discussion of the generalization limits.
[Section 3.3, retained image processing] Although the CNN replaces the gradient computation, the approach still requires image segmentation for the sensor-orientation update and for the fruit-size performance metric. Since the motivation is improved time efficiency, the paper should report the actual servoing rate or computational cost of Deep-3DMTS compared with the baseline. Without this measurement, the central practical benefit remains unvalidated.

minor comments (5)

[Abstract and Introduction] There are several typos, including 'mutli-perspective' in the abstract and 'succes sfully' in the introduction; these should be corrected.
[References] Reference 'Morrision, D.' should be 'Morrison, D.', and the two entries with '(In Press)' should include publication years and volume/page numbers once available.
[Section 3.2, reproducibility] The fine-tuning description omits several training details that affect reproducibility, such as the optimizer (SGD or Adam?), weight decay, batch normalization behavior during fine-tuning, and the exact image preprocessing transformations.
[Section 4, termination criteria] The termination conditions ('magnitude of direction gradient <1.5' or 'proportion of fruit pixels >40%') are introduced without justification or sensitivity analysis; their effect on the reported outcomes should be discussed.
[Table 3] The column header 'Final position ΔBL [mm]' is ambiguous: it should be clarified whether positive values indicate that the baseline is further from the fruit or that Deep-3DMTS is further, and the sign convention should be stated in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Deep-3DMTS is a supervised imitation of the 3DMTS gradient, but the closed-loop equivalence claim is an independent empirical result.

full rationale

The paper's derivation chain is self-contained in the relevant sense. The CNN is trained to regress the direction gradient computed by the nine-camera 3DMTS baseline using reference-camera images from a V-REP simulator (Sections 3.1 and 3.2), and the closed-loop evaluation in Section 4 compares the resulting trajectories to that same baseline via final end-effector position and fruit-size percentage. The training loss (MSE of the gradient) is not the reported performance metric; the final-position and fruit-size comparisons are emergent closed-loop quantities that could in principle diverge from the baseline, so the equivalence claim is not forced by construction. The use of Lehnert et al. (2018) as the teacher and as the baseline is legitimate prior work by one of the authors, but no uniqueness theorem, ansatz, or definitional equivalence is imported from that citation to make the current result tautological. The statistical concerns raised by the small number of trials and the per-image train/validation split are validity and generalizability issues, not circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim does not introduce new physical entities or exotic assumptions. It rests on the fidelity of the simulation, the validity of the 3DMTS gradients as training targets, the suitability of ResNet18 for the regression task, and the accuracy of the fruit segmentation. These are standard domain assumptions for simulation-based deep learning, but they are not independently verified here.

free parameters (2)

Objective function weights w1, w2 = w1=1.0, w2=0.0
Chosen to generate the direction-gradient training targets using only fruit pixel proportion, dropping the mobility term from the original 3DMTS. This affects what the CNN learns and is a hand-set modeling choice.
Termination thresholds = gradient magnitude <1.5, fruit pixel proportion >40%
Used to stop the servoing trials; chosen by the authors and not derived from data or theory.

assumptions (4)

domain assumption The V-REP simulator faithfully represents the protected cropping environment, including leaf occlusions and camera imaging, for the purpose of training and evaluating the CNN.
All training data and all evaluation trials are produced in the same V-REP simulation (Sections 3.1 and 4). The claimed equivalence may not transfer to real environments if the simulator is not representative.
domain assumption The direction gradients computed by the nine-camera 3DMTS algorithm are valid supervisory targets for learning a single-image gradient estimator.
The CNN is trained to imitate 3DMTS gradients, so the correctness of the learned behavior depends on the correctness of those gradients as a ground-truth policy (Section 3.1).
domain assumption A pretrained ResNet18, fine-tuned on synthetic V-REP images, learns a feature representation sufficient for regressing the direction gradient.
The architecture choice and transfer from natural-image pretraining are assumed to be appropriate for this synthetic domain (Section 3.2).
domain assumption The image segmentation used for the performance metric and for the sensor orientation update correctly identifies fruit pixels.
The fruit size metric and the orientation correction rely on segmentation (Sections 3.3 and 4); no segmentation accuracy analysis is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Active Robotic Vision in Agriculture: A Deep Learning Approach to Visual Servoing in Occluded and Unstructured Protected Cropping Environments." pith.science (2026). https://pith.science/paper/UA6XJYCF

@misc{pith2026190801885,
  author       = {Pith},
  title        = {Pith review of: Towards Active Robotic Vision in Agriculture: A Deep Learning Approach to Visual Servoing in Occluded and Unstructured Protected Cropping Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UA6XJYCF}},
  note         = {Machine review of arXiv:1908.01885}
}

read the original abstract

3D Move To See (3DMTS) is a mutli-perspective visual servoing method for unstructured and occluded environments, like that encountered in robotic crop harvesting. This paper presents a deep learning method, Deep-3DMTS for creating a single-perspective approach for 3DMTS through the use of a Convolutional Neural Network (CNN). The novel method is developed and validated via simulation against the standard 3DMTS approach. The Deep-3DMTS approach is shown to have performance equivalent to the standard 3DMTS baseline in guiding the end effector of a robotic arm to improve the view of occluded fruit (sweet peppers): end effector final position within 11.4 mm of the baseline; and an increase in fruit size in the image by a factor of 17.8 compared to the baseline of 16.8 (avg.).

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages

[1]

and Kwok, N

Chen, S., Li, Y. and Kwok, N. M. (2011). Active vision in robotic systems: a survey of recent developments. The International Journal of Robotics Research , 30 (11), 1343-1377. Farrokhsiar, M., Pavlik, G. and Najjaran, H. (2013). An integrated robust probing motion planning an d control scheme: a tube-based mpc approach. Robotics and Autonomous Systems, 6...

work page 2011
[2512]

Morrision, D

IEEE, Stockholm, Sweden. Morrision, D. Corke, P. and Leitner, J. (2018). Multiview picking: next-best-view reaching for improved grasping in clutter. In, 2019 IEEE International Conference on Intelligent Robotics and Systems (IROS), (In Press). IEEE, Macau, China. Perez, T., Sa, I., McCool, C. and Lehnert, C. (2015). A bayesian framework for the asse ssme...

work page 2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

and Kwok, N

Chen, S., Li, Y. and Kwok, N. M. (2011). Active vision in robotic systems: a survey of recent developments. The International Journal of Robotics Research , 30 (11), 1343-1377. Farrokhsiar, M., Pavlik, G. and Najjaran, H. (2013). An integrated robust probing motion planning an d control scheme: a tube-based mpc approach. Robotics and Autonomous Systems, 6...

work page 2011

[2] [2512]

Morrision, D

IEEE, Stockholm, Sweden. Morrision, D. Corke, P. and Leitner, J. (2018). Multiview picking: next-best-view reaching for improved grasping in clutter. In, 2019 IEEE International Conference on Intelligent Robotics and Systems (IROS), (In Press). IEEE, Macau, China. Perez, T., Sa, I., McCool, C. and Lehnert, C. (2015). A bayesian framework for the asse ssme...

work page 2018