See and Switch: Vision-Based Branching for Interactive Robot-Skill Programming

Jan Kristof Behrens; Karla Stepanova; Petr Vanc; V\'aclav Hlav\'a\v{c}

See and Switch: Vision-Based Branching for Interactive Robot-Skill Programming

T0 review · 4 major / 0 minor · reviewed 2026-07-15 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Eye-in-hand images can drive online branching in robot skills taught by non-experts, selecting the right recovery path or flagging new situations that need a demonstration.

desk verdict Solid engineering extension of CIP with a DS-local DINO Switcher and real novice data; the reliability claim is real but scoped to offline user-gated windows, not closed-loop autonomy. read the letter →

arxiv 2603.08057 v2 pith:CPHFDFQ2 submitted 2026-03-09 cs.RO cs.CV

Petr Vanc , Jan Kristof Behrens , Václav Hlaváč , Karla Stepanova This is my paper · ORCID

classification cs.ROcs.CV

keywords programmingbydemonstrationconditionalskillgraphsdecisionstatesvision-basedbranchinganomalydetectioneye-in-handcameraDINOfeaturesinteractiverobotteaching

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Programming robots by demonstration works for fixed motions but breaks when the world changes and a different sequence of actions is required. This paper argues that those conditional tasks can be taught as graphs of short skill parts linked at decision states, and that a vision-based Switcher can choose the right next part from the robot's own camera. At each decision state the Switcher both classifies among the known successor branches and detects when the scene is new, so a human can immediately teach a recovery behavior with kinesthetic guidance, a joystick, or hand gestures. On three board-level manipulation tasks taught by eight novices, the Switcher reaches roughly ninety percent branch-selection accuracy inside carefully timed decision windows and flags anomalies well on a large share of those windows. The result is a practical path from a single demonstration to a growing library of visual conditionals that non-experts can keep extending while the robot is running.

What carries the argument

The Switcher: a decision-state-local classifier built on frozen DINO patch embeddings that jointly scores among competing successor skill parts and raises an anomaly when the current image falls outside the known set, restricting decisions to a short temporal window and to the permitted branch set so that global scene understanding is unnecessary.

What would settle it

Collect a new set of decision-state images in which the decisive object or state is deliberately kept outside the camera field of view, retrain the same Switcher, and check whether branch-selection accuracy collapses to chance while the rest of the pipeline remains unchanged.

Watch

Extended reading notes

Core claim

Within user-gated decision-state windows, a DINO-feature Switcher that operates only on eye-in-hand images and only among the locally permitted successors can select the correct skill branch with up to about 90.7 percent accuracy and detect out-of-distribution scenes at greater than 90 percent accuracy on roughly half of the decision states collected from novice users, thereby making vision-driven conditional programming by demonstration reliable enough for real-robot use.

Load-bearing premise

The visual cue that distinguishes one task variant from another (door open or closed, object present or missing) must actually appear in the eye-in-hand camera image inside the decision-state window; if it does not, correct branching is impossible from vision alone.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a clean systems paper that does what it says: it takes CIP-style conditional task graphs and adds a vision-based Switcher so branch selection and OOD detection can run from eye-in-hand images, plus a modality-agnostic recovery layer (kinesthetic, joystick, gestures). That combination, plus a real novice user study and public code/data, is the actual contribution.

What is new is not the graph model (Eiband et al.) or the anomaly idea (ILeSiA lineage, overlapping authors), but the DS-local frozen-DINO classifier that jointly does multiclass successor selection and novelty scoring, trained only on the permitted successors inside a short context window. They also ship a controlled class-growth experiment (2–8 states) where dinov2-small-attn stays above ~90% longer than SIFT or AEGP, and a user study with 8 novices, 3 tasks, 3 modalities, and hundreds of real-robot rollouts. The offline numbers are credible: ~81.7% branch accuracy overall, ~90.7% after dropping unobservable DS windows, and strong anomaly detection on a large fraction of DS instances. They are honest that limited observability is the dominant failure mode.

The soft spot is exactly the one the stress-test flags, and the paper itself is transparent about it. Headline “reliable switching” is measured offline inside user-gated DS windows chosen so the cue is visible; Table II task success uses manual switching. That isolates the visual classifier from timing and closed-loop errors, which is methodologically clean, but it means the strongest claim does not yet cover automatic online branching on the same data. Abstract/body count mismatches (576 vs ~900 rollouts, 47/79 vs ~43–44/78) are minor bookkeeping noise, not a soundness problem. Free parameters (window length, percentile_keep, attn_keep) are ordinary for this kind of system.

Math is light and appropriate; citations are fair to the CIP/ILeSiA line and to DINO; no circularity. This is for people building interactive PbD and recovery tooling, not for theory. I would send it to peer review: the engineering is real, the data are real, and the limitation is stated. Engage if you care about visual branching in teachable task graphs; skip if you only want closed-loop autonomy claims.

Referee Report

4 major / 0 minor

Summary. The paper presents See & Switch, a programming-by-demonstration framework that represents multi-variant manipulation tasks as graphs of skill parts linked by decision states (DS). A vision-based Switcher, built on frozen DINO features with lightweight heads, uses eye-in-hand images at DS context windows to (i) select among permitted successor skill parts and (ii) flag out-of-distribution scenes that require recovery demonstrations. Teaching is modality-agnostic (kinesthetic, joystick, hand gestures). The system is evaluated on three Robothon board tasks with 8 novice users; the Switcher is assessed offline on user-gated DS windows against SIFT/ORB and AEGP baselines, with a controlled class-growth study and a modality comparison of demonstration effort and replay success under manual switching. Code and data are released.

Significance. If the claims hold under the stated evaluation protocol, the work is a solid incremental systems contribution to interactive PbD: it couples CIP-style conditional task graphs with a practical, DS-local vision classifier and a unified teaching API, and it provides a sizable novice-user real-robot dataset. Strengths include external baselines, a controlled scalability experiment (up to 8 classes), public code/data, and an honest discussion of observability failures. The result is useful for the community even if closed-loop automatic switching remains partially open, because offline DS-local visual branching on novice data is itself a non-trivial empirical target.

major comments (4)

Abstract and Sec. V-B claim 'reliable switching based on visual input for conditional robot-skill programming,' but the primary evidence is offline classification inside user-gated DS windows (explicitly justified in Sec. IV-B.2 to isolate Switcher performance from automatic timing errors). Table II task-success rates further use manual switching. Please either (a) report closed-loop automatic branch-selection and full-task success under system-gated DS insertion on the same novice data, or (b) systematically soften abstract/conclusion language so the central claim matches the offline, user-gated protocol actually evaluated.
Sec. V-B.1 reports overall branch accuracy rising from 81.7% to 90.7% after removing DS windows with insufficient observability (red-marked instances in Figs. 9–11; ~40/78). The abstract headline (90.6%) appears to reflect the filtered figure. State clearly in abstract, results, and conclusion which number is primary, how many DS/windows/samples are excluded and by what objective rule, and report unfiltered accuracy as the main result with filtered as a diagnostic upper bound.
Numerical inconsistencies between abstract and body undermine reproducibility of the headline metrics: abstract cites ~900 rollouts, 90.6% branch accuracy, and >90% anomaly accuracy in 47 of 79 DS; body reports |R|=576, 90.7%, and ~43–44/78 DS (Figs. 9–11). Reconcile rollout counts (demonstrations × replays, filtered unsafe demos), DS counts, and accuracy figures throughout abstract, Sec. IV-B, and Sec. V.
Sec. III-D/E and Sec. V leave free parameters (context length e=10, percentile_keep=0.1, attn_keep, MIL training hyperparameters) largely unablated for sensitivity on the user-study set. Because anomaly thresholds and window placement directly affect both DS creation and reported accuracy, provide at least a short sensitivity analysis or fixed default protocol with justification so the 90% figures are not tied to unreported tuning.

Circularity Check

0 steps flagged · score 1.0 of 10

Empirical systems paper; Switcher accuracies are measured on held-out rollouts and do not reduce by construction to fitted identities.

full rationale

See & Switch is a PbD systems paper whose central claims are empirical: offline multiclass branch-selection and anomaly-detection accuracies of a DS-local DINO Switcher on a newly collected novice-user dataset (576 rollouts; ~78–79 DS windows), plus modality comparison of teaching effort and replay success under manual switching. The Switcher is trained on images from skill-part demonstrations/rollouts and evaluated on held-out rollouts; reported numbers (e.g., 81.7% overall rising to 90.7% after removing unobservable DS windows; >90% anomaly accuracy on ~43–44/78 windows) are ordinary supervised/test metrics, not algebraic identities of fitted parameters. Baselines (SIFT/ORB, Multiclass AEGP) and a controlled class-growth experiment further treat performance as falsifiable measurement. Light author-overlap dependence exists (CIP task-graph/DS logic [5],[8]; ILeSiA-style anomaly scoring [13]), but those citations supply architectural scaffolding, not a uniqueness theorem or a fit that is then renamed as a prediction. No self-definitional loop, no fitted-input-called-prediction, and no load-bearing uniqueness import. Score 1 for minor self-citation without circular derivation; the skeptic’s concern about user-gated offline windows is a validity/scope issue, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The central empirical claim rests on observability of cues in eye-in-hand views, the CIP graph formalism from prior work, DINO as a frozen encoder, and several hand-chosen thresholds (window size, percentile_keep, attention keep fraction). No new physical entities are postulated; the Switcher and modality layer are engineering constructs evaluated only on the authors' board tasks.

free parameters (4)

DS context window length e = 10
Fixed by authors for merging overlapping windows and for all Switcher training/evaluation; directly defines which images enter each DS-local model.
percentile_keep (anomaly threshold) = 0.1
Set to 0.1 in the anomaly experiments (Sec. V-B / Fig. 10); controls the OOD flag rate and therefore anomaly accuracy.
attn_keep fraction = e.g. 0.2
Hyperparameter for attention-gated DINO features (e.g. keep top 20% patches); affects which visual regions drive classification.
MIL head training hyperparameters = 128 / 7e-5 / 1000 epochs
128 hidden units, Adam lr 7e-5, weight decay 1e-3, 1000 epochs; chosen by authors and affect multiclass accuracy of the MIL variant.

assumptions (4)

domain assumption Eye-in-hand RGB images at decision states contain sufficient information to distinguish permitted successor skill parts when the discriminative cue is in view.
Core premise of the Switcher (Sec. III-D); the paper acknowledges failures when the cue is out of view.
domain assumption CIP-style task graphs of skill parts with decision states are an appropriate representation for conditional PbD.
Inherited from Eiband et al.; the framework retains CIP DS logic and anomaly-triggered insertion (Sec. I, III).
domain assumption Frozen DINO patch embeddings plus lightweight heads are adequate for few-shot DS-local classification and novelty scoring on this domain.
Method choice in Sec. III-E; compared empirically to SIFT/ORB and AEGP.
ad hoc to paper User-gated decision-state timing yields a fair offline evaluation of visual branch selection independent of automatic anomaly timing errors.
Explicit design choice in Sec. IV-B.2 to isolate Switcher performance from timing confounds.

invented entities (2)

Vision-based Switcher (DS-local DINO classifier + anomaly score)
purpose: Jointly select the successor skill part and flag OOD views at decision states from eye-in-hand images.
Defined and evaluated only within this paper's datasets; no external benchmark deployment is reported.
Input-modality-abstraction layer for PbD recovery
purpose: Unify kinesthetic, joystick, and gesture commands for demonstration and anomaly approve/pause during execution.
System component introduced in Sec. III-C and evaluated via the multi-modality user study.

how reviews work

0 comments

Cite this review

Pith. "Pith review of See and Switch: Vision-Based Branching for Interactive Robot-Skill Programming." pith.science (2026). https://pith.science/paper/CPHFDFQ2

@misc{pith2026260308057,
  author       = {Pith},
  title        = {Pith review of: See and Switch: Vision-Based Branching for Interactive Robot-Skill Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CPHFDFQ2}},
  note         = {Machine review of arXiv:2603.08057}
}

read the original abstract

Programming by demonstration (PbD) makes robot programming accessible to non-experts, but scaling it to real-world variability remains a challenge for current teaching frameworks, especially when a robot must select suitable task variants online from visual input. We present See & Switch, an interactive teaching-and-execution framework that represents tasks as graphs of skill parts connected by decision states, enabling conditional branching during replay. Its vision-based Switcher uses eye-in-hand images to select the appropriate successor skill part and detect novel situations that require new demonstrations. The framework supports recovery demonstrations during execution through kinesthetic teaching, joystick control, and hand gestures. We evaluate See & Switch on three dexterous manipulation tasks with 8 novice users, collecting approx. 900 real-robot execution rollouts. To isolate visual decision performance from timing errors during decision states, we evaluate the Switcher offline using user-gated decision state windows. In the evaluation within the decision state windows, the method achieves up to 90.6% branch-selection accuracy and detects anomalies with >90% accuracy in 47 of 79 decision states, demonstrating reliable switching based on visual input for conditional robot-skill programming. We provide all code and experiment data at http://imitrob.ciirc.cvut.cz/publications/seeandswitch.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 1 linked inside Pith

[1]

Learning from humans,

A. G. Billardet al., “Learning from humans,”Springer Handb. Robot., pp. 1995–2014, 2016

1995
[2]

A survey of robot learning from demonstration,

B. D. Argallet al., “A survey of robot learning from demonstration,” Robot. Auton. Syst., vol. 57, no. 5, pp. 469–483, 2009

2009
[3]

Incremental learning of gestures by imitation in a humanoid robot,

S. Calinon and A. Billard, “Incremental learning of gestures by imitation in a humanoid robot,” inProc. ACM/IEEE Int. Conf. Human- Robot Interact. (HRI), ser. HRI ’07. New York, NY, USA: Association for Computing Machinery, 2007, p. 255–262

2007
[4]

Billardet al.,Robot Programming by Demonstration

A. Billardet al.,Robot Programming by Demonstration. Berlin, Heidelberg: Berlin, Heidelberg: Springer, 2008, pp. 1371–1394

2008
[5]

Collaborative programming of robotic task decisions and recovery behaviors,

T. Eibandet al., “Collaborative programming of robotic task decisions and recovery behaviors,”Aut. Robots, vol. 47, pp. 229–247, 2023

2023
[6]

Robot learning from demonstration by construct- ing skill trees,

G. Konidariset al., “Robot learning from demonstration by construct- ing skill trees,”Int. J. Robot. Res. (IJRR), vol. 31, no. 3, pp. 360–375, 2012

2012
[7]

Symbolic state estimation with predicates for contact-rich manipulation tasks,

T. Migimatsuet al., “Symbolic state estimation with predicates for contact-rich manipulation tasks,”2022 Int. Conf. on Robot. and Autom. (ICRA), pp. 1702–1709, 2022

2022
[8]

Intuitive programming of conditional tasks by demonstration of multiple solutions,

T. Eibandet al., “Intuitive programming of conditional tasks by demonstration of multiple solutions,”IEEE Robot. Autom. Lett., vol. 4, no. 4, pp. 4483–4490, 2019

2019

Show all 27 references

[9]

Cobt: Collaborative programming of behaviour trees from one demonstration for robot manipulation,

A. Jainet al., “Cobt: Collaborative programming of behaviour trees from one demonstration for robot manipulation,” inProc. IEEE Int. Conf. Robot. Autom. (ICRA), 2024, pp. 12993–12999

2024
[10]

Neural task graphs: Generalizing to unseen tasks from a single video demonstration,

D. Huanget al., “Neural task graphs: Generalizing to unseen tasks from a single video demonstration,” inProc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019, pp. 8565–8574

2019
[11]

Learning robot anomaly recovery skills from multiple time-driven demonstrations,

H. Wuet al., “Learning robot anomaly recovery skills from multiple time-driven demonstrations,”Neurocomputing, vol. 464, pp. 522–532, 2021

2021
[12]

Conditionnet: Learning preconditions and effects for execution monitoring,

D. Sliwowski and D. Lee, “Conditionnet: Learning preconditions and effects for execution monitoring,”IEEE Robot. Autom. Lett. (RA-L), vol. 10, no. 2, pp. 1337–1344, 2025

2025
[13]

Ilesia: Interactive learning of robot situational aware- ness from camera input,

P. Vancet al., “Ilesia: Interactive learning of robot situational aware- ness from camera input,”IEEE Robot. Autom. Lett. (RA-L), vol. 10, no. 10, pp. 10490–10497, 2025

2025
[14]

Learning symbolic failure detection for grasping and mobile manipulation tasks,

P. Hegemannet al., “Learning symbolic failure detection for grasping and mobile manipulation tasks,” inProc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2022, pp. 4302–4309

2022
[15]

Diffusion-pbd: Generalizable robot programming by demonstration with diffusion features,

M. Murrayet al., “Diffusion-pbd: Generalizable robot programming by demonstration with diffusion features,” inProc. IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 2024, pp. 5168–5175

2024
[16]

DINOBot: Robot manipulation via retrieval and alignment with vision foundation models,

N. D. Palo and E. Johns, “DINOBot: Robot manipulation via retrieval and alignment with vision foundation models,” inProc. IEEE Int. Conf. on Rob. and Aut. (ICRA), 2024, pp. 2798–2805

2024
[17]

Emerging Properties in Self-Supervised Vision Transformers,

M. Caronet al., “Emerging Properties in Self-Supervised Vision Transformers,” inIEEE Int. Conf. Comp. Vis. (ICCV), 2021, pp. 9630– 9640

2021
[18]

T. H. Cormenet al.,Introduction to Algorithms. MIT Press, 2009

2009
[19]

Digital robot judge: Building a task-centric performance database of real-world manipulation with electronic task boards,

P. Soet al., “Digital robot judge: Building a task-centric performance database of real-world manipulation with electronic task boards,”IEEE Robot. Autom. Mag., vol. 31, no. 4, pp. 32–44, 2024

2024
[20]

Dinov2: Learning robust visual features without supervision,

M. Oquabet al., “Dinov2: Learning robust visual features without supervision,” 2023

2023
[21]

Siméoniet al., “Dinov3,” 2025

O. Siméoniet al., “Dinov3,” 2025. [Online]. Available: https: //arxiv.org/abs/2508.10104

2025 arXiv
[22]

Object recognition from local scale-invariant features,

D. Lowe, “Object recognition from local scale-invariant features,” in Proc. 7th IEEE Int. Conf. Comput. Vis. (ICCV), 1999, pp. 1150–1157

1999
[23]

MAGSAC++, a fast, reliable and accurate robust estimator,

D. Barathet al., “MAGSAC++, a fast, reliable and accurate robust estimator,” inConf. Comput. Vis. Pattern Recognit. (CVPR), 2020

2020
[24]

Adam:Amethodforstochasticoptimization,

D.P.KingmaandJ.Ba,“Adam:Amethodforstochasticoptimization,” 2017

2017
[25]

Towards effective utilization of mixed-quality demon- strations in robotic manipulation via segment-level selection and optimization,

J. Chenet al., “Towards effective utilization of mixed-quality demon- strations in robotic manipulation via segment-level selection and optimization,” inProc. IEEE Int. Conf. Robot. Autom. (ICRA), 2025, pp. 16884–16891

2025
[26]

Beyond success: Quantifying demonstration quality in learning from demonstration,

M. Bilalet al., “Beyond success: Quantifying demonstration quality in learning from demonstration,” inIEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024, pp. 5120–5127

2024
[27]

How non-experts kinesthetically teach a robot over multiple sessions: diversity in teaching styles and effects on performance,

P. Aliasghariet al., “How non-experts kinesthetically teach a robot over multiple sessions: diversity in teaching styles and effects on performance,”Int. J. Soc. Robot., vol. 16, no. 11, pp. 2079–2105, 2024

Pith tools

Reviewed July 15, 2026 · model on record in the stance chip above.

[1] [1]

Learning from humans,

A. G. Billardet al., “Learning from humans,”Springer Handb. Robot., pp. 1995–2014, 2016

1995

[2] [2]

A survey of robot learning from demonstration,

B. D. Argallet al., “A survey of robot learning from demonstration,” Robot. Auton. Syst., vol. 57, no. 5, pp. 469–483, 2009

2009

[3] [3]

Incremental learning of gestures by imitation in a humanoid robot,

S. Calinon and A. Billard, “Incremental learning of gestures by imitation in a humanoid robot,” inProc. ACM/IEEE Int. Conf. Human- Robot Interact. (HRI), ser. HRI ’07. New York, NY, USA: Association for Computing Machinery, 2007, p. 255–262

2007

[4] [4]

Billardet al.,Robot Programming by Demonstration

A. Billardet al.,Robot Programming by Demonstration. Berlin, Heidelberg: Berlin, Heidelberg: Springer, 2008, pp. 1371–1394

2008

[5] [5]

Collaborative programming of robotic task decisions and recovery behaviors,

T. Eibandet al., “Collaborative programming of robotic task decisions and recovery behaviors,”Aut. Robots, vol. 47, pp. 229–247, 2023

2023

[6] [6]

Robot learning from demonstration by construct- ing skill trees,

G. Konidariset al., “Robot learning from demonstration by construct- ing skill trees,”Int. J. Robot. Res. (IJRR), vol. 31, no. 3, pp. 360–375, 2012

2012

[7] [7]

Symbolic state estimation with predicates for contact-rich manipulation tasks,

T. Migimatsuet al., “Symbolic state estimation with predicates for contact-rich manipulation tasks,”2022 Int. Conf. on Robot. and Autom. (ICRA), pp. 1702–1709, 2022

2022

[8] [8]

Intuitive programming of conditional tasks by demonstration of multiple solutions,

T. Eibandet al., “Intuitive programming of conditional tasks by demonstration of multiple solutions,”IEEE Robot. Autom. Lett., vol. 4, no. 4, pp. 4483–4490, 2019

2019