Pith. sign in

REVIEW 3 major objections 5 minor 15 references

SUNSET is the first ROS2 exemplar whose observable symptoms do not reveal their root cause, forcing self-adaptive managers to investigate and prioritize.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 09:25 UTC pith:KH5UYJMX

load-bearing objection A solid, genuinely useful ROS2 exemplar for ambiguous and concurrent self-adaptation faults; under-validated entropy detection is fixable, and the artifact deserves refereeing. the 3 major comments →

arxiv 2601.13732 v2 pith:KH5UYJMX submitted 2026-01-20 cs.RO

SUNSET - A Sensor-fUsioN based semantic SegmEnTation exemplar for ROS-based self-adaptation

Andreas Wiedholz , Rafael Paintner , Alwin Hoffmann , Carlos Hernandez , Tobias Huber This is my paper
classification cs.RO
keywords self-adaptationROS2exemplarsensor fusionsemantic segmentationuncertaintyroot cause analysisbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces SUNSET, a ROS2-based exemplar for evaluating self-adaptive robotic systems. It claims to be the first robotic exemplar where concurrent uncertainties can have unknown sources, producing a one-to-many mapping between observable symptoms and possible adaptations. This matters because it lets researchers test how managing systems diagnose, prioritize, and resolve failures under ambiguity. The exemplar ships a complete sensor-fusion semantic-segmentation pipeline with a trained ML model, fault injection, a baseline controller, and metrics for both managed and managing system quality.

Core claim

Central claim: SUNSET is the first ROS-based exemplar explicitly designed so that a detected symptom does not determine its cause. A single warning symptom—reduced segmentation performance—can be produced by four distinct faults (degraded RGB input, wrongly applied image enhancement, RGB–depth misalignment, or noisy depth), and three error symptoms can each arise at different nodes; meanwhile several uncertainties can be injected concurrently, including cascading outages. The paper argues this makes SUNSET a valid testbed for architecture-based self-adaptation requiring root-cause analysis and prioritization among self-healing and self-optimizing actions.

What carries the argument

The load-bearing mechanism is the symptom–source decoupling implemented through the mean entropy of the segmentation model's logits. The exemplar exposes an entropy spike as symptom S4 but does not reveal which of four possible faults caused it, creating the 1:n mapping. Around this, the uncertainty injector can combine faults across ROS diagnostic criticality levels, and the baseline MAPE-K managing system demonstrates the expected interface, using redeploy for low-frequency nodes and recalibration for high entropy.

Load-bearing premise

The central assumption is that mean entropy of the segmentation logits reliably signals degraded segmentation for all four S4 fault causes and that the 0.06 threshold cleanly separates healthy from degraded operation; the paper does not quantify how entropy tracks IoU per fault or justify the threshold.

What would settle it

Inject each of the four S4 faults separately while recording logit entropy and per-frame IoU over the same frames. If any fault causes a measurable IoU drop while entropy remains below the 0.06 baseline (or healthy runs cross the threshold), then the symptom detector fails to distinguish that cause from healthy operation, and the claimed 1:n mapping test is not exercised for that fault.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any ROS2 managing system can be plugged into SUNSET and compared on identical, repeatable fault combinations.
  • Evaluations on SUNSET must include root-cause analysis, since a single symptom has multiple candidate adaptations.
  • The concurrent fault combinations allow studying trade-offs between self-healing (restart/redeploy) and self-optimizing (reparametrization) actions under time pressure.
  • The provided baseline numbers (IoU 0.40 with adaptations vs. 0.28 without) give a reference point for future managing systems.
  • Because the pipeline uses three trained models (RGB-only, depth-only, fused), adaptations can change modalities to keep functioning after sensor loss.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The entropy-based symptom detector could transfer to other perception tasks as a generic, architecture-agnostic degradation signal, but its sensitivity to different fault types would need re-validation.
  • The fixed 0.06 threshold and bag-replayed data limit distribution shift; extending SUNSET to perturb model inputs more broadly (e.g., noise, weather) would test whether entropy remains a faithful proxy.
  • The benchmark implicitly pressures managing systems to exploit signals beyond the provided symptom (e.g., topic timing, image statistics) to disambiguate the four S4 causes—making it a test of diagnostic resourcefulness.
  • One could extend SUNSET to other sensor modalities or tasks (e.g., LiDAR-camera fusion) while keeping the same 1:n ambiguity structure.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents SUNSET, a ROS2-based exemplar for evaluating self-adaptive robotic systems. It implements a sensor-fusion semantic segmentation pipeline (RGB and depth) with three trained Deeplab-V3 models, an uncertainty injector that can induce 11 distinct faults, and five observable symptoms: communication outages in the camera/fusion/segmentation nodes (S1–S3), reduced segmentation performance detected via entropy of segmentation logits (S4), and camera blur (S5). The exemplar supports four adaptation types (reparametrization, communication change, component activation/deactivation, redeployment) and includes a MAPE-K baseline controller. The claimed novelty is that S4 exhibits a 1:n symptom-to-root-cause mapping and that concurrent uncertainties can occur, requiring a managing system to prioritize and perform root cause analysis. The paper reports baseline results in terms of IoU, downtime, reaction time, and resolution efficiency, and the code is publicly available.

Significance. If the central behavioral claims hold, SUNSET addresses a real gap in the RSASS exemplar landscape: most existing artifacts avoid concurrent uncertainties or use symptoms with a clear 1:1 mapping. SUNSET offers a modular ROS2-based testbed with a real ML model, separating the managed and managing systems, and includes a publicly available implementation, trained models, fault-injection scripts, and metrics — strengths that lower adoption barriers for comparative evaluation. The entropy-based ambiguous symptom (S4) is an attractive design for forcing root-cause analysis, and the concurrent-failure capability is an important direction for the community. However, the validity of the artifact rests on whether the entropy signal reliably detects all four S4 causes, which is currently not quantitatively demonstrated.

major comments (3)
  1. [§3.3, Fig. 2] The central claim of a 1:n symptom-to-adaptation mapping depends entirely on entropy being a reliable detector of S4. The figure shows only three of the four listed causes (the image-enhancement-on-healthy-images case is not plotted) and provides no separation statistics: there are no error bars, distributions, or ROC/AUC analysis. Consequently there is no evidence that the threshold 0.06 used by the baseline in §4.2 separates healthy from faulty operation, nor that it is robust across the three model variants or across runs. If healthy entropy fluctuates above 0.06, or if any of the four causes fails to raise entropy above threshold, the ambiguous S4 scenario degenerates to a 1:0 or 1:1 mapping, undermining the paper's stated novelty. Please add per-cause entropy distributions, a justified threshold derivation, and sensitivity analysis.
  2. [§4.2, Table 1] The baseline evaluation does not validate the exemplar's diagnosability. The baseline reacts to high entropy only with sensor-fusion recalibration, which is known to be the wrong adaptation for three of the four S4 causes; the paper reports this as a design choice, but it means the reported IoU drop and recovery give no evidence that entropy is an adequate symptom signal or that the intended 1:n challenge is actually implemented. The paper should report per-cause detection rates (true positives for each of the four S4 causes and false positives under healthy conditions) and the outcomes of the injected uncertainties for the three runs. Without this, a reader cannot judge whether SUNSET's ambiguous symptom is empirically realized.
  3. [§3.3, fourth S4 cause] The fourth S4 cause — image enhancement applied to healthy images — is asserted to have 'the same effect' on entropy as a color-space shift, but no empirical evidence is provided. This is not a minor omission: a necessary condition for the claimed four-cause ambiguity is that this cause also raises entropy. Please include a measured curve or statistical comparison in Fig. 2, or explicitly restrict the S4 ambiguity to the other three causes and adjust the claims accordingly.
minor comments (5)
  1. [§3.3] The ordering of S4 causes in the text does not match the legend of Fig. 2. Clarify whether 'Camera images degraded' includes both the deliberate color shift and the image-enhancement-on-healthy-images case, or whether the enhancement case is absent from the plot.
  2. [§4.2] The 1 Hz outage threshold is introduced without justification. State whether this is a design choice influenced by typical ROS topic rates, and provide a sensitivity value or reference.
  3. [Table 1] The table headers are mis-rendered ('u_resolved/executed', 't_react', 't_downtime'). Use readable symbols and define them in the text; also define the metric 'u_resolved/executed' as the ratio of resolved uncertainties to executed adaptations.
  4. [§4.1] The system downtime is measured only when the segmentation node stops sending data. Clarify whether S4 degradations (which do not stop the topic) are excluded from downtime, or whether downtime is intended to capture only S1–S3 outages.
  5. [§5] The conclusion refers to 'unknown sources' for all uncertainties, but only S4 has multiple root causes; S1–S3 have a 1:1 mapping to a specific node via the monitored topic. Please soften or scope the claim to avoid overstating the ambiguity.

Circularity Check

0 steps flagged

No significant circularity: SUNSET is an artifact exemplar whose 1:n symptom-to-adaptation mapping is a design property, and the baseline entropy threshold is a controller parameter rather than a fitted prediction.

full rationale

The paper does not present a derivation in which an output quantity is constructed from its own target. Its central claim is that SUNSET is an exemplar with concurrent uncertainties and a 1:n symptom-to-adaptation mapping; that mapping is explicitly a design feature (§1, §3.3), not a result inferred from data. The baseline controller's S4 detection rule (entropy > 0.06, §4.2) is a hand-specified threshold used to exercise the exemplar, not a parameter fitted to predict IoU or any other target; Figure 2 is illustrative of one run and does not serve as a fitted input to a 'prediction' in the paper. The paper even discloses a limitation that the baseline 'is not able to detect three out of four WARNING uncertainties for S4,' which weakens the demonstration but does not indicate circularity. Self-references to related work by co-authors ([1], [12], [13]) are used to position the contribution relative to existing exemplars and a survey taxonomy; they are not used as an unverified uniqueness theorem to force the design. The reported Table 1 results are descriptive benchmark measurements, not predictions recycled from fitted inputs. Any concerns about the unvalidated entropy threshold or the limited sensitivity of S4 detection are empirical-validation issues, not circular-reasoning issues. The manuscript is self-contained as an exemplar description, and no load-bearing step reduces to its own input.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The central claim rests on two hand-picked baseline thresholds, three undisclosed trained models, and three domain assumptions about ROS2, the dataset, and the entropy-to-quality relationship. No new physical or conceptual entities are introduced.

free parameters (3)
  • Baseline entropy threshold for S4 = 0.06
    The baseline managing system triggers a sensor-fusion recalibration when segmentation entropy exceeds 0.06 (§4.2). This threshold is not derived from task requirements; it was chosen to separate healthy from degraded runs in Fig. 2.
  • Baseline outage frequency threshold = 1 Hz
    The baseline redeploys any node publishing below 1 Hz (§4.2). The value is a design choice for the demo controller, not a property of the exemplar.
  • Per-modality segmentation model weights = not disclosed (trained from scratch on SynDrone)
    Each of the three pipeline configurations uses a separately trained Deeplab-V3/ResNet50 model (§3.2); the weights determine the IoU and entropy baselines but no training details, splits, or hyperparameters are reported.
axioms (3)
  • domain assumption ROS2 lifecycle and diagnostic monitoring behave as specified by ROS2 tooling.
    The adaptation mechanisms rely on LifecycleNode states and diagnostic criticality levels; the paper cites no formal verification.
  • domain assumption The SynDrone dataset provides realistic RGB-D UAV scenes and labels for semantic segmentation.
    Used as the only data basis (§3.1); no analysis of its representativeness is given.
  • domain assumption Entropy of segmentation logits correlates with segmentation quality (IoU) for all injected faults.
    Fig. 2 shows entropy increases under faults, but the relationship to IoU is not quantified per fault; this underpins symptom S4.

pith-pipeline@v1.3.0-alltime-deepseek · 7919 in / 10409 out tokens · 100377 ms · 2026-08-03T09:25:26.816702+00:00 · methodology

0 comments
read the original abstract

The fact that robots are getting deployed more often in dynamic environments, together with the increasing complexity of their software systems, raises the need for self-adaptive approaches. In these environments robotic software systems increasingly encounter (1) failures whose symptoms are easy to observe but root causes might be ambiguous or (2) multiple failures appearing concurrently. We present SUNSET, a ROS2-based exemplar that enables rigorous, repeatable evaluation of architecture-based self-adaptation in such conditions. It implements a sensor fusion semantic-segmentation pipeline driven by a trained Machine Learning (ML) model whose input preprocessing can be perturbed to induce realistic performance degradations. The exemplar exposes five observable failures, each of which can be caused by different faults and supports concurrent failures spanning self-healing and self-optimisation. SUNSET includes the segmentation pipeline, a trained ML model, fault-injection scripts, a baseline controller for further comparisons, and step-by-step integration and evaluation documentation to facilitate reproducible studies. The code is available at https://github.com/XITASO/sunset.

Figures

Figures reproduced from arXiv: 2601.13732 by Alwin Hoffmann, Andreas Wiedholz, Carlos Hernandez, Rafael Paintner, Tobias Huber.

Figure 1
Figure 1. Figure 1: Overview SUNSET. Managed system and additional [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The effect of different uncertainties on the entropy [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Potential adaptations in SUNSET. Blue := Reparametrization, Green := Change of communication, Orange := Activation, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 1 linked inside Pith

  1. [1]

    Elvin Alberts, Ilias Gerostathopoulos, Ivano Malavolta, Carlos Hernández Corbato, and Patricia Lago. 2025. Software architecture-based self-adaptation in robotics. Journal of Systems and Software219 (Jan. 2025), 112258. doi:10.1016/j.jss.2024. 112258

    doi:10.1016/j.jss.2024 2025

  2. [2]

    Elvin Alberts, Ilias Gerostathopoulos, Vincenzo Stoico, and Patricia Lago. 2024. ReBeT: Architecture-based Self-adaptation of Robotic Systems through Behavior Trees. In2024 IEEE International Conference on Autonomic Computing and Self- Organizing Systems (ACSOS). doi:10.1109/ACSOS61780.2024.00018

    arXiv 2024

  3. [4]

    Betty H. C. Cheng, Robert Jared Clark, Jonathon Emil Fleck, Michael Austin Lang- ford, and Philip K. McKinley. 2020. AC-ROS: assurance case driven adaptation for the robot operating system. InProceedings of the 23rd ACM/IEEE Interna- tional Conference on Model Driven Engineering Languages and Systems (MOD- ELS ’20). Association for Computing Machinery, Ne...

    arXiv 2020

  4. [5]

    Gianluca Filippone, Juan Antonio Piñera García, Marco Autili, and Patrizio Pellic- cione. 2024. Handling uncertainty in the specification of autonomous multi-robot systems through mission adaptation. InProceedings of the 19th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS ’24). Association for Computing Machi...

    arXiv 2024

  5. [6]

    Eric Bernd Gil, Ricardo Caldas, Arthur Rodrigues, Gabriel Levi Gomes da Silva, Genaína Nunes Rodrigues, and Patrizio Pelliccione. 2021. Body Sensor Network: A Self-Adaptive System Exemplar in the Healthcare Domain. In2021 International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS). 224–230. doi:10.1109/SEAMS51251.2021.00...

    arXiv 2021

  6. [7]

    Calum Imrie, Rhys Howard, Divya Thuremella, Nawshin Mannan Proma, Tejas Pandey, Paulina Lewinska, Ricardo Cannizzaro, Richard Hawkins, Colin Paterson, Lars Kunze, and Victoria Hodge. 2024. Aloft: Self-Adaptive Drone Controller Testbed. InProceedings of the 19th International Symposium on Software Engi- neering for Adaptive and Self-Managing Systems (SEAMS...

    arXiv 2024

  7. [8]

    Kephart and D.M

    J.O. Kephart and D.M. Chess. 2003. The vision of autonomic computing.Computer 36, 1 (Jan. 2003), 41–50. doi:10.1109/MC.2003.1160055

    Pith/arXiv arXiv 2003

  8. [9]

    Steven Macenski, Tully Foote, Brian Gerkey, Chris Lalancette, and William Woodall. 2022. Robot Operating System 2: Design, architecture, and uses in the wild.Science Robotics7, 66 (May 2022), eabm6074. doi:10.1126/scirobotics. abm6074 Publisher: American Association for the Advancement of Science

    doi:10.1126/scirobotics 2022

  9. [10]

    Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. 2022. Image Segmentation Using Deep Learning: A Survey.IEEE Transactions on Pattern Analysis and Machine Intelligence44, 7 (July 2022), 3523–3542. doi:10.1109/TPAMI.2021.3059968

    arXiv 2022

  10. [11]

    Giulia Rizzoli, Francesco Barbato, Matteo Caligiuri, and Pietro Zanuttigh. 2023. SynDrone – multi-modal UAV dataset for urban scenarios. In2023 IEEE/CVF international conference on computer vision workshops (ICCVW). IEEE Computer Society, Los Alamitos, CA, USA, 2202–2212. doi:10.1109/ICCVW60793.2023.00235

    arXiv 2023

  11. [12]

    Lizeth Tapia Tarifa, Einar Broch Johnsen, and Carlos Hernández Corbato

    Gustavo Rezende Silva, Juliane Päßler, S. Lizeth Tapia Tarifa, Einar Broch Johnsen, and Carlos Hernández Corbato. 2025. ROSA: A Knowledge-based Solution for Robot Self-Adaptation.Frontiers in Robotics and AI12 (May 2025), 1531743. doi:10.3389/frobt.2025.1531743 arXiv:2505.00733 [cs]. Preprint, Jan 20, 2026, Germany Wiedholz et al

    arXiv 2025

  12. [13]

    Lizeth Tapia Tarifa, Ilias Gerostathopoulos, Einar Broch Johnsen, and Car- los Hernández Corbato

    Gustavo Rezende Silva, Juliane Päßler, Jeroen Zwanepol, Elvin Alberts, S. Lizeth Tapia Tarifa, Ilias Gerostathopoulos, Einar Broch Johnsen, and Car- los Hernández Corbato. 2023. SUAVE: An Exemplar for Self-Adaptive Underwater Vehicles. In2023 IEEE/ACM 18th Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS). 181–187. doi:10.11...

    arXiv 2023

  13. [14]

    Maria Tzelepi and Anastasios Tefas. 2021. Semantic Scene Segmentation for Robotics Applications. In2021 12th International Conference on Information, Intel- ligence, Systems & Applications (IISA). 1–4. doi:10.1109/IISA52424.2021.9555526

    arXiv 2021

  14. [15]

    Thomas Vogel. 2018. mRUBiS: an exemplar for model-based architectural self- healing and self-optimization. InProceedings of the 13th International Conference on Software Engineering for Adaptive and Self-Managing Systems (SEAMS ’18). Association for Computing Machinery, New York, NY, USA, 101–107. doi:10. 1145/3194133.3194161

    arXiv 2018

  15. [16]

    2020.An introduction to self-adaptive systems: A contemporary software engineering perspective

    Danny Weyns. 2020.An introduction to self-adaptive systems: A contemporary software engineering perspective. John Wiley & Sons

    2020