Exploring Fully Convolutional Networks for the Segmentation of Hyperspectral Imaging Applied to Advanced Driver Assistance Systems

In\'es del Campo; Javier Echanobe; Jon Guti\'errez-Zaballa; Koldo Basterretxea; M. Victoria Mart\'inez

REVIEW 4 major objections 4 minor 13 references

Exploring Fully Convolutional Networks for the Segmentation of Hyperspectral Imaging Applied to Advanced Driver Assistance Systems

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that convolutional spatial context, not just spectral reflectance, is what makes hyperspectral driving-scene segmentation work, and that a compact U-Net can deliver it on an embedded chip.

desk verdict An honest engineering paper on U-Net for HSI-ADAS whose own ablation quietly guts the "spectral helps" framing, but the deployment data is solid enough to referee. read the letter →

arxiv 2412.03982 v1 pith:R3BHKMC7 submitted 2024-12-05 cs.CV cs.AIcs.LGeess.IV

Jon Gutiérrez-Zaballa , Koldo Basterretxea , Javier Echanobe , M. Victoria Martínez , Inés del Campo This is my paper · ORCID

classification cs.CVcs.AIcs.LGeess.IV

keywords hyperspectralimagingsemanticsegmentationfullyconvolutionalnetworksU-Netadvanceddriverassistancesystemsnear-infraredreflectancemodelquantizationMPSoC

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that adding spatial context through convolutional filters materially improves segmentation of near-infrared hyperspectral driving images compared with using spectral information alone. On the HSI-Drive v1.1 dataset, a compact U-Net reaches 87.66 overall IoU for five classes and 91.50 for three classes, while a spectral-only feedforward network reaches 68.45 and 77.46. The authors also prototype the model with INT8 quantization on a Zynq MPSoC, reaching 27 FPS for inference and 2.55 FPS when the full preprocessing pipeline is included. If true, this supports the viability of hyperspectral scene understanding in embedded automotive systems.

What carries the argument

The load-bearing mechanism is a modified U-Net, an encoder-decoder fully convolutional network with skip connections, fed with 128x128x25 hyperspectral patches. The encoder's 3x3 convolutions over the 25 spectral bands produce spatial-spectral features, and the decoder upsamples them back to per-pixel class scores; overlapping patches are merged to rebuild full images. The comparison baseline is a three-hidden-layer feedforward ANN with a 25-25-100-100-3 structure operating on individual pixels, isolating the effect of spatial context.

What would settle it

Train a spectral-only classifier with comparable effort and capacity on the same HSI-Drive v1.1 splits; if its 5-class overall IoU reaches or exceeds the U-Net's 87.66, or the gap drops below noise, the claim that convolutional spatial context is beneficial for HSI segmentation would be falsified. A simpler check is to rerun both models with several random seeds and report confidence intervals, since the 19-point IoU gap in the 5-class experiment would need to persist.

Watch

Extended reading notes

Core claim

The authors claim that, for semantic segmentation of hyperspectral driving scenes, feeding a convolutional network both spectral reflectance and spatial neighborhood structure yields substantially better segmentation than a purely spectral classifier. Their evidence is the comparison on HSI-Drive v1.1, where the modified U-Net (encoder depth 2, 8 initial filters, 128x128x25 patches) outperforms a three-hidden-layer spectral ANN on all metrics, with the largest relative gains on the minority Road Marks class. They further report that the U-Net degrades only slightly when spectral bands are reduced, whereas a spectral classifier's accuracy varies by nearly 40%, indicating that spatial information can compensate for missing spectral information. The paper frames this as a step toward hyperspectral ADAS, but notes that spectral information is still being overshadowed by spatial information in FCN training, so better fusion of the two is left for future work.

Load-bearing premise

The evidence that spatial context helps rests on comparing the U-Net with a single hand-tuned spectral-only MLP, with no repeated runs or error bars, so the size of the claimed benefit is not quantified.

Editorial extensions

If this is right

In the 3-class experiment, rebuilding images from overlapping 128x128 patches raises overall IoU to 91.50 and improves precision, because pixels near patch borders get missing surrounding context.
The U-Net's IoU advantage over the spectral ANN is largest for Road Marks (69.80 vs 19.86 in the 3-class experiment, and 64.90 vs 16.21 in the 5-class experiment), a minority class that benefits from spatial regularity.
Reducing spectral bands hurts the U-Net far less than it hurts a spectral classifier, so spatial context can compensate for missing spectral information; a single principal-component band costs only 0.75% overall accuracy.
An INT8-quantized U-Net deployed on the Zynq MPSoC reaches 27 FPS inference (487.91 FPS with two DPU cores), and 2.55 FPS when the 353.97 ms preprocessing pipeline is counted.
The 5-class quantized model shows noticeable degradation on some images, so quantization-aware training or finetuning is needed before deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A stronger spectral-only baseline, such as a larger MLP or a one-dimensional convolutional classifier over the 25 bands with the same tuning budget, could narrow the reported gap; the paper does not test this, so the magnitude of the spatial-context benefit is an upper-bound estimate.
Because preprocessing dominates the total latency (353.97 ms vs 36.89 ms for inference), hardware-accelerating the demosaicing and median-filtering steps would be the fastest route to real-time hyperspectral ADAS; the authors recommend this but do not implement it.
The robustness to reduced spectral bands suggests a practical fallback mode: if the hyperspectral sensor degrades, a U-Net trained on a single grayscale-like band could keep the segmentation system running with acceptable accuracy.
The same U-Net plus preprocessing pipeline could be evaluated on other HSI driving datasets or extended with 3D convolutions to fuse spectral and spatial information more explicitly, both directions the authors mention as future work.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful thing here is the deployment work: first U-Net results on HSI-Drive v1.1, a clean INT8 flow through Vitis AI onto a Zynq MPSoC, and honest throughput numbers including the 354 ms preprocessing cost that brings the system to 2.55 FPS. The paper also reports 1000-run stats on three device types, which is more than most papers at this level do. Credit where it's due: the authors flag the all-class results as inconclusive and explicitly say the spectral contribution is overshadowed by spatial information in the FCN. That level of candor is real.

The main comparison, U-Net versus a spectral-only MLP, shows a large gap (87.66 vs 68.45 overall IoU on five classes). The gap is plausible, but there are no error bars and no repeated training runs, so we don't know if it's 20 points or 5. The MLP baseline is hand-tuned and the tuning effort isn't quantified, so I can't tell if it's a weak baseline or a fair one. That's a genuine soft spot, but it's a standard one for this kind of study.

The stress-test note claims the paper's own single-band PCA ablation undermines the central claim. I think that's overstated: the paper says the same thing itself — a one-band U-Net loses only 0.75% overall accuracy, and the conclusion admits spectral information is overshadowed. What the evidence doesn't support is the framing that HSI's spectral richness is what helps the FCN. The system comparison shows spatial context helps a lot; the ablation shows spectral bands add little on top of one PCA component. So the "joint use is beneficial" claim is true only in a weak sense, and the authors almost say so. The real problem is they still conclude that "the incorporation of richer spectral information through HSI improves the segmentation results" without restricting it to purely spectral models. That sentence should have been qualified.

Missing artifacts are the biggest practical issue. No code, no trained models, no preprocessing pipeline released. Given the dataset is the authors' own, independent verification is impossible without those. The measurements themselves look reproducible in principle, and the paper is a decent engineering data point for anyone building HSI-based ADAS perception on embedded FPGAs.

My verdict: conditional accept, leaning toward revision. The engineering is worth refereeing, but the scientific claim needs repeated runs, error bars, and a release of code/data to survive scrutiny. The authors should also either soften the HSI-benefit conclusion or present the ablation as evidence against it, not as an aside.

Referee Report

4 major / 4 minor

Summary. The paper explores whether a fully convolutional U-Net can exploit spatial context for hyperspectral image segmentation in driving scenes, using the HSI-Drive v1.1 dataset with 25 VIS-NIR bands. It reports 3-class and 5-class segmentation results, compares them with a purely spectral three-hidden-layer ANN, and prototypes the U-Net plus the hyperspectral-cube preprocessing pipeline on a Xilinx Zynq MPSoC with INT8 quantization. The authors claim that the joint use of spatial and spectral information is beneficial for segmentation and report 27 FPS inference (2.55 FPS including preprocessing) on the MPSoC.

Significance. If validated, the paper would provide a useful data point for embedded hyperspectral ADAS: a compact U-Net can segment road scenes into drivable/non-drivable areas, and the full pipeline can be deployed on an MPSoC at near-real-time throughput. The strengths are the use of a real driving dataset, a reasonably transparent prototyping workflow, and concrete throughput measurements on three device types. However, the central empirical claim is not yet established because the spectral-only baseline is a single hand-tuned ANN, no repeated runs or error bars are reported, and the paper's own single-band ablation shows that spatial context alone nearly matches the full 25-band U-Net performance.

major comments (4)

[Section 3.2, Table 2] The claim that the U-Net's advantage demonstrates the benefit of spatial information rests on a single spectral-only MLP with architecture 25-25-100-100-3. No standard deviations, repeated runs, or statistical tests are reported for either model, and the MLP hyperparameters were selected by the authors on the same data. The large IoU gap could be partly due to tuning disparity rather than the value of spatial features. Please report mean and standard deviation over multiple training runs (or otherwise quantify variability) and compare against a stronger spectral baseline such as a 1D CNN or a well-tuned spectral classifier with comparable effort.
[Section 3.2, Fig. 3 paragraph] The single-band ablation is more damaging than the weak-baseline concern. The text states that using only one spectral band degrades the U-Net's overall accuracy by 0.75% (first PCA component) or 1.75% (one arbitrary band). If these numbers are measured on the test set, spatial context alone almost reproduces the full 25-band performance, which directly contradicts the conclusion that 'the incorporation of richer spectral information through HSI improves the segmentation results.' Please state explicitly whether this ablation was performed on the validation or test split, provide per-class and IoU metrics for the ablated model, and temper the spectral-benefit claim accordingly.
[Section 4.2, Fig. 4] The deployment claim depends on the INT8 quantized model preserving accuracy, but the paper only says that the 5-class quantized model has 'a noticeable loss of performance on some images' and reports undefined similarity percentages (97.82%, 98.16%, 98.66%) for three example images. Please report quantitative metrics (overall IoU, per-class IoU, or accuracy) of the quantized model on the full test set, alongside the float model, and define how the similarity percentages were computed.
[Section 3, first paragraph] All-class segmentation results are excluded as 'irregular and not concluding,' yet the 10-class task is the original labeling of HSI-Drive v1.1. This exclusion limits the scope of the general claim about HSI segmentation for ADAS. Please state this limitation explicitly in the abstract or conclusions and provide at least a brief quantitative summary of what was observed (for example, mean IoU or which classes failed), so readers can judge how far the proposed approach extends beyond the 3- and 5-class settings.

minor comments (4)

[Section 2.1] Typographical errors: 'Fabri-Perot' should be 'Fabry-Perot' and 'lightning conditions' should be 'lighting conditions.'
[Section 2.1, Table 1] The column header 'M.V eg.' appears malformed and should be 'Painted Metal' or an unambiguous abbreviation; also add a space in 'JeffreysMatusita' throughout.
[Section 3, Equations (1)-(3)] The definitions of 'Overall', 'Mean', and 'Weighted' metrics in Table 2 are not given. Please specify how they are computed from the per-class values (e.g., micro-averaged pixel counts versus macro-averaged class means, and the exact weighting factors).
[Section 3.2, Fig. 3] Figure 3 lacks explicit axis labels and a legend describing which classifier(s) the curves refer to. Please clarify whether the x-axis is the number of spectral bands and whether the curve is only for the ELM or also for the U-Net.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper reports empirical measurements and comparisons rather than deriving conclusions from their own definitions.

full rationale

The paper's claims are empirical: it trains a U-Net and a spectral MLP on the HSI-Drive dataset and compares measured IoU/accuracy values. There is no equation in which an output quantity is defined in terms of the quantity it is said to predict, and no fitted parameter is renamed as a prediction. The only self-citation is the HSI-Drive dataset [1] created by the same group; a dataset used as a measurement target is independent evidence in the relevant sense, and its provenance does not make the comparison circular. The weak MLP baseline and the single-band PCA ablation (0.75% accuracy drop) are substantive threats to the strength of the conclusion that spectral information helps, but they are concerns about experimental control and inference, not about circularity. The paper also explicitly concedes that all-class results are irregular and that the 5-class quantized model degrades noticeably, which further limits the breadth of the claims without introducing a circular step. No step of the paper reduces by construction to its own inputs, so the appropriate score is 0.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

All model components are standard; the only new resource is the HSI-Drive dataset, which is a measurement, not a postulated entity. The paper introduces no new physical entity, force, or conserved quantity. The free parameters are the architecture and class-selection choices that the central comparison depends on.

free parameters (6)

U-Net encoder depth = 2
Selected by grid search over {2,3,4} on a 45-image subset; directly determines model capacity in the central comparison.
U-Net initial filters = 8
Selected from {8,16,32}; the paper chose the 2/8/128 set for an accuracy/complexity trade-off.
U-Net patch size = 128
Evaluated values 64 and 128; 128 was chosen, and test images are tiled into 18 patches.
Patch overlap/tiling = 18 patches (3x6) with overlap
Overlap was chosen to avoid patch-boundary errors, but the overlap degree is not specified.
ANN hidden layer sizes = 25-25-100-100-3
Baseline spectral MLP architecture found by exploration; the strength of this baseline underpins the spatial-info claim.
Class subset = 3 classes and 5 classes
Only favorable subsets are reported; all-class results were excluded as irregular, a selection affecting the generality of the claim.

assumptions (6)

domain assumption Material surfaces in driving scenes have NIR spectral reflectances that are sufficiently distinct to support class separation.
Stated in the abstract and Section 1 as the motivation for HSI in ADAS.
domain assumption The weak labelling of HSI-Drive, which leaves boundary pixels unlabeled, is still suitable for training convolutional networks.
Section 2.1 states unlabeled boundary pixels 'do take part in the training process' without validating this choice.
domain assumption Jeffreys-Matusita distance is a valid proxy for how well a learned classifier will separate the classes.
Used in Section 2.1 to select classes and motivate the experiments.
domain assumption The U-Net architecture is an appropriate inductive bias for this segmentation task.
Section 2.2 adopts U-Net from [9] with modifications; no comparison to other FCN families.
domain assumption The 45-image hyperparameter subset and the 162/57/57 train/validation/test split are representative of the full dataset.
Sections 2.2 and 3 describe the split; no statistical check of representativeness is given.
domain assumption The reported execution times on the MPSoC are representative of steady-state performance.
Section 4 reports means over 1000 runs, but the hardware and software configuration is not fully specified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Exploring Fully Convolutional Networks for the Segmentation of Hyperspectral Imaging Applied to Advanced Driver Assistance Systems." pith.science (2026). https://pith.science/paper/R3BHKMC7

@misc{pith2026241203982,
  author       = {Pith},
  title        = {Pith review of: Exploring Fully Convolutional Networks for the Segmentation of Hyperspectral Imaging Applied to Advanced Driver Assistance Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R3BHKMC7}},
  note         = {Machine review of arXiv:2412.03982}
}

read the original abstract

Advanced Driver Assistance Systems (ADAS) are designed with the main purpose of increasing the safety and comfort of vehicle occupants. Most of current computer vision-based ADAS perform detection and tracking tasks quite successfully under regular conditions, but are not completely reliable, particularly under adverse weather and changing lighting conditions, neither in complex situations with many overlapping objects. In this work we explore the use of hyperspectral imaging (HSI) in ADAS on the assumption that the distinct near infrared (NIR) spectral reflectances of different materials can help to better separate the objects in a driving scene. In particular, this paper describes some experimental results of the application of fully convolutional networks (FCN) to the image segmentation of HSI for ADAS applications. More specifically, our aim is to investigate to what extent the spatial features codified by convolutional filters can be helpful to improve the performance of HSI segmentation systems. With that aim, we use the HSI-Drive v1.1 dataset, which provides a set of labelled images recorded in real driving conditions with a small-size snapshot NIR-HSI camera. Finally, we analyze the implementability of such a HSI segmentation system by prototyping the developed FCN model together with the necessary hyperspectral cube preprocessing stage and characterizing its performance on an MPSoC.

Figures

Figures reproduced from arXiv: 2412.03982 by the authors.

**Figure 1.** Architecture of the modified U-Net. The set of analyzed hyperparameters included: the size of the input image patches, the overlapping between patches, the encoder depth and the number of filters in the first convolutional block. In order to avoid an unaffordable optimization time, we have consulted the typical values of the hyperparameters to be optimized in the literature [10]. This way, a specific range has been… view at source ↗

**Figure 2.** Comparison among the visible (first row), 3-class ground truth (second [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Overall accuracy (%) as a function of the number of spectral channels. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Segmented images produced by the deployed model on the MPSoC. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages

[1]

In: 2021 IEEE Intelligent Vehicles Symposium (IV)

Basterretxea, K., Mart ´ ınez, V., Echanobe, J., Guti´ errez–Zaballa, J., Del Campo, I.: Hsi-drive: A dataset for the research of hyperspectral image processing applied to autonomous driving systems. In: 2021 IEEE Intelligent Vehicles Symposium (IV). pp. 866–873 (2021). https://doi.org/10.1109/IV48863.2021.9575298

arXiv 2021
[2]

In: Proceedings of the Asian Conference on Computer Vision (2020)

Courdier, E., Fleuret, F.: Real-time segmentation networks should be latency aware. In: Proceedings of the Asian Conference on Computer Vision (2020)

work page 2020
[3]

Remote Sensing 11(19), 2220 (2019)

Cui, X., Zheng, K., Gao, L., Zhang, B., Yang, D., Ren, J.: Multiscale spatial- spectral convolutional network with image-based framework for hyperspectral im- agery classification. Remote Sensing 11(19), 2220 (2019)

work page 2019
[4]

International journal of remote sensing 34(7), 2327–2349 (2013)

Forestier, G., Inglada, J., Wemmert, C., Gan¸ carski, P.: Comparison of optical sen- sors discrimination ability using spectral libraries. International journal of remote sensing 34(7), 2327–2349 (2013)

work page 2013
[5]

Water Sa 33(2), 145– 151 (2007)

Govender, M., Chetty, K., Bulcock, H.: A review of hyperspectral remote sensing and its application in vegetation and water resource studies. Water Sa 33(2), 145– 151 (2007)

work page 2007
[6]

Weakly-supervised Semantic Segmentation in Cityscape via Hyperspectral Image

Huang, Y., Huang, E., Chen, L., You, S., Fu, Y., Shen, Q.: Hyperspectral image semantic segmentation in cityscapes. arXiv preprint arXiv:2012.10122 (2020)

work page Pith review arXiv 2020
[7]

https://pypi.org/project/onnx2keras/ (2021)

Malivenko, G.: onnx2keras 0.0.24. https://pypi.org/project/onnx2keras/ (2021)

work page 2021
[8]

https://www.photonfocus.com/ products/camerafinder/camera/mv1-d2048x1088-hs02-96-g2

Photonfocus: MV1-D2048x1088-HS02-96-G2. https://www.photonfocus.com/ products/camerafinder/camera/mv1-d2048x1088-hs02-96-g2

work page

Show all 13 references

[9]

In: International Conference on Medical image computing and computer-assisted intervention

Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomed- ical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)

2015
[10]

Remote Sensing 12(2), 207 (2020)

Wang, S., Chen, W., Xie, S.M., Azzari, G., Lobell, D.B.: Weakly supervised deep learning for segmentation of remote sensing imagery. Remote Sensing 12(2), 207 (2020)

2020
[11]

In: Proceedings of the IEEE International Conference on Computer Vision Workshops

Winkens, C., Sattler, F., Adams, V., Paulus, D.: Hyko: A spectral dataset for scene understanding. In: Proceedings of the IEEE International Conference on Computer Vision Workshops. pp. 254–261 (2017)

2017
[12]

In: VISIGRAPP (5: VISAPP)

Winkens, C., Sattler, F., Paulus, D.: Hyperspectral terrain classification for ground vehicles. In: VISIGRAPP (5: VISAPP). pp. 417–424 (2017)

2017
[13]

https://www.xilinx.com/html_docs/vitis_ai/1_ 4/quantize.html#uim1570695919827 (2021)

Xilinx: Quantizing the model. https://www.xilinx.com/html_docs/vitis_ai/1_ 4/quantize.html#uim1570695919827 (2021)

2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

In: 2021 IEEE Intelligent Vehicles Symposium (IV)

Basterretxea, K., Mart ´ ınez, V., Echanobe, J., Guti´ errez–Zaballa, J., Del Campo, I.: Hsi-drive: A dataset for the research of hyperspectral image processing applied to autonomous driving systems. In: 2021 IEEE Intelligent Vehicles Symposium (IV). pp. 866–873 (2021). https://doi.org/10.1109/IV48863.2021.9575298

arXiv 2021

[2] [2]

In: Proceedings of the Asian Conference on Computer Vision (2020)

Courdier, E., Fleuret, F.: Real-time segmentation networks should be latency aware. In: Proceedings of the Asian Conference on Computer Vision (2020)

work page 2020

[3] [3]

Remote Sensing 11(19), 2220 (2019)

Cui, X., Zheng, K., Gao, L., Zhang, B., Yang, D., Ren, J.: Multiscale spatial- spectral convolutional network with image-based framework for hyperspectral im- agery classification. Remote Sensing 11(19), 2220 (2019)

work page 2019

[4] [4]

International journal of remote sensing 34(7), 2327–2349 (2013)

Forestier, G., Inglada, J., Wemmert, C., Gan¸ carski, P.: Comparison of optical sen- sors discrimination ability using spectral libraries. International journal of remote sensing 34(7), 2327–2349 (2013)

work page 2013

[5] [5]

Water Sa 33(2), 145– 151 (2007)

Govender, M., Chetty, K., Bulcock, H.: A review of hyperspectral remote sensing and its application in vegetation and water resource studies. Water Sa 33(2), 145– 151 (2007)

work page 2007

[6] [6]

Weakly-supervised Semantic Segmentation in Cityscape via Hyperspectral Image

Huang, Y., Huang, E., Chen, L., You, S., Fu, Y., Shen, Q.: Hyperspectral image semantic segmentation in cityscapes. arXiv preprint arXiv:2012.10122 (2020)

work page Pith review arXiv 2020

[7] [7]

https://pypi.org/project/onnx2keras/ (2021)

Malivenko, G.: onnx2keras 0.0.24. https://pypi.org/project/onnx2keras/ (2021)

work page 2021

[8] [8]

https://www.photonfocus.com/ products/camerafinder/camera/mv1-d2048x1088-hs02-96-g2

Photonfocus: MV1-D2048x1088-HS02-96-G2. https://www.photonfocus.com/ products/camerafinder/camera/mv1-d2048x1088-hs02-96-g2

work page