REVIEW 3 major objections 5 minor 1 cited by
End-to-End Steering for Autonomous Vehicles via Conditional Imitation Co-Learning
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Letting specialist branches share features lifts unseen-town driving success by 62%.
desk verdict A novel co-learning architecture that plausibly improves CIL generalization in simulation, but the headline 62% rests on a single unseeded run of a deliberately narrowed benchmark. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the co-learning matrix $C_t$ that linearly mixes the four specialist branches' feature vectors, $\hat{A}_t$, before the action is read out: $A_t = C_t \hat{A}_t$. The diagonal entries of $C_t$ are fixed at 1 and the off-diagonal coefficients are produced by gated hyperbolic tangent units (GTUs), so the network can dynamically decide, per driving scenario, how much each branch's features should influence every other branch's prediction. The paper also introduces a classification-regression hybrid loss (categorical cross-entropy plus mean-squared error on the expected steering) and a co-existence probability matrix loss that nudges output scores toward a Gaussian spatial distribution over steering classes, but it is the GTU-driven co-learning matrix that carries the headline generalization gain.
What would settle it
Take the same GTU co-learning model and the unmodified CIL baseline, and evaluate both on the full route set without dropping multi-turn routes or samples with steering magnitude above 0.8. If the gap in reach-destination success over unseen towns and weathers shrinks to a small margin, the 62% claim is specific to the filtered single-turn, moderate-steering task.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that the conditional imitation learning (CIL) assumption that each navigational-command branch should act independently is the bottleneck to generalization. By computing an $N \times N$ co-learning matrix at each time step and multiplying it with the vector of branch-specific output features, the proposed conditional imitation co-learning (CIC) model lets a branch trained mainly on right turns, for example, draw on features learned from left-turn and straight-driving data. In the benchmark, the GTU-generated co-learning matrix reaches a 75.00% reach-destination success rate in an unseen town with unseen weather, versus 46.25% for the baseline CIL regression model, the 62% improvement cited in the abstract. The paper further reports that classifying steering into nine discrete classes with a hybrid cross-entropy plus mean-squared-error loss improves unseen-environment success by 21% over the regression baseline, while plain classification alone does not beat regression.
Load-bearing premise
The 62% generalization gain is measured on a benchmark that keeps only single-turn routes and drops steering samples with magnitude above 0.8, so the claim depends on those filters matching what 'unseen environment' means in practice.
Editorial extensions
If this is right
- A network trained in one town and two weather conditions reaches 75% success in a new town under new weather conditions, versus 46.25% for the unmodified conditional imitation learning model.
- The co-learning mechanism is output-oriented: it mixes the final branch features rather than hidden-layer parameters, so it can be added atop an existing CIL-style architecture without changing the feature extractor.
- Posing steering regression as classification with a hybrid cross-entropy plus mean-squared-error loss improves unseen-environment success by 21% over the regression baseline, though classification alone does not.
- Existing multi-task sharing methods (mixture of experts, soft parameter sharing, sluice networks) improve over CIL in new towns but none of them reaches the GTU co-learning model's success rates.
Reading between the lines
- Because the benchmark drops multi-turn routes and samples with steering magnitude above 0.8, the reported 62% gain is established only for moderate single-turn maneuvers; extending the evaluation to the full route distribution would test whether the co-learning benefit persists.
- The co-learning matrix idea is not specific to steering: any multi-branch policy or task module that shares an output layer could use the same GTU-generated mixing coefficients, so the mechanism could transfer to other imitation-learning or multi-task settings.
- The spatial-class loss via co-existence probability with a Gaussian kernel is one choice; comparing it to learned kernels or to the sine-wave encoding on the same benchmark would separate the benefit of the spatial prior from the kernel's shape.
- If the GTU coefficients are interpreted per scenario, they may be inspected to see which branch-to-branch transfers matter most, potentially guiding data collection toward under-connected navigation commands.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes conditional imitation co-learning (CIC), an extension of the conditional imitation learning (CIL) architecture of Codevilla et al. (2018) for end-to-end steering. CIC lets the specialist branches (left, right, straight, follow-lane) share output-level information through a co-learning matrix, with coefficients generated either from a manually fixed binary relationship matrix R multiplied by learned coefficients or from gated tanh units (GTUs). The paper also recasts steering regression as classification, adding a hybrid cross-entropy/MSE loss (Eq. 3) and a co-existence-probability loss based on a Gaussian spatial relationship matrix (Eq. 4). Experiments on a modified CoRL benchmark in CARLA report that the GTU-based co-learning model achieves 75.00% reach-destination success in new-town/new-weather conditions versus 46.25% for the CIL regression baseline, a 62% relative improvement, while the hybrid-loss and co-existence-loss variants show smaller gains. The paper concludes that CIC improves generalization robustness in unseen environments.
Significance. If the reported gain is robust, the paper makes a useful empirical contribution by showing that a simple output-level mixing mechanism between CIL specialist branches can yield substantial generalization improvements over the standard architecture, and by providing ablations of classification-based steering losses with baselines including mixture of experts, sluice networks, and soft parameter sharing. The idea of learning branch relationships through GTUs is plausible and the comparison against multiple MTL baselines is informative. However, the central quantitative claim currently rests on a single unseeded run of a deliberately narrowed benchmark, and the hyperparameter selection procedure is not described as respecting a clean train/validation/test split. These issues make the headline 62% claim fragile as it stands.
major comments (3)
- [Section 4, Tables 1 and 4] The headline 62% improvement is computed as (75.00 - 46.25) / 46.25 from single point estimates, with no standard deviations, confidence intervals, or multiple runs/seeds reported. CARLA evaluation includes stochasticity in weather, traffic, and autopilot behavior; a single draw (60/80 versus 37/80 successful routes) is not sufficient evidence for the claimed generalization advantage. Please provide results over multiple seeds and report statistics (mean and variance or a significance test) for all key comparisons.
- [Section 4, benchmark design] The evaluation is restricted to single-turn routes only, and training samples with steering magnitude above 0.8 are discarded. The claim in the abstract and conclusion that the model 'improve[s] autonomous driving success rate in unseen environment' is therefore demonstrated only for moderate single turns in two towns and two new weathers. The decisive test is evaluation on multi-turn routes and sharp-curve scenarios from the full CoRL benchmark; without this, the generalization claim is overstated. Please either extend the evaluation or explicitly scope the claim to the tested distribution.
- [Section 3.3, Section 3.4, Tables 2 and 3] Hyperparameter W for the hybrid loss and the co-existence loss appears to be selected by testing a set of values and reporting the best-performing rows on the same benchmark (W=10 for the hybrid loss, W=0.6 for the co-existence loss), and the binary relationship matrix R in the first co-learning variant is described as manually fine-tuned. This procedure can inflate the reported performance of the proposed variants. Please describe the hyperparameter selection protocol, and ideally use a separate validation set or cross-validation to choose W and R before evaluating on the held-out test routes.
minor comments (5)
- [Abstract] The abstract states '62% on average', but the tables show this exact relative improvement only in the new-town/new-weather cell; the averaging basis over conditions should be clarified.
- [Equation (3)] The symbol y is used for both the continuous steering value and the one-hot vector of the discretized steering; please use distinct notation (e.g., y_true and y_onehot) to avoid ambiguity.
- [Table 4] The row labeled 'stitch network' appears to refer to cross-stitch networks (Misra et al., 2016); please correct the terminology and ensure the reference is cited consistently.
- [Section 4] The paper does not state whether the route definitions, trained models, or code will be made available; providing these would significantly improve reproducibility of the reported success rates.
- [Section 3.4] The choice of Gaussian distribution with sigma^2 = 1 for the co-existence matrix is asserted without sensitivity analysis; please justify this choice or show that performance is stable over a range of sigma values.
Circularity Check
No circularity: the claim is an empirical benchmark comparison against external CARLA scenarios and prior baselines, not a derivation from its own outputs.
full rationale
The paper's central claim is empirical: the CIC co-learning model achieves a 75.00% reach-destination success rate in new-town/new-weather conditions versus 46.25% for CIL regression, quoted as a 62% improvement (Tables 1 and 4). This is a measured benchmark result on an external simulator (CARLA) with prior published baselines, not a chain of derivations whose outputs are fed back as inputs. The co-learning operation At = Ct * Ahat_t and the losses in Equations (3) and (4) are model definitions; they do not presuppose the reported success rates. The only in-paper self-citations (Eraqi et al. 2017, 2022) appear as related work or baselines, not as load-bearing justification for the central claim. Methodological weaknesses—restricting evaluation to single-turn routes, filtering training steering to |s| <= 0.8, reporting no random seeds, and selecting hyperparameters W by comparing test-table entries—are threats to generalization and reproducibility, but they are not circularity: no fitted parameter is renamed as a prediction, and no equation reduces by construction to the target result. The paper is self-contained against external benchmarks, so no circular step is identified.
Assumptions & free parameters
free parameters (7)
- Binary relationship matrix R (co-learning) =
manually fine-tuned, values not reported
- Hybrid loss weight W =
W=10 (best of 5, 10, 15)
- Co-existence loss weight W =
W=0.6 (best of 0.4, 0.6, 0.8)
- Steering discretization step and class count =
9 classes, step 0.2
- Steering training range =
[-0.8, 0.8]
- Gaussian co-existence sigma^2 =
1
- Learning rate and Adam betas =
lr=0.0002, beta1=0.70, beta2=0.85
assumptions (4)
- domain assumption CARLA autopilot labels are a valid expert for imitation learning.
- ad hoc to paper A single-turn CoRL benchmark measures generalization to unseen environments.
- ad hoc to paper Gaussian co-existence matrix with sigma^2=1 captures steering-class spatial structure.
- ad hoc to paper Steering values outside [-0.8,0.8] are irrelevant to the learned task.
invented entities (1)
-
Conditional imitation co-learning matrix C_t with GTU gate outputs
Cite this review
Pith. "Pith review of End-to-End Steering for Autonomous Vehicles via Conditional Imitation Co-Learning." pith.science (2026). https://pith.science/paper/7UOPRI2L
@misc{pith2026241116131,
author = {Pith},
title = {Pith review of: End-to-End Steering for Autonomous Vehicles via Conditional Imitation Co-Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/7UOPRI2L}},
note = {Machine review of arXiv:2411.16131}
}
read the original abstract
Autonomous driving involves complex tasks such as data fusion, object and lane detection, behavior prediction, and path planning. As opposed to the modular approach which dedicates individual subsystems to tackle each of those tasks, the end-to-end approach treats the problem as a single learnable task using deep neural networks, reducing system complexity and minimizing dependency on heuristics. Conditional imitation learning (CIL) trains the end-to-end model to mimic a human expert considering the navigational commands guiding the vehicle to reach its destination, CIL adopts specialist network branches dedicated to learn the driving task for each navigational command. Nevertheless, the CIL model lacked generalization when deployed to unseen environments. This work introduces the conditional imitation co-learning (CIC) approach to address this issue by enabling the model to learn the relationships between CIL specialist branches via a co-learning matrix generated by gated hyperbolic tangent units (GTUs). Additionally, we propose posing the steering regression problem as classification, we use a classification-regression hybrid loss to bridge the gap between regression and classification, we also propose using co-existence probability to consider the spatial tendency between the steering classes. Our model is demonstrated to improve autonomous driving success rate in unseen environment by 62% on average compared to the CIL method.
Figures
Forward citations
Cited by 1 Pith paper
-
A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach
A position/review paper argues data-driven model predictive control is the best route to safe, adaptive, human-like autonomous-driving motion planning, but provides no new derivation or experiment.
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[2]
Bekey, G. A. (2005). Autonomous robots: from biological inspiration to implementation and control . The MIT Press
work page 2005
-
[3]
Bengio, S., Dean, J., Erhan, D., Ie, E., Le, Q., Rabinovich, A., Shlens, J., and Singer, Y. (2013). Using web co-occurrence statistics for improving image categorization. ArXiv preprint , arXiv:1312.5697
work page Pith review arXiv 2013
-
[4]
Bewley, A., Rigley, J., Liu, Y., Hawke, J., Shen, R., Lam, V.-D., and Kendall, A. (2019). Learning to drive from simulation without real world labels. In 2019 International Conference on Robotics and Automation (ICRA) , pages 4817--4823. IEEE
work page 2019
-
[5]
D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L
Bojarski, M., Testa, D. D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016). End to end learning for self-driving cars. arXiv preprint , arXiv:1604.07316
arXiv 2016
-
[6]
Codevilla, F., Miiller, M., Lopez, A., Koltun, V., and Dosovitskiy, A. (2018). End-to-end driving via conditional imitation learning. In Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 1--9
work page 2018
-
[7]
Cui, J., Qiu, H., Chen, D., Stone, P., and Zhu, Y. (2022). Coopernaut: End-to-end driving with cooperative perception for networked vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 17252--17262
work page 2022
-
[8]
Dammen, J. (2019). End-to-end deep learning for autonomous driving. Master's thesis, Norwegian University of Science and Technology
work page 2019
Show all 27 references
-
[9]
N., Fan, A., Auli, M., and Grangier, D
Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D. (2017). Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning (ICML) , pages 933--941. PMLR
2017
-
[10]
Dosovitskiy, A., Ros, G., Codevilla, F., L \'o pez, A., and Koltun, V. (2017). Carla: An open urban driving simulator. In Proceedings of the Conference on Robot Learning (CoRL)
2017
-
[11]
Eraqi, H., Moustafa, M., and Honer, J. (2017). End-to-end deep learning for steering autonomous vehicles considering temporal dependencies. CoRR , abs/1710.03804
2017 arXiv
-
[12]
M., Moustafa, M
Eraqi, H. M., Moustafa, M. N., and Honer, J. (2022). Dynamic conditional imitation learning for autonomous driving. IEEE Transactions on Intelligent Transportation Systems
2022
-
[13]
Hawke, J., Shen, R., Gurau, C., Sharma, S., Reda, D., Nikolov, N., Mazur, P., Micklethwaite, S., Griffiths, N., Shah, A., et al. (2019). Urban driving with conditional imitation learning. arXiv preprint , arXiv:1912.00177
2019 arXiv
-
[14]
Huber, P. J. (1992). Robust estimation of a location parameter. In Breakthroughs in statistics , pages 492--518. Springer, New York
1992
-
[15]
A., Jordan, M
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural Computation , 3(1):79--87
1991
-
[16]
and Ba, J
Kingma, D. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint , arXiv:1412.6980
2014 arXiv
-
[17]
and Genc, Y
Kourbane, I. and Genc, Y. (2021). A hybrid classification-regression approach for 3d hand pose estimation using graph convolutional networks. arXiv preprint , arXiv:2105.10902
2021 arXiv
-
[18]
Liang, X., Wang, T., Yang, L., and Xing, E. (2018). Cirl: Controllable imitative reinforcement learning for vision-based self-driving. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 584--599
2018
-
[19]
Misra, I., Shrivastava, A., Gupta, A., and Hebert, M. (2016). Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3994--4003
2016
-
[20]
Moustafa, M. N. (2005). System and method for pose-angle estimation
2005
-
[21]
Muller, M., Dosovitskiy, A., Ghanem, B., and Koltun, V. (2018). Driving policy transfer via modularity and abstraction. In Conference on Robot Learning (CoRL) , pages 1--14
2018
-
[22]
Prakash, A., Chitta, K., and Geiger, A. (2021). Multimodal fusion transformer for end-to-end autonomous driving. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR)
2021
-
[23]
Rasmus, A., Valpola, H., Honkala, M., Berglund, M., Raiko, T., Santana, E., and Hotz, G. (2016). Learning a driving simulator. CoRR
2016
-
[24]
Rothe, R., Timofte, R., and Gool, L. V. (2015). Dex: Deep expectation of apparent age from a single image. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV) , pages 10--15
2015
-
[25]
Ruder, S., Bingel, J., Augenstein, I., and S gaard, A. (2019). Latent multi-task architecture learning. Proceedings of the AAAI Conference on Artificial Intelligence , 33(01):4822--4829
2019
-
[26]
Tampuu, A., Matiisen, T., Semikin, M., Fishman, D., and Muhammad, N. (2020). A survey of end-to-end driving: Architectures and training methods. IEEE Transactions on Neural Networks and Learning Systems
2020
-
[27]
Yurtsever, E., Lambert, J., Carballo, A., and Takeda, K. (2019). A survey of autonomous driving: Common practices and emerging technologies. arXiv preprint , arXiv:1906.05113
2019 arXiv
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.