Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

End-to-End Steering for Autonomous Vehicles via Conditional Imitation Co-Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read Letting specialist branches share features lifts unseen-town driving success by 62%.

desk verdict A novel co-learning architecture that plausibly improves CIL generalization in simulation, but the headline 62% rests on a single unseeded run of a deliberately narrowed benchmark. read the letter →

arxiv 2411.16131 v1 pith:7UOPRI2L submitted 2024-11-25 cs.AI cs.RO

classification cs.AIcs.RO
keywords AutonomousDrivingEnd-to-endConditionalImitationLearningCo-learningMatrixCo-existenceProbabilitySteeringModelCARLAGeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to improve how end-to-end driving models generalize to unseen towns and weather conditions. It argues that the specialist branches of conditional imitation learning should not be treated as independent: each branch's predictions for one navigational command can benefit from features extracted by branches trained on other commands. The proposed conditional imitation co-learning (CIC) model uses a co-learning matrix, generated by gated tanh units, to mix the branches' output features, and the paper reports that this raises reach-destination success in fully unseen town-plus-weather conditions to 75% from 46.25% for the unmodified model. The paper also tests two ways of casting steering regression as classification, finding that a hybrid cross-entropy plus mean-squared-error loss helps while plain classification does not.

What carries the argument

The load-bearing mechanism is the co-learning matrix $C_t$ that linearly mixes the four specialist branches' feature vectors, $\hat{A}_t$, before the action is read out: $A_t = C_t \hat{A}_t$. The diagonal entries of $C_t$ are fixed at 1 and the off-diagonal coefficients are produced by gated hyperbolic tangent units (GTUs), so the network can dynamically decide, per driving scenario, how much each branch's features should influence every other branch's prediction. The paper also introduces a classification-regression hybrid loss (categorical cross-entropy plus mean-squared error on the expected steering) and a co-existence probability matrix loss that nudges output scores toward a Gaussian spatial distribution over steering classes, but it is the GTU-driven co-learning matrix that carries the headline generalization gain.

What would settle it

Take the same GTU co-learning model and the unmodified CIL baseline, and evaluate both on the full route set without dropping multi-turn routes or samples with steering magnitude above 0.8. If the gap in reach-destination success over unseen towns and weathers shrinks to a small margin, the 62% claim is specific to the filtered single-turn, moderate-steering task.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the conditional imitation learning (CIL) assumption that each navigational-command branch should act independently is the bottleneck to generalization. By computing an $N \times N$ co-learning matrix at each time step and multiplying it with the vector of branch-specific output features, the proposed conditional imitation co-learning (CIC) model lets a branch trained mainly on right turns, for example, draw on features learned from left-turn and straight-driving data. In the benchmark, the GTU-generated co-learning matrix reaches a 75.00% reach-destination success rate in an unseen town with unseen weather, versus 46.25% for the baseline CIL regression model, the 62% improvement cited in the abstract. The paper further reports that classifying steering into nine discrete classes with a hybrid cross-entropy plus mean-squared-error loss improves unseen-environment success by 21% over the regression baseline, while plain classification alone does not beat regression.

Load-bearing premise

The 62% generalization gain is measured on a benchmark that keeps only single-turn routes and drops steering samples with magnitude above 0.8, so the claim depends on those filters matching what 'unseen environment' means in practice.

Editorial extensions

If this is right

  • A network trained in one town and two weather conditions reaches 75% success in a new town under new weather conditions, versus 46.25% for the unmodified conditional imitation learning model.
  • The co-learning mechanism is output-oriented: it mixes the final branch features rather than hidden-layer parameters, so it can be added atop an existing CIL-style architecture without changing the feature extractor.
  • Posing steering regression as classification with a hybrid cross-entropy plus mean-squared-error loss improves unseen-environment success by 21% over the regression baseline, though classification alone does not.
  • Existing multi-task sharing methods (mixture of experts, soft parameter sharing, sluice networks) improve over CIL in new towns but none of them reaches the GTU co-learning model's success rates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the benchmark drops multi-turn routes and samples with steering magnitude above 0.8, the reported 62% gain is established only for moderate single-turn maneuvers; extending the evaluation to the full route distribution would test whether the co-learning benefit persists.
  • The co-learning matrix idea is not specific to steering: any multi-branch policy or task module that shares an output layer could use the same GTU-generated mixing coefficients, so the mechanism could transfer to other imitation-learning or multi-task settings.
  • The spatial-class loss via co-existence probability with a Gaussian kernel is one choice; comparing it to learned kernels or to the sine-wave encoding on the same benchmark would separate the benefit of the spatial prior from the kernel's shape.
  • If the GTU coefficients are interpreted per scenario, they may be inspected to see which branch-to-branch transfers matter most, potentially guiding data collection toward under-connected navigation commands.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes conditional imitation co-learning (CIC), an extension of the conditional imitation learning (CIL) architecture of Codevilla et al. (2018) for end-to-end steering. CIC lets the specialist branches (left, right, straight, follow-lane) share output-level information through a co-learning matrix, with coefficients generated either from a manually fixed binary relationship matrix R multiplied by learned coefficients or from gated tanh units (GTUs). The paper also recasts steering regression as classification, adding a hybrid cross-entropy/MSE loss (Eq. 3) and a co-existence-probability loss based on a Gaussian spatial relationship matrix (Eq. 4). Experiments on a modified CoRL benchmark in CARLA report that the GTU-based co-learning model achieves 75.00% reach-destination success in new-town/new-weather conditions versus 46.25% for the CIL regression baseline, a 62% relative improvement, while the hybrid-loss and co-existence-loss variants show smaller gains. The paper concludes that CIC improves generalization robustness in unseen environments.

Significance. If the reported gain is robust, the paper makes a useful empirical contribution by showing that a simple output-level mixing mechanism between CIL specialist branches can yield substantial generalization improvements over the standard architecture, and by providing ablations of classification-based steering losses with baselines including mixture of experts, sluice networks, and soft parameter sharing. The idea of learning branch relationships through GTUs is plausible and the comparison against multiple MTL baselines is informative. However, the central quantitative claim currently rests on a single unseeded run of a deliberately narrowed benchmark, and the hyperparameter selection procedure is not described as respecting a clean train/validation/test split. These issues make the headline 62% claim fragile as it stands.

major comments (3)
  1. [Section 4, Tables 1 and 4] The headline 62% improvement is computed as (75.00 - 46.25) / 46.25 from single point estimates, with no standard deviations, confidence intervals, or multiple runs/seeds reported. CARLA evaluation includes stochasticity in weather, traffic, and autopilot behavior; a single draw (60/80 versus 37/80 successful routes) is not sufficient evidence for the claimed generalization advantage. Please provide results over multiple seeds and report statistics (mean and variance or a significance test) for all key comparisons.
  2. [Section 4, benchmark design] The evaluation is restricted to single-turn routes only, and training samples with steering magnitude above 0.8 are discarded. The claim in the abstract and conclusion that the model 'improve[s] autonomous driving success rate in unseen environment' is therefore demonstrated only for moderate single turns in two towns and two new weathers. The decisive test is evaluation on multi-turn routes and sharp-curve scenarios from the full CoRL benchmark; without this, the generalization claim is overstated. Please either extend the evaluation or explicitly scope the claim to the tested distribution.
  3. [Section 3.3, Section 3.4, Tables 2 and 3] Hyperparameter W for the hybrid loss and the co-existence loss appears to be selected by testing a set of values and reporting the best-performing rows on the same benchmark (W=10 for the hybrid loss, W=0.6 for the co-existence loss), and the binary relationship matrix R in the first co-learning variant is described as manually fine-tuned. This procedure can inflate the reported performance of the proposed variants. Please describe the hyperparameter selection protocol, and ideally use a separate validation set or cross-validation to choose W and R before evaluating on the held-out test routes.
minor comments (5)
  1. [Abstract] The abstract states '62% on average', but the tables show this exact relative improvement only in the new-town/new-weather cell; the averaging basis over conditions should be clarified.
  2. [Equation (3)] The symbol y is used for both the continuous steering value and the one-hot vector of the discretized steering; please use distinct notation (e.g., y_true and y_onehot) to avoid ambiguity.
  3. [Table 4] The row labeled 'stitch network' appears to refer to cross-stitch networks (Misra et al., 2016); please correct the terminology and ensure the reference is cited consistently.
  4. [Section 4] The paper does not state whether the route definitions, trained models, or code will be made available; providing these would significantly improve reproducibility of the reported success rates.
  5. [Section 3.4] The choice of Gaussian distribution with sigma^2 = 1 for the co-existence matrix is asserted without sensitivity analysis; please justify this choice or show that performance is stable over a range of sigma values.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the claim is an empirical benchmark comparison against external CARLA scenarios and prior baselines, not a derivation from its own outputs.

full rationale

The paper's central claim is empirical: the CIC co-learning model achieves a 75.00% reach-destination success rate in new-town/new-weather conditions versus 46.25% for CIL regression, quoted as a 62% improvement (Tables 1 and 4). This is a measured benchmark result on an external simulator (CARLA) with prior published baselines, not a chain of derivations whose outputs are fed back as inputs. The co-learning operation At = Ct * Ahat_t and the losses in Equations (3) and (4) are model definitions; they do not presuppose the reported success rates. The only in-paper self-citations (Eraqi et al. 2017, 2022) appear as related work or baselines, not as load-bearing justification for the central claim. Methodological weaknesses—restricting evaluation to single-turn routes, filtering training steering to |s| <= 0.8, reporting no random seeds, and selecting hyperparameters W by comparing test-table entries—are threats to generalization and reproducibility, but they are not circularity: no fitted parameter is renamed as a prediction, and no equation reduces by construction to the target result. The paper is self-contained against external benchmarks, so no circular step is identified.

Assumptions & free parameters 7 free parameters · 4 assumptions · 1 invented entities

The central claim rests on several unpaid inputs: a manually tuned relationship matrix, two scanned loss weights, and a fixed steering discretization and training range; the assumption that CARLA autopilot and single-turn routes capture generalizable driving; and a Gaussian co-existence prior fixed by hand. No external benchmark, pretrained model, or independent dataset is used to validate the architecture.

free parameters (7)
  • Binary relationship matrix R (co-learning) = manually fine-tuned, values not reported
    Section 3.2 defines R as a hyperparameter set before training; its entries determine which branch connections exist, and no values or search details are given.
  • Hybrid loss weight W = W=10 (best of 5, 10, 15)
    Section 3.3 and Table 2 report three W values; W=10 is used for the headline CCE plus MSE result, selected on the test benchmark.
  • Co-existence loss weight W = W=0.6 (best of 0.4, 0.6, 0.8)
    Section 3.4 and Table 3 scan W; W=0.6 is the best row and is reported in Table 4.
  • Steering discretization step and class count = 9 classes, step 0.2
    Section 4 fixes discrete steering classes; the number of classes changes the classification problem and is chosen without sensitivity analysis.
  • Steering training range = [-0.8, 0.8]
    Section 4 discards samples outside this range, which removes sharp turns from training; the threshold is chosen by hand.
  • Gaussian co-existence sigma^2 = 1
    Section 3.4 fixes the spatial prior as Gaussian with sigma^2=1; no justification or ablation is given.
  • Learning rate and Adam betas = lr=0.0002, beta1=0.70, beta2=0.85
    Section 4 lists these fixed choices; no sensitivity analysis is provided.
assumptions (4)
  • domain assumption CARLA autopilot labels are a valid expert for imitation learning.
    Section 3.1 collects data from CARLA autopilot with injected steering noise; if the autopilot policy is suboptimal, all baselines are affected equally, but the expert assumption fails for real driving transfer.
  • ad hoc to paper A single-turn CoRL benchmark measures generalization to unseen environments.
    Section 4 modifies the benchmark to only single-turn routes; generalization to multi-turn intersections, roundabouts, or dense traffic is never tested.
  • ad hoc to paper Gaussian co-existence matrix with sigma^2=1 captures steering-class spatial structure.
    Section 3.4 sets Gaussian sigma^2=1 by hand; no sensitivity or justification is given.
  • ad hoc to paper Steering values outside [-0.8,0.8] are irrelevant to the learned task.
    Section 4 discards samples with steering outside [-0.8,0.8], so sharp turns are not learned; this may favor the architecture on the chosen benchmark.
invented entities (1)
  • Conditional imitation co-learning matrix C_t with GTU gate outputs
    purpose: Linearly combines specialist branch features at the output so branches share steering knowledge.
    The matrix is evaluated only on the paper's internal CARLA benchmark; no external dataset, pre-registered prediction, or independent replication is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of End-to-End Steering for Autonomous Vehicles via Conditional Imitation Co-Learning." pith.science (2026). https://pith.science/paper/7UOPRI2L

@misc{pith2026241116131,
  author       = {Pith},
  title        = {Pith review of: End-to-End Steering for Autonomous Vehicles via Conditional Imitation Co-Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7UOPRI2L}},
  note         = {Machine review of arXiv:2411.16131}
}
read the original abstract

Autonomous driving involves complex tasks such as data fusion, object and lane detection, behavior prediction, and path planning. As opposed to the modular approach which dedicates individual subsystems to tackle each of those tasks, the end-to-end approach treats the problem as a single learnable task using deep neural networks, reducing system complexity and minimizing dependency on heuristics. Conditional imitation learning (CIL) trains the end-to-end model to mimic a human expert considering the navigational commands guiding the vehicle to reach its destination, CIL adopts specialist network branches dedicated to learn the driving task for each navigational command. Nevertheless, the CIL model lacked generalization when deployed to unseen environments. This work introduces the conditional imitation co-learning (CIC) approach to address this issue by enabling the model to learn the relationships between CIL specialist branches via a co-learning matrix generated by gated hyperbolic tangent units (GTUs). Additionally, we propose posing the steering regression problem as classification, we use a classification-regression hybrid loss to bridge the gap between regression and classification, we also propose using co-existence probability to consider the spatial tendency between the steering classes. Our model is demonstrated to improve autonomous driving success rate in unseen environment by 62% on average compared to the CIL method.

Figures

Figures reproduced from arXiv: 2411.16131 by the authors.

Figure 1
Figure 1. Network Architecture proposed by (Codevilla [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Training and deployment paths in the work in (Er [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Co-learning between branch 1 and branch 2 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Spatial relationship between the steering classes [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Review of Learning-Based Motion Planning: Toward a Data-Driven Optimal Control Approach

    cs.RO 2025-12 conditional novelty 3.0 of 10

    A position/review paper argues data-driven model predictive control is the best route to safe, adaptive, human-like autonomous-driving motion planning, but provides no new derivation or experiment.

Reference graph

Works this paper leans on

27 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    Bekey, G. A. (2005). Autonomous robots: from biological inspiration to implementation and control . The MIT Press

  3. [3]

    Bengio, S., Dean, J., Erhan, D., Ie, E., Le, Q., Rabinovich, A., Shlens, J., and Singer, Y. (2013). Using web co-occurrence statistics for improving image categorization. ArXiv preprint , arXiv:1312.5697

  4. [4]

    Bewley, A., Rigley, J., Liu, Y., Hawke, J., Shen, R., Lam, V.-D., and Kendall, A. (2019). Learning to drive from simulation without real world labels. In 2019 International Conference on Robotics and Automation (ICRA) , pages 4817--4823. IEEE

  5. [5]

    D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L

    Bojarski, M., Testa, D. D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016). End to end learning for self-driving cars. arXiv preprint , arXiv:1604.07316

  6. [6]

    Codevilla, F., Miiller, M., Lopez, A., Koltun, V., and Dosovitskiy, A. (2018). End-to-end driving via conditional imitation learning. In Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 1--9

  7. [7]

    Cui, J., Qiu, H., Chen, D., Stone, P., and Zhu, Y. (2022). Coopernaut: End-to-end driving with cooperative perception for networked vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 17252--17262

  8. [8]

    Dammen, J. (2019). End-to-end deep learning for autonomous driving. Master's thesis, Norwegian University of Science and Technology

Show all 27 references
  1. [9]

    N., Fan, A., Auli, M., and Grangier, D

    Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D. (2017). Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning (ICML) , pages 933--941. PMLR

  2. [10]

    Dosovitskiy, A., Ros, G., Codevilla, F., L \'o pez, A., and Koltun, V. (2017). Carla: An open urban driving simulator. In Proceedings of the Conference on Robot Learning (CoRL)

  3. [11]

    Eraqi, H., Moustafa, M., and Honer, J. (2017). End-to-end deep learning for steering autonomous vehicles considering temporal dependencies. CoRR , abs/1710.03804

  4. [12]

    M., Moustafa, M

    Eraqi, H. M., Moustafa, M. N., and Honer, J. (2022). Dynamic conditional imitation learning for autonomous driving. IEEE Transactions on Intelligent Transportation Systems

  5. [13]

    Hawke, J., Shen, R., Gurau, C., Sharma, S., Reda, D., Nikolov, N., Mazur, P., Micklethwaite, S., Griffiths, N., Shah, A., et al. (2019). Urban driving with conditional imitation learning. arXiv preprint , arXiv:1912.00177

  6. [14]

    Huber, P. J. (1992). Robust estimation of a location parameter. In Breakthroughs in statistics , pages 492--518. Springer, New York

  7. [15]

    A., Jordan, M

    Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural Computation , 3(1):79--87

  8. [16]

    and Ba, J

    Kingma, D. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint , arXiv:1412.6980

  9. [17]

    and Genc, Y

    Kourbane, I. and Genc, Y. (2021). A hybrid classification-regression approach for 3d hand pose estimation using graph convolutional networks. arXiv preprint , arXiv:2105.10902

  10. [18]

    Liang, X., Wang, T., Yang, L., and Xing, E. (2018). Cirl: Controllable imitative reinforcement learning for vision-based self-driving. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 584--599

  11. [19]

    Misra, I., Shrivastava, A., Gupta, A., and Hebert, M. (2016). Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3994--4003

  12. [20]

    Moustafa, M. N. (2005). System and method for pose-angle estimation

  13. [21]

    Muller, M., Dosovitskiy, A., Ghanem, B., and Koltun, V. (2018). Driving policy transfer via modularity and abstraction. In Conference on Robot Learning (CoRL) , pages 1--14

  14. [22]

    Prakash, A., Chitta, K., and Geiger, A. (2021). Multimodal fusion transformer for end-to-end autonomous driving. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR)

  15. [23]

    Rasmus, A., Valpola, H., Honkala, M., Berglund, M., Raiko, T., Santana, E., and Hotz, G. (2016). Learning a driving simulator. CoRR

  16. [24]

    Rothe, R., Timofte, R., and Gool, L. V. (2015). Dex: Deep expectation of apparent age from a single image. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV) , pages 10--15

  17. [25]

    Ruder, S., Bingel, J., Augenstein, I., and S gaard, A. (2019). Latent multi-task architecture learning. Proceedings of the AAAI Conference on Artificial Intelligence , 33(01):4822--4829

  18. [26]

    Tampuu, A., Matiisen, T., Semikin, M., Fishman, D., and Muhammad, N. (2020). A survey of end-to-end driving: Architectures and training methods. IEEE Transactions on Neural Networks and Learning Systems

  19. [27]

    Yurtsever, E., Lambert, J., Carballo, A., and Takeda, K. (2019). A survey of autonomous driving: Common practices and emerging technologies. arXiv preprint , arXiv:1906.05113

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.