Pith. sign in

REVIEW 4 major objections 5 minor 20 references

Boosting Generalizable Depth Estimation in Endoscopy by Mixture of Lightweight Experts and Intrinsic Image Alignment

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read Routed low-rank experts and intrinsic-image alignment set new endoscopic depth records.

desk verdict A plausible but under-validated incremental contribution to endoscopic depth estimation; the MiLoRE part is solid, the IIA part rests on an untested decomposition, and the 'most accurate' claim is factually wrong on one number. read the letter →

arxiv 2608.00415 v1 pith:DB2V4EDV submitted 2026-08-01 cs.CV cs.AI

classification cs.CVcs.AI
keywords DepthestimationEndoscopyMixtureofexpertsLow-rankadaptationIntrinsicimagedecompositionSelf-supervisedlearningEgo-motionCameraintrinsicsprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EndoMINI is a self-supervised framework that estimates depth, ego-motion, and camera intrinsics from monocular endoscopic video. The paper argues that two design choices fix the main failure modes of prior endoscopic depth models: tissue-feature diversity and illumination/reflectance interference. Instead of one low-rank adapter on a pretrained vision transformer, it routes each input through a top-k mixture of low-rank experts, adapting the network cheaply to different scene characteristics. It also decomposes each frame into reflectance and shading and aligns warped reflectance images during training, so the photometric signal is free of lighting effects. If the claim holds, a single model with about 3.6 million trainable parameters can beat larger baselines on known and unseen endoscopic datasets and predict camera intrinsics to about one percent error.

What carries the argument

MiLoRE is the adapting mechanism. In each Transformer block, three modules (QKV linear and two FFN MLPs) are fine-tuned by routing the input through a top-k selection of LoRA experts, where LoRA means low-rank adaptation: small trainable low-rank updates attached to frozen weights. A router predicts a distribution over experts with an injected noise term, and the output is W0 x plus the weighted sum of selected low-rank updates B_i A_i x. IIA is the lighting-robustness mechanism. A decomposition network with two style MLPs and AdaIN-controlled decoders predicts reflectance R and shading S for source and target frames; the illumination-free loss is the masked photometric error between the war

What would settle it

Shine light from two different positions on a fixed endoscopic scene with known depth and relight the same frames synthetically. If the predicted reflectance images change with lighting, or if the illumination-free alignment loss L_if is not near zero when geometry is unchanged, then the intrinsic decomposition is not providing the claimed lighting invariance. A second check: if removing IIA leaves depth accuracy unchanged on well-lit, low-reflectance frames, the mechanism is not needed there.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that two mechanisms push self-supervised endoscopic depth estimation past prior art: MiLoRE replaces a single low-rank adapter with a gated mixture of low-rank experts, and IIA adds an illumination-free alignment loss from a self-supervised, style-controlled intrinsic decomposition. Trained on SCARED with a pretrained ViT encoder, the model achieves the best supervised depth on SCARED (RelAbs 0.047), best zero-shot depth on Hamlyn (0.140) and SERV-CT (0.083), and camera-intrinsics errors under about 1% (fx 0.51, fy 0.31, cx 1.02, cy 0.98 percent), with 3.6M trainable parameters. Ablations tie the gains to the expert count/top-k and to IIA; replacing

Load-bearing premise

The load-bearing premise is that the intrinsic image decomposition cleanly separates reflectance from shading, so the illumination-free alignment removes lighting effects without throwing away depth-relevant detail.

Editorial extensions

If this is right

  • A single fine-tuned depth foundation model can serve multiple endoscopic settings: the reported zero-shot gains on Hamlyn and SERV-CT suggest less per-hospital retraining.
  • Depth and ego-motion can be estimated when camera intrinsics are unknown, since the network predicts intrinsics to roughly one percent error, removing a practical calibration requirement.
  • The trainable budget stays small (3.6M parameters) while accuracy improves, so the approach fits into a single-GPU fine-tuning budget.
  • Illumination-free reflectance alignment is the proposed route to lighting robustness; if the decomposition stays stable, depth signal should survive light-source motion and specular reflection.
  • The mixture-of-experts routing can be adjusted by changing expert count and top-k, giving a practical knob for balancing capacity and generalization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves decomposition quality unevaluated; a direct test of whether the predicted reflectance is truly illumination-invariant, e.g. on synthetic relighting, would strengthen the mechanism it relies on.
  • The router's expert selection could be read as a soft taxonomy of endoscopic scene appearances; tracking which experts fire on different tissue types might give a free domain-shift or novelty signal during surgery.
  • The same two components could transfer to other non-Lambertian or specular imaging settings, such as cystoscopy, arthroscopy, or open surgical video, where reflectance confounds photometric depth losses.
  • Because IIA is additive to existing appearance-flow and optical-flow alignments, it may compose with other depth backbones or parameter-efficient fine-tuning methods, not only the ViT encoder used here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes EndoMINI, a self-supervised depth/ego-motion estimation framework for endoscopic scenes. It introduces MiLoRE, a mixture-of-experts variant in which LoRA adapters serve as experts with a top-k router, and an intrinsic image alignment (IIA) loss built on a self-supervised intrinsic image decomposition network. The model is fine-tuned on SCARED and evaluated on SCARED, Hamlyn, and SERV-CT, with comparisons to recent baselines including Depth Anything, EndoDAC, MonoPCC, and IID-SfMLearner. The authors report state-of-the-art depth metrics, lower trainable parameters (3.6M), and improved ego-motion/camera-intrinsics prediction. The central claims are that MiLoRE improves cross-scene adaptation and that IIA removes illumination effects.

Significance. The problem is timely and the ingredients are reasonable. MiLoRE is a clean, parameter-efficient combination of MoE and LoRA; the reported 3.6M trainable parameters is a concrete practical advantage. The use of intrinsic decomposition for illumination-robust alignment is plausible, but the paper does not yet demonstrate that the decomposition actually separates reflectance from shading. Because the empirical gains (e.g., SCARED RelAbs 0.047 vs 0.051) are small and no uncertainty estimates are reported, the strength of the evidence is limited. If the decomposition and statistical robustness are validated, the contribution would be useful to endoscopic 3D perception.

major comments (4)
  1. [§3.2, Table 1] The claim of 'most accurate zero-shot depth estimation on both Hamlyn and SERV-CT' is not supported by Table 1. On SERV-CT, EndoMINI's RelAbs (0.083) is worse than Depth Anything's (0.082). On Hamlyn, EndoMINI is best on RelAbs, but differences across methods are small (e.g., 0.140 vs 0.143). No error bars, confidence intervals, or multiple-run statistics are reported, so it is unclear whether the reported gains are significant. Please qualify the claim and add statistical evaluation or at least per-metric consistency analysis.
  2. [§2.3–2.4, Eq. (3)–(5)] The IIA loss is the main novelty for illumination handling, and it relies on the intrinsic decomposition recovering true reflectance. Eq. (3) only enforces R⊗S ≈ I plus style-vector consistency; this is underconstrained and admits a degenerate solution R≈I, S≈1, or a solution where shading remains in R. No visualizations, no quantitative separation metrics, and no relighting-invariance test are provided. Therefore, the improvement of ablation F (IIA on vs. off) could come from the added photometric constraint or extra network capacity rather than from illumination-free alignment. Please include direct evaluation of the decomposition, such as synthetic relighting experiments or reflectance-shading separation metrics.
  3. [§3.4, Table 3] The statement that EndoMINI provides 'the most accurate ego-motion estimation' is contradicted by Table 3: on Seq. 1, EndoMINI's ATE (0.0302) is worse than IID-SfMLearner (0.0296) and EndoDAC (0.0290). No average or standard deviation over the five sequences is given, so the aggregate claim is unsupported. Please report a summary statistic across sequences and clarify whether the camera-intrinsics results are from the same model used for depth evaluation.
  4. [§2.3, Fig. 4] The intrinsic image decomposition network is only sketched. The figure and text do not specify the encoder and decoder architecture, the dimension of the style vectors, whether source and target share weights, how the decomposition is initialized or trained relative to the depth/pose network, or how the gradients from Eq. (5)-(8) flow into the decomposition network. This is a reproducibility concern because the IIA contribution is central to the paper. Please provide architectural details and the exact training schedule.
minor comments (5)
  1. [§3.1] The text says 'five metrics' but then lists only RelAbs, RelSq, RMSE, and RMSElog; δ (threshold accuracy) is used in the tables but omitted from the list.
  2. [Table 1, Hamlyn row] The row for EndoDAC on Hamlyn is marked with citation [13] (AF-SfMLearner) but should cite EndoDAC [4].
  3. [§3.1] Typo: 'all 92672 frames from 21 rectified videos [12] are selected for for evaluation' -- duplicate 'for'.
  4. [Eq. (8)] Eq. (8) writes L = λ1(...) + λ2 Lif + λ3 Lsa + Lrec, but λ4, λ5, λ6 are only defined in Eq. (6). Please restate the full loss with all six weights defined in one place.
  5. [Abstract / §3.2] The term 'supervised depth estimation' is used for SCARED, while the method is described as self-supervised. Please clarify whether 'supervised' refers to fine-tuning with stereo/self-supervision or to the evaluation protocol.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MiLoRE and IIA are empirically validated; the IIA illumination-free claim rests on an untested decomposition assumption, which is a correctness risk rather than a circular step.

full rationale

The paper does not exhibit a circular derivation. The central contributions are (1) MiLoRE, a mixture-of-experts LoRA fine-tuning scheme, and (2) IIA, an intrinsic-image alignment loss. Neither reduces to its own input by construction. MiLoRE is a standard parameter-efficient fine-tuning module whose benefit is tested directly in ablations D and E against MoE and DV-LoRA baselines. IIA is an auxiliary photometric loss on predicted reflectance images (Eq. 5); it is a training objective, not a fitted constant renamed as a prediction, and ablation F provides an on/off comparison. The only self-citation, EndoDAC [4], is used as an architecture source and as a baseline with public weights, and the paper's improvements over it are empirically demonstrated, so this citation is independent support rather than load-bearing circularity. The IIA claim that 'the illumination condition will not influence the reflectance image' depends on an unvalidated assumption that the intrinsic decomposition truly separates reflectance from shading (Eq. 3 is under-constrained and permits degenerate R≈I, S≈1 solutions). This is a genuine limitation and correctness risk, but it is not circularity: the paper does not define reflectance in terms of the depth output, and the IIA loss is not equivalent to the standard photometric loss by construction unless one assumes the degenerate solution, which the paper does not assert. Therefore no specific equation reduces to another by construction, and no fitted parameter is presented as a prediction. The evaluation against SCARED, Hamlyn, and SERV-CT provides external validation.

Assumptions & free parameters 10 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several domain assumptions about the validity of intrinsic image decomposition and flow-based warping in endoscopy, and on hand-chosen hyperparameters (loss weights, rank, expert counts, top-k) that are not swept.

free parameters (10)
  • lambda_1 (IID loss weight) = 0.01
    Chosen by hand to balance the scale of losses; no sensitivity analysis reported.
  • lambda_2 (illumination-free alignment weight) = 0.02
    Hand-set.
  • lambda_3 (smoothness weight) = 0.001
    Hand-set.
  • lambda_4 (optical flow reconstruction weight) = 0.01
    Hand-set.
  • lambda_5 (depth reconstruction weight) = 0.01
    Hand-set.
  • lambda_6 (intrinsic reconstruction weight) = 0.1
    Hand-set; ablations do not vary these weights.
  • LoRA rank r = 4
    Chosen; no analysis of rank sensitivity.
  • Number of FFN experts = 5
    Selected based on ablation A vs B vs C.
  • Number of QKV experts = 3
    Selected based on ablation.
  • Top-k = 2
    Chosen; no analysis of top-k.
assumptions (4)
  • domain assumption An endoscopic image I can be decomposed as the pixel-wise product of reflectance R and shading S (I = R ⊗ S).
    Intrinsic image decomposition is assumed to hold on tissue surfaces; used in Eqs. 3 and 5. The validity on specular or non-Lambertian surfaces is not tested.
  • domain assumption Photometric consistency holds between frames: pixel intensity differences are caused by viewpoint change and illumination, and depth/ego-motion can explain them.
    Core of the self-supervised pipeline, Section 2.1 and losses in Section 2.4.
  • domain assumption Optical flow and appearance flow are available and accurate enough to compute visibility masks and to warp images and intrinsic components.
    Used in Eq. 4 and Eqs. 6-7; inaccurate flow would corrupt the losses.
  • domain assumption A pretrained ViT encoder (Depth Anything) can be effectively adapted to endoscopy with a small number of trainable low-rank adapters and a router.
    Implied by the MiLoRE design in Section 2.2; no transferability guarantees.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Boosting Generalizable Depth Estimation in Endoscopy by Mixture of Lightweight Experts and Intrinsic Image Alignment." pith.science (2026). https://pith.science/paper/DB2V4EDV

@misc{pith2026260800415,
  author       = {Pith},
  title        = {Pith review of: Boosting Generalizable Depth Estimation in Endoscopy by Mixture of Lightweight Experts and Intrinsic Image Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DB2V4EDV}},
  note         = {Machine review of arXiv:2608.00415}
}
read the original abstract

Depth estimation is a significant task for 3D perception in endoscopic surgeries. However, illumination interference and feature diversity in various endoscopic scenes are still challenges for generalizable depth estimation and ego-motion estimation. Based on this, a novel self-supervised framework, EndoMINI, is proposed for depth estimation in endoscopic scenes. Specifically, mixture of low-rank experts (MiLoRE) is proposed to perform parameter-efficient fine-tuning, which can also boost the model adaptation to scenes with different characteristics. Meanwhile, an intrinsic image alignment (IIA) is introduced into the training loss to alleviate the influence of light reflectance in endoscopy with a novel intrinsic image decomposition network. The proposed method is evaluated on SCARED datasets for supervised depth estimation, and two endoscopic datasets, Hamlyn and SERV-CT, for zero-shot depth estimation, compared with state-of-the-art works as well. The experimental results demonstrate outstanding performance of the proposed model and the effects of the main contributions.

Figures

Figures reproduced from arXiv: 2608.00415 by the authors.

Figure 1
Figure 1. shows. Firstly, for endoscopic data, even from the same dataset, the fea￾tures of endoscopic scenes are quite different due to various characteristics of tissues. Therefore, feature encoders usually suffer from corresponding represen￾tation extraction for various scenes and objects. Furthermore, although the ex￾isting method[13] alleviates the illumination inconsistency with appearance flow and optical flow, the ref… view at source ↗
Figure 2
Figure 2. Self-supervised pipeline for endoscopy depth estimation. The proposed pipeline in this work is shown in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Parameter-efficient finetuning on each Transformer block in ViT-B encoder based on mixture of low-rank experts. data. To achieve this, the probability distribution of experts is generated by a router network R based on the input feature at first. Based on this, k experts with highest probability are selected to perform weighted Low-Rank adaptation (LoRA[8]). For the finetuned module with the frozen weight W0 ∈ Rd×k … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Fig.4. Due to the style gap between the reflectance image and the shading image, [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 4
Figure 4. Figure 4: The framework of the proposed intrinsic image decomposition network. In the pipeline, two intrinsic image decomposition decoders generate the cor￾responding reflectance images Rt, Rs and shading images St, Ss from the source image and target image, respectively. The st…
Figure 5
Figure 5. Figure 5: Visualization results. The corresponding ablation settings are in Table. 2. 3.3 Ablation Studies To demonstrate the effects of our main contributions, a series of ablation studies are performed in the experiments additionally. All of ablation results can be found in […

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 11 canonical work pages

  1. [13]

    Medical image analysis77, 102338 (2022)

    Shao, S., Pei, Z., Chen, W., Zhu, W., Wu, X., Sun, D., Zhang, B.: Self-supervised monocular depth and ego-motion estimation in endoscopy: Appearance flow to the rescue. Medical image analysis77, 102338 (2022)

  2. [4]

    In: International Conference on Medical Image Computing and Computer-Assisted Intervention

    Cui, B., Islam, M., Bai, L., Wang, A., Ren, H.: Endodac: Efficient adapting foun- dation model for self-supervised depth estimation from any endoscopic camera. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 208–218. Springer (2024)

  3. [1]

    arXiv preprint arXiv:2101.01133 (2021)

    Allan, M., Mcleod, J., Wang, C., Rosenthal, J.C., Hu, Z., Gard, N., Eisert, P., Fu, K.X., Zeffiro, T., Xia, W., et al.: Stereo correspondence and reconstruction of endoscopic data challenge. arXiv preprint arXiv:2101.01133 (2021)

  4. [2]

    arXiv preprint arXiv:2407.06204 (2024)

    Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., Huang, J.: A survey on mixture of experts. arXiv preprint arXiv:2407.06204 (2024)

  5. [3]

    IEEE Transactions on Medical Imaging40(1), 371–380 (2020)

    Collins, T., Pizarro, D., Gasparini, S., Bourdel, N., Chauvet, P., Canis, M., Calvet, L., Bartoli, A.: Augmented reality guided laparoscopic surgery of the uterus. IEEE Transactions on Medical Imaging40(1), 371–380 (2020)

  6. [5]

    Medical image analysis76, 102302 (2022)

    Edwards, P.E., Psychogyios, D., Speidel, S., Maier-Hein, L., Stoyanov, D.: Serv-ct: A disparity dataset from cone-beam ct for validation of endoscopic 3d reconstruc- tion. Medical image analysis76, 102302 (2022)

  7. [6]

    Giannarou, S., Stoyanov, D., Noonan, D., Mylonas, G., Clark, J., Visentini- Scarzanella, M., Mountney, P., Yang, G.Z.: Hamlyn centre laparoscopic / endo- scopic video datasets, https://hamlyn.doc.ic.ac.uk/vision/

  8. [7]

    In: Proceedings of the IEEE/CVF in- ternational conference on computer vision

    Godard, C., Mac Aodha, O., Firman, M., Brostow, G.J.: Digging into self- supervised monocular depth estimation. In: Proceedings of the IEEE/CVF in- ternational conference on computer vision. pp. 3828–3838 (2019)

Show all 20 references
  1. [8]

    In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=nZeVKeeFYf9

    Hu, E.J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=nZeVKeeFYf9

  2. [9]

    IEEE Journal of Biomedical and Health Informatics pp

    Li, B., Liu, B., Zhu, M., Luo, X., Zhou, F.: Image intrinsic-based unsupervised monocular depth estimation in endoscopy. IEEE Journal of Biomedical and Health Informatics pp. 1–11 (2024) 10 L. Shao et al

  3. [10]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Liu, Y., Li, Y., You, S., Lu, F.: Unsupervised learning for intrinsic image decom- position from a single image. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3248–3257 (2020)

  4. [11]

    Medical image analysis71, 102058 (2021)

    Ozyoruk, K.B., Gokceler, G.I., Bobrow, T.L., Coskun, G., Incetan, K., Almalioglu, Y., Mahmood, F., Curto, E., Perdigoto, L., Oliveira, M., et al.: Endoslam dataset and an unsupervised monocular visual odometry and depth estimation approach for endoscopic videos. Medical image ...

  5. [12]

    IEEE Robotics and Automation Letters6(4), 7225– 7232 (2021)

    Recasens, D., Lamarca, J., Fácil, J.M., Montiel, J., Civera, J.: Endo-depth-and- motion: Reconstruction and tracking in endoscopic videos using depth networks and photometric constraints. IEEE Robotics and Automation Letters6(4), 7225– 7232 (2021)

  6. [14]

    Medical Image Analysis102, 103534 (2025)

    Wang, Z., Zhou, Y., He, S., Li, T., Huang, F., Ding, Q., Feng, X., Liu, M., Li, Q.: Monopcc: Photometric-invariant cycle constraint for monocular depth estimation of endoscopic images. Medical Image Analysis102, 103534 (2025)

  7. [15]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yang, L., Kang, B., Huang, Z., Xu, X., Feng, J., Zhao, H.: Depth anything: Un- leashing the power of large-scale unlabeled data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10371–10381 (2024)

  8. [16]

    Advances in Neural Information Processing Systems37, 21875–21911 (2024)

    Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., Zhao, H.: Depth anything v2. Advances in Neural Information Processing Systems37, 21875–21911 (2024)

  9. [17]

    IEEE Transactions on Medical Imaging43(5), 1934–1944 (2024)

    Yang, Z., Pan, J., Dai, J., Sun, Z., Xiao, Y.: Self-supervised lightweight depth estimation in endoscopy combining cnn and transformer. IEEE Transactions on Medical Imaging43(5), 1934–1944 (2024)

  10. [18]

    In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion

    Zhang, N., Nex, F., Vosselman, G., Kerle, N.: Lite-mono: A lightweight cnn and transformer architecture for self-supervised monocular depth estimation. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. pp. 18537–18546 (2023)

  11. [19]

    In: 2022 international conference on 3D vision (3DV)

    Zhao, C., Zhang, Y., Poggi, M., Tosi, F., Guo, X., Zhu, Z., Huang, G., Tang, Y., Mattoccia, S.: Monovit: Self-supervised monocular depth estimation with a vision transformer. In: 2022 international conference on 3D vision (3DV). pp. 668–678. IEEE (2022)

  12. [20]

    In: 2024 IEEE International Conference on Bioin- formatics and Biomedicine (BIBM)

    Zhou, Y., He, S., Wang, H., Huang, F., Liu, M., Li, Q., Wang, Z.: Improved self-supervised monocular endoscopic depth estimation based on pose alignment- friendly dynamic view selection. In: 2024 IEEE International Conference on Bioin- formatics and Biomedicine (BIBM). pp. 300...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.