Pith. sign in

REVIEW 4 major objections 6 minor 54 references

Opto-ViT: Architecting a Near-Sensor Region of Interest-Aware Vision Transformer Accelerator with Silicon Photonics

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Opto-ViT runs ViT inference on a hybrid silicon-photonic engine, reporting 100.4 KFPS/W with under 1.6% accuracy loss.

desk verdict Plausible and broad architecture with real component-level ideas, but the headline efficiency claim is a black box and two overclaims (first photonic ViT, <1.6% accuracy loss) don't survive the paper's own text. read the letter →

arxiv 2507.07044 v2 pith:MH73UALQ submitted 2025-07-09 cs.AR

Mehrdad Morsali , Chengwei Zhou , Deniz Najafi , Sreetama Sarkar , Pietro Mercati , Navid Khoshavi , Peter Beerel , Mahdi Nikdast
show 2 more authors
Gourav Datta Shaahin Angizi
This is my paper
classification cs.AR
keywords siliconphotonicsvisiontransformeracceleratornear-sensorcomputingmicroringresonatorsVCSELarraysregion-of-interestpruningquantization-awaretrainingWDMmatrixmultiplication
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Opto-ViT is a proposed accelerator that runs Vision Transformers (ViTs) on a hybrid electronic-photonic chip placed near the image sensor. The paper's central claim is that the heavy matrix multiplications of a ViT can be moved into an optical core built from VCSELs and microring resonators, while nonlinear and normalization operations stay in electronics, and that this split preserves accuracy. To cut redundant work, a lightweight Mask Generation Network prunes image patches that fall outside the region of interest before the ViT encoder sees them. Using 8-bit quantization-aware training and a matrix-decomposition trick, the authors report 100.4 KFPS/W, up to 84% energy savings, and under 1.6% accuracy loss across classification, detection, and video tasks. If the energy numbers hold, transformer-based vision becomes practical for always-on edge devices.

What carries the argument

The load-bearing object is the optical core: 64 waveguide arms, 32 wavelength channels, VCSEL-driven optical inputs, microring-resonator banks tuned to weights, and balanced photodetectors that accumulate MAC results. Wavelength-division multiplexing lets one input row multiply many weight columns simultaneously. The paper's key identity is the matrix decomposition $QK^T = (Q W_K^T) X^T$, which removes the need to wait for and re-tune the intermediate key matrix, enabling a five-core pipeline. The Mask Generation Network (MGNet), a single transformer block plus self-attention and a linear head, produces patch-wise binary masks from the current frame alone. Quantization-aware training with a straight-through estimator and symmetric 8-bit quantization keeps the model accurate under photonic precision limits.

What would settle it

Measure the insertion loss, tuning power, and crosstalk of the more than 200 fabricated microring resonators at Q about 5000 and feed the measured values into the same energy model; if the per-MAC energy rises above the simulated figure or the 8-bit resolution at Q about 5000 is not reached under fabrication-process variations, the 100.4 KFPS/W and 84% savings claims fail.

Watch

Extended reading notes

Core claim

The discovery is that ViT inference can be reformulated so that its dominant computation—attention-score and feed-forward matrix multiplications—maps onto a wavelength-division-multiplexed silicon-photonic engine with negligible accuracy cost. Input activations are converted to light amplitudes by VCSEL arrays, weights are imprinted on microring resonators, and balanced photodetectors accumulate the products. The attention operation is rearranged as $QK^T = (Q W_K^T) X^T$ so every weight matrix is tuned into the rings before inference begins, removing the wait for the intermediate key matrix and the associated buffering. Eight-bit quantization during training keeps accuracy within 1.6% of full-precision baselines, while the ROI mask skips up to about 68% of image patches, which in a ViT means those patches' entire downstream computation is skipped. The authors claim this is the first near-sensor, region-aware ViT accelerator using silicon photonics.

Load-bearing premise

The headline efficiency numbers come from an in-house simulator whose optical and electronic energy models are not public, so if those models are optimistic the reported KFPS/W and energy savings do not carry over to real hardware.

Editorial extensions

If this is right

  • At the reported 100.4 KFPS/W, the accelerator is two to three orders of magnitude more energy-efficient than an FPGA or GPU running the same INT8 ViT, which would make transformer inference viable in always-on cameras.
  • Because ViTs process independent patches, an input-side ROI mask skips all downstream computation for pruned patches, giving near-linear energy and latency savings in the backbone.
  • Tuning all attention weight matrices once, via $QK^T = (Q W_K^T) X^T$, removes the intermediate-buffer bottleneck that usually slows attention accelerators.
  • Eight-bit quantization-aware training is sufficient to keep accuracy loss below 1.6% across CIFAR-10, Tiny-ImageNet, COCO detection/segmentation, and ImageNet-VID, so the photonic bit-precision constraint is not a fundamental accuracy blocker.
  • The energy breakdown shows ADC/DAC conversion dominates, so further analog-domain integration would yield the next large efficiency gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same decomposition identity could benefit non-photonic accelerators: any engine with expensive operand loading could restructure attention as $(Q W_K^T) X^T$ to reduce buffer traffic.
  • The MGNet's patch-level masking is a general technique: a transformer that accepts token dropout can use any lightweight saliency model, and training the mask jointly with the backbone might recover the accuracy lost on datasets without bounding-box annotations.
  • The reported 8-bit resolution at Q about 5000 rests on a crosstalk model; a direct measurement of the fabricated resonators would be the decisive test, since the paper reports fabrication but no measured insertion loss, tuning power, or crosstalk.
  • If the energy model holds, the dominant ADC cost suggests that optical-to-digital conversion, not the matrix multiplication itself, is where future near-sensor photonic designs should focus.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Opto-ViT, a hybrid electronic-photonic accelerator for Vision Transformers, combining VCSEL-driven optical inputs with microring-resonator-based matrix multiplications, an electronic unit for nonlinear functions, and a lightweight Mask Generation Network (MGNet) for region-of-interest patch pruning. The authors report up to 100.4 KFPS/W, up to 84% energy savings, and 'less than 1.6% accuracy loss' across classification, detection, and video tasks, based on a bottom-up simulation framework that includes fabricated MRs and circuit-level simulations. The paper also presents a comparison against several prior silicon-photonic accelerators.

Significance. If the reported efficiency and accuracy figures are reproducible, Opto-ViT would represent a substantial advance toward energy-efficient edge inference for vision transformers, with an interesting architectural combination of VCSEL-driven inputs, MR-based MatMul, and decomposition-based mapping to avoid tuning bottlenecks. The authors' effort to fabricate more than 200 MRs and to run circuit-level simulations is a valuable methodological component. However, the central claims are currently undermined by (i) an internal contradiction in the accuracy-loss figure between the abstract and Table I/III, (ii) an unreleased proprietary simulator underpinning the headline performance numbers and the cross-accelerator comparison, and (iii) a novelty assertion contradicted by the paper's own cited reference [22]. These issues must be resolved before the headline results can be fully credited.

major comments (4)
  1. [Abstract and Table I] The abstract and conclusion state that Opto-ViT achieves 'less than 1.6% accuracy loss,' but Table I reports Opto-ViT-B Mask on Tiny-ImageNet at 224×224 at 80.12% versus 84.64% for the non-masked Opto-ViT-B, an absolute drop of 4.52 percentage points, which the paper itself acknowledges ('a 4.5% decrease'). Similarly, in Table III the masked video model drops 1.89 percentage points from the full-precision baseline (54.90 to 53.01 mAP), exceeding the 1.6% bound stated in the text. The headline claim should either be restricted to the cases where it holds (e.g., the non-masked quantized models) or corrected to reflect the actual worst-case accuracy loss.
  2. [Section II (Related Work) and Abstract] The abstract and Section II claim that Opto-ViT is 'the first near-sensor, region-aware ViT accelerator leveraging silicon photonics' and that 'no silicon-photonic-based acceleration method has been developed specifically for vision transformers,' but the Introduction itself cites TRON [22] as 'A silicon-photonics hardware accelerator for vision transformers has been proposed in [22].' This internal contradiction invalidates the current novelty claim. Please rephrase the contribution as the first near-sensor and/or ROI-aware photonic ViT accelerator, and provide a substantive technical comparison with TRON.
  3. [Section IV (Performance Estimation and Performance Comparison)] The headline efficiency figure (100.4 KFPS/W) and the Table IV comparison against LightBulb, HolyLight, HQNNA, Robin, CrossLight, and Lightator are generated by an unreleased 'proprietary simulator' (Section IV). No energy/latency equations, parameter values, or memory/ADC/DAC specifications are given, and the fabricated MRs (Fig. 2c) are described without reporting measured insertion loss, tuning power, crosstalk, or Q-factor values. Consequently, the 8-bit precision assumption at Q=5000 and the end-to-end efficiency numbers cannot be verified or reproduced from the manuscript. Please release the simulator (or a detailed, versioned description) and include measured device characterization to validate the simulation.
  4. [Section IV (Table IV)] The efficiency comparison against prior SiPh accelerators reconstructs each competing design 'to closely match the original' using the authors' own simulator, with no details of the reconstruction (e.g., device models, operating points, dataflow, memory hierarchy). This makes the fairness of the comparison impossible to assess, especially since Lightator at its best (188.24 KFPS/W) already exceeds the proposed Opto-ViT (100.4 KFPS/W). Please provide the reconstruction methodology and, ideally, cross-validate at least one baseline against published numbers from the original papers.
minor comments (6)
  1. [Figures 10 and 11] Figure 10 and Figure 11 appear to show the same plot, although they are captioned as energy and latency, respectively; please verify and correct the figures.
  2. [Section III-B, Eq. (2)] The decomposition notation Q·K^T = Q·(X·W_K)^T = (Q·W_K^T)·X^T is valid but potentially confusing because (Q·W_K^T) still requires computing Q first; please clarify the intended computation order and how the three pre-tuned matrices W_Q, W_K^T, and X^T are combined.
  3. [Section IV (MR Resolution Analysis)] The formula 'Resolution = 1 / max |Pnoise|' is not dimensionally connected to bits; please explain how this quantity maps to 8-bit resolution and what noise level corresponds to one least-significant bit.
  4. [Table IV] The improvement values are expressed in percentages that are difficult to interpret (e.g., '2941.2% (↑)' for HolyLight); please report speedup ratios or clarify the baseline direction.
  5. [Abstract and Table IV] The abstract highlights 100.4 KFPS/W without noting that Table IV reports Lightator at up to 188.24 KFPS/W; the abstract and conclusion should qualify the efficiency claim or at least cite the comparison caveat.
  6. [Section IV (Accuracy Analysis)] In the Tiny-ImageNet mask row, the mask is transferred directly from ImageNet VID, which the authors acknowledge as a domain mismatch; please include the MGNet training details and the mask transfer procedure in the experimental setup so that the 4.5% drop is reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the efficiency figures are simulator outputs rather than fitted targets, and no central claim is defined in terms of its own conclusion.

full rationale

The paper's derivation chain is architectural and algorithmic: the VCSEL-MR optical core implements MatMuls, matrix decomposition avoids waiting on intermediate K matrices, MGNet performs RoI-based patch pruning, and QAT is used for 8-bit precision. None of these steps defines its output in terms of the conclusion it is used to support. The 100.4 KFPS/W figure and Table IV comparison are produced by the authors' proprietary simulator, which is a verifiability and reproducibility limitation, not a circularity: the numbers are simulation outputs rather than fitted parameters renamed as predictions, and no equation in the paper reduces the headline result to its own assumptions. The MGNet component is adopted from the authors' prior work [42], but the paper describes its operation, training objective, and integration rather than treating the citation as a proof of the central claim. No uniqueness theorem from the authors is invoked to forbid alternatives. The accuracy numbers are measured against external benchmarks (CIFAR-10, Tiny-ImageNet, COCO, ImageNet-VID), and the FPGA/GPU comparison uses externally reported platform efficiencies. The abstract's '<1.6% accuracy loss' is inconsistent with the reported 4.5% Tiny-ImageNet mask drop, but that is an accuracy-claim inconsistency, not a circular step. Overall, the derivation is self-contained enough that no specific prediction reduces by construction to its inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical particles or forces. The main assumptions are the accuracy of the MR noise model, the fidelity of the proprietary simulator, the algebraic decomposition benefit, and the universality of 8-bit QAT accuracy retention. The free parameters are device and pruning hyperparameters that directly affect the reported efficiency and accuracy.

free parameters (3)
  • Q-factor of microring resonators = ~5000
    Chosen to achieve 8-bit MR resolution based on the crosstalk model in Section IV; no measured device data are reported to confirm this value.
  • RoI threshold t_reg = not reported
    The sigmoid threshold in MGNet determines the skip ratio and the accuracy-energy trade-off; without a reported value, the patch-pruning results cannot be reproduced.
  • MGNet capacity = embed dim 192 or 384, heads 3 or 6
    Architecture hyperparameters of the RoI network are chosen per task, but no ablation is provided to justify the choice or its effect on accuracy and energy.
assumptions (4)
  • domain assumption The MR noise and resolution model from Duong et al. [41] accurately captures crosstalk and supports 8-bit precision at Q=5000.
    Used in Section IV (MR Resolution Analysis) to justify the device design; no measured MR extinction ratio or crosstalk figures are reported.
  • ad hoc to paper The proprietary in-house simulator faithfully models both the Opto-ViT design and the reconstructed prior accelerators.
    Section IV claims reconstructions of LightBulb, HolyLight, HQNNA, Robin, CrossLight, and Lightator using the authors' framework; this is not externally verifiable.
  • standard math Matrix multiplication associativity in Eq. (2) allows eliminating intermediate K buffering.
    The algebra is valid, but the claimed architectural benefit depends on how Q is pipelined, which is not fully specified.
  • domain assumption Quantization-aware training with symmetric 8-bit quantization preserves accuracy within the reported margins on all evaluated tasks.
    Supported by most tables, but the Tiny-ImageNet mask case shows a 4.5% drop, so the assumption does not hold universally.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Opto-ViT: Architecting a Near-Sensor Region of Interest-Aware Vision Transformer Accelerator with Silicon Photonics." pith.science (2026). https://pith.science/paper/MH73UALQ

@misc{pith2026250707044,
  author       = {Pith},
  title        = {Pith review of: Opto-ViT: Architecting a Near-Sensor Region of Interest-Aware Vision Transformer Accelerator with Silicon Photonics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MH73UALQ}},
  note         = {Machine review of arXiv:2507.07044}
}
read the original abstract

Vision Transformers (ViTs) have emerged as a powerful architecture for computer vision tasks due to their ability to model long-range dependencies and global contextual relationships. However, their substantial compute and memory demands hinder efficient deployment in scenarios with strict energy and bandwidth limitations. In this work, we propose OptoViT, the first near-sensor, region-aware ViT accelerator leveraging silicon photonics (SiPh) for real-time and energy-efficient vision processing. Opto-ViT features a hybrid electronic-photonic architecture, where the optical core handles compute-intensive matrix multiplications using Vertical-Cavity Surface-Emitting Lasers (VCSELs) and Microring Resonators (MRs), while nonlinear functions and normalization are executed electronically. To reduce redundant computation and patch processing, we introduce a lightweight Mask Generation Network (MGNet) that identifies regions of interest in the current frame and prunes irrelevant patches before ViT encoding. We further co-optimize the ViT backbone using quantization-aware training and matrix decomposition tailored for photonic constraints. Experiments across device fabrication, circuit and architecture co-design, to classification, detection, and video tasks demonstrate that OptoViT achieves 100.4 KFPS/W with up to 84% energy savings with less than 1.6% accuracy loss, while enabling scalable and efficient ViT deployment at the edge.

Figures

Figures reproduced from arXiv: 2507.07044 by the authors.

Figure 1
Figure 1. (a) Masked inference in Vision transformer (ViT), (b) Transformer [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. (a) Overall architecture of Opto-ViT. (b) Architecture of an optic [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. illustrates a simple example of performing a MatMul between two 3×3 matrices in an optical core. As shown and previously described, the column elements of the second matrix, denoted as W, are tuned into the MRs within each arm of the core, with each arm corresponding to a column of the W matrix. Next, the elements of the input matrix X are applied row by row as vectors to the VCSEL driver, modulating them into light… view at source ↗
Figures from the paper (5 more)
Figure 7
Figure 7. Figure 7: Proposed bottom-up evaluation framework. [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 6
Figure 6. Figure 6: Matrix splitting and hardware mapping methodology in Opto-ViT. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 8
Figure 8. Figure 8: Breakdown of energy consumption for processing different ViT models [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Processing delay breakdown for various ViT models processed with [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 11
Figure 11. Figure 11: Processing latency of the accelerator when running the baseline ViT [PITH_FULL_IMAGE:figures/full_fig_p007_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 44 canonical work pages

  1. [22]

    Tron: Transformer neural network acceleration with non-coherent silicon photonics,

    S. Afifi, F. Sunny, M. Nikdast, and S. Pasricha, “Tron: Transformer neural network acceleration with non-coherent silicon photonics,” in Proceedings of the Great Lakes Symposium on VLSI 2023 , 2023, pp. 15–21

    work page 2023

  2. [1]

    A 100,000 fps vision sensor with embedded 535gops/w 256 × 256 simd processor array,

    S. J. Carey, A. Lopich, D. R. Barr, B. Wang, and P. Dudek, “A 100,000 fps vision sensor with embedded 535gops/w 256 × 256 simd processor array,” in Symposium on VLSI . IEEE, 2013

    work page 2013

  3. [2]

    A 0.5-v real-time computational cmos image sensor with programmable kernel for feature extraction,

    T.-H. Hsu, Y .-R. Chen, R.-S. Liu, C.-C. Lo, K.-T. Tang, M.-F. Chang, and C.-C. Hsieh, “A 0.5-v real-time computational cmos image sensor with programmable kernel for feature extraction,” IEEE JSSC, vol. 56, pp. 1588–1596, 2020

    work page 2020

  4. [3]

    4.9 a 1ms high-speed vision chip with 3d-stacked 140gops column-parallel pes for spatio-temporal image processing,

    T. Yamazaki et al., “4.9 a 1ms high-speed vision chip with 3d-stacked 140gops column-parallel pes for spatio-temporal image processing,” in 2017 IEEE International Solid-State Circuits Conference (ISSCC) . IEEE, 2017, pp. 82–83

    work page 2017

  5. [4]

    Cmp-pim: an energy-efficient comparator-based processing-in-memory neural network accelerator,

    S. Angizi, Z. He, A. S. Rakin, and D. Fan, “Cmp-pim: an energy-efficient comparator-based processing-in-memory neural network accelerator,” in Proceedings of the 55th Annual Design Automation Conference , 2018, pp. 1–6

    work page 2018

  6. [5]

    Deep mapper: A multi-channel single-cycle near-sensor dnn accelerator,

    M. Morsali, S. Tabrizchi, M. Liehr, N. Cady, M. Imani, A. Roohi, and S. Angizi, “Deep mapper: A multi-channel single-cycle near-sensor dnn accelerator,” in 2023 IEEE International Conference on Rebooting Computing (ICRC). IEEE, 2023, pp. 1–5

    work page 2023

  7. [6]

    Macsen: A processing-in-sensor architecture integrating mac operations into image sensor for ultra-low-power bnn-based intelligent visual perception,

    H. Xu, Z. Li, N. Lin, Q. Wei, F. Qiao, X. Yin, and H. Yang, “Macsen: A processing-in-sensor architecture integrating mac operations into image sensor for ultra-low-power bnn-based intelligent visual perception,” IEEE TCAS II , vol. 68, pp. 627–631, 2020

    work page 2020

  8. [7]

    Senputing: An ultra-low-power always-on vision percep- tion chip featuring the deep fusion of sensing and computing,

    H. Xu et al., “Senputing: An ultra-low-power always-on vision percep- tion chip featuring the deep fusion of sensing and computing,” IEEE TCASI, 2021

    work page 2021

Show all 54 references
  1. [8]

    Appcip: Energy-efficient approximate convolution- in-pixel scheme for neural network acceleration,

    S. Tabrizchi et al., “Appcip: Energy-efficient approximate convolution- in-pixel scheme for neural network acceleration,” IEEE JETCAS , pp. 225–236, 2023

    2023

  2. [9]

    Pisa: A non-volatile processing-in-sensor accelerator for imaging systems,

    S. Angizi et al. , “Pisa: A non-volatile processing-in-sensor accelerator for imaging systems,” IEEE TETC, 2023

    2023

  3. [10]

    Mr-pipa: An integrated multi-level rram (hfo x) based processing-in-pixel accelerator,

    M. Abedin et al., “Mr-pipa: An integrated multi-level rram (hfo x) based processing-in-pixel accelerator,” IEEE JXCDC, 2022

    2022

  4. [11]

    Pinsim: A processing in-and near-sensor simulator to model intelligent vision sensors,

    S. Tabrizchi, M. Morsali, D. Pan, S. Angizi, and A. Roohi, “Pinsim: A processing in-and near-sensor simulator to model intelligent vision sensors,” IEEE Computer Architecture Letters , 2024

    2024

  5. [12]

    Apris: Approximate processing reram in-sensor architecture enabling artificial-intelligence-powered edge,

    S. Tabrizchi, R. Gaire, M. Morsali, M. Liehr, N. Cady, S. Angizi, and A. Roohi, “Apris: Approximate processing reram in-sensor architecture enabling artificial-intelligence-powered edge,” IEEE Transactions on Emerging Topics in Computing , 2024

    2024

  6. [13]

    Enabling normally-off in situ computing with a magneto- electric fet-based sram design,

    D. Najafi, M. Morsali, R. Zhou, A. Roohi, A. Marshall, D. Misra, and S. Angizi, “Enabling normally-off in situ computing with a magneto- electric fet-based sram design,” IEEE Transactions on Electron Devices, vol. 71, no. 4, pp. 2742–2748, 2024

    2024

  7. [14]

    Pixel-level processing: why, what, and how?

    A. E. Gamal, D. X. D. Yang, and B. A. Fowler, “Pixel-level processing: why, what, and how?” in Sensors, Cameras, and Applications for Digital Photography, vol. 3650. SPIE, 1999, pp. 2–13

    1999

  8. [15]

    A reconfigurable convolution-in-pixel cmos image sensor architecture,

    R. Song, K. Huang, Z. Wang, and H. Shen, “A reconfigurable convolution-in-pixel cmos image sensor architecture,” IEEE TCSVT , 2022

    2022

  9. [16]

    Pipsim: A behavior-level modeling tool for cnn processing-in-pixel accelerators,

    A. Roohi, S. Tabrizchi, M. Morsali, D. Z. Pan, and S. Angizi, “Pipsim: A behavior-level modeling tool for cnn processing-in-pixel accelerators,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 43, no. 1, pp. 141–150, 2023

    2023

  10. [17]

    Attention is all you need,

    A. Vaswani et al. , “Attention is all you need,” in Advances in Neural Information Processing Systems, I. Guyon et al., Eds., vol. 30. Curran Associates, Inc., 2017

    2017

  11. [18]

    Emerging properties in self-supervised vision transformers,

    M. Caron, H. Touvron, I. Misra, H. J ´egou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 9650–9660

    2021

  12. [19]

    Deepcompress- vit: Rethinking model compression to enhance efficiency of vision transformers at the edge,

    S. Ahmed, A. Al Arafat, D. Najafi, A. Mahmood, M. N. Rizve, M. Al Nahian, R. Zhou, S. Angizi, and A. S. Rakin, “Deepcompress- vit: Rethinking model compression to enhance efficiency of vision transformers at the edge,” in Proceedings of the Computer Vision and Pattern Recognit...

    2025

  13. [20]

    Vita: A vision transformer inference accelerator for edge applications,

    S. Nag, G. Datta, S. Kundu, N. Chandrachoodan, and P. A. Beerel, “Vita: A vision transformer inference accelerator for edge applications,” in 2023 IEEE International Symposium on Circuits and Systems (ISCAS) . IEEE, 2023, pp. 1–5

    2023

  14. [21]

    Retransformer: Reram-based processing-in-memory architecture for transformer acceleration,

    X. Yang, B. Yan, H. Li, and Y . Chen, “Retransformer: Reram-based processing-in-memory architecture for transformer acceleration,” in Proceedings of the 39th International Conference on Computer-Aided Design, 2020, pp. 1–9

    2020

  15. [23]

    An energy/illumination-adaptive cmos image sensor with reconfigurable modes of operations,

    J. Choi, S. Park, J. Cho, and E. Yoon, “An energy/illumination-adaptive cmos image sensor with reconfigurable modes of operations,” IEEE Journal of Solid-State Circuits , vol. 50, no. 6, pp. 1438–1450, 2015

    2015

  16. [24]

    AI edge devices using computing-in-memory and processing-in-sensor: from system to device,

    T.-H. Hsu et al. , “AI edge devices using computing-in-memory and processing-in-sensor: from system to device,” in IEDM, 2019

    2019

  17. [25]

    A near-sensor processing accelerator for approximate local binary pattern networks,

    S. Angizi, M. Morsali, S. Tabrizchi, and A. Roohi, “A near-sensor processing accelerator for approximate local binary pattern networks,” IEEE Transactions on Emerging Topics in Computing , vol. 12, no. 1, pp. 73–83, 2023

    2023

  18. [26]

    Robin: A robust optical binary neural network accelerator,

    F. P. Sunny, A. Mirza, M. Nikdast, and S. Pasricha, “Robin: A robust optical binary neural network accelerator,” ACM TECS, no. 5s, pp. 1–24, 2021

    2021

  19. [27]

    Neuro-photonix: Enabling near- sensor neuro-symbolic ai computing on silicon photonics substrate,

    D. Najafi, H. E. Barkam, M. Morsali, S. Jeong, T. Das, A. Roohi, M. Nikdast, M. Imani, and S. Angizi, “Neuro-photonix: Enabling near- sensor neuro-symbolic ai computing on silicon photonics substrate,” IEEE Transactions on Circuits and Systems for Artificial Intelligence , 2025

    2025

  20. [28]

    Crosslight: A cross- layer optimized silicon photonic neural network accelerator,

    F. Sunny, A. Mirza, M. Nikdast, and S. Pasricha, “Crosslight: A cross- layer optimized silicon photonic neural network accelerator,” in DAC. IEEE, 2021, pp. 1069–1074

    2021

  21. [29]

    Silicon photonics codesign for deep learning,

    Q. Cheng et al. , “Silicon photonics codesign for deep learning,” Pro- ceedings of the IEEE , vol. 108, pp. 1261–1282, 2020

    2020

  22. [30]

    Oisa: Architecting an optical in-sensor accelerator for efficient visual computing,

    M. Morsali, S. Tabrizchi, D. Najafi, M. Imani, M. Nikdast, A. Roohi, and S. Angizi, “Oisa: Architecting an optical in-sensor accelerator for efficient visual computing,” in 2024 Design, Automation & Test in Europe Conference & Exhibition (DATE) . IEEE, 2024, pp. 1–6

    2024

  23. [31]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/ forum?id=YicbFdNTTy

    2021

  24. [32]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2019

    2019 arXiv

  25. [33]

    Holylight: A nanophotonic accelerator for deep learning in data centers,

    W. Liu, W. Liu, Y . Ye, Q. Lou, Y . Xie, and L. Jiang, “Holylight: A nanophotonic accelerator for deep learning in data centers,” in DATE. IEEE, 2019, pp. 1483–1488

    2019

  26. [34]

    Light- bulb: A photonic-nonvolatile-memory-based accelerator for binarized convolutional neural networks,

    F. Zokaee, Q. Lou, N. Youngblood, W. Liu, Y . Xie, and L. Jiang, “Light- bulb: A photonic-nonvolatile-memory-based accelerator for binarized convolutional neural networks,” in DATE. IEEE, 2020, pp. 1438–1443

    2020

  27. [35]

    Hardware-software co-design of slimmed optical neural networks,

    Z. Zhao, D. Liu, M. Li, Z. Ying, L. Zhang, B. Xu, B. Yu, R. T. Chen, and D. Z. Pan, “Hardware-software co-design of slimmed optical neural networks,” in ASP-DAC. IEEE, 2019, pp. 705–710

    2019

  28. [36]

    Lightator: An optical near-sensor accelerator with compressive acquisition enabling versatile image pro- cessing,

    M. Morsali, B. Reidy, D. Najafi, S. Tabrizchi, M. Imani, M. Nikdast, A. Roohi, R. Zand, and S. Angizi, “Lightator: An optical near-sensor accelerator with compressive acquisition enabling versatile image pro- cessing,” arXiv preprint arXiv:2403.05037 , 2024

    2024 arXiv

  29. [37]

    Silicon microring resonators,

    W. Bogaerts, P. De Heyn, T. Van Vaerenbergh, K. De V os, S. Ku- mar Selvaraja, T. Claes, P. Dumon, P. Bienstman, D. Van Thourhout, and R. Baets, “Silicon microring resonators,” Laser & Photonics Reviews , pp. 47–73, 2012

    2012

  30. [38]

    Reusing softmax hardware unit for gelu computation in transformers,

    C. Peltekis, K. Alexandridis, and G. Dimitrakopoulos, “Reusing softmax hardware unit for gelu computation in transformers,” in 2024 IEEE 6th International Conference on AI Circuits and Systems (AICAS), 2024, pp. 159–163

    2024

  31. [39]

    [Online]

    (2011) Ncsu eda freepdk45. [Online]. Available: http://www.eda.ncsu. edu/wiki/FreePDK45

    2011

  32. [40]

    Synopsys design compiler, product version 14.9.2014,

    Synopsys, Inc., “Synopsys design compiler, product version 14.9.2014,” 2014

    2014

  33. [41]

    A case study of signal-to-noise ratio in ring-based optical networks-on-chip,

    L. H. K. Duong, M. Nikdast, S. Le Beux, J. Xu, X. Wu, Z. Wang, and P. Yang, “A case study of signal-to-noise ratio in ring-based optical networks-on-chip,” IEEE Design & Test , vol. 31, 2014

    2014

  34. [42]

    Energy-efficient & real-time computer vision with intelligent skipping via reconfigurable CMOS image sensors,

    M. A.-A. Kaiser, S. Sarkar, P. A. Beerel, A. R. Jaiswal, and G. Datta, “Energy-efficient & real-time computer vision with intelligent skipping via reconfigurable CMOS image sensors,” arXiv preprint arXiv:2409.17341, 2024

    2024 arXiv

  35. [43]

    Quantization and training of neural networks for efficient integer-arithmetic-only inference,

    B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko, “Quantization and training of neural networks for efficient integer-arithmetic-only inference,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2...

    2018

  36. [44]

    Estimating or propagating gradients through stochastic neurons for conditional computation,

    Y . Bengio, N. L ´eonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308.3432, 2013

    2013 arXiv

  37. [45]

    I-vit: Integer-only quantization for efficient vision transformer inference,

    Z. Li and Q. Gu, “I-vit: Integer-only quantization for efficient vision transformer inference,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 17 065–17 075

    2023

  38. [46]

    Imagenet-21k pretraining for the masses,

    T. Ridnik, E. Ben-Baruch, A. Noy, and L. Zelnik-Manor, “Imagenet-21k pretraining for the masses,” arXiv preprint arXiv:2104.10972 , 2021

    2021 arXiv

  39. [47]

    Convolutional deep belief networks on cifar-10,

    A. Krizhevsky and G. Hinton, “Convolutional deep belief networks on cifar-10,” Unpublished manuscript, vol. 40, no. 7, pp. 1–9, 2010

    2010

  40. [48]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in Proc. Conference on Computer Vision and Pattern Recognition , 2009, pp. 248–255

    2009

  41. [49]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13 . Springer,...

    2014

  42. [50]

    Exploring plain vision transformer backbones for object detection,

    Y . Li, H. Mao, R. Girshick, and K. He, “Exploring plain vision transformer backbones for object detection,” in European conference on computer vision . Springer, 2022, pp. 280–296

    2022

  43. [51]

    Mask r-cnn,

    K. He, G. Gkioxari, P. Doll ´ar, and R. Girshick, “Mask r-cnn,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 2961–2969

    2017

  44. [52]

    Imagenet large scale visual recognition challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet large scale visual recognition challenge,” International journal of computer vision, vol. 115, pp. 211–252, 2015

    2015

  45. [53]

    A silicon photonic accelerator for convolutional neural networks with heterogeneous quantization,

    F. Sunny, M. Nikdast, and S. Pasricha, “A silicon photonic accelerator for convolutional neural networks with heterogeneous quantization,” in GLSVLSI, 2022, pp. 367–371

    2022

  46. [54]

    Eq-vit: Algorithm-hardware co-design for end-to- end acceleration of real-time vision transformer inference on versal acap architecture,

    P. Dong et al. , “Eq-vit: Algorithm-hardware co-design for end-to- end acceleration of real-time vision transformer inference on versal acap architecture,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , vol. 43, no. 11, pp. 3949–3960, 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.