{"id":"57482986-5493-49cc-b02c-dbe092d97aba","arxiv_id":"2411.13050","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A co-designed chip and training method, Topkima-Former, selects the top-k attention scores inside an in-memory converter to speed up softmax by up to 15x with a 0.4-1.2% accuracy drop.","lead":"Transformers are slow and power-hungry because softmax, which decides which words matter, needs to process many values. This paper proposes a chip circuit that finds only the five biggest attention scores directly inside the memory array, cutting softmax latency by up to 15x with only a small accuracy dip.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 0.4-1.2% accuracy drop is for global top-k, but the hardware selects per-sub-crossbar top-k; Fig. 4(c) shows this degrades accuracy, and the final 256x256 hardware accuracy is never numerically reported.","rationale":"The reader's verdict is already CONDITIONAL, and this stress-test reinforces it rather than moving it. The most load-bearing weakness is the mismatch between the software accuracy experiment and the hardware selection rule: the paper's headline accuracy numbers are for global top-k, while the described 256x256 design uses per-sub-crossbar top-k. This is not a speculative failure mode; the paper itself documents in Fig. 4(c) that sub-top-k changes the selected indices and degrades accuracy in the 128x128 case. For the 256x256 case, the claim that accuracy is 'comparable' is unquantified, so the actual hardware accuracy is unknown. The comparator/PVT and alpha-worst-case concerns the reader raised are real but secondary: PVT corners are reported for the arbiter path, and alpha is an average that can be characterized empirically. The sub-top-k issue directly affects the central accuracy deliverable and is acknowledged within the manuscript, making it the cleanest load-bearing concern. A concrete simulation using the hardware selection rule would settle whether the final hardware preserves the headline accuracy. The macro-level speedup and energy-efficiency comparisons may still be valid, so the appropriate outcome remains a conditional acceptance with the accuracy claim made conditional on an end-to-end sub-top-k evaluation.","tokens_in":11027,"tokens_out":6316,"duration_ms":66005,"concrete_test":"Run TFCBP training and inference exactly as in the paper but replace the global top-5 rule with the hardware sub-top-k rule: for each attention head, split the 384 scores into a 256-column block and a 128-column block, retain the top-3 of the first block and the top-2 of the second block, mask all other scores before softmax, and use 5-bit Q and 4-bit/15-level K^T quantization as described in Section III-A. Report accuracy for ViT/CIFAR-10, ViT/CIFAR-100, DistilBERT/SQuAD, and BERT-base/SQuAD, and compare with both the paper's global top-5 result and the untruncated baseline. If any model/dataset pair loses more than 0.2 additional percentage points relative to global top-5, the abstract's 0.4-1.2% accuracy-drop range must be restated for the actual sub-top-k hardware configuration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim ('0.4% to 1.2% reduction' at k=5) is obtained from a software global-top-k experiment, but the proposed hardware does not implement global top-k. Because a 256x256 crossbar is smaller than the 64x384 K^T matrix, the paper splits one head into two sub-crossbars and selects top-3 in one and top-2 in the other (Section IV-B), with no crossbar-global arbitration. The paper's own Fig. 4(c) shows this sub-top-k rule differs from global top-k (e.g., selecting [127,128], [255,256], [384] instead of [380,...,384]) and causes accuracy loss for 128x128. For the 256x256 configuration it only says accuracy is 'comparable' to global top-k; no numerical result is reported. Since the final hardware accuracy is the conjunction of the software top-k approximation, sub-top-k fragmentation, 4-bit K^T precision, 5-bit quantization, and IMA/distribution errors, the headline 0.4-1.2% drop cannot be attached to the full Topkima-Former unless the hardware selection rule is evaluated end-to-end. This is the load-bearing gap: if the 256x256 sub-top-k loss is nontrivial, the headline accuracy claim is overstated even though the macro speedup numbers may stand.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Topkima-Former, a hardware-software co-design for transformer inference that accelerates the softmax operation by combining top-k activation selection with in-memory ADC (IMA). The circuit-level contribution is a decreasing-ramp IMA with an arbiter-encoder that identifies the k largest dot-product results in Q·K^T without an explicit sorting network. The algorithm-level contribution is a top-k forward, complete-backward training scheme (TFCBP) that keeps the accuracy loss at k=5 between 0.4% and 1.2% on ViT, DistilBERT, and BERT-base. The architecture-level contribution is a scale-free attention that folds the 1/sqrt(dk) scaling into WQ. The paper reports macro-level speedups of 15x and 8x versus a conventional softmax macro and a digital top-k softmax macro, and system-level speedups of 1.8x-84x and energy-efficiency improvements of 1.3x-35x over prior IMC accelerators.","tokens_in":11277,"tokens_out":9838,"duration_ms":86064,"significance":"If the claims hold, the work offers a practical way to remove a known bottleneck in transformer inference. The decreasing-ramp IMA idea is clean, and the combination of top-k selection with the ADC ramp eliminates explicit sorting, which is a genuine contribution. The paper honestly reports the accuracy degradation of sub-top-k fragmentation in Fig. 4(c) but does not numerically report the final hardware accuracy. The use of SPICE simulation for the circuit and NeuroSim for the system is commendable, and the accuracy evaluation is performed on held-out data (Fig. 3). The TFCBP training is simple and appears to be effective in reducing the accuracy drop compared to prior top-k approaches. The scale-free attention is a known but useful trick. The overall co-design story is coherent and of interest to the IMC and transformer-acceleration community.","major_comments":[{"comment":"The headline accuracy claim ('0.4% to 1.2% reduction' at k=5) is obtained from a software experiment with global top-k (Fig. 3). The proposed hardware, however, selects per-sub-crossbar top-k with no crossbar-global arbitration: the 256x256 configuration uses sub-top-k 3 and 2 for the two halves of K^T. Fig. 4(c) shows that sub-top-k differs from global top-k, and the text only states that the 256x256 accuracy is 'comparable' to global top-k without reporting a numerical value. Since the final hardware accuracy is the conjunction of the software top-k approximation, sub-top-k fragmentation, 4-bit K^T precision, 5-bit quantization, and IMA errors, the reported 85.1% accuracy after IMA error injection does not establish the end-to-end accuracy. Please report the numerical accuracy for the actual 256x256 sub-top-k configuration for all four datasets, or explicitly state the combined accuracy drop.","section":"Section IV-B, Fig. 4(c)"},{"comment":"The macro-level SPICE simulations are performed in 65 nm CMOS, while the system-level NeuroSim evaluation and the comparison in Table I are reported at 32 nm. The paper does not describe how the SPICE-derived parameters (Tarb, Tima, α, and energy values) are scaled from 65 nm to 32 nm, or why this scaling does not change the speedup and EE conclusions. Without a scaling methodology, the consistency between the macro-level and system-level numbers is unclear. Please either present the SPICE macro at the target 32 nm technology or provide a transparent scaling model.","section":"Section IV-B and Table I"},{"comment":"The early-stopping factor α is given as 0.31 averaged over the dataset, and the latency model uses this average in Tima,arb. The claimed 15x and 8x speedups of topkima-SM depend critically on this value. If the distribution of α has a heavy tail (e.g., inputs with many scores near the top-k threshold), the average may not be representative. Please report the distribution or the worst-case α across the dataset and show the sensitivity of the speedup claim to α.","section":"Section IV-B, Eq. (4)"},{"comment":"The paper reports that after injecting IMA errors in the SW simulation, the BERT-base SQuAD accuracy drops from 86.7% to 85.1% (a 1.6% absolute drop). This exceeds the 'less than 1.2%' top-k-only drop stated in Section IV-A. The relationship between these numbers and the headline 0.4-1.2% claim should be clarified: how much of the 1.6% drop is due to top-k approximation, quantization, and IMA errors, respectively?","section":"Section IV-B"}],"minor_comments":[{"comment":"There is a typo: 'tokima softmax macro' should be 'topkima softmax macro'.","section":"Abstract"},{"comment":"The description of the arbiter-encoder timing is terse; please clarify what happens if two SAs fire in the same cycle and whether the ranking among them is arbitrary.","section":"Section III-A, Fig. 2"},{"comment":"The latency numbers in Fig. 4(a) appear to be in microseconds, while the text uses nanoseconds for Tima; please ensure the units are consistent and clearly labeled.","section":"Section IV-B, Fig. 4(a)"},{"comment":"The abstract says softmax 'comprises up to 40% of the total latency' while the introduction says 'up to 40% inference time'; please unify the phrasing.","section":"Abstract and Introduction"},{"comment":"The claim that the sorting operation consumes at least 75% of the latency is not backed by a specific reference or a quantitative example; please provide the basis for this estimate.","section":"Section II-B"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid engineering contribution with a plausible circuit idea. The key issue is that the headline accuracy claim is not yet tied to the actual hardware configuration; the authors need to provide the numerical sub-top-k accuracy. The technology-node scaling also needs clarification. I believe these are fixable within a revision. The paper may be a good fit for the journal, but the quantitative claims should be tightened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Topkima is a real hardware trick: flip the ramp direction in the IMA so larger MAC values trip the comparator earlier, then use an arbiter-encoder to grab the top-k without sorting. That's genuinely new relative to the cited prior art, and the SPICE macro plus the honest sub-top-k analysis in Fig. 4(c) give it credibility. The scale-free rewrite is an exact algebraic change, cleanly folded into WQ, and the TFCBP training tweak is simple but reasonable.\n\nThe soft spots are the usual simulation-paper ones, plus one that matters. The abstract's 0.4-1.2% accuracy drop comes from a software global-top-k experiment. The hardware splits each head into sub-crossbars and selects top-3/top-2 locally. The paper does show in Fig. 4(c) that sub-top-k 3+2 stays close to global top-k, but the exact number for the final 256x256 configuration never appears in the text. That should be stated: report end-to-end accuracy for the actual hardware (sub-top-k, 4-bit K^T, quantization, injected circuit error) rather than the software-only headline.\n\nThe bigger methodology gap is the 65nm-to-32nm transition. The SPICE macro is 65nm; the system-level table is 32nm, and no scaling factor is given. For a paper whose quantitative claims are the main event, that needs an explicit statement. The alpha=0.31 early-stopping factor is an average; real-time claims should show the distribution or a worst-case number. And while they handle ties in the arbiter, the circuit's functional correctness across PVT is asserted only via delay measurements, not end-to-end simulation.\n\nNone of these kill the core idea. The macro speedups (15x vs conventional, 8x vs digital top-k) are plausible given the sorting elimination and early stopping, and the sub-top-k analysis is a good-faith disclosure of a real limitation. The system-level comparisons to other accelerators are noisy but not egregiously unfair.\n\nWho should read it: people building IMC accelerators for transformers, and anyone worrying about softmax overhead on edge hardware. It's an incremental systems contribution, not a scientific breakthrough, but it deserves a serious referee. I'd send it out with a request to report the end-to-end accuracy number, explain the node scaling, and add worst-case analysis. That's a revise-and-resubmit, not a reject.","headline":"A genuinely new IMA circuit trick for top-k softmax; the accuracy-reporting gap is real but fixable, and the core contribution deserves review.","tokens_in":11897,"tokens_out":4595,"would_cite":true,"duration_ms":44626,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Topkima-Former claims that embedding top-k selection into the analog-to-digital converters inside an SRAM compute-in-memory array removes sorting latency and cuts softmax macro latency about 15x versus conventional and 8x versus digital…","keywords":["in-memory computing","top-k softmax","transformer acceleration","SRAM compute-in-memory","ramp ADC","quantization-aware training","scale-free attention","winner-take-all"],"falsifier":"Feed attention-score vectors into a circuit-level model of the decreasing-ramp IMA with 5-bit conversion, deliberately including ties and cases where the $k$th and $(k{+}1)$th largest scores differ by less than one LSB, and count how often the indices reported by the arbiter-encoder differ from the exact top-k; a non-negligible mismatch rate would break the claim that top-k is obtained without sorting latency in hardware.","tokens_in":10777,"feed_emoji":"⚡","tokens_out":13711,"duration_ms":117148,"temperature":0.7,"pith_summary":"Topkima-Former sets out to show that the softmax bottleneck in transformer attention, which can consume up to 40% of inference latency, can be attacked at the circuit level by selecting only the $k$ largest attention scores inside the analog-to-digital converter of an SRAM compute-in-memory array. The proposed topkima macro uses a decreasing ramp so that larger MAC voltages trip the comparator first, and a counter stops conversion once $k$ columns have fired, eliminating the separate sorting step that slows digital top-k designs and shrinking the softmax workload from $d$ terms to $k$. A modified training scheme, top-k forward with complete backward propagation, keeps the accuracy cost of $k=5$ to a reported 0.4-1.2% across ViT, DistilBERT, and BERT-base on CIFAR-10, CIFAR-100, and SQuAD. The paper reports that the resulting softmax macro is about 15x faster and 30x more energy-efficient than a conventional softmax macro, about 8x faster and 3x more energy-efficient than a digital top-k softmax macro, and that the full Topkima-Former system is 1.8x-84x faster and 1.3x-35x more energy-efficient than earlier in-memory accelerators. This matters because softmax overhead grows with sequence length, so the payoff should increase for long-context models.","feed_headline":"Circuit trick finds top attention scores, cuts softmax delay 15x","feed_subtitle":"It skips sorting, keeps only five scores, and reports accuracy loss under 1.2% on ViT, DistilBERT, and BERT-base.","key_machinery":"The load-bearing mechanism is the topkima macro: a dual-10T SRAM compute-in-memory array for $Q \\cdot K^T$ whose column ADC is a decreasing-ramp converter that doubles as a winner-take-all selector. A common ramp starts higher than every MAC voltage and falls one step per cycle; the column with the largest value crosses first, its sense amplifier latches, and the crossing time becomes the digitized value while the order of crossings gives the rank. An arbiter-encoder, using the same request/acknowledge handshake as event-based vision sensors, reports which columns fired, and a counter halts the conversion once $k$ have fired, so no sorting network is needed. Two supporting pieces carry the accuracy and architecture claims: TFCBP training, which uses only top-k activations in the forward pass but full gradients in the backward pass, and a scale-free rewrite that folds the $1/\\sqrt{d_k}$ normalization of attention into $W_Q$, removing per-element scaling hardware.","core_discovery":"At its core, the paper claims that top-k selection, normally a sorting problem, can be merged into the analog-to-digital conversion step of a compute-in-memory macro at almost no extra cost. The macro computes $Q \\cdot K^T$ in the analog domain, then uses a decreasing ramp so the column with the largest MAC voltage crosses the sense-amplifier threshold first; an arbiter-encoder records the fired columns and a counter stops the ramp after $k$ firings. The values and identities of the top-k scores emerge from the conversion itself, and only those $k$ values proceed to a digital softmax core. The paper's experiments at $k=5$ show a 0.4-1.2% accuracy drop for ViT, DistilBERT, and BERT-base on the tested datasets, macro-level speedups of about 15x over a conventional softmax macro and 8x over a digital top-k softmax macro, and energy savings of about 30x and 3x respectively. The paper also reports 1.8x-84x speedup and 1.3x-35x energy-efficiency gains for the full system when the scale-free attention trick is included.","pith_inferences":["The reported early-stopping factor $\\alpha$ of about 0.31 is an average; an input with many near-top scores would stop the ramp later and shrink the 15x macro speedup, so the distribution of stopping cycles is worth measuring.","Because physical crossbars force $K^T$ to be split, global top-k becomes sub-top-k per array, and the paper's own data show an accuracy cost from that split; the real hardware accuracy is therefore worse than the headline 0.4-1.2% algorithm-level number.","The same winner-take-all-in-ADC idea could plausibly extend to other cheap-attention schemes such as ReLU attention or sparsemax, or to any layer where only the largest activations matter.","TFCBP has only been exercised at $k=5$ on three models; whether smaller $k$ becomes viable with more training is an open question, though the CIFAR-10 result at $k=1$ hints that the ceiling may be higher."],"forward_implications":["Softmax cost scales with $k$ instead of the full sequence dimension $d$, so the speedup from this design grows as transformer sequence lengths increase.","Digital top-k softmax is shown to be unattractive because sorting dominates its latency; embedding selection in the ADC is what removes that overhead.","The attention scaling factor can be absorbed into the trained $W_Q$ weights, eliminating a per-element division from hardware.","At $k=5$ with TFCBP training, accuracy loss stays within 0.4-1.2% on the tested ViT, DistilBERT, and BERT-base workloads.","System-level comparisons with earlier in-memory accelerators report 1.8x-84x speedup and 1.3x-35x energy-efficiency improvements."],"supporting_citations":[{"why":"Supplies the ramp in-memory ADC macro that the paper modifies by reversing the ramp direction from increasing to decreasing.","marker":"[6]"},{"why":"Provides the arbiter-encoder handshake that records which columns fired first and stops conversion after $k$ firings.","marker":"[16]"},{"why":"Defines the digital top-k softmax baseline whose sorting overhead motivates the new approach.","marker":"[3]"},{"why":"Supplies the quantization-aware-training idea and the earlier RRAM in-memory accelerator used as a comparison point.","marker":"[1]"},{"why":"Provides the system-level in-memory computing simulation setup used for architecture latency and energy results.","marker":"[5]"},{"why":"Supplies the digital softmax core that computes probabilities from the $k$ selected scores.","marker":"[17]"},{"why":"Introduces the scale-free attention approach that the paper rewrites to fold the scaling factor into $W_Q$.","marker":"[21]"},{"why":"Provides the softmax hardware latency model and the evidence that softmax cost rises sharply with sequence length.","marker":"[13]"},{"why":"Documents that softmax can consume up to 40% of transformer inference latency, the problem this design targets.","marker":"[7]"}],"fun_headline_variants":["Top-k ADC macro speeds softmax 15x, trims accuracy 1.2%","Merging top-k into ADC gives 15x faster softmax, <1.2% accuracy loss","Topkima softmax macro: 15x speedup, 8x vs digital top-k, 1.2% max loss","In-memory top-k selection cuts softmax latency 15x, keeps accuracy high","Top-k analog conversion skips sorting, speeds softmax 15x for transformers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the analog comparator and arbiter-encoder reliably picking out the true top-k MAC results on every input under process, voltage, and temperature variation and ties, while the split crossbars preserve enough global top-k structure for the headlined accuracy to hold.","fun_headline_variants_meta":{"raw":{"variants":["Top-k ADC macro speeds softmax 15x, trims accuracy 1.2%","Merging top-k into ADC gives 15x faster softmax, <1.2% accuracy loss","Topkima softmax macro: 15x speedup, 8x vs digital top-k, 1.2% max loss","In-memory top-k selection cuts softmax latency 15x, keeps accuracy high","Top-k analog conversion skips sorting, speeds softmax 15x for transformers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000657,"raw_usage":{"total_tokens":3093,"prompt_tokens":1114,"completion_tokens":1979,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":730,"completion_tokens_details":{"reasoning_tokens":1851}},"tokens_in":730,"tokens_out":1979,"duration_ms":20107,"temperature":1.0,"reasoning_tokens":1851,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:54:00.154892+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed attention-score vectors into a circuit-level model of the decreasing-ramp IMA with 5-bit conversion, deliberately including ties and cases where the $k$th and $(k{+}1)$th largest scores differ by less than one LSB, and count how often the indices reported by the arbiter-encoder differ from the exact top-k; a non-negligible mismatch rate would break the claim that top-k is obtained without sorting latency in hardware.","supporting_citations":[{"cited_title":"A 16K current- based 8T SRAM compute-in-memory macro with decoupled read/write and 1-5bit column ADC,","cited_arxiv_id":null,"evidence_quote":"Supplies the ramp in-memory ADC macro that the paper modifies by reversing the ramp direction from increasing to decreasing."},{"cited_title":"A 240×180 130 db 3 µs latency global shutter spatiotemporal vision sensor,","cited_arxiv_id":null,"evidence_quote":"Provides the arbiter-encoder handshake that records which columns fired first and stops conversion after $k$ firings."},{"cited_title":"A length adaptive algorithm-hardware co-design of transformer on fpga through sparse attention and dynamic pipelining,","cited_arxiv_id":null,"evidence_quote":"Defines the digital top-k softmax baseline whose sorting overhead motivates the new approach."},{"cited_title":"ReTransformer: ReRAM-based processing-in-memory architecture for transformer acceleration,","cited_arxiv_id":null,"evidence_quote":"Supplies the quantization-aware-training idea and the earlier RRAM in-memory accelerator used as a comparison point."},{"cited_title":"DNN+ NeuroSim V2. 0: An end-to-end benchmarking framework for compute-in-memory accelerators for on-chip training,","cited_arxiv_id":null,"evidence_quote":"Provides the system-level in-memory computing simulation setup used for architecture latency and energy results."},{"cited_title":"Hardware-aware softmax approximation for deep neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the digital softmax core that computes probabilities from the $k$ selected scores."},{"cited_title":"Tron: Transformer neural network acceleration with non-coherent silicon photonics,","cited_arxiv_id":null,"evidence_quote":"Introduces the scale-free attention approach that the paper rewrites to fold the scaling factor into $W_Q$."},{"cited_title":"Efficient softmax hardware architecture for deep neural networks,","cited_arxiv_id":null,"evidence_quote":"Provides the softmax hardware latency model and the evidence that softmax cost rises sharply with sequence length."},{"cited_title":"Softermax: Hardware/software co-design of an efficient softmax for transformers,","cited_arxiv_id":null,"evidence_quote":"Documents that softmax can consume up to 40% of transformer inference latency, the problem this design targets."}],"review_version":1}