SoftNeuro: Fast Deep Inference using Multi-platform Optimization

Gabriel Ogawa; Hiroshi Miyake; Hitoshi Matsuo; Masaki Hilaga; Tatsuya Kawaguchi; Yasuhiro Kuroda; Yusuke Kozawa

RECORD

SoftNeuro: Fast Deep Inference using Multi-platform Optimization

Masaki Hilaga, Yasuhiro Kuroda, Hitoshi Matsuo, Tatsuya Kawaguchi, Gabriel Ogawa, Hiroshi Miyake, Yusuke Kozawa

Machine Learning · Submitted 12 Oct 2021

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Experiments show that the proposed framework achieves both fast inference and efficient tuning.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2110.06037 (version 1)

Pith IDNAW3KPCR, Pith's permanent record of this paper and its review

CitationGabriel Ogawa, Hiroshi Miyake, Hitoshi Matsuo, Masaki Hilaga, Tatsuya Kawaguchi, Yasuhiro Kuroda, Yusuke Kozawa. "SoftNeuro: Fast Deep Inference using Multi-platform Optimization." arXiv:2110.06037 (2021). https://pith.science/paper/NAW3KPCR

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 4,727 input tokens.

What the abstract itself states, as probabilities: claims first 0.10 · states limitation 0.06 · human participants 0.02 · controlled comparison 0.07 · randomized assignment 0.01 · states quantitative result 0.05 · compares with prior results 0.10 · mentions code or data availability 0.03.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

Faster inference of deep learning models is highly demanded on edge devices and even servers, for both financial and environmental reasons. To address this issue, we propose SoftNeuro, a novel, high-performance inference framework with efficient performance tuning. The key idea is to separate algorithmic routines from network layers. Our framework maximizes the inference performance by profiling various routines for each layer and selecting the fastest path. To efficiently find the best path, we propose a routine-selection algorithm based on dynamic programming. Experiments show that the proposed framework achieves both fast inference and efficient tuning.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Faster inference of deep learning models is highly demanded on edge devices and even servers, for both financial and environmental reasons.	background 1.00	0.05	0.02	asserted 1.00
s1	To address this issue, we propose SoftNeuro, a novel, high-performance inference framework with efficient performance tuning.	contribution 1.00	0.80	0.11	asserted 0.96
s2	The key idea is to separate algorithmic routines from network layers.	method 0.84	0.55	0.04	asserted 1.00
s3	Our framework maximizes the inference performance by profiling various routines for each layer and selecting the fastest path.	method 0.95	0.83	0.14	asserted 1.00
s4	To efficiently find the best path, we propose a routine-selection algorithm based on dynamic programming.	method 0.68	0.88	0.05	asserted 1.00
s5	Experiments show that the proposed framework achieves both fast inference and efficient tuning.	result 1.00	0.95	0.84	asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.98.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

Faster inference of deep learning models is highly demanded on edge devices and even servers, for both financial and environmental reasons. To address this issue, we propose SoftNeuro, a novel, high-performance inference framework with efficient performance tuning. The key idea is to separate algorithmic routines from network layers. Our framework maximizes the inference performance by profiling various routines for each layer and selecting the fastest path. To efficiently find the best path, we propose a routine-selection algorithm based on dynamic programming. Experiments show that the proposed framework achieves both fast inference and efficient tuning.

Human review

How Pith checks papers → · Full audit of this paper →

How the literature uses it

3 arXiv papers cite this in 4 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it

Uses its method or tool 3 papers

Although the prediction would be a bottleneck, optimized by SoftNeuro®[SoftNeuro® is developed by Morpho, Inc.] [ref] , it can be done within 0.2 seconds with one node.

Surrogate Modeling for Computationally Expensive Simulations of Supernovae in High-Resolution Galaxy Simulations · Discussion · 0.98
For inference, computationally expensive architectures in our deep learning models (e.g. 3D-CNNs) are optimized to the supercomputer Fugaku by SoftNeuro® [ref] developed by Morpho, Inc.

3D-Spatiotemporal Forecasting the Expansion of Supernova Shells Using Deep Learning toward High-Resolution Galaxy Simulations · Prediction for the SN Explosion · 1.00
Additionally, the inference of ML in our framework is optimized for two main architectures, which are ONNX [ref] for x86 architecture and SoftNeuro [ref] for ARM architecture.

ASURA-FDPS-ML: Star-by-star Galaxy Simulations Accelerated by Surrogate Modeling for Supernova Feedback · Speed-up · 0.99

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.