{"id":"d94ff56c-ebd2-4d84-a31c-df7f3e4aab62","arxiv_id":"2411.19114","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PREBA uses an FPGA preprocessing accelerator and MIG-aware dynamic batching to raise partitioned-GPU inference throughput 3.7x over a CPU baseline.","lead":"PREBA offloads AI inference data preprocessing from the CPU to an FPGA and adds a MIG-aware dynamic batching system, reporting 3.7x throughput and 3.4x tail-latency improvements on an NVIDIA A100 partitioned into seven GPU slices. The paper matters because it identifies CPU preprocessing as the main bottleneck when GPUs are partitioned for inference, a practical problem for GPU rental and AI-as-a-service providers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No accuracy validation for FPGA-preprocessed inputs: if the DPU output differs numerically from the OpenCV/Librosa baseline, model accuracy could shift, undermining the like-for-like validity of the reported 3.7x throughput and other speedups.","rationale":"The paper's central contribution is a hardware/software co-design for an AI inference server. In an inference server, the returned prediction is the product; throughput/latency are secondary. The entire evaluation (Figures 17–21) reports performance, power, and cost but never accuracy. This is a critical omission because the DPU is a fundamentally different implementation of the preprocessing pipeline. Xilinx Vitis libraries, while functionally correct, implement operations like resize and STFT with their own precision and rounding, which can differ from OpenCV and Librosa. For example, OpenCV's resize uses specific interpolation kernels and rounding; a fixed-point FPGA implementation may use different coefficients. Similarly, Librosa's mel spectrogram applies a specific window and FFT normalization; FPGA floating-point FFTs may produce slightly different values. These small numerical differences are known to affect deep model predictions, especially for high-accuracy transformer models like Swin-Transformer and Conformer. If PREBA changes the input distribution, the impressive throughput gains may not represent a valid improvement over the CPU baseline: the system could simply be serving a different, potentially degraded task. The test I propose settles this directly: compare accuracy on standard benchmarks. This is a routine validation in inference-serving papers and is absent here. My concern does not attack the implementation's internal consistency; it targets the missing link between 'we preprocess faster' and 'the inference is correct.' Since the reader already issued CONDITIONAL and this concern is a primary condition, I recommend keeping CONDITIONAL (no change).","tokens_in":23611,"tokens_out":6363,"duration_ms":54445,"concrete_test":"For each of the six workloads, feed a fixed evaluation set (ILSVRC-2012 validation for MobileNet/SqueezeNet/Swin-Transformer; LibriSpeech test-clean for CitriNet/Conformer) through both the CPU baseline pipeline (OpenCV/Librosa) and the DPU pipeline. Compute the end-to-end model accuracy (top-1 for CV, word error rate for ASR) for each pipeline. If the absolute difference is within a pre-registered tolerance (e.g., ≤0.5% top-1 or ≤0.5% WER), the correctness assumption holds; if accuracy degrades beyond that, the speedup numbers must be revised to report performance at equal accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central performance claims are end-to-end but only metrics are throughput/latency/power/cost; Section 6 reports no accuracy for any of the six workloads. PREBA replaces CPU-side OpenCV/Librosa preprocessing with FPGA Vitis Vision/DSP implementations (Section 4.2). These libraries are not guaranteed bit-exact to the CPU references: JPEG decoding, resizing interpolation, FFT, Mel-filterbank and mean/variance normalization can differ in rounding, fixed-point precision, and algorithmic details. For pretrained ImageNet and LibriSpeech models, a small shift in input distribution can change predictions. The paper's 3.7x throughput improvement is only meaningful if the FPGA pipeline produces inputs that yield the same model accuracy as the CPU pipeline. The absence of any accuracy comparison (e.g., top-1 accuracy, word error rate) leaves the central claim unsupported on correctness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies NVIDIA MIG-based AI inference servers and argues that CPU-side data preprocessing becomes a critical bottleneck as the GPU is partitioned into many vGPUs. To address this, PREBA offloads image and audio preprocessing to an FPGA-based DPU and adds a MIG-aware dynamic batching system that chooses batch size and queueing time from profiled latency/throughput curves. The system is implemented on an A100 with an Alveo U55C FPGA and evaluated on six vision and audio models. The reported results are an average 3.7x throughput improvement, 3.4x tail-latency reduction, 3.5x energy-efficiency improvement, and 3.0x cost-efficiency improvement over a CPU-preprocessing baseline, with an ablation study separating the DPU and dynamic batching contributions.","tokens_in":23818,"tokens_out":4007,"duration_ms":38055,"significance":"If the reported results are valid, PREBA addresses a real and increasingly important problem: MIG makes CPU preprocessing a bottleneck, and prior work on MIG inference has largely ignored the preprocessing stage. The paper's strengths are that it is built end-to-end on commodity hardware, uses six diverse workloads, includes a clear 'Ideal' oracle as an upper bound, and provides an ablation study that separates the hardware and software contributions. The main weakness is that the paper never demonstrates that the FPGA-preprocessed inputs preserve model accuracy relative to the CPU baseline, which is essential for interpreting the speedups as like-for-like. The dynamic batching heuristic also needs stronger justification. These issues are fixable with additional measurements and analysis, so the work is potentially a solid systems contribution if they are addressed.","major_comments":[{"comment":"The paper does not validate that FPGA-preprocessed inputs are functionally equivalent to the CPU baseline's OpenCV/Librosa outputs. Section 4.2 describes DPU units implemented with Vitis Vision and DSP libraries, and Section 6 reports only throughput, latency, power, and cost (Figures 17-21), with no accuracy comparison for any of the six workloads. JPEG decoding, resizing interpolation, FFT/Mel-filterbank, and mean/variance normalization can differ numerically between the CPU and FPGA implementations due to rounding, fixed-point precision, and algorithmic details, and for pretrained ImageNet and LibriSpeech models such differences can change predictions. The reported 3.7x throughput improvement is only meaningful if the DPU produces inputs that yield the same model accuracy as the CPU pipeline. Please add an accuracy comparison (e.g., top-1 accuracy on ILSVRC-2012 for vision models and word error rate on LibriSpeech for audio models) or a direct numerical equivalence check (e.g., maximum absolute or relative error against CPU reference outputs).","section":"§4.2 and §6"},{"comment":"All performance results are reported without run-to-run variance, confidence intervals, or the number of repetitions. Since the central claims are empirical (3.7x throughput, 3.4x tail latency, 3.5x energy efficiency, 3.0x cost efficiency), a single measurement per configuration leaves open the possibility that the improvements are within noise, especially for tail-latency metrics. Please report at least three to five independent runs per configuration and show mean and variance (or error bars) for the main throughput and latency figures.","section":"§6, Figures 17-20"},{"comment":"The baseline is a single CPU-based preprocessing implementation using OpenCV and Librosa on a 32-core EPYC server. The paper generalizes from this to claim that CPU-side preprocessing is the critical bottleneck in MIG inference, but it does not compare against a more optimized CPU pipeline (e.g., multi-threaded OpenCV, vectorized preprocessing, NVIDIA DALI, or GPU-side preprocessing). Section 3.3 reports that sustaining a 1g.5gb(7x) A100 would require 393 CPU cores for CitriNet, but this number depends on the efficiency of the specific CPU implementation. Without a state-of-the-art or at least a reasonably tuned CPU baseline, the 75.6% throughput drop and the 3.7x speedup may overstate the severity of the preprocessing bottleneck. Please add a comparison against a stronger CPU preprocessing baseline or explicitly scope the claim to the evaluated baseline.","section":"§3.3 and §6.1"},{"comment":"The analytical model for Time_queue rests on two assertions that are not fully justified. First, it claims that Time_knee is approximately constant at 35 ms regardless of audio input length, but Figure 15 shows this only for three audio models on 1g.5gb(7x), and the supporting data for other MIG configurations and models is not shown. Second, the paper sets Time_queue = Time_knee / (number of vGPUs) without a derivation or a sensitivity study; this division is presented as a heuristic, yet the batching section calls it an analytical model. The end-to-end results are encouraging, but the paper should either provide a short queueing-theoretic justification for the division by vGPU count or present a sensitivity analysis showing that the exact choice of this divisor does not materially affect the reported results.","section":"§4.3, Figure 15"}],"minor_comments":[{"comment":"The caption contains a typo: 'Tansformer' should be 'Transformer'.","section":"Figure 21"},{"comment":"Table 1 reports FPGA resource utilization but not the operating frequency or the achieved preprocessing throughput per CU, so it is difficult to assess the efficiency of the DPU implementation. Please add the clock frequency and the per-unit throughput or latency numbers.","section":"Table 1"},{"comment":"The Time_queue formula is described in prose rather than as a numbered equation; please write it explicitly as an equation (e.g., Time_queue = Time_knee / V for V vGPUs) and relate it to the notation in Figure 15.","section":"§4.3"},{"comment":"The text says the baseline uses 'Librosa' but reference [2] is 'LibrosaCpp'. Please clarify whether the baseline is the Python Librosa library or the C++ port, since numerical differences between these implementations could affect the claimed baseline preprocessing cost.","section":"§5, Software"},{"comment":"The definition of 'Ideal' as a system with no preprocessing overhead is clear, but the statement that PREBA achieves '91.6% of the performance of Ideal for 5 out of the 6 studied models' would benefit from a per-model table so that the outlier (presumably Conformer large or CitriNet) is visible.","section":"Section 6.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's main novel angle is the combination of FPGA preprocessing offload with MIG-aware dynamic batching. The closest prior work (TrainBox, DLBooster) targets training, not MIG inference, so the novelty is defensible. The main risk is that the speedup claims are presented without any accuracy validation, which is a correctness issue rather than a tuning issue. Adding accuracy equivalence measurements and a stronger CPU baseline should be feasible within the manuscript's scope and would make the paper publishable. I would also encourage the editor to ask for at least a brief comparison with DALI or a GPU-based preprocessing baseline, because the current single-baseline comparison makes the external validity of the 3.7x claim unclear."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead PREBA. The core observation is sound and the implementation is real: on an A100 partitioned into 1g.5gb(7x), CPU-side preprocessing becomes the bottleneck as the number of vGPU instances scales, and offloading it to an FPGA DPU plus a MIG-aware dynamic batching system gives a measured 3.7x end-to-end throughput gain. The characterization (Figures 5-9) is clearly presented, and the ablation in Figure 22 does a good job separating the DPU effect (101% gain) from the dynamic batching effect (54% further gain). Prior work did not identify this MIG-specific preprocessing scaling problem, so the novelty claim holds.\n\nThe soft spot is the one the stress-test note puts its finger on: there is no accuracy validation for FPGA-generated inputs. Section 4.2 uses Vitis Vision/DSP libraries, Section 6 reports throughput, latency, power, and cost, but never checks top-1 accuracy or WER against the OpenCV/Librosa CPU pipeline. JPEG decode, resize interpolation, FFT, Mel-filterbank, and normalization can all differ numerically in fixed-point or rounding behavior, and pre-trained ImageNet/LibriSpeech models can be sensitive to that. Without this check, the 3.7x number is not strictly like-for-like. This is a real gap, but it is addressable in a revision.\n\nSmaller issues: no error bars or run-to-run variance; only one CPU baseline. Both minor, common in systems papers. The Time_queue 'guarantee' in Section 4.3 is stated more formally than the argument supports, but it is a heuristic that works in evaluation. The profiling-based Batch_knee selection is not circular because the final claims are measured end-to-end against a baseline, not predicted from the fitted curves.\n\nI agree with the reader's conditional verdict. The central design is plausible, the engineering is substantial, and the missing accuracy check is the main thing standing between this and a clean accept. A serious referee should ask for that experiment and then the paper would likely hold up.\n\nRecommendation: send to peer review, with a request for accuracy validation and some variance reporting. I'd cite a revised version.","headline":"Real implementation and a clear MIG preprocessing characterization, but the missing accuracy check for FPGA-preprocessed inputs has to be answered before the 3.7x claim is fully like-for-like.","tokens_in":24308,"tokens_out":2513,"would_cite":false,"duration_ms":23932,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PREBA shows that CPU-side data preprocessing, not the GPU, is the main bottleneck in MIG-based AI inference servers, and that an FPGA data processing unit plus a MIG-aware dynamic batching system removes it.","keywords":["Multi-Instance GPU","AI inference serving","FPGA data preprocessing","dynamic batching","tail latency","GPU utilization","MIG scheduling","hardware/software co-design"],"falsifier":"Compare end-to-end accuracy for the six workloads under CPU and DPU preprocessing on identical inputs: top-1 accuracy for MobileNet, SqueezeNet, and Swin-Transformer; word error rate for CitriNet and the two Conformers. A material accuracy gap (for example, more than about 0.5 points) would mean the measured 3.7x throughput does not translate into equally correct inference.","tokens_in":23441,"feed_emoji":"⚡","tokens_out":6608,"duration_ms":54587,"temperature":0.7,"pith_summary":"NVIDIA's Multi-Instance GPU (MIG) splits one large GPU into several smaller virtual GPUs (vGPUs), each capable of running an independent inference server. This paper claims that the resulting servers are not GPU-bound but preprocessing-bound: every activated vGPU multiplies the CPU-side work of decoding, resizing, and spectrogram computation, and the CPU cannot keep up. PREBA is the proposed fix, a hardware/software co-design that moves all data preprocessing onto an FPGA-based data processing unit and pairs it with a batching system whose parameters are chosen from per-model, per-vGPU profiling. In a real A100 system with six workloads, PREBA reports a 3.7x throughput gain, a 3.4x tail-latency reduction, a 3.5x energy-efficiency gain, and a 3.0x cost-efficiency gain over a CPU-preprocessing baseline. If these numbers hold, MIG becomes a practical way to serve latency-critical AI at high utilization.","feed_headline":"FPGA preprocessing unblocks partitioned GPU inference","feed_subtitle":"Moving data prep off the CPU lifts throughput 3.7x and cuts tail latency 3.4x on MIG servers.","key_machinery":"The load-bearing object is the batch knee, $\\mathit{Batch_{knee}}$: the largest batch size before a vGPU's tail latency starts spiking with little throughput gain. The DPU's computing units (CUs), each a chain of functional units for decode-resize-crop-normalize or resample-mel-spectrogram-normalize, are the hardware mechanism that makes small batches cheap; the audio path splits into two CU types so the normalize stage (which needs all samples) does not serialize independent requests. The software half is a dynamic batching system that profiles throughput-versus-latency curves per model and MIG configuration, bucketizes variable audio lengths into queues, and computes $\\mathit{Time_{queue}}$ analytically as $\\mathit{Time_{knee}}$ divided by the vGPU count.","core_discovery":"The paper's central claim is that in a MIG inference server the scarce resource is not GPU compute but CPU preprocessing throughput. With seven vGPUs active, preprocessing demand scales sevenfold while CPU utilization saturates near 90%, causing a 75.6% end-to-end throughput drop in the authors' characterization. PREBA removes that bottleneck with an FPGA DPU whose computing units are deliberately optimized for single-input latency (rather than batch throughput), so requests are preprocessed immediately on arrival and the downstream batching stage has free choice of batch size. On top of that, the dynamic batching system sets the maximum batch size at the 'knee' of the tail-latency curve ($\\mathit{Batch_{knee}}$), a point that is smaller for fine-grained vGPUs, and sets the queueing window as $\\mathit{Time_{knee}}$ divided by the number of vGPUs. The measured result is that PREBA sustains 91.6% of an oracle 'Ideal' system's throughput on five of six models, while the CPU baseline loses 77.2% on average.","pith_inferences":["If FPGA and CPU preprocessing are numerically identical, the same offload recipe could apply to tokenization and other light text preprocessing, but this paper does not test that claim.","The DPU→CPU→GPU data path adds tens of microseconds per request; at very high request rates or with multiple DPUs on one PCIe root complex, a direct DPU-to-GPU path (P2P) would likely be needed, which the paper notes but does not implement.","The $\\mathit{Time_{queue}} = \\mathit{Time_{knee}} / \\#\\text{vGPUs}$ rule assumes roughly even traffic across queues; bursty or skewed arrivals may require an adaptive controller rather than the fixed analytical formula."],"forward_implications":["A MIG server with PREBA can reach over 91% of an oracle system's throughput, so AIaaS providers can partition a single A100 into seven vGPUs without paying the preprocessing tax.","Because $\\mathit{Batch_{knee}}$ depends on vGPU size, model, and input length, the profiling step must be repeated per configuration; the paper provides a low-cost recipe for doing so.","Variable-length audio workloads should be batched by length bucket rather than by arrival count, which is what PREBA's per-bucket queues do.","Energy efficiency rises because the DPU displaces CPU cycles, even though GPU power consumption increases with higher utilization."],"supporting_citations":[{"why":"Defines NVIDIA's Multi-Instance GPU, the partitioning feature the paper characterizes and targets.","marker":"[63]"},{"why":"A100 datasheet: the GPU whose seven-GPC MIG slices are used in all experiments.","marker":"[61]"},{"why":"Prior MIG inference scheduling work that disables preprocessing, the baseline PREBA extends.","marker":"[43]"},{"why":"Analyzes data stalls in DNN training, establishing preprocessing as a throughput bottleneck and motivating the offload.","marker":"[56]"},{"why":"NVIDIA Triton Inference Server: the reference serving pipeline (preprocess, batch, execute) that PREBA reimplements.","marker":"[58]"},{"why":"Prior FPGA offload of data preprocessing for training, the closest hardware approach to PREBA's DPU.","marker":"[69]"},{"why":"SLA-aware batching for inference that PREBA's dynamic batching system builds on.","marker":"[23]"},{"why":"Supplies the cost-efficiency metric (throughput time over CAPEX plus OPEX) used in the TCO evaluation.","marker":"[50]"}],"fun_headline_variants":["MIG servers' CPU preprocessing bottleneck solved with FPGA co-design","FPGA accelerator breaks MIG's preprocessing bottleneck, boosts throughput 3.7x","PREBA: FPGA preprocessing for MIG inference, 3.7x throughput gain","Offload data prep to FPGA: MIG inference gets 3.7x faster","Beat MIG preprocessing with FPGA: 3.7x throughput, 3.4x latency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The speedups are measured without verifying that the FPGA's JPEG, resize, and Mel-spectrogram outputs are numerically equivalent to the CPU's OpenCV/Librosa outputs, so model accuracy could shift even though throughput and latency improve.","fun_headline_variants_meta":{"raw":{"variants":["MIG servers' CPU preprocessing bottleneck solved with FPGA co-design","FPGA accelerator breaks MIG's preprocessing bottleneck, boosts throughput 3.7x","PREBA: FPGA preprocessing for MIG inference, 3.7x throughput gain","Offload data prep to FPGA: MIG inference gets 3.7x faster","Beat MIG preprocessing with FPGA: 3.7x throughput, 3.4x latency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000689,"raw_usage":{"total_tokens":3122,"prompt_tokens":944,"completion_tokens":2178,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":2069}},"tokens_in":560,"tokens_out":2178,"duration_ms":12141,"temperature":1.0,"reasoning_tokens":2069,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:31:19.964451+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare end-to-end accuracy for the six workloads under CPU and DPU preprocessing on identical inputs: top-1 accuracy for MobileNet, SqueezeNet, and Swin-Transformer; word error rate for CitriNet and the two Conformers. A material accuracy gap (for example, more than about 0.5 points) would mean the measured 3.7x throughput does not translate into equally correct inference.","supporting_citations":[{"cited_title":"Multi-Instance GPU","cited_arxiv_id":null,"evidence_quote":"Defines NVIDIA's Multi-Instance GPU, the partitioning feature the paper characterizes and targets."},{"cited_title":"NVIDIA A100","cited_arxiv_id":null,"evidence_quote":"A100 datasheet: the GPU whose seven-GPC MIG slices are used in all experiments."},{"cited_title":"PARIS and ELSA: An Elastic Scheduling Algorithm for Reconfigurable Multi-GPU Inference Servers","cited_arxiv_id":null,"evidence_quote":"Prior MIG inference scheduling work that disables preprocessing, the baseline PREBA extends."},{"cited_title":"Analyzing and Mitigating Data Stalls in DNN Training","cited_arxiv_id":null,"evidence_quote":"Analyzes data stalls in DNN training, establishing preprocessing as a throughput bottleneck and motivating the offload."},{"cited_title":"NVIDIA Triton Inference Server","cited_arxiv_id":null,"evidence_quote":"NVIDIA Triton Inference Server: the reference serving pipeline (preprocess, batch, execute) that PREBA reimplements."},{"cited_title":"TrainBox: An Extreme-Scale Neural Network Training Server Architecture by Sys- tematically Balancing Operations","cited_arxiv_id":null,"evidence_quote":"Prior FPGA offload of data preprocessing for training, the closest hardware approach to PREBA's DPU."},{"cited_title":"Lazy Batching: An SLA-aware Batching System for Cloud Machine Learning Inference","cited_arxiv_id":null,"evidence_quote":"SLA-aware batching for inference that PREBA's dynamic batching system builds on."},{"cited_title":"E3: Energy-efficient Microservices on SmartNIC- accelerated Servers","cited_arxiv_id":null,"evidence_quote":"Supplies the cost-efficiency metric (throughput time over CAPEX plus OPEX) used in the TCO evaluation."}],"review_version":1}