{"id":"5ac0d909-9dd5-453b-af0a-e392fa75cd2c","arxiv_id":"2412.04307","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A public benchmark and unified test conditions for compressing intermediate features of large models, with two image-codec baselines evaluated.","lead":"This paper builds a test dataset of intermediate features from three large models (DINOv2, Llama3, and SD3) across five tasks, along with unified bitrate and accuracy evaluation conditions. It benchmarks two image codecs adapted as baselines, giving the feature coding community a shared starting point.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline-specific truncation in Table 4 violates the paper's 'unified test conditions' and may drive the VTM-over-Hyperprior benchmark conclusion.","rationale":"I agree with the reader's CONDITIONAL verdict and with the observation that truncation ranges differ between baselines, but the reader's stated weakest assumption is split-point representativeness. The truncation mismatch is more load-bearing because it is an internal inconsistency with the paper's own 'unified test conditions' contribution, not merely an external representativeness question. A first benchmark can defensibly fix a small set of models and split points, but a benchmark whose two headline baselines are evaluated under different lossy pre-processing is not yet a fair standardized testbed. The concrete test uses the released code to settle the effect: if the ranking persists under identical truncation, the core benchmark message stands; if it flips, the benchmark tables need revision. The dataset, feature distribution analysis, and BPFP proposal retain value, and the issue is addressable, so no verdict change beyond the existing CONDITIONAL is warranted.","tokens_in":13952,"tokens_out":5743,"duration_ms":60031,"concrete_test":"Re-run the two baselines on Cls, Seg, and Dpt with identical truncation ranges per task (for example, use the VTM ranges [-20,20] for Cls and Seg and [-20,20] for SPDM4 for both methods), keeping quantization and all other settings fixed, and recompute the rate-accuracy table and R-AD curves. If VTM still dominates at matched BPFP, the codec-level conclusion survives; if Hyperprior's accuracy rises or the ranking flips, the published comparison is confounded by preprocessing choices.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Sec. 5, the paper claims 'unified test conditions' that enable 'fair comparisons across future feature coding studies,' but Sec. 6.1.1 states that 'different truncations are used for the two baselines for their distinct coding strategies,' and Table 4 gives different truncation ranges for the same task: Cls and Seg use [-20,20] for VTM versus [-5,5] for Hyperprior, and SPDM4 uses [-20,20] versus [-10,10]. Truncation is a lossy pre-processing step, so the two baselines are coding different inputs at the same BPFP. Consequently, the Table 3 result that 'the VTM baseline outperforms the Hyperprior baseline' can be an artifact of Hyperprior's more aggressive truncation rather than of codec capability. Because the benchmark's central contribution is a standardized, fair testbed, allowing each method to choose its own truncation range undermines that contribution. The dataset and analysis remain useful, and the issue is fixable by fixing truncation and quantization as part of the test conditions, but as written the 'unified' claim is internally inconsistent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new research area, large model feature coding, and contributes a dataset (LaMoFC) of intermediate features extracted from three large models (DINOv2, Llama3, SD3) across five tasks (image classification, semantic segmentation, depth estimation, common sense reasoning, text-to-image synthesis). It proposes a new bitrate metric, Bits Per Feature Point (BPFP), and defines task-specific evaluation heads. Two baseline codecs are introduced by adapting VVC (VTM) and Hyperprior to feature inputs, and a benchmark is reported. The stated goal is to provide a standardized testbed and unified test conditions to enable fair comparisons in future feature coding research.","tokens_in":14174,"tokens_out":3418,"duration_ms":34431,"significance":"If the described dataset and test conditions were fully realized, this would be a valuable community resource. The public release of code and data is a concrete strength, as is the proposal of BPFP to decouple bitrate from pixel-based conventions. The dataset covers model types and modalities that are genuinely under-explored in the feature coding literature. However, the central claim of 'unified test conditions' is currently undermined by baseline-specific pre-processing, and the benchmark comparison is therefore not as clean as advertised. The dataset and BPFP contribution can stand after revision, but the benchmark conclusions need to be re-examined or re-framed.","major_comments":[{"comment":"The paper claims in Sec. 5 that it establishes unified test conditions enabling fair comparisons, but Sec. 6.1.1 states that 'different truncations are used for the two baselines for their distinct coding strategies,' and Table 4 shows different truncation ranges for the same task (e.g., Cls: [-20,20] for VTM vs [-5,5] for Hyperprior; Seg: [-20,20] vs [-5,5]; SPDM4: [-20,20] vs [-10,10]). Since truncation is a lossy pre-processing step, the two baselines are coding different inputs at the same reported BPFP, so the head-to-head comparison in Table 3 is not a fair test of codec capability. Specifically, the conclusion that the VTM baseline outperforms the Hyperprior baseline may be an artifact of Hyperprior's more aggressive truncation rather than of the coding method itself. Fixing truncation ranges as part of the test conditions, or reporting results across a common set of truncation ranges, is necessary to support the 'unified' and 'fair comparison' claims.","section":"Sec. 6.1.1, Table 4"},{"comment":"The source data selection is biased toward samples that are already correctly handled by the feature-extraction model. For Cls, the text says 'each representing a unique class accurately classified by DINOv2,' and for CSR, 'chosen for their longest input prompts and correct prediction by Llama3.' This guarantees a 100% oracle accuracy for these tasks, which inflates the initial accuracy and removes headroom for measuring accuracy degradation. It also may not reflect the full distribution of inputs encountered in real deployments, where misclassified or ambiguous samples are common. The authors should justify this choice explicitly or include a stratified sample with both correctly and incorrectly classified examples, and analyze the impact on the reported rate-accuracy curves.","section":"Sec. 4.1.3"},{"comment":"No error bars, confidence intervals, or repeated-run statistics are reported anywhere in the benchmark. Given the small sample sizes (80 for depth estimation, 100 for segmentation, 500 for the others), the differences between the two baselines at comparable bitrates (e.g., Cls at BPFP~2: 99.80 vs 93.20; Seg at BPFP~1.7: 80.42 vs 77.96) may be within sampling variability. Reporting standard deviations across multiple random seeds or bootstrap intervals would substantially strengthen the empirical claims and help readers assess whether the observed differences are significant.","section":"Sec. 4.1 and Table 3"}],"minor_comments":[{"comment":"The notation SPDS, SPDM, SPG, and SPH is introduced in prose but Table 1 abbreviates these as split points without a cross-reference to the definitions; consider adding a dedicated column or a footnote in Table 1 to improve readability.","section":"Sec. 4.1.2"},{"comment":"Figures 2 and 3 are qualitative visualizations of distributions and DCT energy. To make the redundancy analysis more concrete, consider adding a quantitative measure such as the fraction of energy in the DC coefficient or a spatial autocorrelation statistic.","section":"Sec. 4.2"},{"comment":"The acronyms R-A and D-A are used in headings and figure captions but are not expanded in the main text; please define them at first use in Sec. 6.2 and Sec. 6.3.","section":"Sec. 6.2"},{"comment":"The sentence 'Different truncations are used for the two baselines for their distinct coding strategies' is an explicit acknowledgment of the fairness issue; even if this is a deliberate design choice, the paper should explain why this is acceptable under the 'unified test conditions' claim, rather than leaving the reader to reconcile the two statements.","section":"Sec. 6.1.1"}],"recommendation":"major_revision","confidential_remarks":"The dataset and code release are genuine contributions, but the 'unified test conditions' claim is the central scaffolding of the paper and is contradicted by the baseline-specific truncation ranges. The authors should carefully rework this part of the paper, ideally by fixing truncation/quantization as part of the evaluation protocol. If they instead retain baseline-specific settings, they should either drop the 'fair comparisons' language or present the baseline comparison as an exploratory study rather than a standardized benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful thing: this is the first public benchmark that takes feature coding beyond small CNNs and gives the community a concrete testbed of features from DINOv2, Llama3, and SD3 across five tasks, with code and data released. The BPFP bitrate unit is a simple and sensible replacement for BPP when the coded input is a feature tensor, not pixels. The distribution and DCT analyses (Figs. 2-3) make a real point: transformer features have different redundancy statistics than ResNet/Mask R-CNN features, so the old benchmark features are not necessarily representative. The two baselines are honest starting points, not claimed to be competitive codecs, and the cross-task generalizability experiment is a worthwhile addition.\n\nThe main soft spot lands exactly where the stress-test note points. Section 5 advertises 'unified test conditions,' but Section 6.1.1 and Table 4 show that the two baselines use different truncation ranges for the same task — e.g., Cls and Seg use [-20,20] for VTM and [-5,5] for Hyperprior. Truncation is lossy preprocessing, so the two codecs are compressing different inputs at the same BPFP. The Table 3 result that VTM outperforms Hyperprior is therefore not a clean comparison, and the 'unified' claim is internally inconsistent. This is fixable: specify a single truncation range per task (or per feature type) as part of the test conditions, and require all methods to use it; or report sensitivity across ranges. I would not call the central dataset contribution broken, but the benchmark comparison as presented needs the fix.\n\nOther issues are real but minor. The Cls and CSR sample sets are filtered to correctly classified samples, which biases difficulty upward; sample sizes are small (80–500) and there are no error bars. The split-point choice is defended by downstream task needs, though transfer to other models remains an open question. None of this is fatal, and the paper is transparent about the choices.\n\nOverall, the dataset and BPFP convention are worth having, and the paper is a legitimate contribution to coding for machines. It deserves a serious referee pass. I would accept it for review with a request to fix the truncation inconsistency before publication. I would likely cite it for the dataset if the truncated test conditions get cleaned up.","headline":"First large-model feature coding benchmark; useful dataset and BPFP metric, but the 'unified test conditions' are undercut by baseline-specific truncation ranges.","tokens_in":14693,"tokens_out":2309,"would_cite":true,"duration_ms":20534,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a public dataset, unified test conditions, and baseline codecs to launch feature coding for large models.","keywords":["feature coding","large models","benchmark","test dataset","BPFP","DINOv2","Llama3","Stable Diffusion 3"],"falsifier":"Extract features from other split points of the same models, for example DINOv2's 10th block or Llama3's 8th decoder layer, run the two baseline codecs, and compare the value distributions and rate-accuracy curves with Table 3 and Figure 2 of the paper; a large mismatch would show the dataset is not representative of large-model features in general.","tokens_in":13792,"feed_emoji":"🧠","tokens_out":7829,"duration_ms":73958,"temperature":0.7,"pith_summary":"Large models are increasingly split across cloud and edge, and the feature tensors passed between segments must be compressed to save bandwidth and storage. This paper argues that existing feature-coding research, built on small convolutional networks and visual data, does not transfer to Transformer-based large models, and that the field lacks the shared resources needed to develop. To fill that gap, it constructs a public dataset of features from DINOv2, Llama3, and Stable Diffusion 3, covering five tasks and both visual and textual modalities, and it fixes unified test conditions including a new bitrate metric, bits per feature point (BPFP). It then evaluates two adapted image codecs on this testbed, showing substantial bitrate savings but visible accuracy loss at low rates and weak correlation between feature distortion and task accuracy for text. The paper's central claim is that large-model feature coding deserves its own research program, one that can now be pursued against a common benchmark.","feed_headline":"Large-model feature coding gets its first shared testbed","feed_subtitle":"Three model families, five tasks, two data modalities, plus baselines to beat.","key_machinery":"The load-bearing object is the proposed testbed: five feature classes extracted at fixed split points of three large models, namely FCls and FSeg from the 40th DINOv2 ViT block, FDpt as a concatenation of the 10th, 20th, 30th, and 40th blocks, FCSR from the 32nd Llama3 decoder layer, and FTTI from the Stable Diffusion 3 VAE decoder input. The evaluation protocol centers on BPFP, bits per feature point, which counts coding bits on the feature tensor itself rather than on source pixels, making bitrate comparable across visual, textual, and generative features. The baseline pipeline is pre-processing (truncation to a central range, 10-bit uniform quantization, packing into a 2D YUV-400 frame), a core codec (VTM intra coding or Hyperprior), and post-processing (unpacking and de-quantization). The fixed split points do the argumentative work: they define what counts as the feature in a distributed large-model system, so the dataset's representativeness rests entirely on those choices.","core_discovery":"The paper's central discovery is that the features large models exchange have statistics that existing feature-coding benchmarks do not capture. The 40th-block DINOv2 features, 32nd-layer Llama3 features, and Stable Diffusion 3 VAE-decoder-input features show wider, more asymmetric value ranges, sparser and more concentrated distributions, and redundancy patterns (strong vertical correlation for vision transformers, weak spatial correlation for text) that differ from ResNet and Mask R-CNN features used in earlier work. Under two baseline codecs, a handcrafted VVC Intra pipeline and a learned Hyperprior pipeline, these features can be compressed to a fraction of their original 32-bit size, but accuracy drops sharply below a bitrate threshold, and feature mean squared error is not a reliable predictor of task accuracy except for text-to-image synthesis. The intended conclusion is that large-model feature coding is a distinct problem: its data, metrics, and codecs need to be designed specifically for these representations, and the dataset plus test conditions provided here are offered as the common ground for doing so.","pith_inferences":["Extension: the observed vertical redundancy in DINOv2 features suggests a codec that predicts along the token dimension, or a learned transform matched to patch-based features, could beat DCT-based packing; the paper identifies the property but does not test such a codec.","Extension: BPFP could plausibly be applied to embeddings and latents beyond the three models, but the paper's dataset does not show how the metric behaves on other large-model families, so that transfer remains untested.","Extension: a codec trained on 40th-block DINOv2 features may waste bits on early-block features, since the paper's distribution analysis shows layer-dependent ranges; a cross-layer generalizability test would extend the existing cross-task analysis."],"forward_implications":["Future feature-coding papers can report results on the same five feature classes, making rate-accuracy numbers directly comparable across methods and papers.","BPFP replaces bits-per-pixel as the natural bitrate unit for feature coding, including non-visual features where pixels do not exist.","The benchmark shows that off-the-shelf image codecs lose task accuracy at low bitrates, so new codecs exploiting the redundancy structure of transformer features are needed.","Because feature MSE does not track task accuracy for textual features, semantic distortion metrics must be developed before LLM feature coding can be properly optimized."],"supporting_citations":[{"why":"Supplies the DINOv2 vision model whose 40th-block features define the Cls, Seg, and Dpt feature classes in the dataset.","marker":"[33]"},{"why":"Supplies Llama3, the 8B model whose 32nd-decoder-layer features define the CSR feature class.","marker":"[11]"},{"why":"Supplies Stable Diffusion 3, the model whose VAE-decoder-input features define the TTI feature class.","marker":"[12]"},{"why":"Provides the learned Hyperprior codec used as one of the two baseline compression methods.","marker":"[1]"},{"why":"Provides VVC, whose VTM Intra implementation is used as the handcrafted baseline codec.","marker":"[2]"},{"why":"Supplies the ImageNet source images for the image classification feature set.","marker":"[8]"},{"why":"Supplies the train-test split used to select NYU-Depth-v2 source data for the depth estimation feature set.","marker":"[22]"},{"why":"Supports the choice of the Llama3 split point by showing features at that layer integrate with other large models.","marker":"[45]"}],"fun_headline_variants":["First benchmark for coding features from large models","Large-model feature coding gets its own test suite","New dataset and test conditions for feature coding","Baselines and benchmark for compressing model features","Unified test conditions for large-model feature coding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole testbed rests on the assumption that features extracted at the chosen split points, the 40th DINOv2 block, the 32nd Llama3 decoder layer, and the SD3 VAE decoder input, are representative of what real distributed systems will exchange.","fun_headline_variants_meta":{"raw":{"variants":["First benchmark for coding features from large models","Large-model feature coding gets its own test suite","New dataset and test conditions for feature coding","Baselines and benchmark for compressing model features","Unified test conditions for large-model feature coding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000329,"raw_usage":{"total_tokens":1840,"prompt_tokens":957,"completion_tokens":883,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":814}},"tokens_in":573,"tokens_out":883,"duration_ms":9094,"temperature":1.0,"reasoning_tokens":814,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:31:35.519424+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Extract features from other split points of the same models, for example DINOv2's 10th block or Llama3's 8th decoder layer, run the two baseline codecs, and compare the value distributions and rate-accuracy curves with Table 3 and Figure 2 of the paper; a large mismatch would show the dataset is not representative of large-model features in general.","supporting_citations":[{"cited_title":"Scaling recti- fied flow transformers for high-resolution image synthesis","cited_arxiv_id":null,"evidence_quote":"Supplies Stable Diffusion 3, the model whose VAE-decoder-input features define the TTI feature class."},{"cited_title":"Sullivan, and Jens-Rainer Ohm","cited_arxiv_id":null,"evidence_quote":"Provides VVC, whose VTM Intra implementation is used as the handcrafted baseline codec."},{"cited_title":"ImageNet: a large-scale hierarchical image database","cited_arxiv_id":null,"evidence_quote":"Supplies the ImageNet source images for the image classification feature set."},{"cited_title":"NExT-GPT: Any-to-any multimodal LLM","cited_arxiv_id":null,"evidence_quote":"Supports the choice of the Llama3 split point by showing features at that layer integrate with other large models."}],"review_version":1}