Pith. sign in

REVIEW 5 major objections 5 minor 21 references

FoQuS claims a small, trajectory-selected coreset can replace the full AMR training set.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

FoQuS uses three training-dynamics scores to select a 1% to 30% coreset for AMR and claims higher accuracy than ten existing coreset methods.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Useful relative comparison of coreset heuristics on AMR, but the central claim is unverified: no full-training baseline appears anywhere in the paper. the 5 major comments →

arxiv 2509.08300 v1 pith:GPNK6NCE submitted 2025-09-10 cs.LG cs.AI

FoQuS: A Forgetting-Quality Coreset Selection Framework for Automatic Modulation Recognition

Yao Lu , Chunfeng Sun , Dongwei Xu , Yun Lin , Qi Xuan , Guan Gui This is my paper
classification cs.LG cs.AI
keywords automatic modulation recognitioncoreset selectiontraining dynamicsforgetting scoresignal-to-noise ratiodata-efficient learningcross-architecture generalizationwireless signal classification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes FoQuS, a coreset selection method for automatic modulation recognition (AMR), and claims that training on just 1% to 30% of the original data can approximate full-dataset training accuracy. It works by running one full training pass, recording each sample's prediction trajectory, and combining three complementary scores: how often a sample is forgotten, how persistently it errors, and a measure of its learning quality. A tiered sampling step then mixes high-, medium-, and low-scoring samples to preserve class and SNR diversity. If correct, this gives signal-processing engineers a cheap way to iterate on model architectures and hyperparameters without repeatedly training on massive signal datasets.

Core claim

FoQuS (Forgetting-Quality Score) establishes that the information needed to train an AMR model well can be concentrated in a small subset of examples, identified from training dynamics during a single full-dataset run. The per-sample score combines a forgetting score (counts of correct-to-incorrect flips), a persistent error score (counts of sustained incorrect predictions), and a quality score (mean correctness minus a scaled mean loss). Instead of selecting the global top-k samples, the method splits the dataset into three tiers by score and draws a specified proportion from each tier. On RML2016.10a, Sig2019-12, and RML2018.01a, the resulting coresets outperform ten image-oriented coreset

What carries the argument

The central object is the FoQuS score, a scalar per-sample importance measure computed as S_FoQuS = S_forget/(T-1) + S_persist_err/(T-1) + S_quality. S_forget counts the epochs where a sample's prediction flips from correct to incorrect; S_persist_err counts consecutive incorrect-to-incorrect transitions; S_quality combines accumulated correctness and cross-entropy loss. The score is computed during a single full training run, then samples are partitioned into three tiers and drawn proportionally to form the coreset. This machinery turns a training trajectory into a data-selection rule that balances decision-boundary samples, hard examples, and easy informative examples.

Load-bearing premise

The load-bearing premise is that the correctness and loss trajectories recorded during one full training run by one architecture are stable enough to transfer to other architectures and fresh initializations; if those trajectories mostly reflect idiosyncratic inductive biases, a coreset selected this way will only help the model that recorded them.

What would settle it

Compare two same-size coresets on the same target model: one selected by FoQuS using the target architecture's own trajectories, and one selected using a different architecture's trajectories. If the cross-architecture coreset is substantially worse, the transferability premise fails. Also, train a FoQuS-selected coreset at the 30% rate to convergence and compare with full-dataset training; if the accuracy gap remains large, the central claim that a small subset approximates full training is undercut.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Model developers can validate new architectures or hyperparameters using coresets of 1% to 30% of the data, cutting repeated training cost while keeping accuracy close to full training.
  • Coresets selected with one architecture transfer to other architectures, including LSTM, SigNet, and IQFormer, so a single selection can support multiple downstream models.
  • Because the selection requires one full training run plus cheap trajectory bookkeeping, it fits into standard SGD training loops without additional labels or model changes.
  • The tiered sampling deliberately retains low-SNR and hard examples, addressing the failure mode of image-based coresets that oversample easy high-SNR samples.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If trajectory transferability holds beyond the tested architectures, FoQuS could serve as a reusable data-curation step: compute scores once on a strong model, then use the same small coreset for all future model development on that dataset.
  • The same score combination could extend to other structured-signal classification tasks with a nuisance parameter analogous to SNR, such as speaker, sonar, or radar-emitter identification, where class balance alone is insufficient.
  • The largest reported gains occur at the extreme 1% sampling rate, suggesting a natural testable extension: measuring how coreset usefulness changes with dataset size, model capacity, and label noise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes FoQuS, a coreset-selection framework for automatic modulation recognition (AMR). FoQuS first trains a model on the full dataset and records per-sample prediction correctness and loss trajectories. It computes three scores (forgetting, persistent error, and quality), combines them into a single FoQuS score, partitions samples into three tiers, and samples a predetermined proportion from each tier to form the coreset. The authors evaluate FoQuS on RML2016.10a, Sig2019-12, and RML2018.01a under sampling rates of 1%–30%, using CNN1D/CNN2D for in-architecture tests and several cross-architecture settings, reporting that FoQuS outperforms 10 existing coreset-selection baselines in most settings. The paper also includes ablations showing that the full three-metric combination outperforms single or pairwise combinations.

Significance. If the claims are validated, FoQuS would be a domain-specific coreset-selection method that could reduce the cost of repeated model development for AMR while preserving accuracy. The paper has concrete strengths: it benchmarks three AMR datasets against ten baselines, includes cross-architecture experiments, and presents a method that is simple enough to be reproduced. However, the central claim that FoQuS 'approximates the effect of full training' is not directly tested, and the SNR motivation stated in the introduction is not examined because all experiments use only high-SNR data. The absence of a full-training baseline and the lack of variance reporting prevent the current results from supporting the paper's central contribution.

major comments (5)
  1. [§IV.B, Tables I–IV; Eq. (1)] The stated objective in Eq. (1) is to make L_S(θ_S) close to L_D(θ_D), and the abstract claims FoQuS 'approximates the effect of full training.' However, no result for a model trained on the full dataset is reported anywhere. Tables I–IV only compare coreset-selection methods against each other. For example, Table I (Sig2019-12, CNN2D, 1%) reports FoQuS accuracy 0.1900, but without the full-training value one cannot tell whether this is a good approximation or a major degradation. Add full-training accuracy (and ideally loss) for each architecture/dataset/budget and report the gap. This baseline is needed to support the central contribution.
  2. [§IV.A; §§I–II] The introduction and related work motivate FoQuS by the SNR structure of wireless signals, claiming that image-based methods may oversample high-SNR samples and degrade in noisy environments. Yet §IV.A states 'we uniformly use high SNR data for experiments,' and no experiment varies SNR or reports the SNR composition of the coresets. Consequently the domain-specific claim that FoQuS is better suited to AMR than image coreset methods because of SNR is untested. Add experiments at low/mixed SNR and, if possible, compare SNR distributions of the selected subsets.
  3. [§III.B, Eqs. (7)–(8); Algorithm 1] The text says the three metrics are 'normalized' before addition, but Eq. (8) only divides the first two scores by T−1 and leaves S_quality unnormalized. S_quality itself is a weighted combination of correctness count and cumulative loss with an unexplained β=0.1. Also, Algorithm 1 computes the final FoQuS score inside the epoch loop (line 8), although S_forget and S_persist_err are defined as sums over t=2..T. Please specify the exact normalization, define the tier proportions and β, and correct the algorithm so the score is computed after all epochs. This is needed for reproducibility.
  4. [§IV.B, Table IV] Cross-architecture generalization to SigNet and IQFormer is reported only for FoQuS; no Uniform or other baseline is shown for these architectures. Since the claim is that FoQuS has 'good cross-architecture generalization,' the comparison should include at least Uniform and one or two competitive baselines under the same settings; otherwise it is possible that all methods generalize similarly to these architectures.
  5. [§IV.B, Tables I–IV] All results are means of 3 runs with no standard deviations, confidence intervals, or significance tests. Many of the reported advantages over the best baseline are small (e.g., Table II, CNN1D RML2018.01a: FoQuS average 0.6861 vs k-Center Greedy 0.6722; at 5% 0.6321 vs 0.6250). Without variance information the 'outperforms' claim is not statistically supported. Report variance or per-seed values, and consider a paired test across methods.
minor comments (5)
  1. [Throughout] There are several typos and inconsistent terms: 'core set' vs 'coreset,' 'forgetfulness score' vs 'forgetting score,' and 'certained SNR samples' in Section II. A careful proofread is needed.
  2. [Eq. (1)] In the definitions of L_D(θ_D) and L_S(θ_S), the loss ℓ(θ;x,y) should be ℓ(θ_D;x,y) and ℓ(θ_S;x,y), respectively, to be consistent with the notation.
  3. [§IV.A] The experimental setup does not specify the exact high-SNR cutoff, nor the optimizer, learning rate, batch size, or the number of epochs T used for the selection run. These details are needed for reproducibility.
  4. [§III.B, Eq. (7)] The choice of β=0.1 is stated as a default but no sensitivity analysis or justification is given. Since β controls the relative weight of cumulative loss in the quality score, a brief sensitivity check would strengthen the paper.
  5. [Table III] The LSTM cross-architecture results are noticeably lower than the CNN results in some settings (e.g., CNN1D→LSTM at 1% on RML2016.10a is below 0.19 for all methods). This is not discussed; a sentence explaining why would help the reader interpret the cross-architecture claims.

Circularity Check

0 steps flagged

No significant circularity: FoQuS is an independent heuristic whose score is not defined in terms of the target it claims to approximate.

full rationale

The coreset score in Section III-B is computed from recorded per-sample training trajectories (Eqs. 2-7) and combined in Eq. (8); it is not defined in terms of the full-training loss difference of Eq. (1), nor is any fitted parameter renamed as a prediction. The method's transferability is tested directly in Table III (selection with model A, evaluation with model B), which is the right empirical check for the concern that trajectories are architecture-specific. The self-citations that are present ([15], [5]) are used only as related work and as an evaluation model, not as load-bearing justification, so they do not create circularity. The absence of a full-training baseline in Tables I-IV is a real evidence gap for the abstract's claim that FoQuS 'approximates the effect of full training,' but a missing comparison is not a circular reduction: nothing in the paper's equations makes the reported accuracy equal to the selection score by construction. The undisclosed tier proportions and beta=0.1 are heuristic choices rather than fitted parameters that are then presented as predictions. Overall, the central derivation is self-contained as a heuristic and is benchmarked against external selection methods, so there is no specific circular step to quote.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests on a few hand-set parameters (beta, tier proportions, SNR cutoff, selection epochs) and several unproven domain assumptions about the transferability of training dynamics and the sufficiency of high-SNR evaluation. No new physical entities are introduced.

free parameters (4)
  • beta scaling factor = 0.1
    In Eq. 7, beta weights the accumulated loss term in the quality score. It is hand-set without justification and directly changes the FoQuS ordering.
  • tier proportions
    Algorithm 1 step 12 says 'draw a proportion of samples from each tier' but the proportions are never specified. This controls the composition of the coreset and is a major unstated degree of freedom.
  • high-SNR cutoff
    Section IV.A says 'we uniformly use high SNR data for experiments' but never defines the threshold. This filter determines which samples are eligible for selection and evaluation, directly affecting every reported number.
  • selection-model training epochs T
    The score in Eqs. 2-7 depends on the number of epochs T used to train the selection model, but T is never stated. Different T would give different trajectories and thus different coresets.
axioms (4)
  • domain assumption A model's per-sample prediction and loss trajectory on the full training set is a meaningful measure of importance for training a fresh model from scratch.
    The entire method in Section III.B relies on this premise. The paper tests it indirectly via cross-architecture experiments, but it is not derived or proven.
  • domain assumption High-SNR samples alone are sufficient to evaluate the method and to demonstrate its advantage in noisy environments.
    Section IV.A restricts all experiments to 'high SNR' data, while the Introduction claims the method addresses low-SNR sample loss. This assumption contradicts the stated motivation and is untested.
  • ad hoc to paper Adding the three scores with equal weight (after dividing by T-1 for two of them and scaling the third by beta) is a valid combination.
    Eq. 8 sums S_forget/(T-1), S_persist_err/(T-1), and S_quality with no derivation or scale matching, despite the text claiming normalization. The choice of equal weights and beta=0.1 is arbitrary.
  • domain assumption Class-balanced coresets with samples drawn from three score tiers preserve the data distribution needed for AMR training.
    Section IV.B states all coresets are class-balanced, but no evidence is given that class balance plus tiered sampling covers the SNR or other distributional properties of the original data.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of \emph{FoQuS}: A Forgetting-Quality Coreset Selection Framework for Automatic Modulation Recognition." pith.science (2026). https://pith.science/paper/GPNK6NCE

@misc{pith2026250908300,
  author       = {Pith},
  title        = {Pith review of: \emphFoQuS: A Forgetting-Quality Coreset Selection Framework for Automatic Modulation Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GPNK6NCE}},
  note         = {Machine review of arXiv:2509.08300}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Deep learning-based Automatic Modulation Recognition (AMR) model has made significant progress with the support of large-scale labeled data. However, when developing new models or performing hyperparameter tuning, the time and energy consumption associated with repeated training using massive amounts of data are often unbearable. To address the above challenges, we propose \emph{FoQuS}, which approximates the effect of full training by selecting a coreset from the original dataset, thereby significantly reducing training overhead. Specifically, \emph{FoQuS} records the prediction trajectory of each sample during full-dataset training and constructs three importance metrics based on training dynamics. Experiments show that \emph{FoQuS} can maintain high recognition accuracy and good cross-architecture generalization on multiple AMR datasets using only 1\%-30\% of the original data.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 14 canonical work pages

  1. [1]

    Data-driven deep learning for automatic modulation recognition in cognitive radios,

    Y . Wang, M. Liu, J. Yang, and G. Gui, “Data-driven deep learning for automatic modulation recognition in cognitive radios,”IEEE Transac- tions on Vehicular Technology, vol. 68, no. 4, pp. 4074–4077, 2019

    2019

  2. [2]

    Adversarial attacks in modulation recognition with convolutional neural networks,

    Y . Lin, H. Zhao, X. Ma, Y . Tu, and M. Wang, “Adversarial attacks in modulation recognition with convolutional neural networks,”IEEE Transactions on Reliability, vol. 70, no. 1, pp. 389–401, 2020

    work page 2020

  3. [3]

    Radio machine learning dataset generation with gnu radio,

    T. J. O’shea and N. West, “Radio machine learning dataset generation with gnu radio,” inProceedings of the GNU Radio Conference, vol. 1, no. 1, 2016

    work page 2016

  4. [4]

    Over-the-air deep learning based radio signal classification,

    T. J. O’Shea, T. Roy, and T. C. Clancy, “Over-the-air deep learning based radio signal classification,”IEEE Journal of Selected Topics in Signal Processing, vol. 12, no. 1, pp. 168–179, 2018. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 5 TABLE III CROSS-ARCHITECTURE GENERALIZATION OFFoQuS. THE BEST RESULTS ARE MARKED INBOLDFACE. FOR EACH EXP...

    work page arXiv 2018

  5. [5]

    Signet: A novel deep learning framework for radio signal classification,

    Z. Chen, H. Cui, J. Xiang, K. Qiu, L. Huang, S. Zheng, S. Chen, Q. Xuan, and X. Yang, “Signet: A novel deep learning framework for radio signal classification,”IEEE TCCN, vol. 8, no. 2, pp. 529–541, 2021

    work page 2021

  6. [6]

    Active learning for convolutional neural networks: A core-set approach,

    O. Sener and S. Savarese, “Active learning for convolutional neural networks: A core-set approach,”arXiv preprint arXiv:1708.00489, 2017

    Pith/arXiv arXiv 2017

  7. [7]

    Active learning by acquiring contrastive examples,

    K. Margatina, G. Vernikos, L. Barrault, and N. Aletras, “Active learning by acquiring contrastive examples,”arXiv preprint arXiv:2109.03764, 2021

    Pith/arXiv arXiv 2021

  8. [8]

    Deep learning on a data diet: Finding important examples early in training,

    M. Paul, S. Ganguli, and G. K. Dziugaite, “Deep learning on a data diet: Finding important examples early in training,”NeurIPS, vol. 34, pp. 20 596–20 607, 2021

    work page 2021

  9. [9]

    An empirical study of example forgetting during deep neural network learning,

    M. Toneva, A. Sordoni, R. T. d. Combes, A. Trischler, Y . Bengio, and G. J. Gordon, “An empirical study of example forgetting during deep neural network learning,”arXiv preprint arXiv:1812.05159, 2018

    Pith/arXiv arXiv 2018

  10. [10]

    Moderate coreset: A universal method of data selection for real-world data-efficient deep learning,

    X. Xia, J. Liu, J. Yu, X. Shen, B. Han, and T. Liu, “Moderate coreset: A universal method of data selection for real-world data-efficient deep learning,” inICLR, 2022

    work page 2022

  11. [11]

    Dataset distillation meets provable subset selection,

    M. Tukan, A. Maalouf, and M. Osadchy, “Dataset distillation meets provable subset selection,”arXiv preprint arXiv:2307.08086, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Online core- set selection for rehearsal-based continual learning,

    J. Yoon, D. Madaan, E. Yang, and S. J. Hwang, “Online core- set selection for rehearsal-based continual learning,”arXiv preprint arXiv:2106.01085, 2021

    Pith/arXiv arXiv 2021

  13. [13]

    Non-uniform class-wise coreset selection: Characterizing category difficulty for data- efficient transfer learning,

    H. Zhang, Z. Xing, W. Yang, C. Ma, W. Tan, and B. Yan, “Non-uniform class-wise coreset selection: Characterizing category difficulty for data- efficient transfer learning,”arXiv preprint arXiv:2504.13234, 2025

    work page arXiv 2025

  14. [14]

    Deepcore: A comprehensive library for coreset selection in deep learning,

    C. Guo, B. Zhao, and Y . Bai, “Deepcore: A comprehensive library for coreset selection in deep learning,” inInternational Conference on Database and Expert Systems Applications. Springer, 2022, pp. 181– 195

    work page 2022

  15. [15]

    Rk-core: An established methodology for exploring the hierarchical structure within datasets,

    Y . Lu, Y . Huang, J. Nie, Z. Chen, and Q. Xuan, “Rk-core: An established methodology for exploring the hierarchical structure within datasets,” in ICASSP. IEEE, 2024, pp. 3150–3154

    work page 2024

  16. [16]

    Coreset selection for object detection,

    H. Lee, S. Kim, J. Lee, J. Yoo, and N. Kwak, “Coreset selection for object detection,” inCVPR, 2024, pp. 7682–7691

    work page 2024

  17. [17]

    Fedcs: Coreset selection for federated learning,

    C. Hao, W. Xie, D. Li, H. Qin, H. Ye, L. Fang, and Y . Li, “Fedcs: Coreset selection for federated learning,” inCVPR, 2025, pp. 15 434–15 443

    work page 2025

  18. [18]

    Contextual diversity for active learning,

    S. Agarwal, H. Arora, S. Anand, and C. Arora, “Contextual diversity for active learning,” inECCV. Springer, 2020, pp. 137–153

    work page 2020

  19. [19]

    Herding dynamical weights to learn,

    M. Welling, “Herding dynamical weights to learn,” inICML, 2009, pp. 1121–1128

    work page 2009

  20. [20]

    Selection via proxy: Efficient data selection for deep learning,

    C. Coleman, C. Yeh, S. Mussmann, B. Mirzasoleiman, P. Bailis, P. Liang, J. Leskovec, and M. Zaharia, “Selection via proxy: Efficient data selection for deep learning,”arXiv preprint arXiv:1906.11829, 2019

    Pith/arXiv arXiv 1906

  21. [21]

    Iqformer: A novel transformer-based model with multi-modality fusion for automatic mod- ulation recognition,

    M. Shao, D. Li, S. Hong, J. Qi, and H. Sun, “Iqformer: A novel transformer-based model with multi-modality fusion for automatic mod- ulation recognition,”IEEE TCCN, 2024

    work page 2024

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.