{"id":"82e73602-88b2-44d5-8af0-51b65aecf55e","arxiv_id":"2411.14972","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Open-Amp generates synthetic guitar-effects training data from crowd-sourced neural amp captures, and models trained on this data transfer to real effects classification and unseen analog pedal emulation.","lead":"This paper presents Open-Amp, an open-source toolkit that creates large synthetic datasets of guitar amplifier and pedal sounds by running user-uploaded neural emulations. The authors train an audio-effects encoder on this data and show it transfers to real classification tasks, and a one-to-many model that can mimic unseen analog pedals.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on an underspecified evaluation protocol: Table I has no stated train/test splits or variance, and Table II's cross-dataset label spaces appear incompatible; without released code the central claim is not verifiable.","rationale":"I read the paper as proposing a data-augmentation framework whose headline value is that crowd-sourced LSTM captures can replace or supplement real-device recordings for training generalizable audio-effects models. The framework itself is sensible, and the emulation/enrolment experiment provides some supporting evidence: freezing a pretrained TCN and fine-tuning only the embedding on a few seconds of a real analog pedal approaches a from-scratch one-to-one model, which is a meaningful transfer signal even if the captures are imperfect. I do not therefore make fidelity of the Proteus Tone Pack models the main objection; even biased captures could be useful augmentation, and the enrolment experiment directly tests transfer to real analog devices. The softer spot is the classification evidence. Table I is the basis for 'new state-of-the-art results', but the paper does not specify the train/test splits, classifier training details, or seed variance. The original FxNet paper used four subset combinations (Mono/Poly x Discrete/Continuous), and the table reproduces those labels, but it is not stated that the MLP and KNN used the same splits. The overall margin is 1.0 percentage point, and in two of eight conditions the MLP is worse than FxNet; without error bars, the headline is fragile. Table II's cross-dataset numbers cannot be interpreted without a label mapping, and the Open30 test set appears to overlap with training devices. These are concrete, checkable omissions rather than conjectures about data quality. The reader's conditional verdict is appropriate; I would keep it conditional, with the condition being release of code and exact evaluation protocols.","tokens_in":9227,"tokens_out":8085,"duration_ms":79098,"concrete_test":"Re-run Table I via the released Open-Amp code with the exact FxNet train/test splits from [34] and 10 random seeds, reporting mean and standard deviation of overall accuracy; if the Open-Amp MLP's mean is not at least one standard error above 86.9%, the SOTA claim fails. In the same reproduction, remove the 30 Open30 devices from the Open-Amp encoder's training set and re-evaluate Table II; if the Open30 accuracy drops materially, the claimed transferability to unseen datasets needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is not the fidelity of the crowd-sourced LSTM captures, but the evaluation protocol for the central 'state-of-the-art' claim. In Section III-A, Table I is presented as a comparison to the FxNet baseline from [34], yet the paper never states which train/test splits were used for the KNN/MLP classifiers, how the MLP was optimized, or how many seeds were averaged. The margins are small (overall 87.9% vs 86.9%; the KNN is worse in all rows, and the MLP loses on Mono-Cont.-to-Mono-Disc. and Poly-Cont.-to-Poly-Disc.), so even a modest protocol mismatch could reverse the conclusion. Table II is even less specified: GFX (13 distortion classes), EGFX (12 effects spanning distortion/reverb/modulation/delay), EGDB (6 amplifiers) and Open30 (30 devices) have incompatible label spaces, yet the table reports MLP accuracy for every train/test pair without defining a class mapping or whether the classifier is trained on source labels and tested on target labels. The 99.8% Open-Amp-to-Open30 result is also uninformative because Open30 devices were randomly selected from the same Proteus Tone Pack used to train the Open-Amp encoder and may not have been held out. Without code or an exact protocol, the claim that Open-Amp 'achieves new state-of-the-art results on multiple guitar effects classification tasks' is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Open-Amp, a Python framework that generates synthetic audio effects data by rendering audio through crowdsourced neural network emulations of guitar amplifiers and effects pedals, sourced from the GuitarML Proteus Tone Pack. The authors use this framework to train a contrastive guitar-effects encoder and evaluate it on existing datasets (GFX, EGFX, EGDB) plus a synthetic Open30 set, reporting state-of-the-art classification accuracy versus the FxNet baseline. They also train a one-to-many TCN-based effects model with FiLM conditioning on synthetic devices and demonstrate enrollment of unseen analog EGFX pedals by fine-tuning only a learned embedding.","tokens_in":9516,"tokens_out":4585,"duration_ms":45036,"significance":"If the claims are upheld, this is a valuable contribution: it provides a scalable, open-source alternative to physical-device data collection for audio effects research, and the code release is a concrete asset to the community. The use of external datasets (GFX, EGFX, EGDB) for evaluation is a strength, as is the enrollment experiment on real analog pedals, which goes beyond purely synthetic validation. However, the central 'state-of-the-art' claim currently rests on an underspecified and in places internally inconsistent evaluation protocol (Tables I and II), and the reported gains are small and lack variance estimates. The idea is promising, but the evidence as presented is not yet sufficient to support the abstract's claims.","major_comments":[{"comment":"The evaluation protocol for the central classification claim is underspecified. The paper does not state how the train/test splits for the GFX subsets were constructed (e.g., per-clip random split, per-guitar or per-conditioning split, split ratio), how the KNN classifier was configured (k, distance metric, feature normalization), or how the MLP was optimized (optimizer, number of epochs, learning rate, regularization, early stopping, number of seeds averaged). The overall accuracy gain over the FxNet baseline is only 1.0 percentage point (87.9% vs 86.9%), the KNN is worse in all rows, and the MLP loses on two of eight rows. Without confidence intervals or significance tests, the claimed improvement is not established.","section":"Section III-A, Table I"},{"comment":"The cross-dataset evaluation in Table II is not interpretable as described. The datasets have incompatible label spaces: GFX has 13 distortion classes, EGFX has 12 effects across distortion/reverb/modulation/delay, EGDB has 6 amplifiers, and Open30 has 30 devices. The text says an MLP classifier is fit with 'various permutations of training and test set,' but it never specifies how a classifier trained on, for example, 13 GFX classes can be evaluated on 12 EGFX classes or 6 EGDB classes. No class mapping or relabeling procedure is given. If the intended procedure is instead to train an encoder on the source dataset and then train a separate MLP on the target dataset, that needs to be stated explicitly; as written, the numbers in Table II do not support the claim that the Open-Amp encoder 'performs best on all datasets.'","section":"Section III-A, Table II"},{"comment":"The 99.8% Open-Amp-to-Open30 result is not evidence of transferability. Open30 consists of 30 devices 'randomly selected' from the same Proteus Tone Pack used to train the Open-Amp encoder, and the paper does not state that these 30 devices were excluded from the contrastive training set. Without a clear hold-out guarantee, the near-perfect accuracy likely reflects that the encoder was trained on the same device identities. This result should be either removed or re-run with explicitly held-out devices.","section":"Section III-A, Open30"},{"comment":"The framework's validity depends on the fidelity of the crowdsourced LSTM captures as emulations of the physical devices they are named after, yet the paper provides no verification of this fidelity. All models are described as single-layer LSTMs with hidden size 40, but no comparison of any capture against a real device recording is reported, and no reference to existing validation of GuitarML captures is given. The authors should either provide a quantitative fidelity check (e.g., ESR or spectral loss for a capture versus the real device) or explicitly discuss this as a limitation, since the transferability claims rest on the assumption that these synthetic models adequately represent real analog and digital effects.","section":"Section III, first paragraph"}],"minor_comments":[{"comment":"The table would benefit from a note that the Open-Amp row uses the 100,000-iteration encoder, not the 200,000-iteration encoder from Table I; this is mentioned in the text but is easy to miss and could confuse readers.","section":"Section III-A, Table II"},{"comment":"The selection of the five devices in Table III is based on the Emb-64 model's combined loss; this selection procedure should be stated in the caption or text so the reader understands that the displayed spread is not uniformly random across all 394 devices.","section":"Section III-B, Table III"},{"comment":"The enrollment results in Fig. 2 show no error bars or statistical significance tests; since the differences between the baseline and embedding methods are sometimes small (especially at low data fractions), the authors should add variance information or state how many runs were averaged.","section":"Section III-B, Fig. 2"},{"comment":"The term 'Open30' is close to 'Open-Amp' and could be confused with the framework itself; consider renaming it to something like 'Proteus-30' or 'OpenAmp-30' for clarity.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely and the code release is a positive feature, but the evaluation protocol for the main SOTA claim is not yet at a publishable standard. The Table II cross-dataset protocol is especially problematic; if the authors cannot specify a valid label mapping or a clear encoder-transfer procedure, they should re-frame the experiment or remove it. The small margin in Table I and the absence of variance make the headline claim fragile. These issues are fixable with additional detail and re-analysis, so I recommend major revision rather than rejection. I would also encourage the editor to ask for a comparison of the one-to-many model against the prior zero-shot amplifier modeling work [23] if feasible, given the authors claim their model is 'similar.'"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The central idea here is real: use crowd-sourced neural amp captures as a synthetic data source for audio effects. That is new, and it is useful. The Open-Amp package is released, it renders audio online during training, and the diversity of devices (160 captures from the GuitarML Proteus Tone Pack) is far beyond what Pedalboard or DASP offer for distortion. I think the paper deserves a serious referee.\n\nThe strongest evidence is the one-to-many emulation experiment. Pretraining a TCN on synthetic devices and then enrolling real analog pedals from EGFX works: with limited training data, the embedding-initialized models match or beat one-to-one baselines. That is a meaningful transfer result and it is not circular, because the target pedals are real analog hardware.\n\nThe classification experiments are more of a mixed bag. The contrastive encoder does beat FxNet on the GFX task overall (87.9 vs 86.9), with far fewer parameters and without seeing GFX during training. That is a fair data point. But the margin is 1 point, the KNN variant loses across the board, and the paper gives no seeds, error bars, or MLP optimization details. The stress-test note is right: the SOTA claim is not established as written.\n\nTable II is the weakest part. The label spaces of GFX, EGFX, EGDB, and Open30 are incompatible (13, 12, 6, and 30 classes), yet the paper reports cross-dataset MLP accuracy without explaining how the classifier maps between them. That is a protocol gap, not a small omission. Also, Open30 is just a random subset of the same Proteus Tone Pack used to train the encoder, so the 99.8% accuracy there is not evidence of generalization to unseen data. The paper should stop calling it that.\n\nOne other thing: the paper says 65 parameter-conditioned models are each treated as five separate models, which should give 420 synthetic devices, not the stated 394. Minor arithmetic discrepancy, but worth fixing.\n\nThe unverified fidelity of user-uploaded LSTM captures is a real concern, but it is partly mitigated by the EGFX enrollment result. If the synthetic captures were badly biased, it is hard to see why the learned embedding would transfer to real analog pedals as well as it does.\n\nOverall: the framework is a contribution, the experiments point in the right direction, and the missing details are fixable. I would send this to peer review, but with a clear request for code/data, error bars, an explicit account of the cross-dataset label mapping, and a more modest claim than \"new state-of-the-art\" until those are in place.","headline":"A genuinely useful synthetic data framework built on crowd-sourced neural amp captures; the core idea is solid, but the SOTA claim is oversold and the evaluation protocol needs more detail before the numbers can be trusted.","tokens_in":10050,"tokens_out":2647,"would_cite":true,"duration_ms":28541,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-Amp shows that a crowdsourced collection of learned emulations of guitar amps and pedals can be turned into synthetic training data that transfers to real audio-effects tasks, including classification and the emulation of unseen…","keywords":["audio effects","synthetic data","data augmentation","guitar amplifier modelling","contrastive learning","one-to-many modelling","guitar effects classification","neural emulation"],"falsifier":"A direct test would be to take a physical distortion pedal with no capture in the tone library, record clean/processed audio pairs, then compare enrolling that pedal into the Open-Amp foundation model against training a dedicated per-device model on the same pairs; if the enrolment route does not match the per-device model at comparable data sizes, the claimed transferability to unseen analog effects is refuted.","tokens_in":9016,"feed_emoji":"🎸","tokens_out":9549,"duration_ms":85280,"temperature":0.7,"pith_summary":"Open-Amp turns a crowd-sourced collection of neural emulations of guitar amplifiers and distortion pedals into a source of synthetic training audio. The paper's central claim is that machine-learning models trained on this synthetic data transfer to real audio-effects tasks: a guitar-effects encoder trained only on Open-Amp renders outperforms the published baselines on most evaluated classification settings and generalises across datasets better than encoders trained on the domain-specific datasets themselves. The paper also claims that a one-to-many guitar-effects model trained on Open-Amp synthetic devices can enrol previously unseen analog pedals through its learned embedding space, matching dedicated per-device models particularly when training data is scarce. If these claims hold, large-scale training for audio-effects models no longer requires physical access to hundreds of devices, since user-contributed captures can stand in for them.","feed_headline":"Crowdsourced amp captures beat dedicated guitar-effects datasets","feed_subtitle":"An encoder trained on synthetic renders of 160 user-created models beats classifiers trained on real-device datasets.","key_machinery":"The load-bearing object is the Open-Amp rendering pipeline: a Python package that takes arbitrary input audio, runs it through a collection of user-uploaded recurrent neural-network captures of guitar amplifiers and pedals (59 amplifiers and 101 pedals, all single-layer LSTMs with 40 hidden units), and produces effect-processed audio online during training via multiprocess dataloading. Because the effects are software models rather than fixed recordings, the input signal can be chosen freely and augmented on the fly. The classification experiments use a contrastive learning objective in which two different clips processed by the same effect form a positive pair and clips from other effects form negatives. The emulation experiments use a temporal convolutional network with feature-wise linear modulation conditioned on a learned lookup table of 394 device embeddings, so enrolling a new device means freezing the network and fine-tuning only a new embedding vector.","core_discovery":"The paper's discovery is that a diverse collection of learned emulations of real guitar gear can be used as a synthetic data source that transfers to real-device tasks. Concretely, the authors claim that a contrastively trained encoder, fed audio rendered through 160 user-created amplifier and pedal models, achieves new state-of-the-art accuracy on multiple guitar-effects classification benchmarks and beats encoders trained on the domain-specific datasets themselves. They further claim that a one-to-many guitar-effects model trained on 394 synthetic device variants can embed three unseen analog distortion pedals and reproduce their sound comparably to a dedicated one-to-one model trained on recordings of that same pedal, with the advantage growing when training data is scarce.","pith_inferences":["If the user-submitted captures are faithful, the key bottleneck for audio-effects foundation models shifts from collecting physical hardware to auditing capture quality and coverage; model zoos could be evaluated as data assets rather than rebuilt device by device.","The same embedding-enrolment recipe is plausibly testable on time-based effects such as delay, reverb, or modulation, where the contrastive encoder's effect identity would need to disentangle the dry input from time-varying processing.","A stronger external validation than the reported benchmarks would be to enrol directly recorded analog hardware that has no corresponding capture in the tone library and measure whether the trained embeddings still separate it correctly.","Because input audio is arbitrary, Open-Amp could be used to stress-test effect models with out-of-distribution signals, revealing whether learned effect embeddings are device-specific or partly input-dependent."],"forward_implications":["The state-of-the-art classification results are achieved without the encoder ever seeing the benchmark's original recordings, so transfer from synthetic renders is the source of the performance.","The Open-Amp encoder generalises across datasets better than encoders trained on the target dataset itself, so the framework can serve as pretraining data.","The foundation-enrolment results show that an unseen analog pedal can be added to a model with only a fraction of the paired data a per-device model needs, which matters when hardware access or recording time is limited.","Because rendering happens online, the input signal can be any audio, so downstream tasks such as transcription or automatic mixing can draw effect-processed training material without committing to a fixed dataset."],"supporting_citations":[{"why":"Supplies the GFX benchmark and the FxNet baseline that the Open-Amp encoder is compared against for classification.","marker":"[34]"},{"why":"Supplies the EGFX dataset of real analog distortion, modulation, delay, and reverb effects used for enrolling unseen devices.","marker":"[35]"},{"why":"Provides the SimCLR contrastive objective used to train the Open-Amp guitar-effects encoder.","marker":"[39]"},{"why":"Provides the unprocessed electric-guitar audio that Open-Amp renders through the effect models during training.","marker":"[38]"},{"why":"Defines the one-to-many amplifier-modelling task and the baseline that the Open-Amp foundation model is compared with.","marker":"[23]"},{"why":"Underpins the recurrent-neural-network architecture used for the user-created device captures.","marker":"[8]"},{"why":"Supplies the EGDB dataset used in the cross-dataset encoder evaluation.","marker":"[24]"},{"why":"Supplies the multi-resolution spectral loss that, with the error-to-signal ratio, trains the one-to-many foundation model.","marker":"[43]"}],"fun_headline_variants":["Open-Amp: Synthetic amp data beats real-device datasets","Crowdsourced amp emulations power SOTA guitar effect models","On-demand synthetic audio effect data from user-driven models","160 user amp models outtrain dedicated guitar effect datasets","User-created amp models transfer to unseen analog pedals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the user-uploaded emulations in the tone-library collection actually reproduce the sound of the real amplifiers and pedals they claim to represent, and the paper reports no fidelity check; if many of them are inaccurate, the synthetic data will not reflect real hardware and the reported transfer may not generalise beyond this particular collection.","fun_headline_variants_meta":{"raw":{"variants":["Open-Amp: Synthetic amp data beats real-device datasets","Crowdsourced amp emulations power SOTA guitar effect models","On-demand synthetic audio effect data from user-driven models","160 user amp models outtrain dedicated guitar effect datasets","User-created amp models transfer to unseen analog pedals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000623,"raw_usage":{"total_tokens":2857,"prompt_tokens":889,"completion_tokens":1968,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":1886}},"tokens_in":505,"tokens_out":1968,"duration_ms":13561,"temperature":1.0,"reasoning_tokens":1886,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:39:22.327349+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to take a physical distortion pedal with no capture in the tone library, record clean/processed audio pairs, then compare enrolling that pedal into the Open-Amp foundation model against training a dedicated per-device model on the same pairs; if the enrolment route does not match the per-device model at comparable data sizes, the claimed transferability to unseen analog effects is refuted.","supporting_citations":[{"cited_title":"Guitar effects recognition and parameter estimation with convolutional neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the GFX benchmark and the FxNet baseline that the Open-Amp encoder is compared against for classification."},{"cited_title":"EGFxSet: Electric guitar tones processed through real effects of distortion, modulation, delay and reverb,","cited_arxiv_id":null,"evidence_quote":"Supplies the EGFX dataset of real analog distortion, modulation, delay, and reverb effects used for enrolling unseen devices."},{"cited_title":"A simple framework for contrastive learning of visual representations,","cited_arxiv_id":null,"evidence_quote":"Provides the SimCLR contrastive objective used to train the Open-Amp guitar-effects encoder."},{"cited_title":"Automatic tablature transcription of electric guitar recordings by estimation of score- and instrument-related parameters,","cited_arxiv_id":null,"evidence_quote":"Provides the unprocessed electric-guitar audio that Open-Amp renders through the effect models during training."},{"cited_title":"Towards zero-shot amplifier modeling: One-to-many amplifier modeling via tone embedding control","cited_arxiv_id":"2407.10646","evidence_quote":"Defines the one-to-many amplifier-modelling task and the baseline that the Open-Amp foundation model is compared with."},{"cited_title":"Real-time guitar amplifier emulation with deep learning,","cited_arxiv_id":null,"evidence_quote":"Underpins the recurrent-neural-network architecture used for the user-created device captures."},{"cited_title":"Towards automatic transcription of polyphonic electric guitar music: A new dataset and a multi-loss transformer model,","cited_arxiv_id":null,"evidence_quote":"Supplies the EGDB dataset used in the cross-dataset encoder evaluation."},{"cited_title":"auraloss: Audio focused loss functions in PyTorch,","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-resolution spectral loss that, with the error-to-signal ratio, trains the one-to-many foundation model."}],"review_version":1}