{"id":"e22d352e-c3ca-4ff1-80e7-a46663365f0d","arxiv_id":"1908.07590","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A feature-based semantic filter improves precision over naive keyword retrieval for deciding when to add sound effects to radio stories, evaluated on new crowdsourced datasets.","lead":"This paper tests a two-stage system that retrieves candidate sound-effect keywords from story text, then uses a classifier with hand-designed linguistic features to decide whether a sound should actually play. The work is an early step toward automatically producing radio dramas, but the evidence is limited by small datasets and the absence of a direct baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Balanced test sampling inflates headline precision: on the real 1,251-negative/336-positive distribution, Table 4 implies precision ≈0.39, not 0.70, and no retrieval baseline is run.","rationale":"The paper is a self-described preliminary study and makes a useful contribution in framing the sound-effect-triggering decision as a classification problem with interpretable features. The crowdsourced annotation effort and the ablation study are legitimate evidence that the hand-designed features carry signal. However, the most load-bearing weakness is not label reliability per se but the evaluation protocol: precision is computed on a 50/50 sample, which is far from the real candidate distribution reported in Table 3. Since precision is the metric the authors say they care most about, reporting it on a balanced sample substantially overstates the operational benefit. The reader's concern about balanced sampling being unrepresentative is correct, but the sharper formulation is that the reported precision value is a direct artifact of prevalence, and the missing baseline makes the robustness claim untestable. This does not invalidate the preliminary finding, but it means the paper should be conditioned on re-analysis on the full distribution and a direct retrieval baseline. Because the reader's verdict is already CONDITIONAL, my stress-test does not change it.","tokens_in":6654,"tokens_out":5698,"duration_ms":130533,"concrete_test":"Rerun the 5-fold evaluation on all 1,587 unanimous-labeled sentences (336 positive + 1,251 negative), reporting precision, recall, and F1 at the model's default threshold; also run an always-accept and a BM25 tag-retrieval baseline on the same gold set. If the recomputed precision is ≈0.39 and the margin over the actual retrieval baseline is small or insignificant, the abstract and Section 4.2 should be revised to say the model improves discrimination on a balanced sample rather than achieving robust retrieval precision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central precision claim rests on a balanced evaluation set: 336 positives are paired with 336 negatives sampled from 1,251 negative sentences (Section 4.1). Precision 0.7022 and recall 0.7718 on that balanced set imply a false-positive rate of about 0.327 on the negative half (FP ≈ 110 of 336 negatives). Applying that false-positive rate to the full 1,251 negatives yields roughly 410 false positives, so operational precision on the actual unanimous-label distribution is about 259/(259+410) ≈ 0.39, not 0.70. The paper's statement that 'our model is verified to obtain more robust results than a simple retrieval model' is also not backed by an actual retrieval baseline on the same data; it is inferred only from beating the 50% chance level of an always-play classifier on an artificial 50/50 sample. Thus the headline 'robust retrieval results' overstates what Table 4 demonstrates: a classification improvement on a balanced sample, not retrieval precision in the intended application.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses automatic sound-effect insertion for radio stories. It proposes a retrieval-based framework that first finds candidate trigger phrases via tag-based matching and then uses a semantic inference classifier (SVM or XGBoost) with handcrafted features (special words, part of speech, syntactic relations) to filter false positives. Two crowdsourced datasets are collected: a first dataset of 1,393 stories is used for statistical analysis and feature design, and a second dataset of 632 new stories yields 2,069 candidate sentences labeled by three crowd workers, with only unanimous labels retained. Experiments on a balanced sample of 336 positive and 336 negative sentences report all-feature precision 0.7022, recall 0.7718, and F1 0.7313; ablations identify special words as the most important feature group and show that a now-words feature hurts performance. Additional heuristic rules raise precision to 0.7544 at a substantial recall cost. The paper claims that the model is more robust than simple tag-based retrieval.","tokens_in":7055,"tokens_out":6398,"duration_ms":499165,"significance":"If the central claim were substantiated, this would be a useful contribution to an understudied cross-modal retrieval task: automatic sound-effect insertion for stories, with two annotated datasets and a transparent feature-based pipeline. The ablation study is internally consistent, the crowdsourcing procedure is clearly described, and the heuristic rules provide a practical precision-recall trade-off. However, the evidence as presented does not establish the headline claim of robust retrieval relative to naive retrieval: no retrieval baseline is run on the same evaluation set, and the reported precision is measured on an artificially balanced sample rather than on the natural distribution of candidate triggers. These are fixable within a revision, so the paper has potential, but the current claims outrun the experiments.","major_comments":[{"comment":"The central claim that the proposed model is more robust than a simple retrieval model is not demonstrated. The only support offered is the sentence in §4.2: \"Since we use half-and-half positive and negative test data, our model is verified to obtain more robust results than a simple retrieval model.\" Beating 50% accuracy on a balanced sample is not equivalent to beating a tag-based retrieval baseline, and no such baseline (BM25 or any other retrieval method) is evaluated on the same 672 sentences or on the 2,069 candidate sentences. The paper should either include a direct baseline (e.g., accepting all candidate triggers, or ranking by BM25 score and thresholding) on the same evaluation data, or the conclusions must be restated as claims about classifier performance on a balanced sample rather than about retrieval precision in the intended application.","section":"§4.1–§4.2, Tables 4–5"},{"comment":"The balanced evaluation design inflates the reported precision relative to the application. The test set contains 336 positives and 336 negatives, but the unanimous-label population in Table 3 has 336 positives and 1,251 negatives. For the all-feature SVM in Table 4, precision 0.7022 and recall 0.7718 imply roughly 110 false positives among the 336 negatives (false-positive rate ≈ 0.327). Scaling that false-positive rate to the full 1,251 negatives gives approximately 410 false positives, yielding an operational precision of about 259/(259+410) ≈ 0.39 rather than 0.70. Because the paper states that precision is the metric of most interest, precision should be reported on the natural distribution or adjusted for prevalence, and the claim that the model \"successfully decreases the false positive rate\" needs to be quantified against a retrieval baseline on that distribution.","section":"§4.1, Table 4"},{"comment":"The final model configuration appears to be selected post hoc on the evaluation folds. The paper removes the now-words feature because Table 4 shows that excluding it improves results, and it adds the quotation/colon and simile/metaphor rules because Table 5 shows a precision gain on the same evaluation setup. No nested cross-validation or separate held-out validation set is used to separate model selection from evaluation. Consequently, the reported precision and F1 values are likely optimistic, and the claimed generalization beyond the training stories is not fully supported. The authors should either perform feature and rule selection on a validation split and evaluate only once on a test split, or explicitly present the selection as exploratory and validate the final choices on an untouched test set.","section":"§4.2 and §5, Tables 4–5"}],"minor_comments":[{"comment":"The statement in §1 that \"over 40% triggers suggested by the retrieved results should not be added with sound effects\" appears inconsistent with the statistics in §2, where 1.64 of 6.25 candidate triggers per story are non-confident, i.e., about 26.2%. If the 40% figure refers only to scene-triggered effects (where 42.1% are non-confident), that should be stated explicitly.","section":"§1 and §2"},{"comment":"The column heading \"Feature Excluded\" is confusing because the first row is \"None,\" which actually represents the full feature set. Consider renaming the rows to indicate the excluded feature explicitly (e.g., \"No exclusion,\" \"Special words,\" \"Action words,\" \"Now words,\" \"POS,\" \"Syntactic\") and clarify in the text that excluding now words improves performance while including them worsens it.","section":"Table 4"},{"comment":"The text says the overall results of SVM and XGBoost are very similar, but only SVM results are reported. Reporting both, or at least providing the XGBoost numbers in an appendix, would make the claim checkable.","section":"§4.2"},{"comment":"No confidence intervals or significance tests are reported for the 5-fold cross-validation differences. Given that some ablation differences are small (e.g., POS vs. None in Table 4), it would be helpful to know whether the observed gaps are stable across folds.","section":"§4.1 and §4.2"},{"comment":"The paper does not specify how the 16 tag queries were used to retrieve candidate sentences from the 632 new stories (e.g., exact tag matching, BM25, threshold settings), nor what retrieval recall was achieved. This information is needed to assess the end-to-end pipeline and to interpret the subsequent classification results.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a short SIGIR paper on a genuinely understudied task. The main problem is not the idea or the dataset construction but the mismatch between the claims and the evaluation: the missing retrieval baseline and the balanced-sampling precision inflation are the two load-bearing issues. Both can be addressed by adding a baseline and reporting prevalence-adjusted or naturally distributed metrics. The post hoc feature selection is also a concern but is fixable by using a validation split. I do not see grounds for rejection; the contribution is potentially publishable after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a genuine but narrow preliminary study that introduces a new task—filtering candidate sound-effect triggers in radio stories—and two new crowdsourced datasets. The feature analysis is honest and the ablation is internally consistent. But the headline precision number is overstated. The evaluation uses a balanced sample (336 positives vs. 336 negatives drawn from 1,251 negatives), so the reported 0.70 precision does not reflect the real operating distribution. Your stress-test math holds: the implied false-positive rate on negatives is about 0.327, which would put precision at roughly 0.39 on the actual 1,251-negative set, not 0.70. Also, the paper claims to beat \"a simple retrieval model\" but never runs one on the same data. That is a load-bearing gap.\n\nWhat is actually new: the task formulation, the two datasets (one for feature inspiration, one for evaluation), and the observation that over 40% of keyword triggers do not correspond to real sound effects. The feature categories (special words, POS, syntactic) are hand-crafted but grounded in the annotated data, and the ablation showing special words matter most is plausible. Given the small scale, the work is a reasonable first step.\n\nSoft spots: the balanced sampling issue is the big one. The 'now words' feature being dropped after it hurt performance is a post hoc adjustment, though the separate test stories partially mitigate circularity. The datasets and feature lexicons are not released, which limits reproducibility. The crowdsourced labels (only unanimous labels) are not checked against expert producers, but for a preliminary study that is a minor issue.\n\nWho this is for: people working on text-to-audio retrieval or automated audiobook/radio production. It deserves a serious referee, but the authors need to run a real retrieval baseline and report precision on the actual distribution, ideally with a precision-recall curve. I would engage with it if asked, but I would not cite it in my own work.\n\nRecommendation: send to peer review; expect major revision before acceptance.","headline":"A legitimate narrow preliminary study whose headline precision claim is inflated by balanced test sampling and a missing retrieval baseline.","tokens_in":7393,"tokens_out":2678,"would_cite":false,"duration_ms":123508,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A semantic inference layer over candidate triggers cuts false positives in retrieval-based sound-effect insertion for radio stories, reaching F1 0.7313 and precision 0.7022.","keywords":["cross-modal retrieval","sound effects","radio stories","semantic inference","feature engineering","crowdsourcing","tag-based retrieval","text-to-speech"],"falsifier":"Have professional radio producers independently label the same 672 sentences and compare their decisions with the unanimous crowd labels; if expert agreement with the crowd is low, or if the SVM's precision measured against expert positives is no better than plain keyword retrieval, the paper's central claim is falsified.","tokens_in":6458,"feed_emoji":"🎙️","tokens_out":6435,"duration_ms":56804,"temperature":0.7,"pith_summary":"The paper targets the labor-intensive step of adding sound effects to radio stories, proposing a retrieval-based pipeline that first retrieves candidate sound effects from text tags and then applies a semantic inference model to decide whether a candidate trigger actually calls for a sound in context. The central finding is that this two-stage design substantially cuts false positives: on a balanced, crowdsourced set of 672 sentences with unanimous three-labeler agreement, the best feature-based classifier reaches precision 0.7022 and F1 0.7313, whereas more than 40% of naive keyword-retrieval triggers are false positives. The authors analyze which context features matter most, showing that special words such as action and subjunctive markers are the strongest signals, while a category of time words actually hurts performance. They also show that heuristic rules can push precision to 0.7544 at the cost of recall, which they argue suits production use where an inappropriate sound is worse than a missed one. If the result holds, it suggests an automatic text-to-radio pipeline is within reach.","feed_headline":"Context filter lifts sound-effect precision past 0.70","feed_subtitle":"Semantic inference on trigger context removes ambiguous keyword matches, reaching F1 0.73.","key_machinery":"The central object is the semantic inference model layered on top of a tag-based retrieval system. A candidate trigger is any phrase the retrieval step matches to a sound-tag database; the inference model must decide whether that trigger is semantically active in its sentence. The mechanism is a feature vector assembled from three families: special-word counts (subjunctive markers like 'plan' or 'like', action words like 'knock' or 'cry', weather words, negative words, and time words), one-hot part-of-speech encodings for the trigger and adjacent words, and one-hot dependency-parse relations indicating whether the trigger is a subject, object, or modifier. These features feed a standard SVM (or XGBoost) classifier, making the decision rule interpretable and cheap to deploy.","core_discovery":"The paper's discovery is that the gap between literal keyword matches and semantically active sound effects can be closed by a shallow, interpretable classifier over sentence context. Given a candidate trigger returned by tag-based retrieval, the model predicts whether the sound is 'happening' in the story using counts of special words (subjunctive, action, weather, negative, time), part-of-speech roles of the trigger and its neighbors, and dependency-parsing relations. Trained on 336 positive and 336 negative sentences labeled unanimously by three crowd workers, an SVM with these features achieves precision 0.7022, recall 0.7718, accuracy 0.7195, and F1 0.7313; ablations show that removing special words lowers precision by about 8 points and F1 by about 5 points, with action words the single most important feature. The paper also reports that including 'now'-type time words hurts all metrics, and that hand-added rules (e.g., no sound after 'as if' or a simile) improve precision to 0.7544 while cutting recall to 0.6337.","pith_inferences":["The same disambiguation step could be dropped into other cross-modal retrieval pipelines, such as background-music selection for video or sound for audiobooks, where literal tag matches are also noisy.","If professional producers rather than crowd workers set ground truth, the model's precision may shift; a producer-labeled test set would be the natural next validation.","The strong performance of simple lexical features suggests that neural models trained on larger data might inherit the same cues, making these features useful priors rather than obsolete heuristics.","The precision-recall trade-off rules could be tuned per story genre or audience, since children's stories and news pieces likely tolerate different rates of missed versus wrong sound effects."],"forward_implications":["If correct, an automatic radio-story production pipeline becomes plausible: text-to-speech narration plus retrieval-based sound effects with a semantic filter, reducing manual dubbing effort.","Tag-based retrieval systems for other audio and video content can adopt the same feature-filtering layer to suppress false triggers without retraining the underlying retriever.","The precision-oriented heuristic rules give producers an explicit trade-off knob: sacrifice recall for precision when an inappropriate sound is more costly than a missed one.","The feature-ablation findings point to special words, especially action words, as the highest-value signals, which can guide feature engineering in similar text-to-sound tasks.","Since the best model still uses only one sentence of context, the paper's own future-work direction suggests that multi-sentence or neural context models are the next step to improve recall without losing precision."],"supporting_citations":[{"why":"BM25 retrieval is the baseline that generates candidate triggers and whose false positives motivate the semantic inference layer.","marker":"[8]"},{"why":"Shows sound effects in radio stories enhance imagery and attention, establishing why the task matters.","marker":"[9]"},{"why":"Semantic annotation and retrieval of music and sound effects is the direct precursor of the tag-based sound database.","marker":"[12]"},{"why":"A comprehensive survey of cross-modal retrieval that frames the text-to-sound application.","marker":"[13]"},{"why":"Provides the sparse auditory representation method used for content-based sound retrieval.","marker":"[5]"},{"why":"Demonstrates large-scale content-based audio retrieval from text queries, supporting the feasibility of tag-based retrieval.","marker":"[2]"}],"fun_headline_variants":["Semantic context filter lifts sound-effect precision to 0.70","Action words key to precise sound-effect autofill in radio","SVM context model hits 0.70 precision on radio sound effects","Heuristic rules trade recall for precision in sound-effect retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the unanimous votes of three crowdsourced labelers reliably capture whether a sound effect should actually play, and that the 336 positive and 336 negative sentences sampled for the balanced evaluation resemble the distribution of real radio stories; if those labels or that sample are unrepresentative, the reported precision and F1 do not transfer to production.","fun_headline_variants_meta":{"raw":{"variants":["Semantic context filter lifts sound-effect precision to 0.70","Action words key to precise sound-effect autofill in radio","SVM context model hits 0.70 precision on radio sound effects","Heuristic rules trade recall for precision in sound-effect retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1345,"prompt_tokens":918,"completion_tokens":427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":354}},"tokens_in":534,"tokens_out":427,"duration_ms":5349,"temperature":1.0,"reasoning_tokens":354,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:01:01.603779+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have professional radio producers independently label the same 672 sentences and compare their decisions with the unanimous crowd labels; if expert agreement with the crowd is low, or if the SVM's precision measured against expert positives is no better than plain keyword retrieval, the paper's central claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BM25 retrieval is the baseline that generates candidate triggers and whose false positives motivate the semantic inference layer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows sound effects in radio stories enhance imagery and attention, establishing why the task matters."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Semantic annotation and retrieval of music and sound effects is the direct precursor of the tag-based sound database."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the sparse auditory representation method used for content-based sound retrieval."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates large-scale content-based audio retrieval from text queries, supporting the feasibility of tag-based retrieval."}],"review_version":1}