{"id":"030c3661-6416-495d-8707-ff854cfa2bc8","arxiv_id":"2504.18950","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On relevant BBC Rewind queries, a zero-shot pipeline of PyAnnote diarisation plus ECAPA or TitaNet embeddings reaches about 86 to 87% P@1 and tolerates many distortions.","lead":"Researchers show that pre-trained speaker recognition models can retrieve a known person's voice from a noisy, decades-old BBC archive with roughly 86% precision at rank one on a filtered set of relevant queries. The system, built from off-the-shelf diarization and embedding models, also degrades gracefully under added noise, bit-depth reduction, and mild reverberation, which matters for media archives and forensic search.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation may include each query file in the search index, so the reported P@1 could reflect self-retrieval rather than retrieval of unseen archive files.","rationale":"The reader's weakest assumption concerned the reliability of the manual AVP/AoP/SP categorization and the lack of released annotations. That is a legitimate evaluation-quality concern. However, the more directly load-bearing issue is the query-in-index leakage: the query files are drawn from the same 12,594-file archive that is searched, and the manuscript never states that each query's own file is removed before ranking. If it is not removed, the top-scoring file for a query will frequently be the query file itself, simply because the system is matching a recording against itself. This would inflate P@1 regardless of label quality and would undermine the specific claim that the systems retrieve target speakers effectively across the archive, as opposed to recognizing the query file. I keep the reader's CONDITIONAL verdict because the concern is testable and the paper's internal evidence (P@1 around 86 rather than near 100) even hints that self-retrieval may not be the whole story; but the condition must be sharpened: the authors need to either confirm query-file exclusion in the evaluation protocol or rerun the numbers with each query file removed. If the exclusion is already in the code and the numbers are unchanged, the concern is resolved and the original conditional acceptance logic stands. If not, the central effectiveness result is not yet supported.","tokens_in":21451,"tokens_out":5928,"duration_ms":69483,"concrete_test":"Rerun the Table IV evaluation after explicitly removing each query's own fileID from the ranked candidate list (e.g., set score[query_fileID] = -inf before ranking), and recompute P@1 for ECAPA-TDNN-SB and TitaNet-L-NeMo on the 451-file AVP∪AoP set. If the code or a released evaluation script already performs this exclusion, report that and show the unchanged numbers; if P@1 drops materially or the exclusion is absent from the code, the reported 86.3/86.9 overstates performance on unseen archive files.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central effectiveness claim (P@1 of 86.3 and 86.9 on AVP∪AoP in Table IV) rests on a retrieval evaluation over the BBC Rewind archive, but the paper never states that a query's own video file is excluded from the ranked archive. Section V-A says the 523 query files were 'randomly selected from the Rewind corpus', and Section II-D scores 'each archive file' with no hold-out note. If the query file itself remains in the index, then its speaker embedding is scored against a near-identical embedding derived from the same recording, making it a trivially relevant top candidate. The headline numbers would then measure how often the system can re-identify the query recording, not how well it retrieves previously unseen files containing the target speaker. The fact that P@1 is 86 rather than near 100 suggests the authors may have excluded query files in code, but the omission of any such statement in the manuscript is a load-bearing gap: without it, the paper's central claim that pre-trained models provide 'effective speaker retrieval' on the archive is not established. This concern is independent of the annotation-quality issue raised by the reader; it affects even perfectly clean ground-truth labels.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses speaker retrieval on large, aged media archives, using the BBC Rewind corpus (12,594 Northern Ireland files, 409 hours, 1948–1979) as a testbed. The proposed system performs diarisation, extracts segment- and speaker-level embeddings with pre-trained models (x-vector, ECAPA-TDNN, TitaNet), and ranks archive files by cosine similarity. To handle unreliable synopsis metadata, the authors manually categorize 523 query files into Audio-Visual Presence (AVP), Silent Presence (SP), and Audio-only Presence (AoP), and use the AVP∪AoP subset (Q†, 451 files) for headline evaluation. In the clean setup, ECAPA-TDNN-SB and TitaNet-L-NeMo achieve P@1 of 86.3% and 86.9% on Q† (Table IV); the paper also reports robustness experiments against additive noise, bit-depth reduction, sampling-rate reduction, and reverberation. The central claims are that pre-trained models provide effective and robust speaker retrieval with no training or fine-tuning on the target archive, and that the framework generalizes beyond BBC Rewind.","tokens_in":21621,"tokens_out":5415,"duration_ms":57519,"significance":"If the evaluation protocol is sound, the paper makes a useful practical contribution: it demonstrates that off-the-shelf speaker embeddings and diarisation, combined with a metadata-driven query-selection procedure, can retrieve speakers from a difficult archival corpus without any target-domain training. The work also provides a systematic robustness comparison across several distortion types, which is valuable for practitioners. The paper is clear about the distinction between AVP/SP/AoP labels and proposes a sensible way to reduce label noise. The results are reproducible in principle because the pre-trained model checkpoints and the BBC Rewind corpus are public, though query-file annotations are not released. The main significance depends on whether the retrieval evaluation truly excludes the query files themselves and whether the manual label categorization is reliable; both points require clarification.","major_comments":[{"comment":"The manuscript never states whether each query's own video file is excluded from the archive index during retrieval. The query set is described as 523 files 'randomly selected from the Rewind corpus' (Section V-A), and Eq. (6) in Section II-D scores 'each archive file' with no hold-out note. If a query's own file remains in the index, its speaker embedding is derived from the same recording as the query, making it a near-identical, trivially relevant candidate. The reported P@1 values (e.g., 86.3% for ECAPA-TDNN-SB on Q† in Table IV) would then measure, at least in part, re-identification of the query recording rather than retrieval of previously unseen files containing the target speaker. This is load-bearing for the central effectiveness and robustness claims in Tables III--V and Figs. 9--10. The authors must state explicitly whether query files were removed from the archive before scoring; if they were removed in the implementation, that statement is missing and must be added.","section":"Section V-A / Section II-D"},{"comment":"The construction of Q† depends on a manual categorization of the 523 query videos into AVP, SP, and AoP categories, but no annotation protocol, annotator training, or inter-annotator agreement is reported. The paper says only that the authors 'thoroughly examined both the visual and auditory contents' (Section V-C). Because Q† is the test set for all headline results, the reliability of this categorization is critical: if the identification of silent-presence files is imperfect, the reported P@1 values overstate system quality. The authors should provide a detailed annotation guideline, report inter-annotator statistics on at least a subset, and state how borderline cases were resolved. The cumulative analysis in Fig. 6, which sorts files by system performance before manual labeling, also risks biasing the subsequent exclusion of SP files; this should be acknowledged or the annotation should be performed independently of the system scores.","section":"Section V-C (Table III)"},{"comment":"Several hyperparameters and method choices are selected on the same query set that is later used for evaluation, without a separate validation split. Table II compares uniform, linear, softmax (with τ tuned over 1, 5, 10, 15), and ranking weighting schemes on the 523-file query set and then adopts linear weighting for the remaining experiments; this is a model-selection step performed on the test data. Similarly, Fig. 7 sweeps the fusion weight λ in Eq. (10) on Q† and reports that λ=0.5 gives the best P@K. Such in-sample selection can inflate performance and complicates the interpretation of the reported numbers. The paper should either use a hold-out validation set for these choices or explicitly disclose that the reported results include development-on-test selection. This is not fatal to the core methodology, but it must be disclosed for a rigorous evaluation.","section":"Section V-B and Fig. 7"}],"minor_comments":[{"comment":"The abbreviation AvgRPR is used in Tables VII--VIII and X but is never defined; it should be spelled out (e.g., average relative performance reduction) when first used.","section":"Section VI-B"},{"comment":"The caption says 'TitNet-L-NeMo'; this should be 'TitaNet-L-NeMo'.","section":"Fig. 10 caption"},{"comment":"The temperature values evaluated for softmax weighting are τ = 1, 5, 10, 15; the paper does not explain why this grid is chosen, and a wider range may be needed to support the claim that linear weighting is superior to any softmax setting.","section":"Section II-C / Table II"},{"comment":"The query selection is described as 'randomly selected' but no random seed or selection procedure is given; this limits reproducibility of the exact query set. Releasing the query file IDs would be helpful.","section":"Section V-A"},{"comment":"The count of 5,800 distinct names is produced by spaCy NER, but no error analysis of the NER output is provided; a small manual validation of name extraction accuracy would strengthen the metadata statistics.","section":"Section III-B"}],"recommendation":"major_revision","confidential_remarks":"The most important issue to resolve before acceptance is the missing statement about whether query files are excluded from the retrieval index. If the query files were not excluded, the headline P@1 values and all robustness figures would need to be recomputed; if they were excluded, a single sentence in Section V-A would fix the problem. The annotation-protocol concern is also significant, as the Q† criterion defines the entire evaluation. The paper is otherwise a solid systems paper with useful robustness experiments, but the evaluation protocol needs to be tightened and disclosed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one if you care about speaker retrieval on messy real-world archives. The punchline: the paper shows off-the-shelf diarisation plus speaker embeddings can reach 86–87% P@1 on a 50-year-old broadcast archive without training on it. That is a genuinely useful result for media indexing and forensic search. But there is a load-bearing ambiguity that must be resolved before the numbers mean anything: the manuscript never says whether each query's own video file is removed from the archive before ranking. Section V-A says the 523 query files were randomly selected from the Rewind corpus, and Section II-D scores “each archive file” with no hold-out note. If a query file is still in the index, the system is partly re-identifying the same recording, not retrieving unseen files with the target speaker. The fact that P@1 is 86 rather than near 100 hints the authors may have excluded query files in code, but they don't say so. A referee must ask for an explicit statement and, better, a released query/index split.\n\nWhat the paper does well: it is a careful system study with real coverage. The architecture is assembled from known parts (PyAnnote diarisation, x-vector/ECAPA/TitaNet embeddings, cosine ranking), but the comparison across five embedding variants on a 409-hour archive is not something I've seen before. The duration-based weighting in Eq. (3) is a simple contribution, and the robustness suite (noise at several SNRs, bit-depth reduction, sampling-rate mismatch, real and synthetic reverberation) is systematic and exposes model-specific weaknesses, like TitaNet-L collapsing under reverberation. The discussion of synopsis-derived labels and the Silent Presence/Audio-Visual Presence/Audio-only Presence taxonomy is thoughtful and honest.\n\nSoft spots, in proportion: the query-exclusion omission is the biggest and must be fixed. The manual review of 523 videos into those categories is described without a released protocol or inter-annotator agreement, which leaves the 15% SP removal open to reproducibility questions. The softmax temperature and fusion lambda in Eq. (4) and Eq. (10) are tuned on the same test set, and no error bars are given, so the differences between TitaNet-L and ECAPA-TDNN are not clearly beyond noise. Robustness experiments only distort queries, not the archive, so the generalization claims in the conclusion outrun the evidence. These are all addressable in revision.\n\nWho this is for: researchers working on speaker retrieval, media-archive search, and robustness of speaker embeddings. It deserves a serious referee, but the referee should read the evaluation setup closely. My vote: conditional acceptance, with the index-exclusion question as a mandatory revision.\n\nBest,\n[Your name]","headline":"A useful empirical speaker-retrieval study on BBC Rewind, but the central P@1 numbers rest on a missing statement about whether query files stay in the search index; a referee should demand that clarification before trusting the results.","tokens_in":22202,"tokens_out":1903,"would_cite":false,"duration_ms":20735,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-trained speaker models retrieve named speakers from a 1948-1979 BBC archive with 86-87% top-one accuracy.","keywords":["speaker retrieval","speaker diarisation","speaker embeddings","media archives","zero-shot retrieval","metadata-derived labels","noise robustness","reverberation"],"falsifier":"Have two independent annotators, blind to the retrieval results, label every one of the 523 query files as audio-visual presence, audio-only presence, or silent presence; if their labels disagree with the paper's assignments on more than a few percent of files, the reported 86-87% top-one accuracy on the refined query set is not reproducible.","tokens_in":21203,"feed_emoji":"🎙️","tokens_out":9192,"duration_ms":77360,"temperature":0.7,"pith_summary":"The paper argues that speaker retrieval in large, aged, weakly annotated audio archives can be done without any training or fine-tuning on the archive itself, by chaining pre-trained speaker diarisation with pre-trained speaker embeddings. It takes on two 'in the wild' problems: journalist-written synopses name people who may be visible but silent, and the recordings span quiet studios to street noise, overlapping speech, and 31 years of changing technology. After filtering the query set down to the 451 files where the named person is actually audible, the ECAPA-TDNN and TitaNet-Large systems rank the correct file first for 86.3% and 86.9% of queries on the BBC Rewind Northern Ireland subset. Systematic distortion tests show the pipeline absorbs additive noise, 8-bit audio, and 8 kHz bandwidth, while 4 kHz bandwidth and strong reverberation degrade it sharply, with model-specific failure points.","feed_headline":"Speaker search on a 1948-79 archive hits 86-87% top-one accuracy","feed_subtitle":"Zero-shot diarisation and embeddings find speakers in noisy aged recordings without training on the target archive","key_machinery":"The engine of the system is a four-stage chain: diarisation splits an archive file into speaker-labelled segments; a pre-trained embedding model converts each segment into a fixed-length vector; duration-proportional weights aggregate those vectors into one speaker embedding per detected speaker, with linear weighting chosen over uniform, softmax, and rank weighting because it gave the best retrieval with no hyperparameter; and retrieval ranks a file by the maximum cosine similarity between the query embedding and any speaker embedding in that file. A second mechanism, the evaluation taxonomy that separates synopsis-derived names into audio-visual presence, audio-only presence, and silent presence, is what lets the authors measure the system on labels that actually correspond to an audible voice.","core_discovery":"On the refined query set of 451 files (19.3 hours) whose synopsis names correspond to an audible voice, the best pre-trained systems retrieve the correct file at rank one for 86.3% (ECAPA-TDNN) and 86.9% (TitaNet-Large) of queries, with top-three precision around 84%. The paper concludes that speaker retrieval in the wild is achievable with zero in-domain training: pre-trained diarisation segments each file, a pre-trained embedding model represents each segment, duration-weighted averaging forms one embedding per detected speaker, and a file is scored by the maximum cosine similarity between the query and that file's speaker embeddings. It further shows that segment-level retrieval matches speaker-level accuracy, that interpolating the scores of the two best embeddings gives a small consistent gain, and that the silent-presence label category explains most of the residual failures on the unfiltered query set.","pith_inferences":["A practical system should use query-side audio verification before trusting the ranked list, because 15% of raw synopsis-derived queries in this archive are silent-presence and produce near-zero retrieval accuracy; the paper's own cumulative analysis implies this filter is what makes the headline numbers hold.","The near-perfect precision on the top 60% of queries suggests residual errors are dominated by label and prominence effects rather than acoustics, so a query set engineered to be uniformly difficult would be a sharper test of embedding quality.","The TitaNet-Large reverberation collapse, despite its reverberation-augmented training, invites a testable fix: prepend dereverberation or adapt the embeddings with reverberant augmentation and measure whether its 3.8% top-one accuracy at two seconds of reverberation recovers toward the ECAPA system's 33.3%.","The metadata recipe of named-entity extraction plus manual presence auditing should transfer to other broadcast archives, but each corpus would need its own audit because the mix of audio-visual, audio-only, and silent-presence entries depends on how journalists wrote the synopses."],"forward_implications":["Deploying the pipeline on a new archive requires no manual annotation or fine-tuning; only archive audio and weak name metadata are needed to produce ranked speaker searches.","On a query set whose labels genuinely correspond to an audible voice, top-one precision of 86-87% is the realistic operating point, while on unfiltered metadata the number falls to about 75% because silent-presence queries act as noise.","Segment-level scoring is nearly as accurate as speaker-level scoring, so the same framework can return the timestamps where the target speaker appears with little accuracy cost.","The distortion tests map the operating envelope: babble and white noise are the hardest noise types, music the easiest; 8 kHz and 8-bit audio are tolerable, 4 kHz audio is catastrophic; ECAPA-TDNN holds up to roughly one second of reverberation time while TitaNet-Large collapses above 0.25 seconds.","Simply interpolating ECAPA-TDNN and TitaNet-Large scores improves P@1, P@3, P@5, and P@10 consistently, indicating the two models carry complementary information."],"supporting_citations":[{"why":"supplies the pre-trained diarisation pipeline that turns each archive file into speaker-labelled segments, the first stage of the retrieval chain.","marker":"[36]"},{"why":"extends the diarisation model to overlap-aware resegmentation, which matters for the multi-talker and overlapping speech common in this archive.","marker":"[37]"},{"why":"defines the x-vector embedding baseline that the paper benchmarks against ECAPA-TDNN and TitaNet.","marker":"[26]"},{"why":"introduces ECAPA-TDNN, one of the two top-performing embedding models in the retrieval results.","marker":"[29]"},{"why":"introduces TitaNet, whose large variant achieves the highest P@1 in the benchmark.","marker":"[30]"},{"why":"is the BBC Rewind archive, the target corpus with 12,594 Northern Ireland files spanning 1948 to 1979.","marker":"[31]"},{"why":"supplies the in-the-wild speaker-recognition training data that makes zero-shot transfer to noisy archives plausible.","marker":"[33]"}],"fun_headline_variants":["Speaker retrieval in noisy archives: zero-shot hits 87% top-1","Zero-shot speaker ID on 1948-79 BBC archive reaches 87%","No training on archive: 87% speaker retrieval accuracy","Zero-shot speaker search on aged BBC audio: 87% top-1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline accuracy numbers assume the manual classification of which query files actually contain the named speaker's voice is correct, and that classification was done by the authors without a published annotation protocol or a second annotator.","fun_headline_variants_meta":{"raw":{"variants":["Speaker retrieval in noisy archives: zero-shot hits 87% top-1","Zero-shot speaker ID on 1948-79 BBC archive reaches 87%","No training on archive: 87% speaker retrieval accuracy","Zero-shot speaker search on aged BBC audio: 87% top-1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001042,"raw_usage":{"total_tokens":4400,"prompt_tokens":979,"completion_tokens":3421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":3341}},"tokens_in":595,"tokens_out":3421,"duration_ms":24856,"temperature":1.0,"reasoning_tokens":3341,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:05:18.199638+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent annotators, blind to the retrieval results, label every one of the 523 query files as audio-visual presence, audio-only presence, or silent presence; if their labels disagree with the paper's assignments on more than a few percent of files, the reported 86-87% top-one accuracy on the refined query set is not reproducible.","supporting_citations":[{"cited_title":"pyannote.audio: neural building blocks for speaker diarization,","cited_arxiv_id":null,"evidence_quote":"supplies the pre-trained diarisation pipeline that turns each archive file into speaker-labelled segments, the first stage of the retrieval chain."},{"cited_title":"End-to-end speaker segmentation for overlap-aware resegmentation,","cited_arxiv_id":null,"evidence_quote":"extends the diarisation model to overlap-aware resegmentation, which matters for the multi-talker and overlapping speech common in this archive."},{"cited_title":"X- vectors: Robust dnn embeddings for speaker recognition,","cited_arxiv_id":null,"evidence_quote":"defines the x-vector embedding baseline that the paper benchmarks against ECAPA-TDNN and TitaNet."},{"cited_title":"ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,","cited_arxiv_id":null,"evidence_quote":"introduces ECAPA-TDNN, one of the two top-performing embedding models in the retrieval results."},{"cited_title":"TitaNet: Neural model for speaker representation with 1d depth-wise separable convolutions and global context,","cited_arxiv_id":null,"evidence_quote":"introduces TitaNet, whose large variant achieves the highest P@1 in the benchmark."},{"cited_title":"(2024) BBC Rewind","cited_arxiv_id":null,"evidence_quote":"is the BBC Rewind archive, the target corpus with 12,594 Northern Ireland files spanning 1948 to 1979."},{"cited_title":"V oxCeleb: A Large-Scale Speaker Identification Dataset,","cited_arxiv_id":null,"evidence_quote":"supplies the in-the-wild speaker-recognition training data that makes zero-shot transfer to noisy archives plausible."}],"review_version":1}