{"id":"959853a2-6179-40d7-8ef3-91b13a553028","arxiv_id":"2505.05681","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An automatic pipeline turns unlabeled capuchin videos and audio descriptions into clean training pairs, then fine-tunes X-CLIP with LoRA to improve video-text retrieval and zero-shot behavior classification over pre-trained baselines.","lead":"This paper builds a retrieval tool for wild capuchin monkey videos by cleaning noisy field recordings with large language models and fine-tuning a video-text model with a small amount of automatically extracted training data. For behavioral scientists, it offers a way to find relevant clips in thousands of hours of raw footage without manually labeling the whole archive.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Selection of LoRA hyperparameters on the same 177-pair test set used for the headline Hits@5 comparison inflates the reported uplift; no validation split or holdout is described.","rationale":"The central claim is an empirical one: LoRA fine-tuning on automatically cleaned pairs yields large retrieval gains over raw X-CLIP. The strongest evidence is the Hits@5 table, but that table is produced by a selection procedure on the test set itself, which breaks the independence needed for the evidence to support the claim. I identify this as the most load-bearing concern because even a perfect cleaning pipeline cannot rescue an evaluation protocol that selects the reported model on the same data used to measure it. This concern is more fundamental than the reader's weakest assumption about LLaMA/BLIP-2 cleaning reliability: a partial failure of cleaning would weaken the method, whereas test-set selection directly undermines the validity of the measured uplift. The reader's rationale did mention 'test-set selection of hyperparameters' as one of several issues, so there is partial agreement, but the reader's stated weakest assumption was cleaning quality rather than this selection bias. I credit the paper for excluding test videos from training, using 166 unique test videos disjoint from the training videos, and reporting per-behavior NDCG, all of which are good practices. These do not, however, address the selection problem. If an honest validation split were introduced and the gains persisted, the claim would be credible; if not, the reported uplift is likely inflated. No fatal flaw in the method itself is apparent, and the cleaning design is a reasonable engineering contribution, so REJECT is not warranted. The appropriate verdict remains CONDITIONAL: the empirical claims should not be taken at face value until the evaluation protocol is corrected.","tokens_in":17609,"tokens_out":5608,"duration_ms":56929,"concrete_test":"Hold out a validation split, for example 5-fold on the 177 gold pairs or a newly annotated set of about 100 pairs. For each fold, select the LoRA rank and layer configuration by avg(Top-1/2/3) on the validation portion only, evaluate the selected model once on the held-out portion, and average Hits@5 and NDCG@5 over folds. Separately, re-run the BLIP-2 threshold selection over 0.25-0.40 on the validation portion without consulting test retrieval results, and report how the chosen threshold affects Hits@5. If the cross-validated Hits@5 falls below the reported 0.16 (16-frame) or 0.15 (8-frame) by more than about 20% relative, the headline uplift should be revised downward and the central claim restated as conditional on test-set selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline 167%/114% Hits@5 uplift is the output of a model-selection procedure run on the same gold test set from which the uplift is computed. Section 4.2 states that LoRA rank (1, 2, 4, 8) and layer placement (Upper, Bottom, Vertical) were varied and that 'we define the best model as the one achieving the highest average across Top-1, Top-2, and Top-3 accuracies, i.e., avg(Top-1, Top-2, Top-3). The best performing models are shown in Table 2.' No validation split or holdout is mentioned anywhere in the paper. With 3 layer configurations times 4 ranks = 12 candidate configurations per frame count, selecting the configuration that maximizes a Top-1/2/3 average on the test set and then reporting Hits@5, Hits@10, and NDCG@5 from the same test set (Tables 3-4) is a multiple-comparisons selection on the evaluation data. Raw X-CLIP baselines are not selected this way, so the comparison is unfair: the reported uplift is an upper-bound estimate of what the method would achieve under an honest protocol, not an unbiased estimate. The paper's own observation that 'the optimal model varies by Top-K' (Section 4.2) shows the rankings are noisy; picking the best on the test set can inflate Hits@5 materially, especially at n=177 where one additional hit changes Hits@5 by roughly 0.006. Additionally, the BLIP-2 threshold of 0.32 in Section 3.1.5 was 'obtained through visual inspection: we tried several threshold values and checked the retrieval results.' If that retrieval check was performed on the gold set, the cleaning hyperparameter is also test-informed. The paper must specify the data used for both threshold selection and model selection, and provide a protocol in which both are chosen on a validation split before any test evaluation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper tackles video-text retrieval of capuchin monkey behaviors from raw field footage. It constructs training pairs by transcribing audio with Whisper, filtering transcripts with LLaMA 3.2 (quality score, ethogram-based behavior detection, translation, name removal), and retaining clip-text pairs whose BLIP-2 max-frame cosine similarity exceeds 0.32. It then fine-tunes X-CLIP with LoRA, adding a contrastive loss term that uses a text embedding computed without the Prompt Generator so that retrieval embeddings are independently computable. On a manually annotated 177-pair test set, the fine-tuned 16-frame and 8-frame models achieve Hits@5 of 0.16 and 0.15, respectively, versus at most 0.06 and 0.07 for raw X-CLIP baselines, and per-behavior NDCG@5 for several behaviors is substantially higher than for raw models; zero-shot classification also improves. The authors claim this is the first demonstration of fine-tuning a video-text contrastive model on automatically cleaned, unlabeled primate footage.","tokens_in":17913,"tokens_out":7307,"duration_ms":76739,"significance":"If the evaluation were unbiased, this would be a useful applied contribution: it shows a practical route to turning noisy field video and weak audio descriptions into training signal for an ethology-specific retrieval model, and it provides a benchmark with manual labels for future work. The design choice of filtering with BLIP-2 rather than X-CLIP reduces same-model circularity, and the comparison against several raw X-CLIP variants is appropriate. The core limitation is the evaluation protocol: LoRA hyperparameters and possibly the BLIP-2 threshold are selected on or with reference to the test data, so the reported gains are optimistic upper bounds. With a proper validation split, the qualitative conclusion that fine-tuning helps over raw models would likely survive, but the reported magnitudes would need revision.","major_comments":[{"comment":"The LoRA rank and layer placement are selected on the same 177-pair test set used for the headline results in Tables 3 and 4. Section 4.2 defines the best model as the one maximizing avg(Top-1, Top-2, Top-3) on the test set, and no validation split or holdout is described; with 3 layer placements and 4 ranks there are 12 candidate configurations per frame count. The paper's own observation that \"the optimal model varies by Top-K\" indicates that the ranking is noisy, so selecting on the test set and then reporting Hits@5, Hits@10, and NDCG@5 from that same set can inflate the reported uplift materially; at n=177, one additional hit changes Hits@5 by about 0.006. Because the raw X-CLIP baselines are not subjected to the same selection procedure, the comparison is unfair and the headline 167%/114% Hits@5 improvements are upper-bound estimates rather than unbiased estimates. The authors should either introduce a validation split for model selection or report the full set of configurations with appropriate multiple-comparison awareness.","section":"Section 4.2, Table 2, Appendix A"},{"comment":"The BLIP-2 filtering threshold of 0.32 is described as \"obtained through visual inspection: we tried several threshold values and checked the retrieval results.\" The manuscript does not specify which data were used for this retrieval check. If the threshold was tuned on the same 177-pair test set, then the pipeline itself involves selection on the evaluation data, further biasing the reported gains; if a separate development set was used, this should be stated explicitly. This step is load-bearing because the threshold determines the training pairs, so its selection protocol must be transparent.","section":"Section 3.1.5"},{"comment":"The manual test set was constructed by using the pipeline's own LLaMA 3.2 behavior labels to select instances per behavior type. This stratified selection may bias the test distribution toward clips whose transcripts are easy for the pipeline to classify, making the test set non-representative of the raw video population. The authors should either sample the test set independently of the pipeline, or provide an analysis of how the LLaMA-based stratification affects the reported metrics.","section":"Section 4.1"},{"comment":"The central comparisons are reported without any measure of uncertainty. At n=177, the difference between Hits@5 0.16 and 0.06 corresponds to roughly 18 additional hits, which may be real, but the absence of confidence intervals or significance tests makes it difficult to assess how much of the gap is attributable to the fine-tuning versus the test-set selection described above. Please report bootstrap confidence intervals or otherwise characterize the variability of the retrieval metrics.","section":"Tables 3 and 4"}],"minor_comments":[{"comment":"The two contrastive losses are both written as CL(c,t) in the equation and the surrounding text, even though one uses the prompt-generator-bypassing text embedding and the other uses the projector output; please introduce distinct notation (e.g., CL_ret and CL_zs).","section":"Section 3.2, Eq. (6)"},{"comment":"The pipeline is called \"agentic\" and \"an agent,\" but it is a fixed sequence of LLM/VLM calls without reasoning, tool use, or memory; consider using a less loaded term such as \"LLM-based filtering pipeline.\"","section":"Section 3.1"},{"comment":"The phrase \"several versions os raw X-CLIP pretrained models\" contains a typo (\"os\" should be \"of\").","section":"Section 1, contributions list"},{"comment":"The caption does not state whether the displayed clips are the top-1 retrieval outputs or hand-picked examples; please clarify the selection protocol for the qualitative results.","section":"Figure 6"},{"comment":"The zero-shot classification improvements are reported as percentages relative to the best raw model, but the absolute Top-1 accuracies are only 0.12-0.14; please frame these small absolute gains in the discussion and avoid overclaiming.","section":"Section 4.3.2"},{"comment":"The phrase \"as proven by the computed metrics\" overstates what a single test set can establish; suggest \"supported by\" instead.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the application is genuinely useful. The main blocker is test-set-based model selection; please ask the authors to re-run with a validation split before further review. I would also ask for clarification of the BLIP-2 threshold tuning data. The manuscript would benefit from a statement about data availability and release of the pipeline prompts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the central claim—that fine-tuning X-CLIP on automatically cleaned clip-text pairs improves retrieval for capuchin behaviors—is plausible, but the headline uplift numbers are inflated because the LoRA configuration was selected on the same test set used to measure performance.\n\nWhat's actually new: this is the first video-text retrieval setup for capuchin behavior, and the pipeline is a sensible combination of existing parts: Whisper for transcription, LLaMA for filtering/translation, BLIP-2 for hard filtering, then LoRA fine-tuning of X-CLIP. The loss modification that adds the prompt-generator-bypass embedding is a small but genuinely useful trick for retrieval, since it makes clip and text embeddings independently computable. The authors also chose BLIP-2 rather than X-CLIP for filtering, which reduces same-model circularity. The qualitative gap between near-random raw baselines and the fine-tuned model is convincing that fine-tuning helps, and the benchmark itself is a real contribution for primatology.\n\nThe soft spots are real and they land on the headline numbers. Section 4.2 describes selecting the best model by average Top-1/2/3 on the same 177-pair test set used for the Hits@5/10 and NDCG tables. With 12 candidate configs per frame count (4 ranks × 3 layer placements), that's multiple comparisons on the evaluation data. The raw X-CLIP baselines are not selected this way, so the reported 167% and 114% uplifts are upper-bound estimates, not unbiased ones. The paper's own note that the optimal model varies by Top-K is a symptom of that noisy selection. To make matters worse, the BLIP-2 threshold of 0.32 was chosen by visual inspection of retrieval results, and the paper doesn't say whether that inspection used the test set or a separate development set. If the former, it's another test-informed hyperparameter. At n=177, a single hit moves Hits@5 by roughly 0.006, so these issues matter quantitatively.\n\nNone of this kills the paper. The direction of the result is likely right, and the protocol issues are fixable: add a validation split for hyperparameter selection, pre-register or justify the threshold, report confidence intervals, and release code and data. As it stands, the exact uplift numbers should not be taken at face value.\n\nWho this is for: behavioral ecologists and applied vision researchers working with raw field video. It deserves a serious referee, but the evaluation needs a major cleanup before the numbers are citable.","headline":"Plausible and useful proof-of-concept, but the advertised Hits@5 gains are inflated because LoRA was selected on the same test set used to measure performance.","tokens_in":18556,"tokens_out":2983,"would_cite":false,"duration_ms":29570,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a video-text model on automatically cleaned clip-transcript pairs extracted from raw footage produces large retrieval gains on capuchin behavior data, lifting Hits@5 by 167% (16-frame) and 114% (8-frame) over raw pre-trained…","keywords":["video-text retrieval","contrastive learning","LoRA fine-tuning","X-CLIP","capuchin monkey behavior","ethogram","noisy data cleaning","multimodal LLM agents"],"falsifier":"Retrain the same LoRA setup on the unfiltered noisy clip-transcript pairs (or with the BLIP-2 threshold removed) and compare Hits@5 on the same test set; if retrieval gains persist, the cleaning pipeline is not the cause of the improvement. A second check would be to replace LLaMA's behavior labels with random labels during cleaning and see whether NDCG@5 collapses.","tokens_in":17386,"feed_emoji":"🐒","tokens_out":4924,"duration_ms":49561,"temperature":0.7,"pith_summary":"This paper claims that a video-text contrastive model can be adapted to a highly specialized domain—wild capuchin monkey behavior—using only raw, unlabeled videos and the imperfect audio descriptions recorded by field collaborators. The authors build an automatic pipeline that transcribes the audio, uses a large language model to filter and translate the transcripts, and uses an image-text model to discard clips whose visual content does not match the text. The surviving clip-text pairs are used to fine-tune X-CLIP with LoRA. On a manually labeled test set, the fine-tuned model raises Hits@5 from 0.06 to 0.16 (16-frame) and from 0.07 to 0.15 (8-frame), while raw pre-trained models score near random. If correct, this provides a path to retrieval tools for animal behavior research without expensive manual annotation.","feed_headline":"Auto-cleaned clips lift monkey behavior retrieval by 167%","feed_subtitle":"LoRA fine-tuning of X-CLIP on weak audio transcripts turns near-zero Hits@5 into usable ranking.","key_machinery":"The load-bearing mechanism is a two-stage pipeline. First, an agentic data treatment step uses Whisper to transcribe audio, LLaMA 3.2 to score, translate, and behavior-classify transcripts against an ethogram, and BLIP-2 to compute a max-over-frames cosine similarity between clip frames and transcript, keeping only pairs above a 0.32 threshold. Second, X-CLIP is fine-tuned with LoRA on the surviving pairs, with a modified loss that adds a contrastive term on the text embedding computed without the Prompt Generator, so the text and video embeddings can be computed independently at retrieval time.","core_discovery":"The central discovery is that low-rank fine-tuning of a pre-trained video-text model on automatically cleaned, weakly supervised clip-text pairs transfers the model to a new domain where the raw model fails completely. Concretely, the fine-tuned X-CLIP reaches NDCG@5 above 0.4 for most of the tested ethogram behaviors, whereas raw X-CLIP variants are mostly at zero; Hits@5 improves by 167% for the 16-frame model and 114% for the 8-frame model. The paper further shows that the gains appear in retrieval, and to a lesser degree in zero-shot classification, and that qualitative retrievals capture behaviors such as nursing, threatening, eating, and swinging on a vine.","pith_inferences":["The same agentic cleaning recipe—transcribe, LLM-filter with an ethogram, image-text similarity filter—could transfer to other species or field sites that have audio notes, provided an ethogram exists.","The 0.32 BLIP-2 threshold was picked by visual inspection on this dataset; on a new domain it would need recalibration, since it controls the precision/recall trade-off of the training pairs.","A possible blind spot is that cleaning via LLaMA and BLIP-2 may teach the model to recognize transcript style rather than behavior content; a stronger test would measure generalization to novel ethogram descriptions not seen in the pipeline.","If the method holds, it offers a cheaper route to domain-specific retrieval than full annotation, with the caveat that final validation still requires a small manually labeled set."],"forward_implications":["Researchers can build behavior retrieval systems from raw field footage without manual labels: audio notes from collaborators, when cleaned, provide the training signal.","The method scales to the larger 13,060-video, 284-hour collection: more raw footage should yield more clean pairs and better fine-tuning.","Behavior ranking becomes usable: most ethogram behaviors reach NDCG@5 above 0.4, so a researcher can query text and retrieve relevant clips.","Zero-shot classification also improves, so the fine-tuned model can serve as a domain-specific backbone for downstream classifiers.","Rare behaviors such as scratch and sexual remain hard, indicating where more data or different prompts are needed."],"supporting_citations":[{"why":"It supplies the video-text model being fine-tuned.","marker":"[10]"},{"why":"It provides the LoRA method used for parameter-efficient fine-tuning.","marker":"[9]"},{"why":"It transcribes the raw audio into timestamped text to form initial clip-transcript pairs.","marker":"[37]"},{"why":"It computes the clip-text cosine similarity used to filter noisy pairs.","marker":"[22]"},{"why":"It powers the quality scoring, behavior classification, translation, and name removal steps.","marker":"[8]"},{"why":"It justifies the choice of LoRA placement and rank by showing superior few-shot adaptation for vision-language models.","marker":"[19]"},{"why":"It defines the contrastive loss and dual-encoder formulation that X-CLIP extends.","marker":"[7]"},{"why":"It provides the ethogram of capuchin behaviors used in prompts and test labels.","marker":"[39]"}],"fun_headline_variants":["Weak audio to clean pairs: LoRA fine-tune boosts monkey hits by 167%","Monkey video retrieval jumps 167% with LoRA on auto-cleaned clips","167% Hits@5 gain: LoRA fine-tune on auto-cleaned monkey clips","Zero to useful: auto-cleaned clips lift monkey retrieval 167%","Auto-cleaned clips turn raw monkey videos into retrievable behaviors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole training signal rests on the assumption that the automatic cleaning steps—LLaMA's transcript decisions and BLIP-2's 0.32 similarity threshold chosen by visual inspection—separate genuinely aligned clip-text pairs from the many noisy pairs; if they do not, the model learns from mislabeled pairs.","fun_headline_variants_meta":{"raw":{"variants":["Weak audio to clean pairs: LoRA fine-tune boosts monkey hits by 167%","Monkey video retrieval jumps 167% with LoRA on auto-cleaned clips","167% Hits@5 gain: LoRA fine-tune on auto-cleaned monkey clips","Zero to useful: auto-cleaned clips lift monkey retrieval 167%","Auto-cleaned clips turn raw monkey videos into retrievable behaviors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000898,"raw_usage":{"total_tokens":3874,"prompt_tokens":954,"completion_tokens":2920,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":2816}},"tokens_in":570,"tokens_out":2920,"duration_ms":22494,"temperature":1.0,"reasoning_tokens":2816,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:59:48.263046+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same LoRA setup on the unfiltered noisy clip-transcript pairs (or with the BLIP-2 threshold removed) and compare Hits@5 on the same test set; if retrieval gains persist, the cleaning pipeline is not the cause of the improvement. A second check would be to replace LLaMA's behavior labels with random labels during cleaning and see whether NDCG@5 collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the LoRA method used for parameter-efficient fine-tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the video-text model being fine-tuned."},{"cited_title":"Radford, J","cited_arxiv_id":null,"evidence_quote":"It transcribes the raw audio into timestamped text to form initial clip-transcript pairs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It computes the clip-text cosine similarity used to filter noisy pairs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the ethogram of capuchin behaviors used in prompts and test labels."}],"review_version":1}