{"id":"d432d431-9358-4bc7-ba85-0a8d31914c4e","arxiv_id":"2506.19058","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"On the TalentCLEF 2025 benchmark, zero-shot prompting gave the best multilingual job-title matching result, while fine-tuned classification gave the best job-skill prediction result.","lead":"This paper reports a team's results in the 2025 TalentCLEF competition on matching job titles and predicting job skills in multiple languages. It compares three standard AI approaches and finds that the best method depends on the task, which matters for builders of automated hiring and labor-market tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Task A 'prompting beats contrastive' rests on a 0.013 MAP gap over 100 queries per language, with no significance test and one language favoring contrastive; the winning run is also post-deadline.","rationale":"The single most load-bearing assumption is that the small test-set differences in Section 4 reflect real method ordering. This assumption underlies both the abstract's claim that prompting is best for Task A and the broader 'task-specific' conclusion; if it fails, the paper's main positive evidence for the central claim disappears. The reader's weakest_assumption (no variance estimates; 100 titles per language; gap could reverse) identifies exactly this concern, so I agree. My stress test adds two concrete aggravating details: Table 3 shows the German language pair actually favors contrastive, and footnote 2 says the winning prompt result was obtained after the submission deadline. A second, more objective inconsistency exists in the abstract's 'largest multilingual language models perform best for both tasks': Table 4 shows the best Task B system, mdeberta-v3-base (276M), is the smallest model tested, and Section 4.2 says so explicitly. This should be corrected, but it is not the central load-bearing issue because the paper's nuanced conclusion already says model choice is task-specific. The conditional verdict remains appropriate; no verdict change is needed.","tokens_in":11089,"tokens_out":8967,"duration_ms":87353,"concrete_test":"Using the released code or predictions, compute per-query average precision for gte-Qwen2-7B-instruct and contrastive m-e5-large on the 100 English, Spanish, and German test titles. Run a paired bootstrap (10,000 resamples, stratifying by language) on the mean MAP difference and a sign test on the per-language direction. Also apply the same procedure to the Task B top two systems (mdeberta-v3-base classification vs gte-Qwen2-7B-instruct prompting). If either 95% confidence interval includes 0 or the sign test is not significant, the paper should state the methods are statistically indistinguishable on that task rather than claiming a winner.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline Task A result (zero-shot gte-Qwen2-7B-instruct at 0.493 MAP vs fine-tuned contrastive m-e5-large at 0.480, Table 2) is not robustly supported. Section 2.1 puts the gold test subset at 100 titles per language, and Section 4 reports no variance estimates, confidence intervals, or significance tests. The per-language breakdown in Table 3 makes the fragility visible: the prompt model leads on English (0.537 vs 0.511) and Spanish (0.495 vs 0.484), but trails on German (0.442 vs 0.446). The headline ordering therefore rests on an average gap of about 0.013 MAP over at most 300-400 queries, which a paired bootstrap over per-query AP scores would very likely show to be within sampling noise. The Task B headline ordering (classification 0.290 vs prompting 0.283, Table 4) is similarly small. Footnote 2 also states that the winning gte prompt results were obtained after the submission deadline, so the 'best' Task A system is not the officially submitted model. If the Task A ordering is noise, the paper's central conclusion that prompt-based matching excels over fine-tuned contrastive matching loses its main supporting evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This working note describes NLPnorth's submissions to TalentCLEF 2025 Task A (multilingual job-title matching) and Task B (job-title-based skill prediction). The authors compare fine-tuned classification (MaChAmp with cross-entropy), fine-tuned contrastive learning (InfoNCE on ESCO-derived title/description pairs), and zero-shot prompting with instruction-tuned embedders, the largest being gte-Qwen2-7B-instruct. On Task A test data they report 0.492 MAP averaged over English, Spanish, and German for zero-shot prompting, marginally ahead of 0.480 for fine-tuned contrastive matching; on Task B they report 0.290 MAP for fine-tuned classification with mdeberta-v3-base versus 0.283 for prompting. The paper concludes that model choice should be task-specific rather than one-size-fits-all, and that prompting is competitive for title matching while discriminative training leads for skill prediction.","tokens_in":11362,"tokens_out":10295,"duration_ms":93860,"significance":"Taken at face value, the paper offers a useful side-by-side comparison of three dominant NLP paradigms on an external shared-task benchmark, with the notable finding that a 7B zero-shot embedder is competitive with fine-tuned specialist models on multilingual title retrieval. The authors release code and preprocessing scripts and construct a unified ESCO-derived training corpus, which supports reproducibility; hyperparameters are selected on development data rather than on test labels. The main contribution is as a system description, and the empirical rankings are directly supported by the reported MAP scores. However, the headline differences are small, are estimated on a 100-query-per-language test subset, and are presented without any uncertainty quantification, so the paper's central conclusions should be treated as provisional pending additional statistical analysis.","major_comments":[{"comment":"The headline Task A result—zero-shot gte-Qwen2-7B-instruct at 0.492/0.493 MAP versus fine-tuned contrastive m-e5-large at 0.480—is estimated on a gold subset of only 100 titles per language (Section 2.1) and is reported without confidence intervals or significance tests. The per-language breakdown in Table 3 shows the prompting model trailing on German (0.442 vs 0.446), so the 0.013 average gap could plausibly be sampling noise. Please add paired bootstrap or per-query AP significance testing over the roughly 300 queries, and revise the abstract and conclusions to match the outcome. Please also reconcile the reporting discrepancy: the abstract gives 0.492 for the en/es/de average, while Table 3 gives 0.493 for the same average, which does not match the arithmetic of the row's per-language values.","section":"Section 4.1, Tables 2 and 3"},{"comment":"The best Task A prompt-based run (gte-Qwen2-7B-instruct) was obtained after the submission deadline; the officially submitted model was m-e5-large-instruct. Because the abstract and conclusion present the post-deadline run as the paper's best result without mentioning this, the paper conflates the official shared-task ranking with exploratory post-deadline experimentation. Please state this caveat in the abstract and clearly label post-deadline rows in Tables 2 and 3.","section":"Footnote 2, Section 4.1"},{"comment":"The abstract claims that 'the largest multilingual language models perform best for both tasks,' but Section 4.2 states that model size has 'a less clear impact' and that 'the smallest language model performs best' for Task B, and Table 4 shows the best test model is mdeberta-v3-base with 276M parameters. Section 1 also states that 'contrastive learning excels on multilingual title matching,' whereas Section 4.1 and the abstract crown prompting. These internal inconsistencies need to be aligned with the actual results before the paper's task-specific conclusion can be evaluated.","section":"Abstract, Section 1, Section 4.2, Table 4"},{"comment":"The text says 'For the test predictions, we select the best performing model from each category.' Please clarify explicitly whether this selection was made on development MAP or on test MAP. If test labels influenced model selection, the reported test scores are circular; if development was used, state this in the text. The same clarification is needed for both tasks.","section":"Section 4.1 and Section 4.2"},{"comment":"The Task B conclusion that classification 'excels' over prompting rests on a test MAP gap of only 0.007 (0.290 vs 0.283). Although the validation gap is larger (0.267 vs 0.222), no significance test accompanies the test comparison, so the stated superiority is not established. Please add a significance or stability analysis or soften the claim to 'competitive.'","section":"Section 4.2, Table 4"},{"comment":"The test-set model selection wording and the interleaved validation/test columns in Table 2 make it hard to determine which models were actually evaluated on the test split and which numbers are validation-only. In particular, the gte-Qwen2-7B-instruct test scores are post-deadline, and it is unclear whether the 'best performing model from each category' was chosen using development or test labels. Please make the selection protocol and the table structure explicit.","section":"Section 4.1, Table 2"}],"minor_comments":[{"comment":"Section 4.1 contains the typo 'bath datasplits' (should be 'both data splits'), and Section 4.2 contains 'a the smallest language model' (should be 'the smallest language model').","section":"Section 4.1, Section 4.2"},{"comment":"The tables interleave validation and test columns without clear group headers, making it difficult to tell which models were evaluated on the test split. Add explicit column groups and use a dash for unmeasured cells.","section":"Tables 2 and 4"},{"comment":"The per-category analysis covers only the mapped subset, since Table 5 shows that 77.4--87.8% of validation titles are unmapped to ESCO major groups. The text's statement that 'most models do not perform well' on unmapped titles is hard to reconcile with the 'Unmapped Titles' row in Table 6, which shows MAP values of 0.426--0.617. Please clarify what that row represents and discuss the coverage limitation.","section":"Section 5, Tables 5 and 6"},{"comment":"The instruction-tuned embedders gte-Qwen2-7B-instruct and Linq-Embed-Mistral are not cited; please add references for both models.","section":"Section 3.3"},{"comment":"For Task B, the statement that 'the optimal negative ratio for taskB was 1:1' does not say whether this ratio was selected on development data; please state the selection protocol.","section":"Section 3.1"},{"comment":"The 'Avg. (en,es,de)' column value of 0.493 for gte-Qwen2-7B-instruct is inconsistent with the abstract's 0.492 and with the arithmetic of the row's per-language values (0.537, 0.496, 0.442 average to 0.492). Correct the rounding or the table entry.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"This is a workmanlike shared-task system description, and the empirical material—external benchmark anchoring, dev-based hyperparameter selection, and code release—is in good shape. The main issues are that the headline conclusions are drawn from very small test gaps on a 100-query-per-language subset without uncertainty quantification, that the best Task A result is post-deadline while being presented as the main result, and that several claims in the abstract and introduction contradict the results tables. These are addressable in revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on Zhang and van der Goot's TalentCLEF 2025 working note. It's exactly what a shared-task report should look like: transparent, three paradigms compared on the same tasks, ESCO-derived training data, and code released. The side-by-side comparison of classification, contrastive, and zero-shot prompting on both job-title matching and job-skill prediction is the genuinely useful piece, and the per-category ESCO analysis gives practitioners a sense of where each method struggles.\n\nThe empirical rankings are believable as measured, and the protocol is careful: hyperparameters like negative ratios, batch size, and learning rate were tuned on dev data, not the test labels. I also want to give credit for the honest admission that the best Task A prompting run (gte-Qwen2-7B) was obtained after the deadline.\n\nNow the soft spots. The headline claim—prompting beats contrastive on Task A—rests on a 0.493 vs 0.480 MAP gap over only 100 gold titles per language. No error bars, no significance test. Per-language, the prompt model actually trails on German. That ordering could easily flip on another sample. Same for the Task B margin (0.290 vs 0.283). The stress-test note is right on this, and the paper doesn't address variance at all, not even with a caveat. That is the biggest weakness.\n\nSecond, the text contradicts itself: the Introduction says contrastive excels on title matching, while the abstract and Section 4 say prompting wins on test. That needs fixing—it's sloppy.\n\nThird, the conclusion that \"the largest multilingual language models perform best for both tasks\" is not supported for Task B, where the smallest classification model (mdeberta-v3-base, 276M) is the winner. This is a minor overgeneralization but still misleading.\n\nNone of these are load-bearing errors: the methods are sound, the measurements are transparent, and the released scripts are a real resource. For a working note, this is solid and usable. But the central \"prompting wins\" claim should be framed as a provisional observation until a significance test or a larger evaluation settles it. The positive contribution—the comparison itself and the preprocessing code—survives.\n\nWho is this for? Practitioners in HR-tech and NLP-for-job-market research who want a concrete model-selection reference. As a referee, I'd ask for confidence intervals or at least a paired bootstrap, a fix of the intro/test inconsistency, and a tempering of the \"largest is best\" claim. All addressable. I'd say yes to peer review: the paper deserves serious referee time, and with minor revisions it would be acceptable for a workshop or short paper.\n\nRecommendation: engage with it, but don't base your own model choices on the Task A ordering until it's pinned down.","headline":"Useful shared-task comparison of three paradigms for job-title and skill matching, but the headline Task A result is a 0.013 MAP gap over 100 titles per language with no significance test—treat the ordering as provisional.","tokens_in":11890,"tokens_out":2641,"would_cite":true,"duration_ms":25839,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot prompting with a 7-billion-parameter embedder wins multilingual job-title matching, while fine-tuned classification wins job-to-skill prediction.","keywords":["computational job market analysis","NLP for Human Resources","job title matching","job-skill matching","classification","contrastive learning","prompting","large language models"],"falsifier":"Compute confidence intervals for the difference in MAP between gte-Qwen2-7B-instruct and m-e5-large contrastive on Task A by bootstrapping over the 100 gold-standard test titles per language (or by scoring the full 5,000-title test set if gold labels become available); if the interval crosses zero, the claim that prompting is the best title-matching method is falsified.","tokens_in":10873,"feed_emoji":"💼","tokens_out":9218,"duration_ms":78955,"temperature":0.7,"pith_summary":"This paper reports a systematic comparison of three modeling paradigms on two tasks from the TalentCLEF 2025 shared task: multilingual job-title matching (Task A) and job-title-based skill prediction (Task B). The authors find that the best method differs by task: zero-shot prompting with a 7-billion-parameter instruction-tuned embedder (gte-Qwen2-7B-instruct) reaches the highest mean average precision of 0.492 on Task A test data averaged over English, Spanish, and German, while a fine-tuned classification model (mdeberta-v3-base) reaches 0.290 on Task B test data. The broader point is that there is no one-size-fits-all model for labor-market matching: prompt-based methods can rival supervised systems for title retrieval without task-specific training, but the direct supervised signal still wins for predicting skills. If correct, this gives practitioners a concrete task-by-task guide to choosing systems and suggests that instruction-tuned LLM embedders are now strong off-the-shelf tools for multilingual job title retrieval.","feed_headline":"Zero-shot prompting wins title matching; fine-tuning wins skills","feed_subtitle":"Best test MAP: 0.492 for zero-shot prompting on titles, 0.290 for fine-tuned classification on skills.","key_machinery":"The argument rests on three competing mechanisms. Classification reformulates ranking as binary relevance prediction: each query is paired with sampled negatives, an encoder is fine-tuned with cross-entropy through the MaChAmp toolkit, and the softmax score ranks candidates. Contrastive learning constructs positive pairs from ESCO (preferred title to description, preferred title to alternative title) and fine-tunes multilingual sentence embedders with the InfoNCE loss so that related titles and skills sit close in cosine space. Prompting skips training entirely: instruction-tuned LLM embedders (m-e5-large-instruct, Linq-Embed-Mistral, gte-Qwen2-7B-instruct) encode the query and candidates with a task prefix and rank by cosine similarity. The paper also builds a unified ESCO-derived corpus of titles, alternative labels, and multilingual descriptions, which supplies the training signal for the contrastive models.","core_discovery":"On the paper's own terms, the central discovery is a task-dependent ordering of methods. For multilingual job-title matching, the zero-shot prompt-based embedder gte-Qwen2-7B-instruct achieves the best test MAP of 0.492 averaged over English, Spanish, and German, outperforming the best fine-tuned contrastive model (m-e5-large, 0.480) and all classification models; it also leads on English–Spanish and English–Chinese cross-lingual transfer. For job-title-based skill prediction, fine-tuned classification with mdeberta-v3-base achieves the best test MAP of 0.290, ahead of contrastive and prompting alternatives. The paper concludes that instruction-tuned prompting is effective for title matching and that classification excels at skill prediction, so model choice should be task-specific rather than universal.","pith_inferences":["The one-point test gap between prompting (0.493) and contrastive (0.480) on Task A is not tested for significance; with only 100 gold titles per language, a bootstrap or paired test could plausibly erase the gap, so the headline ordering should be treated as provisional.","Read the abstract's 'largest models win both tasks' as applying to Task A: in Task B the best classifier, mdeberta-v3-base, is a 276M-parameter model, so model size is not the consistent driver across both tasks.","The strong zero-shot cross-lingual results suggest a cheap extension: prompting with instruction-tuned embedders could be tested on languages absent from ESCO altogether, not just Chinese, to see whether the transfer advantage holds.","Because the contrastive models were the only ones trained on ESCO descriptions, the comparison between contrastive and prompting partly confounds training signal with method; a prompt-based model given the same descriptions might close the gap further."],"forward_implications":["For multilingual job-title matching, zero-shot prompting with a 7B instruction-tuned embedder is competitive with, and here superior to, fine-tuned contrastive systems, so teams without training data or compute for fine-tuning can still deploy a strong title matcher.","For skill prediction, the direct supervised signal from classification matters more: fine-tuned classification should be the default until prompting methods close the gap.","Cross-lingual transfer works best through the largest prompt-based embedder, which leads on English–Spanish and English–Chinese pairs even though Chinese was unseen during training.","Model choice should be task-specific: the winning approach for title matching is not the winning approach for skill prediction, so evaluations should report the two tasks separately.","Skill prediction is harder than title matching across all methods, as evidenced by the much lower MAP scores on Task B."],"supporting_citations":[{"why":"Defines the TalentCLEF 2025 shared tasks, the data splits, and the MAP evaluation that all results are measured against.","marker":"[51]"},{"why":"ESCO taxonomy is the source of the extra titles, alternative labels, and multilingual descriptions used to build contrastive training pairs.","marker":"[26]"},{"why":"Supplies ESCOXLM-R, the multilingual taxonomy-pretrained encoder evaluated as a classification and contrastive backbone.","marker":"[53]"},{"why":"Source of the m-e5 model family used for classification, contrastive fine-tuning, and prompting.","marker":"[54]"},{"why":"Provides mdeberta-v3-base, the fine-tuned classification backbone that achieves the best Task B test MAP.","marker":"[55]"},{"why":"Provides the par-m-mpnet-base-v2 multilingual sentence embedder used as a baseline.","marker":"[56]"},{"why":"Defines the InfoNCE objective used to fine-tune all contrastive models with in-batch negatives.","marker":"[58]"},{"why":"MaChAmp is the toolkit used to fine-tune all classification models with cross-entropy loss.","marker":"[52]"}],"fun_headline_variants":["Zero-shot wins title match; fine-tune wins skills","For titles: prompting wins; for skills: classification wins","Task-specific winner: prompt for titles, classify for skills","Prompting best for title match, classification for skills","No universal winner: prompt beats on titles, fine-tune on skills"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline claim that zero-shot prompting is the best Task A method depends on the assumption that the 100 gold-standard test titles per language are representative and that the reported margin over fine-tuned contrastive learning (0.493 vs 0.480 MAP) is real rather than sampling noise, since no variance estimates or significance tests are reported.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot wins title match; fine-tune wins skills","For titles: prompting wins; for skills: classification wins","Task-specific winner: prompt for titles, classify for skills","Prompting best for title match, classification for skills","No universal winner: prompt beats on titles, fine-tune on skills"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000921,"raw_usage":{"total_tokens":3970,"prompt_tokens":987,"completion_tokens":2983,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":2900}},"tokens_in":603,"tokens_out":2983,"duration_ms":20440,"temperature":1.0,"reasoning_tokens":2900,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:38:09.318524+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute confidence intervals for the difference in MAP between gte-Qwen2-7B-instruct and m-e5-large contrastive on Task A by bootstrapping over the 100 gold-standard test titles per language (or by scoring the full 5,000-title test set if gold labels become available); if the interval crosses zero, the claim that prompting is the best title-matching method is falsified.","supporting_citations":[{"cited_title":"Gasco, H","cited_arxiv_id":null,"evidence_quote":"Defines the TalentCLEF 2025 shared tasks, the data splits, and the MAP evaluation that all results are measured against."},{"cited_title":"le Vrang, A","cited_arxiv_id":null,"evidence_quote":"ESCO taxonomy is the source of the extra titles, alternative labels, and multilingual descriptions used to build contrastive training pairs."},{"cited_title":"Zhang, Y","cited_arxiv_id":null,"evidence_quote":"Source of the m-e5 model family used for classification, contrastive fine-tuning, and prompting."}],"review_version":2}