REVIEW 3 major objections 3 minor 24 references
A 355,315-resume public dataset with ESCO occupations, quarter-level dates, and education levels shows that a reasoning LLM extractor comes within 1.1-2.7 points of the annotator-agreement ceiling.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 06:47 UTC pith:RXEJGHQ5
load-bearing objection A valuable public dataset with an overstated headline: the 1.1 pp gap is against LLM-generated labels; the human-reference gap is 2.7 pp. the 3 major comments →
JobHop v2: A Large-Scale Career Trajectory Dataset from Unstructured Resumes
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that a carefully engineered pipeline—zero-shot prompting with strict JSON schema, high-reasoning local model inference, a retry loop that recovers all JSON parse failures, a five-stage cleaning process, and a two-step ESCO code assignment policy—can convert roughly 400,000 noisy, pseudonymized multilingual resumes into a clean, open dataset of 355,315 career trajectories. The best evaluated extractor achieves a 100% JSON parse rate and, under a revised graduated partial-credit scoring protocol, lands within 1.1-2.7 percentage points of the pairwise label-agreement ceiling on three complementary reference sets. The paper interprets this near-ceiling performance as
What carries the argument
The load-bearing artifact is the JobHop v2 dataset itself, produced by a reasoning-controlled LLM extraction pipeline: a rule-rich zero-shot prompt with a strict JSON schema, greedy decoding at high reasoning effort, and a multi-step retry mechanism that recovers all initial JSON failures. The companion evaluation mechanism is the revised protocol—field-importance-weighted partial-credit scoring with length-ratio-gated string similarity and flexible multilingual date matching—along with the 'label agreement' row, defined as symmetric pairwise agreement among three reference annotation sets, which serves as the interpretive ceiling that extractors are compared against. The two-step ESCO assig
Load-bearing premise
The load-bearing premise is that the 'label agreement' row—computed as pairwise agreement among hand, edited, and agent annotations, two of which were generated by an LLM—is a fair proxy for the human inter-annotator ceiling; if the AI-generated labels share inductive biases with the extractor, the 1.1-2.7 point gap is partly LLM-LLM agreement.
What would settle it
Have two independent human annotators label the same 200 benchmark resumes under the same schema without LLM assistance, then compute pairwise agreement using the same graduated partial-credit protocol; if the human-only agreement is lower than the reported label-agreement values (86.6/88.5/89.1), the extractor's true distance from the human ceiling is wider than claimed, whereas similar or higher values would confirm the ceiling interpretation.
If this is right
- Career-trajectory research gets a public benchmark roughly two orders of magnitude larger than prior open resume datasets, enabling training and evaluation of next-occupation prediction and occupational embedding models.
- The 100% JSON parse rate after retry offers a practical template for high-throughput structured extraction from noisy, multilingual documents running entirely on local infrastructure.
- The 1.1-2.7 point gap to the agreement ceiling suggests that further accuracy gains may come more from improving annotation conventions than from scaling models further.
- The open release of dataset and code allows independent replication and extension to other labour markets, though the paper notes generalization beyond the Flemish three-language corpus remains untested.
- The privacy design—pseudonymization, removal of location fields, and quarter-level date coarsening—demonstrates a route to openly sharing career data derived from sensitive administrative records.
Where Pith is reading between the lines
- Because two of the three reference annotation sets were generated by an LLM, the near-ceiling result may partly reflect agreement between language models rather than a purely human benchmark; a human-only annotation study would be the decisive check, as the paper itself cautions.
- The deliberate 6.9% of work experiences left 'unknown' in ESCO assignment may make downstream occupational analyses less reliable for nonstandard or marginalized roles; researchers should model this missingness explicitly rather than ignoring it.
- The same masked-text, retry-based extraction architecture could transfer to other pseudonymized administrative corpora, such as health or education records, where privacy constraints also require fully local processing—an extension the paper does not claim.
- The pairwise preference for v2 over v1 was judged by an LLM, so a human end-user evaluation of extraction quality would be a natural next test before treating the 68.3% win rate as the final measure of practical improvement.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces JobHop v2, a large-scale career trajectory dataset constructed by LLM-based extraction from ~440,000 pseudonymized, multilingual resumes provided by VDAB. The released dataset contains 355,315 resumes, 1,993,291 work experiences, and 923,981 education entries, annotated with ESCO occupational codes, quarter-level temporal information, and a normalized five-level education taxonomy. The paper also presents the extraction pipeline (reasoning-controlled GPT-OSS-120B with retry, achieving 100% JSON parse rate), a five-stage cleaning and normalization workflow, and an evaluation protocol comparing five extractors against three annotation references. The central quantitative claim is that the best extractor (GPT-OSS-120B, high reasoning) trails the 'inter-annotator agreement ceiling' by only 1.1–2.7 percentage points. A blind LLM-judged pairwise comparison also finds v2 preferred over v1 on 68.3% of resumes.
Significance. JobHop v2, if released as described, would be the largest fully public, open, end-to-end extracted career-trajectory dataset with rich metadata, filling a clear gap in the literature. The paper's strengths include the reproducible extraction pipeline, the released dataset and code, the use of real unstructured resumes, and the multi-faceted evaluation. The three-reference evaluation design is thoughtful, but two of the three references are generated by the same LLM (Claude Sonnet), which undermines the interpretation of the 'ceiling' and the headline gap. The ESCO mapping relies on an unnamed commercial classifier, and the evaluation weights are hand-chosen without sensitivity analysis. These issues are fixable by revision.
major comments (3)
- [§4.3, Table 2, Abstract] The headline claim that GPT-OSS-120B (high) trails the 'inter-annotator agreement ceiling' by only 1.1–2.7 pp is overstated. The 1.1 pp gap is against the Agent reference, which was generated by Claude Sonnet, and the Edited reference is also Claude-revised; only the Hand column is fully human. The 'Label agreement' row is symmetric pairwise agreement among the three sets, so the Agent-column ceiling is inflated by LLM–LLM agreement and is not an upper bound on extractor quality. A model that matches human labels on cases where the LLM sets err could exceed it. The paper concedes this in §4.3 ('scores against them should not be read as a fully independent measure of quality'), yet the abstract and conclusion still headline the 1.1–2.7 range. Restricting to the Hand column gives a 2.7 pp gap (83.9 vs 86.6); this should be reported as the primary figure, or a genuine human inter-annotator
- [§3.4] The ESCO occupation-code assignment uses a 'commercial classifier' that is never named or specified. This is a black-box component in the pipeline. The paper claims to release code for reproducible research, but the ESCO codes cannot be independently regenerated or audited without the classifier's identity and version (or an open alternative). Additionally, the paper states that resume contents were never transmitted to external services; the authors should clarify whether this commercial classifier is hosted locally or accessed via API, since the title/description text passed to it is resume content. Please disclose the classifier and its deployment.
- [§4.1] The evaluation protocol introduces several hand-chosen components: field-importance weights (Title/Company 1.5×, dates 1.3×, etc.), composition weights (0.6·M̄ + 0.2·R_entry + 0.2·P_entry), and string-similarity thresholds (90/75/50, etc.). These choices materially affect the numbers in Table 2 and hence the 'gap to ceiling' claim. No justification or sensitivity analysis is provided. The authors should either derive the weights from an explicit cost model or demonstrate that the relative ranking and the gap to the ceiling are stable across a range of plausible weight settings.
minor comments (3)
- [§3.3 and Figure 3b] Quality filtering removes resumes with >20 work-experience entries, but Figure 3b and its caption state that trajectory lengths range 'from 1 to over 20 jobs.' Please clarify whether the filter applies before merging (so post-merge lengths can exceed 20) or correct the figure/text to reflect the actual distribution.
- [§4.2] The paper refers to 'Hand annotations' as multi-annotator labels but does not report the number of annotators, the annotation procedure, or the actual human inter-annotator agreement. Since the paper uses the term 'inter-annotator agreement ceiling,' reporting the measured human agreement would strengthen the validation and help readers interpret the label-agreement row.
- [§4.4] The head-to-head comparison against JobHop v1 relies on a single LLM judge (Kimi K2.6). The limitations paragraph acknowledges this, but the abstract and conclusion repeat the 68.3% win rate without the caveat. Consider reporting this result as 'as judged by an LLM' or adding a sensitivity check with a second judge model.
Circularity Check
Headline 1.1–2.7 pp gap to the 'inter-annotator agreement ceiling' is partly LLM–LLM agreement because two of the three reference sets defining the ceiling are LLM-generated.
specific steps
-
self definitional
[§4.3, Table 2 and accompanying text; §4.2 reference-set definitions]
"The Label agreement row reports the symmetric pairwise agreement of each reference set with the other two, and acts as the annotator-agreement ceiling for its column: because the human and agent labels themselves disagree at this level, no extractor should be expected to exceed it ... GPT-OSS-120B at high reasoning is the strongest system: it comes closest to the ceiling on all three reference sets, at 83.9 vs. 86.6 (Hand), 86.2 vs. 88.5 (Edited), and 88.0 vs. 89.1 (Agent), a gap of only 1.1–2.7 percentage points"
The 'inter-annotator agreement ceiling' is defined as pairwise agreement among three reference sets, two of which (Edited and Agent) are produced by Claude Sonnet (§4.2). The Agent column, which yields the 1.1 pp lower bound, measures agreement between an LLM-generated reference and an LLM extractor (GPT-OSS-120B), so the headline gap partially reduces to LLM–LLM agreement rather than human-level quality. The paper concedes that 'scores against them should not be read as a fully independent measure of quality,' yet the abstract and conclusion still headline the 1.1–2.7 range. Only the Hand column (2.7 pp gap) is fully human-referenced, making the lower bound an artifact of the LLM-composed reference set.
full rationale
The dataset construction and extraction pipeline are self-contained and evaluated against real human annotations in the Hand column, so the core contribution is not circular. However, the headline quantitative claim—trailing the inter-annotator agreement ceiling by only 1.1–2.7 pp—is partly driven by LLM–LLM agreement: the Edited and Agent reference sets are Claude Sonnet outputs, and the ceiling is defined as pairwise agreement among the three sets. The 1.1 pp lower bound comes from the Agent column, which is not an independent measure of extraction quality, as the paper itself notes. The 2.7 pp Hand-only gap is the honest human-referenced figure. This is a disclosed partial circularity in the evaluation framing, not a definitional derivation, so a score of 4 is appropriate.
Axiom & Free-Parameter Ledger
free parameters (5)
- τ1 (primary ESCO confidence threshold) =
0.45
- τ2 (rescue ESCO confidence threshold) =
0.85
- Evaluation field-importance weights =
Title/Company 1.5; dates 1.3; description 1.2; location/type 1.0; degree 0.9
- Evaluation composition weights =
0.6·similarity + 0.2·entry recall + 0.2·entry precision; count penalty constants 0.3, 0.5
- Cleaning thresholds =
>20 work entries removed; duration >40y removed; merge gap ≤1 quarter; parse-artifact year >2025 rejected
axioms (8)
- domain assumption ESCO taxonomy v1.1.2 is the correct external standard for occupational coding
- domain assumption VDAB pseudonymization and the added protections (location removal, quarter-level date coarsening) are sufficient to prevent re-identification
- domain assumption Resumes from VDAB job-seekers are a faithful representation of career trajectories
- ad hoc to paper The three-annotation-set 'label agreement' is a valid proxy for the inter-annotator agreement ceiling
- domain assumption Kimi K2.6 is a reliable blind judge of extraction accuracy and completeness
- ad hoc to paper The unnamed 'commercial classifier' maps titles/descriptions to ESCO codes correctly
- domain assumption LLM instruction 'extract only what is explicitly present' adequately controls hallucination
- ad hoc to paper Quality filtering rules do not systematically bias the released corpus
read the original abstract
Large-scale, richly annotated career trajectory data underpins workforce planning, job recommendation, and labour market analysis, yet publicly available datasets are either small, closed to independent use, or built from pre-standardized occupational codes with LLM-synthesized rather than authentic free text. We present JobHop~v2, an improved version of the publicly available JobHop dataset, constructed through end-to-end large language model (LLM) extraction from a corpus of ${\sim}440{,}000$ pseudonymized, multilingual resumes provided by VDAB, the Flemish Public Employment Service. The released dataset comprises $355{,}315$ career trajectories annotated with ESCO occupational codes, quarter-level temporal information, and normalized five-level education attainment, broadening both the coverage and the annotation richness of the original release. Relative to v1, JobHop~v2 introduces a redesigned extraction pipeline based on reasoning-controlled LLM inference with a retry mechanism (achieving a 100% JSON parse rate), a richer extraction schema, and a revised evaluation protocol scored against three complementary annotation baselines. Evaluated against these baselines, our best extractor comes closest to the inter-annotator agreement ceiling among all compared models, trailing it by only 1.1-2.7 percentage points. The dataset and code are publicly released to support reproducible career-trajectory research.
Figures
Reference graph
Works this paper leans on
-
[1]
J.-J. Decorte, J. Van Hautte, J. Deleu, C. Develder, T. De- meester, Career path prediction using resume repre- sentation learning and skill-based matching, arXiv preprint arXiv:2310.15636 (2023)
Pith/arXiv arXiv 2023
-
[2]
E. Senger, Y. Campbell, R. van der Goot, B. Plank, Karrierewege: A large scale career path prediction dataset, 2024. URL: https://arxiv.org/abs/2412.14612. arXiv:2412.14612
Pith/arXiv arXiv 2024
-
[3]
T. Du, A. Kanodia, H. Brunborg, K. Vafa, S. Athey, Labor-llm: Language-based occupational representa- tions with large language models, 2024
2024
- [4]
-
[5]
M. Yamashita, T. Tran, D. Lee, Openresume: Advanc- ing career trajectory modeling with anonymized and synthetic resume datasets, in: 2024 IEEE International Conference on Big Data (BigData), 2024, pp. 6697–6706. doi:10.1109/BigData62323.2024.10825519
arXiv 2024
-
[6]
K. Vafa, E. Palikot, T. Du, A. Kanodia, S. Athey, D. Blei, CAREER: A foundation model for labor sequence data, 2024. URL: https://openreview.net/forum?id= 4i1MXH8Sle
2024
-
[7]
R. Zha, Y. Sun, C. Qin, L. Zhang, T. Xu, H. Zhu, E. Chen, Toward unified representation learning for career mobility analysis with trajectory hypergraph, ACM Transactions on Information Systems 42 (2024). URL: https://doi.org/10.1145/3651158. doi:10.1145/ 3651158
doi:10.1145/3651158 2024
-
[8]
L. Li, H. Jing, H. Tong, J. Yang, Q. He, B.-C. Chen, Nemo: Next career move prediction with contex- tual embedding, in: Proceedings of the 26th In- ternational Conference on World Wide Web Com- panion, WWW ’17 Companion, International World Wide Web Conferences Steering Committee, Repub- lic and Canton of Geneva, CHE, 2017, p. 505–513. URL: https://doi.or...
arXiv 2017
-
[9]
European Commission, Directorate-General for Em- ployment, Social Affairs and Inclusion, M. Le Vrang, A. Papantoniou, E. Pauwels, P. Fannes, D. Vandensteen, J. De Smedt, ESCO: Boosting job matching in Europe with semantic interoperability, Computer 47 (2014) 57–64. doi:10.1109/MC.2014.283
-
[10]
Q. Meng, H. Zhu, K. Xiao, L. Zhang, H. Xiong, A hi- erarchical career-path-aware neural network for job mobility prediction, in: Proceedings of the 25th ACM SIGKDD international conference on knowledge dis- covery & data mining, 2019, pp. 14–24
2019
-
[11]
Schellingerhout, V
R. Schellingerhout, V. Medentsiy, M. de Rijke, Ex- plainable career path predictions using neural mod- els, in: Proceedings of the RecSys in HR 2022 Work- shop (co-located with RecSys 2022), CEUR Workshop Proceedings, 2022. URL: https://ceur-ws.org/Vol-3218/ RecSysHR2022-paper_6.pdf
2022
-
[12]
C. Qin, H. Zhu, T. Xu, C. Zhu, C. Ma, E. Chen, H. Xiong, An enhanced neural network approach to person-job fit in talent recruitment, ACM Transactions on In- formation Systems 38 (2020). URL: https://doi.org/10. 1145/3376927. doi:10.1145/3376927
doi:10.1145/3376927 2020
-
[13]
Moore, S
W. Moore, S. Pedlow, P. Krishnamurty, K. Wolter, I. Chicago, National longitudinal survey of youth 1997 (nlsy97), National Opinion Research Center, Chicago, IL 254 (2000) 22
1997
-
[14]
U. C. Bureau, U. B. of Labor Statistics, Current popu- lation survey, U.S. Census Bureau, 2023. URL: https: //www.census.gov/programs-surveys/cps.html
2023
-
[15]
C. H. Ayishathahira, C. Sreejith, C. Raseek, Combi- nation of neural networks and conditional random fields for efficient resume parsing, in: 2018 Interna- tional CET Conference on Control, Communication, and Computing (IC4), IEEE, Thiruvananthapuram, In- dia, 2018, pp. 388–393
2018
-
[16]
N. Li, B. Kang, T. De Bie, Llm4jobs: unsupervised occu- pation extraction and standardization leveraging large language models, 2023. doi:10.48550/arXiv.2309. 09708, arXiv:2309.09708 [cs]
-
[17]
N. Li, B. Kang, T. De Bie, Skillgpt: a restful API service for skill extraction and standardization using a large language model, 2023. doi: 10.48550/arXiv.2304. 11060, arXiv:2304.11060 [cs]
-
[18]
Sainz, I
O. Sainz, I. García-Ferrero, R. Agerri, O. L. de Lacalle, G. Rigau, E. Agirre, GoLLIE: Annotation guidelines improve zero-shot information-extraction, in: In- ternational Conference on Learning Representations (ICLR), 2024. URL: https://openreview.net/forum?id= Y3wpuxd7u9
2024
-
[19]
M. De-Arteaga, A. Romanov, H. Wallach, J. Chayes, C. Borgs, A. Chouldechova, S. Geyik, K. Kenthapadi, A. T. Kalai, Bias in bios: A case study of semantic repre- sentation bias in a high-stakes setting, in: Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, Association for Computing Machinery, New York, NY, USA, 2019, pp...
arXiv 2019
-
[20]
A. Salinas, P. Shah, Y. Huang, R. McCormack, F. Morstatter, The unequal opportunities of large language models: Examining demographic biases in job recommendations by chatgpt and llama, in: Proceedings of the 3rd ACM Conference on Eq- uity and Access in Algorithms, Mechanisms, and Optimization, EAAMO ’23, Association for Com- puting Machinery, New York, N...
arXiv 2023
-
[21]
W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, I. Stoica, Efficient mem- ory management for large language model serving with PagedAttention, in: Proceedings of the 29th Sym- posium on Operating Systems Principles (SOSP ’23), Association for Computing Machinery, New York, NY, USA, 2023. doi:10.1145/3600006.3613165
arXiv 2023
-
[22]
Model release and accompanying technical report
OpenAI, gpt-oss-120b and gpt-oss-20b model card, https://openai.com/index/introducing-gpt-oss/, 2025. Model release and accompanying technical report
2025
-
[23]
A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, et al., The llama 3 herd of models, 2024. arXiv:2407.21783
Pith/arXiv arXiv 2024
-
[24]
Gemma Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, et al., Gemma 2: Improving open language models at a practical size, 2024. arXiv:2408.00118
Pith/arXiv arXiv 2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.