pith. sign in

arxiv: 1907.06556 · v1 · pith:G4HKVDGRnew · submitted 2019-07-15 · 💻 cs.IR

Should we Embed? A Study on the Online Performance of Utilizing Embeddings for Real-Time Job Recommendations

Markus Reiter-Haas , Emanuel Lacic , Tomislav Duricic , Valentin Slawicek , Elisabeth Lex This is my paper

Pith reviewed 2026-05-24 21:16 UTC · model grok-4.3

classification 💻 cs.IR
keywords job recommendationsembeddingsclick-through rateonline evaluationpersonalizationreal-time recommendationsuser interactions
0
0 comments X

The pith

Embeddings from recent interactions achieve the highest click-through rates for similar job recommendations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper reports an online experiment on the Studo Jobs platform that tests embeddings for real-time job recommendations in two scenarios. For suggesting jobs similar to one a user has viewed, embeddings built from the most recent interaction deliver the best click-through rate. For personalizing the jobs shown on a user's homepage, the strongest results come from combining embeddings that reflect both how often and how recently the user has interacted with job postings.

Core claim

In the online study, embeddings based on the most recent user interaction achieve the best online performance in terms of Click-Through Rate when recommending similar jobs. To personalize the job postings shown on a user's homepage, combining embeddings based on the frequency and recency with which a user interacts with job postings results in the best online performance.

What carries the argument

Embeddings generated from user interaction histories with job postings, using variants that emphasize the most recent interaction, interaction frequency, recency, or a combination of frequency and recency.

If this is right

  • Embeddings from the most recent interaction improve CTR for similar-job recommendations under real-time constraints.
  • Combined frequency-and-recency embeddings improve CTR for homepage personalization under real-time constraints.
  • Different recommendation scenarios on the same platform benefit from different embedding construction strategies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The pattern suggests that temporal aspects of user behavior matter differently depending on whether the task is similarity matching or broad personalization.
  • Platforms could test whether weighting recent interactions more heavily in other domains yields comparable CTR gains.

Load-bearing premise

The online experiment on the Studo Jobs platform sufficiently isolates the effect of the embedding strategies on CTR without significant confounding factors from the platform's user base or other system components.

What would settle it

A replication of the A/B test on a different job platform that finds no significant difference in CTR between the embedding strategies would falsify the reported performance advantages.

Figures

Figures reproduced from arXiv: 1907.06556 by Elisabeth Lex, Emanuel Lacic, Markus Reiter-Haas, Tomislav Duricic, Valentin Slawicek.

Figure 1
Figure 1. Figure 1: Analysis of incorporating job embeddings to recommend similar jobs. The reported results show a daily CTR and the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Online performance of job embeddings when used to [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
read the original abstract

In this work, we present the findings of an online study, where we explore the impact of utilizing embeddings to recommend job postings under real-time constraints. On the Austrian job platform Studo Jobs, we evaluate two popular recommendation scenarios: (i) providing similar jobs and, (ii) personalizing the job postings that are shown on the homepage. Our results show that for recommending similar jobs, we achieve the best online performance in terms of Click-Through Rate when we employ embeddings based on the most recent interaction. To personalize the job postings shown on a user's homepage, however, combining embeddings based on the frequency and recency with which a user interacts with job postings results in the best online performance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 3 minor

Summary. The paper reports results from an online A/B study on the Studo Jobs platform evaluating embedding-based strategies for two real-time job recommendation scenarios: (i) similar-job recommendations and (ii) homepage personalization. It finds that recency-only embeddings achieve the highest CTR for similar jobs, while a combination of frequency- and recency-based embeddings performs best for homepage personalization.

Significance. If the reported CTR ordering holds under the described traffic split and observation windows, the work supplies concrete, production-derived guidance on when to prefer recency versus combined interaction signals when deploying embeddings in job recommenders. The live-platform setting and explicit comparison of aggregation choices constitute a practical contribution to the cs.IR literature on embedding utilization under latency constraints.

minor comments (3)
  1. [§3] §3 (Experimental Setup): the description of how user sessions are assigned to variants and whether cross-device users are deduplicated should be expanded to confirm that the CTR differences can be attributed solely to the embedding strategy.
  2. [Table 2] Table 2: report the number of impressions and the p-values (or confidence intervals) for each pairwise CTR comparison so that readers can judge the statistical reliability of the claimed ordering.
  3. [§4.2] §4.2: the precise definition of the 'combined' embedding (e.g., weighted sum, concatenation followed by projection, or separate scoring) is stated only at a high level; an equation or pseudocode would remove ambiguity.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive summary and significance assessment of our online A/B study on embedding strategies for job recommendations. The recommendation of minor_revision is noted. No major comments were enumerated in the report.

Circularity Check

0 steps flagged

Empirical online study with no derivation chain or self-referential predictions

full rationale

The manuscript presents results from a live A/B experiment on the Studo Jobs platform comparing CTR across embedding strategies for similar-job and homepage personalization scenarios. No equations, fitted parameters, uniqueness theorems, or ansatzes are introduced; the central claims are direct empirical measurements of online performance under described traffic splits and observation windows. Because the work contains no predictive derivation that could reduce to its own inputs, the analysis is self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The claim relies on the assumption that the platform's online experiment accurately reflects the impact of different embedding methods without external biases.

axioms (1)
  • domain assumption CTR measures the effectiveness of job recommendations
    The study uses CTR as the key performance indicator for the online study.

pith-pipeline@v0.9.0 · 5660 in / 1238 out tokens · 33371 ms · 2026-05-24T21:16:33.614352+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages · 2 internal anchors

  1. [1]

    F. Abel, A. Benczúr, D. Kohlsdorf, M. Larson, and R. Pálovics. Recsys challenge 2016: Job recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems, pages 425–426. ACM, 2016

    work page 2016

  2. [2]

    F. Abel, Y . Deldjoo, M. Elahi, and D. Kohlsdorf. Recsys challenge 2017: Offline and online evaluation. In Proceedings of the Eleventh ACM Conference on Recommender Systems, pages 372–373. ACM, 2017

    work page 2017

  3. [3]

    S. T. Al-Otaibi and M. Ykhlef. A survey of job recommender systems. Interna- tional Journal of Physical Sciences , 7(29):5127–5142, 2012

    work page 2012

  4. [4]

    J. R. Anderson, D. Bothell, M. D. Byrne, S. Douglass, C. Lebiere, and Y . Qin. An integrated theory of the mind. Psychological review, 111(4):1036, 2004

    work page 2004

  5. [5]

    Barkan and N

    O. Barkan and N. Koenigstein. Item2vec: neural item embedding for collaborative filtering. In 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2016

    work page 2016

  6. [6]

    J. Beel, M. Genzmehr, S. Langer, A. Nürnberger, and B. Gipp. A comparative analysis of offline and online evaluations and discussion of research paper reco- mmender system evaluation. In Proceedings of the international workshop on reproducibility and replication in recommender systems evaluation, pages 7–14. ACM, 2013

    work page 2013

  7. [7]

    P. G. Campos, F. Díez, and I. Cantador. Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols. User Modeling and User-Adapted Interaction, 24(1-2):67–119, 2014

    work page 2014

  8. [8]

    Chandramouli, J

    B. Chandramouli, J. J. Levandoski, A. Eldawy, and M. F. Mokbel. Streamrec: a real-time recommender system. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of data , pages 1243–1246. ACM, 2011

    work page 2011

  9. [9]

    Eksombatchai, P

    C. Eksombatchai, P. Jindal, J. Z. Liu, Y . Liu, R. Sharma, C. Sugnet, M. Ulrich, and J. Leskovec. Pixie: A system for recommending 3+ billion items to 200+ million users in real-time. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, pages 1775–1784. International World Wide Web Conferences Steering Committee, 2018

    work page 2018

  10. [10]

    George and S

    T. George and S. Merugu. A scalable collaborative filtering framework based on co-clustering. In Fifth IEEE International Conference on Data Mining (ICDM’05), pages 4–pp. IEEE, 2005

    work page 2005

  11. [11]

    Huang, B

    Y . Huang, B. Cui, W. Zhang, J. Jiang, and Y . Xu. Tencentrec: Real-time stream rec- ommendation in practice. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data , pages 227–238. ACM, 2015

    work page 2015

  12. [12]

    Kenthapadi, B

    K. Kenthapadi, B. Le, and G. Venkataraman. Personalized job recommendation system at linkedin: Practical challenges and lessons learned. In Proc. of ACM RecSys’17

    work page

  13. [13]

    Kowald, S

    D. Kowald, S. C. Pujari, and E. Lex. Temporal effects on hashtag reuse in twitter: A cognitive-inspired hashtag recommendation approach. In Proc. of WWW’17

    work page

  14. [14]

    Neighborhood Troubles: On the Value of User Pre-Filtering To Speed Up and Enhance Recommendations

    E. Lacic, D. Kowald, and E. Lex. Neighborhood troubles: On the value of user pre-filtering to speed up and enhance recommendations. arXiv preprint arXiv:1808.06417, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  15. [15]

    Lacic, D

    E. Lacic, D. Kowald, D. Parra, M. Kahr, and C. Trattner. Towards a scalable social recommender engine for online marketplaces: The case of apache solr. In Proceedings of the 23rd International Conference on World Wide Web , pages 817–822. ACM, 2014

    work page 2014

  16. [16]

    Beyond Accuracy Optimization: On the Value of Item Embeddings for Student Job Recommendations

    E. Lacic, D. Kowald, M. Reiter-Haas, V . Slawicek, and E. Lex. Beyond accuracy optimization: On the value of item embeddings for student job recommendations. arXiv preprint arXiv:1711.07762, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  17. [17]

    Le and T

    Q. Le and T. Mikolov. Distributed representations of sentences and documents. In Proc. of ICML’14

    work page

  18. [18]

    Linden, B

    G. Linden, B. Smith, and J. York. Amazon. com recommendations: Item-to-item collaborative filtering. IEEE Internet computing, (1):76–80, 2003

    work page 2003

  19. [19]

    Mikolov, I

    T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119, 2013

    work page 2013

  20. [20]

    Musto, G

    C. Musto, G. Semeraro, M. de Gemmis, and P. Lops. Learning word embeddings from wikipedia for content-based recommender systems. In European Conference on Information Retrieval, pages 729–734. Springer, 2016

    work page 2016

  21. [21]

    M. J. Pazzani and D. Billsus. Content-based recommendation systems. In The adaptive web, pages 325–341. Springer, 2007

    work page 2007

  22. [22]

    V .-T. Phi, L. Chen, and Y . Hirate. Distributed representation based recommender systems in e-commerce. In DEIM F orum, 2016

    work page 2016

  23. [23]

    Rafter, K

    R. Rafter, K. Bradley, and B. Smyth. Personalised retrieval for online recruitment services. In The BCS/IRSG 22nd Annual Colloquium on Information Retrieval (IRSG 2000), Cambridge, UK, 5-7 April, 2000 , 2000

    work page 2000

  24. [24]

    A. Said, J. Lin, A. Bellogín, and A. de Vries. A month in the life of a production news recommender system. In Proceedings of the 2013 workshop on Living labs for information retrieval evaluation, pages 7–10. ACM, 2013

    work page 2013

  25. [25]

    J. Yuan, W. Shalaby, M. Korayem, D. Lin, K. AlJadda, and J. Luo. Solving cold- start problem in large-scale recommendation engines: A deep learning approach. In 2016 IEEE International Conference on Big Data (Big Data) , pages 1901–1910. IEEE, 2016

    work page 2016

  26. [26]

    Zhang and X

    C. Zhang and X. Cheng. An ensemble method for job recommender systems. In Proceedings of the Recommender Systems Challenge , page 2. ACM, 2016

    work page 2016