pith. sign in

arxiv: 1906.09569 · v1 · pith:U6MD5IU5new · submitted 2019-06-23 · 💻 cs.CL · cs.DL· cs.HC

Systematic improvement of user engagement with academic titles using computational linguistics

Nim Dvir , Ruti Gafni This is my paper

Pith reviewed 2026-05-25 17:48 UTC · model grok-4.3

classification 💻 cs.CL cs.DLcs.HC
keywords user engagementacademic titlescomputational linguisticsNLPtitle optimizationinformation interactionA/B testingsentiment analysis
0
0 comments X

The pith

Academic titles modified for novelty, familiarity, and emotionality via NLP receive higher engagement scores.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that three attributes—novelty, familiarity, and emotionality—drive user engagement with academic titles, and that an NLP model can modify titles to improve these attributes and thereby increase engagement. In a pilot study with 216 participants using A/B testing, modified titles scored significantly higher on selection, evaluation, and the User Engagement Scale. This matters because it indicates that information interactions can be improved solely through wording adjustments using computational linguistics, without altering the underlying content. A sympathetic reader would see this as a method to make academic work more accessible and selected by users.

Core claim

The authors developed a model based on novelty, familiarity, and emotionality to systematically improve academic titles using computational linguistics, natural language processing, word frequency, sentiment analysis, and lexical substitution. A between-group pilot study (n=216) comparing original and modified titles found that the treatment titles had significantly higher scores for information use and user engagement measured by selection and the User Engagement Scale. The results provide empirical support that engaging content can be systematically evaluated and produced.

What carries the argument

The NLP-based optimization model that adjusts titles according to the three attributes of novelty, familiarity, and emotionality using word frequency analysis, sentiment analysis, and lexical substitution.

If this is right

  • The modified titles lead to higher user selection and evaluation of the information.
  • Computational linguistics provides a useful approach for optimizing information interactions.
  • The insights can inform the development of digital content strategies.
  • User engagement with digital information depends upon the wording being used.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This optimization approach could be extended to other forms of academic content such as abstracts or paper summaries.
  • Testing the model on non-academic titles like news headlines would check if the three attributes generalize.
  • Integrating the model into author tools might help researchers improve title engagement before submission.
  • If additional attributes beyond the three are important, the current model may miss some engagement factors.

Load-bearing premise

The three attributes identified in the literature review—novelty, familiarity, and emotionality—are the main and sufficient drivers of engagement with academic titles.

What would settle it

A larger-scale replication study in which the modified titles do not produce significantly higher User Engagement Scale scores than the originals would falsify the central claim.

Figures

Figures reproduced from arXiv: 1906.09569 by Nim Dvir, Ruti Gafni.

Figure 1
Figure 1. Figure 1: The "sticky words" model We also argue that these words can be systematically recognized using word frequency, sentiment analysis and semantic substitution. R2. CAN UE WITH INFORMATION BE SYSTEMATICALLY EVALUATED, CREATED OR MANIPULATED? Based on the previous question, we suggest the UE can be manipulated based on the wording used (its delivery). We argue that these words can be employed to evaluate and im… view at source ↗
Figure 2
Figure 2. Figure 2: The computational linguistic process of compiling a dataset [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Selection results for original vs. treat [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
read the original abstract

This paper describes a novel approach to systematically improve information interactions based solely on its wording. Following an interdisciplinary literature review, we recognized three key attributes of words that drive user engagement: (1) Novelty (2) Familiarity (3) Emotionality. Based on these attributes, we developed a model to systematically improve a given content using computational linguistics, natural language processing (NLP) and text analysis (word frequency, sentiment analysis and lexical substitution). We conducted a pilot study (n=216) in which the model was used to formalize evaluation and optimization of academic titles. A between-group design (A/B testing) was used to compare responses to the original and modified (treatment) titles. Data was collected for selection and evaluation (User Engagement Scale). The pilot results suggest that user engagement with digital information is fostered by, and perhaps dependent upon, the wording being used. They also provide empirical support that engaging content can be systematically evaluated and produced. The preliminary results show that the modified (treatment) titles had significantly higher scores for information use and user engagement (selection and evaluation). We propose that computational linguistics is a useful approach for optimizing information interactions. The empirically based insights can inform the development of digital content strategies, thereby improving the success of information interactions.elop more sophisticated interaction measures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript proposes an NLP-based model to systematically improve academic titles by enhancing novelty, familiarity, and emotionality through word frequency analysis, sentiment analysis, and lexical substitution. It describes a pilot between-group A/B test with 216 participants comparing original and modified titles on selection and evaluation using the User Engagement Scale, claiming significantly higher scores for the modified titles.

Significance. If the empirical results can be substantiated with full statistical reporting and controls, the work could offer a practical, literature-grounded method for optimizing digital content engagement. It bridges computational linguistics with information science and has potential applications in academic publishing and digital strategies. The approach is novel in its systematic application but currently lacks the evidentiary detail to assess impact.

major comments (3)
  1. [Abstract / Pilot Study] Abstract / Pilot Study: The report of 'significantly higher scores for information use and user engagement' provides no statistical details such as p-values, effect sizes, exact sample allocation, exclusion criteria, or power analysis. This under-specification is load-bearing for the central claim that the model systematically improves engagement.
  2. [Methods / Experimental Design] Methods / Experimental Design: The A/B test lacks an ablation component or direct measurement of changes in novelty, familiarity, and emotionality scores between original and modified titles. Without this, it is not possible to confirm that improvements are due to the modeled attributes rather than unmodeled factors such as title length or syntactic changes.
  3. [Literature Review / Model Design] Literature Review / Model Design: The assumption that the three attributes are primary and sufficient drivers is stated but the pilot includes no control arm altering titles along other dimensions (e.g., length or complexity) to test sufficiency. This underpins the 'systematic' improvement claim.
minor comments (2)
  1. [Abstract] The final sentence of the abstract appears truncated ('elop more sophisticated interaction measures.') and should be corrected.
  2. [Methods] The manuscript would benefit from reporting the exact lexical substitution rules or sentiment lexicon used, and from including reproducibility materials such as the title modification code or examples of before/after titles.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive report. We address each major comment below with explanations grounded in the manuscript and pilot design. Revisions will be made where they strengthen reporting without altering the core claims of this preliminary study.

read point-by-point responses

  1. Referee: [Abstract / Pilot Study] The report of 'significantly higher scores for information use and user engagement' provides no statistical details such as p-values, effect sizes, exact sample allocation, exclusion criteria, or power analysis. This under-specification is load-bearing for the central claim that the model systematically improves engagement.

    Authors: We agree that expanded statistical reporting will improve clarity. The full manuscript reports independent-samples t-tests with p < 0.05 for the engagement measures and equal allocation (108 participants per group). We will revise the abstract and add a statistical analysis subsection detailing effect sizes (Cohen's d), exclusion criteria (failed attention checks), and post-hoc power analysis. revision: yes

  2. Referee: [Methods / Experimental Design] The A/B test lacks an ablation component or direct measurement of changes in novelty, familiarity, and emotionality scores between original and modified titles. Without this, it is not possible to confirm that improvements are due to the modeled attributes rather than unmodeled factors such as title length or syntactic changes.

    Authors: We acknowledge the value of explicit before-after attribute measurements. The model applies targeted substitutions, and attribute scores were computed for all titles using the described NLP pipeline. We will add a table in the revised manuscript reporting the mean changes in novelty, familiarity, and emotionality scores across the stimulus set to directly link modifications to the modeled attributes. revision: partial

  3. Referee: [Literature Review / Model Design] The assumption that the three attributes are primary and sufficient drivers is stated but the pilot includes no control arm altering titles along other dimensions (e.g., length or complexity) to test sufficiency. This underpins the 'systematic' improvement claim.

    Authors: The three attributes were chosen after an interdisciplinary literature review establishing their roles in engagement; the pilot evaluates the resulting computational model rather than exhaustively testing sufficiency. The systematic aspect derives from the reproducible NLP procedure. We maintain that additional control conditions are not required to support the preliminary findings and will not revise the manuscript on this point. revision: no

Circularity Check

0 steps flagged

No circularity: model derived from external literature and tested empirically on new data

full rationale

The paper's derivation chain begins with an interdisciplinary literature review identifying three attributes (novelty, familiarity, emotionality), then applies standard NLP techniques (word frequency, sentiment analysis, lexical substitution) to modify titles, followed by an A/B test on new title data (n=216) measuring user engagement. No equations, fitted parameters, or predictions reduce to self-inputs by construction. No self-citations are load-bearing for the central claim, no uniqueness theorems are imported, and no ansatz is smuggled via citation. The empirical pilot results stand as independent evidence rather than a renaming or self-definition of prior inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that novelty, familiarity, and emotionality are the key engagement drivers; no free parameters or invented entities are introduced in the abstract.

axioms (1)
  • domain assumption Novelty, familiarity, and emotionality are the primary attributes driving user engagement with content wording.
    Identified via interdisciplinary literature review and used as the basis for the optimization model.

pith-pipeline@v0.9.0 · 5761 in / 1217 out tokens · 44064 ms · 2026-05-25T17:48:56.408281+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages

  1. [1]

    L., & McKay, J

    https://doi.org/10.1002/pra2.2016.14505301150 Systematic Evaluation and Improvement of User Engagement 512 O’Brien, H. L., & McKay, J. (2018). Modeling antecedents of user engagement. In The handbook of communication engagement (pp. 73–88). https://doi.org/10.1002/9781119167600.ch6 O’Brien, H. L., & Toms, E. G. (2008). What is user engagement? A conceptua...

    work page doi:10.1002/pra2.2016.14505301150 2016

  2. [2]

    L., & Toms, E

    https://doi.org/10.1002/asi.20801 O’Brien, H. L., & Toms, E. G. (2013). Examining the generalizability of the User Engagement Scale (UES) in exploratory search. Information Processing & Management, 49(5), 1092–1107. https://doi.org/10.1016/j.ipm.2012.08.005 Overbeeke, K., Djajadiningrat, T., Hummels, C., Wensveen, S., & Prens, J. (2003). Let’s make things...

    work page doi:10.1002/asi.20801 2013