Pith. sign in

REVIEW 4 major objections 5 minor 22 references

HARE: a Flexible Highlighting Annotator for Ranking and Exploration

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HARE shows that a token-level highlighting workflow plus post-processing can rank clinical documents by mobility relevance, with Spearman correlation up to 0.899.

desk verdict HARE is a solid tool paper where the segment-fidelity claim outruns the evaluation, but the code and comparison are worth a serious referee. read the letter →

arxiv 1908.11302 v1 pith:HAZTFKEW submitted 2019-08-29 cs.CL cs.AIcs.IR

Denis Newman-Griffis , Eric Fosler-Lussier This is my paper · ORCID
classification cs.CLcs.AIcs.IR
keywords token-levelrelevancetaggingdocumentrankinghighlightingannotationclinicalnarrativesmobilityinformationViterbismoothingembeddingcomparisoncorpustriage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces HARE, a supervised workflow that converts token-level relevance annotations into highlighted document segments and a ranked list of documents by relevance to a target criterion. It claims that with a small annotated corpus—400 physical-therapy records labeled for mobility descriptions—a feed-forward tagger using standard embeddings can rank unseen documents with high agreement to human rankings. The paper also finds that post-processing, especially Viterbi smoothing with transition counts taken from the training data, changes ranking quality more than the choice of static versus contextualized embeddings. If correct, HARE provides a lightweight route to corpus triage in domains that lack established vocabularies, and a way to compare annotation models by qualitative output rather than a single score.

What carries the argument

The load-bearing mechanism is the combination of a binomial token annotator and a Viterbi smoother. The annotator is a feed-forward network that maps embedding features to a relevance probability for each token; because token scores are treated as conditionally independent given the features, the raw output is noisy at segment boundaries. The smoother decodes the most likely sequence of relevant/irrelevant states using a transition matrix counted from the training data, then converts the path probabilities into per-token conditional relevance scores. Those smoothed scores feed three document scorers—Segments+Tokens, SumScores, and Density—which turn the token annotations into a document ranking.

What would settle it

Take a new clinical corpus with gold mobility annotations and compare HARE's document ranking against a sequence tagger that models token transitions contextually; if the context-aware model ranks documents markedly better while token-level $F_2$ is unchanged, HARE's conditional-independence-plus-static-transition assumption is the bottleneck.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that a token-level relevance tagger trained with simple embeddings, followed by configurable post-processing, can produce high-fidelity highlighting of mobility-relevant segments and high-quality document rankings without task-specific linguistic resources. In the reported experiments, static in-domain FastText embeddings match or beat ELMo and clinicalBERT on both token annotation and ranking correlation, and Viterbi smoothing—decoding the most likely relevant/irrelevant state sequence using transition probabilities counted from training data—raises Spearman $\rho$ for contextualized models from 0.771 to 0.886 (ELMo) and from 0.689 to 0.844 (BERT) while barely changing token-level $F_2$. The practical discovery is that ranking correlation and segment quality, rather than token $F_2$ alone, reveal how well a model captures document-level relevance.

Load-bearing premise

The load-bearing premise is that token relevance can be judged from local embedding features plus a fixed transition table learned from training data, so if real mobility passages have longer-range structure that those averages miss, the highlighted segments and rankings will be systematically off.

Editorial extensions

If this is right

  • A new corpus can be triaged without a domain ontology: label a few hundred documents for the construct of interest, train the token tagger, and rank the rest by highlighted segments.
  • Viterbi smoothing should be treated as a standard post-processing step for ranking tasks, since it improved Spearman $\rho$ for all three embedding types in the reported experiments while leaving token $F_2$ essentially unchanged.
  • Contextualized embeddings are not automatically better for this workflow: static in-domain embeddings produced the highest ranking correlation, so embedding choice should be validated on ranking quality rather than assumed.
  • The same binary pipeline extends to multi-label and multi-class settings by training independent relevance taggers or switching to multi-class cross-entropy, as described in the paper's appendix.
  • Qualitative tools—threshold curves, score distributions, and per-token lexicalization—allow model comparisons to happen before costly manual annotation review.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension is replacing the static transition counts with context-dependent transition probabilities; the paper itself flags its fixed transitions as likely to degrade smoothing, and a dynamic version would show whether sequence structure is the remaining bottleneck.
  • Ranking metrics may be a more honest evaluation target than token $F_2$ for triage tasks, since two models with nearly identical $F_2$ can rank documents differently; future comparisons would be more informative reporting both.
  • The ranking interface could be embedded in an active-learning loop: use the ranker to select the least-certain documents for expert review, retrain, and repeat, which would lower the annotation cost of reaching a usable ranking.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents HARE, a supervised token-level relevance tagging and visualization system, applied to identifying mobility information in 400 clinical notes. The pipeline consists of embedding-based feedforward annotation (static embeddings, ELMo, BERT), post-processing (thresholding, collapsing adjacent segments, Viterbi smoothing), scoring and ranking methods, and web-based interfaces for viewing and comparing model outputs. The experiments compare three embedding sources and the effect of Viterbi smoothing using ten-fold cross-validation, reporting token-level precision, recall, and F-2 as well as Spearman's rho for document ranking. The central claim is that HARE, with minimal tuning, produces a high-quality document ranking and captures mobility-relevant segments with high fidelity, in support of exploratory analysis of new clinical corpora.

Significance. If the claims hold, HARE fills a practical niche: most annotation tools target manual annotation, while HARE explicitly supports early-stage organic exploration, model comparison, and interactive post-processing. The evaluation is anchored in externally annotated gold tokens rather than self-generated labels, so the main ranking and annotation claims are not circular in a problematic sense. The system is open-source, the mobility corpus is a realistic clinical use case with public availability of the code, and the qualitative tools (threshold curves, score distributions, lexicalization) are genuinely useful for understanding model behavior. The main value is therefore methodological and practical. The main reservations, detailed below, concern the mismatch between the segment-fidelity claim and the token-level evidence, and the lack of statistical grounding for the reported differences.

major comments (4)
  1. [Section 4, Table 2] The abstract and Section 4 state that HARE is able to "capture mobility-likely document segments with high fidelity," but no segment-level evaluation is reported. Table 2 reports only token-level precision, recall, and F-2. Because the relevant output for the exploratory-analysis use case is the contiguous highlighted segments, and because Section 3.2 explicitly modifies segment boundaries through collapsing and Viterbi smoothing, segment fidelity should be measured directly (for example, segment-level precision/recall/F1, with an explicit matching criterion for partial overlaps). Without such an evaluation, the segment-capture part of the central claim is asserted rather than demonstrated. Either add a segment-level evaluation or rephrase the claim as qualitative.
  2. [Section 3.4.1] The Segments+Tokens scorer is defined as multiplying the number of relevant segments by "a large constant" and adding the number of relevant tokens. No value is given for this constant, and the resulting ranking can depend critically on it: if it is too small relative to plausible token counts, documents with many relevant tokens can outrank documents with more segments, and if it is too large, token counts become irrelevant except as a tie-breaker. Since this scorer is the default and is used in the main ranking results, the exact constant used in the experiments should be reported, along with a brief sensitivity check showing that the reported Spearman correlations are stable over a plausible range of constants.
  3. [Section 3.1 and Appendix A] The hyperparameter description states that "all hyperparameters discussed in this section were tuned on held-out development data in cross-validation experiments," and the paper reports the best settings. However, it is not stated whether the development data are nested within each training fold or are drawn from the same documents that later appear in the test folds. If the latter, the reported numbers are selected rather than predictive results, and the comparison between embedding methods could be optimistically biased. Please clarify the tuning protocol: state where the development data came from relative to the cross-validation folds, and if per-fold selection was used, describe the selection rule and its effect on the reported point estimates.
  4. [Table 2] All quantitative comparisons in Table 2 are given as point estimates. No confidence intervals, per-fold standard deviations, or significance tests are reported for token-level F-2 or for Spearman's rho. The differences that motivate the paper's conclusions—for example, static embeddings outperforming ELMo and BERT on ranking, and Viterbi smoothing substantially improving rho for the contextualized models—could plausibly be within noise. I request at least per-fold summary statistics for the macro-averaged precision/recall/F-2, and a paired test or confidence intervals for the ranking correlations, so that the relative claims about embedding sources and post-processing can be evaluated.
minor comments (5)
  1. [Appendix B] The text contains an unresolved reference "Figure ??" when discussing the comparison of ranking methods. The actual figure appears to be Figure 10; the reference should be corrected.
  2. [Section 3.4.1 vs Appendix B] Section 3.4.1 names Segments+Tokens as the default scorer, but Appendix B states that "SumScores provided the best overall ranking correlation." These statements are compatible if the default is chosen for interpretability, but that rationale should be stated explicitly to prevent an apparent inconsistency.
  3. [Section 3.2, Eqs. (1)-(2)] The notation in equations (1) and (2) is terse. In particular, Wj,i is described as the likelihood of being in state j at time i given ri-1 and ti, but the role of the observed token ti in conditioning the transition and the final score si is not fully spelled out. A one-sentence explanation of the normalizing denominator would aid reproducibility.
  4. [Figure 3] The caption for Figure 3 does not explain the meaning of the colored or boxed elements, so the effect of "collapse one blank" is hard to parse from the figure alone. A short descriptive caption should be added.
  5. [Section 3.1.3] The phrase "binomial softmax distribution" is nonstandard; what appears to be intended is a two-class softmax (i.e., binary logistic) output distribution. The wording could be aligned with standard usage.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: HARE's ranking and tagging claims are evaluated against external gold annotations with held-out cross-validation.

full rationale

The core empirical claims in Section 4 are anchored to external gold-standard token annotations from the Thieu et al. (2017) mobility corpus, with ten-fold document-level cross-validation. The token tagger is a feedforward DNN trained on those gold labels, and ranking quality is measured as Spearman correlation between model-produced rankings and gold-produced rankings under fixed scoring functions; this is a held-out evaluation, not a quantity that is forced by construction. The Viterbi transition probabilities in Section 3.2 are counted from training folds and applied to held-out folds, which is ordinary parameter estimation rather than circular reuse of the test labels. The choice of Segments+Tokens as the default ranking method even though Appendix B reports SumScores as best is a stated design preference ('for its clear interpretation'), not a fitted value disguised as a prediction. The paper does overstate segment-level fidelity because Table 2 reports only token-level precision/recall/F-2 and no segment-level precision/recall is provided; however, that is an unsupported-evidence or rigor concern, not a circularity. Self-citations to prior mobility extraction work (Newman-Griffis and Zirikly, 2018; Newman-Griffis et al., 2019) only motivate the difficulty of mobility extraction and are not load-bearing for the results. No equation or claimed prediction reduces to an input of the same derivation, so no circularity step can be exhibited.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The core scientific load is modest: the system relies on standard ML components and a single annotated dataset. The main fitted quantities are hyperparameters and transition counts, plus an unstated ranking constant. No new entities are invented.

free parameters (8)
  • Context window size for static embeddings = 10 words each side
    Chosen without a stated tuning process in Section 3.1.2; affects feature extraction for the static embedding model.
  • Negative sampling ratio = 0.75
    Tuned on development data (Appendix A, Figure 9a); controls the balance of irrelevant tokens sampled per epoch.
  • Relevant class weight = 2:1
    Tuned on development data (Appendix A, Figure 9d); adjusts the loss to emphasize relevant tokens.
  • Dropout rate = 0.6
    Tuned on development data (Appendix A, Figure 9c); applied to the DNN input.
  • Hidden layer configuration = 3 layers of size 300
    Tuned on development data (Appendix A, Figure 9e); defines the capacity of the token annotation model.
  • Binarization threshold = 0.5
    Used for all reported results (Table 2) but configurable; affects precision/recall and ranking.
  • Viterbi transition matrix = Counted from training data
    Estimated from labeled transitions in the training fold (Section 3.2), introducing a fitted component into post-processing.
  • Ranking constant in Segments+Tokens scorer = Unspecified 'large constant'
    Section 3.4.1 states a large constant is used to weight segment count, but its value is not given, leaving the ranking formula underdetermined.
assumptions (4)
  • domain assumption Token predictions are conditionally independent given the embedding features.
    Explicitly stated in Section 3.1.3; this modeling choice ignores token sequence context and motivates the Viterbi smoothing post-process.
  • domain assumption Gold annotations in the Thieu et al. (2017) dataset are reliable.
    The evaluation in Section 4 treats expert token-level mobility annotations as ground truth without inter-annotator agreement analysis.
  • domain assumption Pretrained embeddings (FastText, ELMo, clinicalBERT) transfer to clinical mobility text.
    The system relies on off-the-shelf or lightly adapted embeddings (Section 3.1.2) and does not validate that the domains match the target corpus.
  • domain assumption Document relevance can be computed as a function of token-level relevance scores.
    All three ranking methods (Section 3.4.1) reduce document relevance to token or segment statistics, which may miss discourse-level factors like information redundancy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HARE: a Flexible Highlighting Annotator for Ranking and Exploration." pith.science (2026). https://pith.science/paper/HAZTFKEW

@misc{pith2026190811302,
  author       = {Pith},
  title        = {Pith review of: HARE: a Flexible Highlighting Annotator for Ranking and Exploration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HAZTFKEW}},
  note         = {Machine review of arXiv:1908.11302}
}
read the original abstract

Exploration and analysis of potential data sources is a significant challenge in the application of NLP techniques to novel information domains. We describe HARE, a system for highlighting relevant information in document collections to support ranking and triage, which provides tools for post-processing and qualitative analysis for model development and tuning. We apply HARE to the use case of narrative descriptions of mobility information in clinical data, and demonstrate its utility in comparing candidate embedding features. We provide a web-based interface for annotation visualization and document ranking, with a modular backend to support interoperability with existing annotation tools. Our system is available online at https://github.com/OSU-slatelab/HARE.

Figures

Figures reproduced from arXiv: 1908.11302 by the authors.

Figure 1
Figure 1. HARE workflow for working with a set of documents; outlined boxes indicate automated compo￾nents, and gray boxes signify user interfaces. validation experiments. We report the best settings here, and provide full comparison of hyperparam￾eter settings in Appendix A. 3.1.1 Preprocessing Different domains exhibit different patterns in to￾ken and sentence structure that affect preprocess￾ing. In clinical text, tokeniza… view at source ↗
Figure 2
Figure 2. Precision, recall, and F-2 when varying bi [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Illustration of Viterbi smoothing [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (5 more)
Figure 6
Figure 6. Figure 6: Ranking interface [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 7
Figure 7. Figure 7: Distribution of token relevance scores on mo [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 8
Figure 8. Figure 8: Embedding model selection results, by F-2 on cross validation development set. Default settings for [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Hyperparameter tuning results, measuring F-2 on development set in cross-validation experiments. For [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Comparison of ranking methods used for model scores and gold scores. Scores given are Spearman’s [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 12 canonical work pages

  1. [1]

    Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019. https://www.aclweb.org/anthology/W19-1909 Publicly Available Clinical BERT Embeddings . In Clinical NLP Workshop, pages 72--78. ACL

    work page 2019

  2. [2]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. https://doi.org/1511.09249v1 Enriching Word Vectors with Subword Information . TACL, 5:135--146

    arXiv 2017

  3. [3]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://www.aclweb.org/anthology/N19-1423 Bert: Pre-training of deep bidirectional transformers for language understanding . In NAACL-HLT, pages 4171--4186. ACL

    work page 2019

  4. [4]

    Ingrid E Fisher, Margaret R Garnsey, and Mark E Hughes. 2016. https://doi.org/10.1002/isaf.1386 Natural Language Processing in Accounting, Auditing and Finance: A Synthesis of the Literature with a Roadmap for Future Research . Intelligent Systems in Accounting, Finance and Management, 23(3):157--214

    work page doi:10.1002/isaf.1386 2016

  5. [5]

    Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing . To appear

    work page 2017

  6. [6]

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. http://arxiv.org/abs/1901.08746 BioBERT: a pre-trained biomedical language representation model for biomedical text mining . arXiv preprint arXiv:1901.08746, pages 1--8

    arXiv 2019

  7. [7]

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. http://arxiv.org/abs/1301.3781 Efficient Estimation of Word Representations in Vector Space . arXiv preprint arXiv:1301.3781, pages 1--12

    arXiv 2013

  8. [8]

    Denis Newman-Griffis and Ayah Zirikly. 2018. http://aclweb.org/anthology/W18-2301 Embedding Transfer for Low-Resource Medical Named Entity Recognition: A Case Study on Patient Mobility . In BioNLP, pages 1--11. ACL

    work page 2018

Show all 22 references
  1. [9]

    Denis Newman-Griffis, Ayah Zirikly, Guy Divita, and Bart Desmet. 2019. Classifying the reported ability in clinical mobility descriptions . In BioNLP

    2019

  2. [10]

    Philip V Ogren. 2006. https://www.aclweb.org/anthology/N06-4006 Knowtator: A Prot \' e g \' e plug-in for annotated corpus construction . In NAACL-HLT, pages 273--275, New York City, USA. ACL

    2006

  3. [11]

    Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. http://www.aclweb.org/anthology/D14-1162 Glove: Global vectors for word representation . In EMNLP, pages 1532--1543. ACL

    2014

  4. [12]

    Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. https://doi.org/10.18653/v1/N18-1202 Deep Contextualized Word Representations . In NAACL-HLT, pages 2227--2237, New Orleans, Louisiana. ACL

    2018 doi

  5. [13]

    S Trent Rosenbloom, Joshua C Denny, Hua Xu, Nancy Lorenzi, William W Stead, and Kevin B Johnson. 2011. https://doi.org/10.1136/jamia.2010.007237 Data from clinical notes: a perspective on the tension between structure and flexible documentation . JAMIA, 18(2):181--186

    2011 arXiv

  6. [14]

    Guergana K Savova, James J Masanz, Philip V Ogren, Jiaping Zheng, Sunghwan Sohn, Karin C Kipper-Schuler, and Christopher G Chute. 2010. https://doi.org/10.1136/jamia.2009.001560 Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evalu...

    2010 arXiv

  7. [15]

    Steven J Skube, Elizabeth A Lindemann, Elliot G Arsoniadis, Mari Akre, Elizabeth C Wick, and Genevieve B Melton. 2018. Characterizing Functional Health Status of Surgical Patients in Clinical Notes . In AMIA Joint Summits, pages 379--388. AMIA

    2018

  8. [16]

    Ergin Soysal, Jingqi Wang, Min Jiang, Yonghui Wu, Serguei Pakhomov, Hongfang Liu, and Hua Xu. 2018. https://doi.org/10.1093/jamia/ocx132 CLAMP – a toolkit for efficiently building customized clinical natural language processing pipelines . JAMIA, 25(3):331--336

    2018 doi

  9. [17]

    Pontus Stenetorp, Sampo Pyysalo, Goran Topi \' c , Tomoko Ohta, Sophia Ananiadou, and Jun'ichi Tsujii. 2012. https://www.aclweb.org/anthology/E12-2021 brat: a Web-based Tool for NLP-Assisted Text Annotation . In EACL, pages 102--107. ACL

    2012

  10. [18]

    Jane Taggart, Siaw-Teng Liaw, and Hairong Yu. 2015. https://doi.org/https://doi.org/10.1016/j.ijmedinf.2015.09.008 Structured data quality reports to improve EHR data quality . Int J Med Info, 84(12):1094--1098

    2015 doi

  11. [19]

    Thanh Thieu, Jonathan Camacho, and Pei-Shu Ho et al. 2017. https://doi.org/10.1109/BIBM.2017.8218042 Inductive identification of functional status information and establishing a gold standard corpus: A case study on the Mobility domain . In BIBM, pages 2300--2302. IEEE

    2017

  12. [20]

    Yonghui Wu, Mike Schuster, and Zhifeng Chen et al. 2016. http://arxiv.org/abs/1609.08144 Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation . arXiv preprint arXiv:1609.08144

    2016 arXiv

  13. [21]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  14. [22]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.