{"id":"61bd9fcf-8a64-4e2c-ba17-fa391366d722","arxiv_id":"1908.11302","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":8,"one_line_summary":"HARE is a web-based annotation and ranking system that uses token-level relevance scores to triage and explore clinical documents, demonstrated on mobility descriptions.","lead":"This paper presents HARE, an open-source tool that highlights relevant words in medical notes and ranks whole documents by their relevance. It helps researchers compare different AI text models by visualizing where each model finds important information.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Segment-fidelity claim is unsupported: only token-level F-2 is reported, no segment-level precision/recall, so 'high fidelity' segment capture is asserted rather than demonstrated.","rationale":"The reader's CONDITIONAL verdict already identifies missing details and reproducibility issues. My concern is distinct but compatible: the central claim includes high-fidelity segment capture, yet the evaluation only measures token-level performance. This strengthens the condition for acceptance rather than changing the verdict. The ranking part of the claim is supported by Spearman's rho, and the conditional-independence/static-transition concern raised by the reader is a real modeling limitation but is empirically mitigated by the reported smoothing improvements. The missing segment-level evaluation is the weakest link because it directly undercuts one of the two central capabilities claimed. I therefore recommend keeping the CONDITIONAL verdict, with an explicit requirement for segment-level evaluation before the segment-fidelity claim is accepted.","tokens_in":8430,"tokens_out":7214,"duration_ms":71493,"concrete_test":"Using the released HARE code and the 400-note mobility corpus, reproduce the best static or ELMo model with binarization threshold 0.5 and no collapsing, then compute segment-level precision, recall, and F1 by matching predicted contiguous relevant segments to gold contiguous mobility segments (using exact span match or a boundary-overlap metric). Repeat with collapsing k=1 and with Viterbi smoothing enabled. Compare these segment-level scores to the token-level F-2 in Table 2. If segment-level F1 is substantially lower than token-level F-2, the 'high fidelity' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim in Section 4 has two parts: document ranking and segment capture. Ranking is supported by Spearman's rho, but segment capture is never measured at the segment level. Table 2 reports token-level precision, recall, and F-2 only. Token-level F-2 around 84 with recall near 94 and precision near 60 can be achieved even when predicted relevant tokens are scattered across many short spurious segments, or when gold segments are fragmented. The qualitative examples in Figures 4-5 and the 'collapsing adjacent segments' post-processor in Section 3.2 suggest the authors are aware of this, but no segment-level evaluation is provided. Therefore, the statement 'capture mobility-likely document segments with high fidelity' is asserted, not demonstrated. If the intended meaning is qualitative, the wording should say so; if quantitative, segment-level precision/recall/F1 are required. This is load-bearing because the exploratory-analysis use case depends on segments being meaningful, not just on individual tokens being labeled correctly.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HARE, a supervised token-level relevance tagging and visualization system, applied to identifying mobility information in 400 clinical notes. The pipeline consists of embedding-based feedforward annotation (static embeddings, ELMo, BERT), post-processing (thresholding, collapsing adjacent segments, Viterbi smoothing), scoring and ranking methods, and web-based interfaces for viewing and comparing model outputs. The experiments compare three embedding sources and the effect of Viterbi smoothing using ten-fold cross-validation, reporting token-level precision, recall, and F-2 as well as Spearman's rho for document ranking. The central claim is that HARE, with minimal tuning, produces a high-quality document ranking and captures mobility-relevant segments with high fidelity, in support of exploratory analysis of new clinical corpora.","tokens_in":8670,"tokens_out":3982,"duration_ms":38570,"significance":"If the claims hold, HARE fills a practical niche: most annotation tools target manual annotation, while HARE explicitly supports early-stage organic exploration, model comparison, and interactive post-processing. The evaluation is anchored in externally annotated gold tokens rather than self-generated labels, so the main ranking and annotation claims are not circular in a problematic sense. The system is open-source, the mobility corpus is a realistic clinical use case with public availability of the code, and the qualitative tools (threshold curves, score distributions, lexicalization) are genuinely useful for understanding model behavior. The main value is therefore methodological and practical. The main reservations, detailed below, concern the mismatch between the segment-fidelity claim and the token-level evidence, and the lack of statistical grounding for the reported differences.","major_comments":[{"comment":"The abstract and Section 4 state that HARE is able to \"capture mobility-likely document segments with high fidelity,\" but no segment-level evaluation is reported. Table 2 reports only token-level precision, recall, and F-2. Because the relevant output for the exploratory-analysis use case is the contiguous highlighted segments, and because Section 3.2 explicitly modifies segment boundaries through collapsing and Viterbi smoothing, segment fidelity should be measured directly (for example, segment-level precision/recall/F1, with an explicit matching criterion for partial overlaps). Without such an evaluation, the segment-capture part of the central claim is asserted rather than demonstrated. Either add a segment-level evaluation or rephrase the claim as qualitative.","section":"Section 4, Table 2"},{"comment":"The Segments+Tokens scorer is defined as multiplying the number of relevant segments by \"a large constant\" and adding the number of relevant tokens. No value is given for this constant, and the resulting ranking can depend critically on it: if it is too small relative to plausible token counts, documents with many relevant tokens can outrank documents with more segments, and if it is too large, token counts become irrelevant except as a tie-breaker. Since this scorer is the default and is used in the main ranking results, the exact constant used in the experiments should be reported, along with a brief sensitivity check showing that the reported Spearman correlations are stable over a plausible range of constants.","section":"Section 3.4.1"},{"comment":"The hyperparameter description states that \"all hyperparameters discussed in this section were tuned on held-out development data in cross-validation experiments,\" and the paper reports the best settings. However, it is not stated whether the development data are nested within each training fold or are drawn from the same documents that later appear in the test folds. If the latter, the reported numbers are selected rather than predictive results, and the comparison between embedding methods could be optimistically biased. Please clarify the tuning protocol: state where the development data came from relative to the cross-validation folds, and if per-fold selection was used, describe the selection rule and its effect on the reported point estimates.","section":"Section 3.1 and Appendix A"},{"comment":"All quantitative comparisons in Table 2 are given as point estimates. No confidence intervals, per-fold standard deviations, or significance tests are reported for token-level F-2 or for Spearman's rho. The differences that motivate the paper's conclusions—for example, static embeddings outperforming ELMo and BERT on ranking, and Viterbi smoothing substantially improving rho for the contextualized models—could plausibly be within noise. I request at least per-fold summary statistics for the macro-averaged precision/recall/F-2, and a paired test or confidence intervals for the ranking correlations, so that the relative claims about embedding sources and post-processing can be evaluated.","section":"Table 2"}],"minor_comments":[{"comment":"The text contains an unresolved reference \"Figure ??\" when discussing the comparison of ranking methods. The actual figure appears to be Figure 10; the reference should be corrected.","section":"Appendix B"},{"comment":"Section 3.4.1 names Segments+Tokens as the default scorer, but Appendix B states that \"SumScores provided the best overall ranking correlation.\" These statements are compatible if the default is chosen for interpretability, but that rationale should be stated explicitly to prevent an apparent inconsistency.","section":"Section 3.4.1 vs Appendix B"},{"comment":"The notation in equations (1) and (2) is terse. In particular, Wj,i is described as the likelihood of being in state j at time i given ri-1 and ti, but the role of the observed token ti in conditioning the transition and the final score si is not fully spelled out. A one-sentence explanation of the normalizing denominator would aid reproducibility.","section":"Section 3.2, Eqs. (1)-(2)"},{"comment":"The caption for Figure 3 does not explain the meaning of the colored or boxed elements, so the effect of \"collapse one blank\" is hard to parse from the figure alone. A short descriptive caption should be added.","section":"Figure 3"},{"comment":"The phrase \"binomial softmax distribution\" is nonstandard; what appears to be intended is a two-class softmax (i.e., binary logistic) output distribution. The wording could be aligned with standard usage.","section":"Section 3.1.3"}],"recommendation":"major_revision","confidential_remarks":"The main issue is the disconnect between the segment-fidelity claim and the token-level evaluation; this is fixable with a segment-level evaluation or an appropriately qualified claim. The unresolved figure reference in Appendix B suggests that the appendix figures may need checking. Given the practical value of the system and the external gold annotations, I see no irreparable flaw, but the statistical grounding should be improved before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHARE is a modest, genuinely useful systems paper. The new thing is the combination—token-level relevance tagging, Viterbi smoothing, document ranking, and visualization in one open-source package—plus a clean head-to-head comparison of static, ELMo, and BERT embeddings on clinical mobility notes. The authors don't overclaim about the science; they present it as a tool for exploratory analysis, and the tool appears to work. The Spearman rankings (0.86–0.90 after smoothing) and token-level F-2 around 84 are credible for a lightly tuned DNN.\n\nWhat's best: the paper ships code, reports hyperparameter sweeps honestly in the appendix, and the authors flag their own limitation about static transition probabilities in Viterbi smoothing. That's good practice.\n\nThe soft spot is real and load-bearing. The abstract and intro say the tagger 'captures mobility-likely document segments with high fidelity.' The evaluation never measures segments. Table 2 gives token-level precision, recall, and F-2 only. Token-level recall near 94 can coexist with fragmented or scattered predicted segments. The viewer and the collapsing post-processor address segment structure, but there is no segment-level precision/recall/F1 anywhere. If 'high fidelity' means qualitative, fine, but then say so. If it means quantitative, the missing metric matters because the exploration use case depends on segments being meaningful.\n\nMinor issues: the 'large constant' in Segments+Tokens is unspecified (Section 3.4.1); Appendix B refers to 'Figure ??'; no confidence intervals or significance tests on the Spearman correlations; and hyperparameters are tuned on dev folds from the same cross-validation, which the paper states, so it is more a caveat than a fatal flaw. No data release is mentioned, though the code is public.\n\nOverall, this is a legitimate tool paper. The segment-fidelity claim needs either a quantitative segment-level evaluation or a downgrade to qualitative language. A serious referee should see it before publication, but it is not a desk reject. I would send it to review and ask for the segment-level numbers and a precise constant.","headline":"HARE is a solid tool paper where the segment-fidelity claim outruns the evaluation, but the code and comparison are worth a serious referee.","tokens_in":9164,"tokens_out":2271,"would_cite":false,"duration_ms":20768,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HARE shows that a token-level highlighting workflow plus post-processing can rank clinical documents by mobility relevance, with Spearman correlation up to 0.899.","keywords":["token-level relevance tagging","document ranking","highlighting annotation","clinical narratives","mobility information","Viterbi smoothing","embedding comparison","corpus triage"],"falsifier":"Take a new clinical corpus with gold mobility annotations and compare HARE's document ranking against a sequence tagger that models token transitions contextually; if the context-aware model ranks documents markedly better while token-level $F_2$ is unchanged, HARE's conditional-independence-plus-static-transition assumption is the bottleneck.","tokens_in":8246,"feed_emoji":"🩺","tokens_out":9112,"duration_ms":76321,"temperature":0.7,"pith_summary":"This paper introduces HARE, a supervised workflow that converts token-level relevance annotations into highlighted document segments and a ranked list of documents by relevance to a target criterion. It claims that with a small annotated corpus—400 physical-therapy records labeled for mobility descriptions—a feed-forward tagger using standard embeddings can rank unseen documents with high agreement to human rankings. The paper also finds that post-processing, especially Viterbi smoothing with transition counts taken from the training data, changes ranking quality more than the choice of static versus contextualized embeddings. If correct, HARE provides a lightweight route to corpus triage in domains that lack established vocabularies, and a way to compare annotation models by qualitative output rather than a single score.","feed_headline":"Highlighting tagger ranks clinical notes by mobility relevance","feed_subtitle":"A simple tagger plus Viterbi smoothing triages an unlabeled clinical corpus without a domain vocabulary.","key_machinery":"The load-bearing mechanism is the combination of a binomial token annotator and a Viterbi smoother. The annotator is a feed-forward network that maps embedding features to a relevance probability for each token; because token scores are treated as conditionally independent given the features, the raw output is noisy at segment boundaries. The smoother decodes the most likely sequence of relevant/irrelevant states using a transition matrix counted from the training data, then converts the path probabilities into per-token conditional relevance scores. Those smoothed scores feed three document scorers—Segments+Tokens, SumScores, and Density—which turn the token annotations into a document ranking.","core_discovery":"On its own terms, the paper's central claim is that a token-level relevance tagger trained with simple embeddings, followed by configurable post-processing, can produce high-fidelity highlighting of mobility-relevant segments and high-quality document rankings without task-specific linguistic resources. In the reported experiments, static in-domain FastText embeddings match or beat ELMo and clinicalBERT on both token annotation and ranking correlation, and Viterbi smoothing—decoding the most likely relevant/irrelevant state sequence using transition probabilities counted from training data—raises Spearman $\\rho$ for contextualized models from 0.771 to 0.886 (ELMo) and from 0.689 to 0.844 (BERT) while barely changing token-level $F_2$. The practical discovery is that ranking correlation and segment quality, rather than token $F_2$ alone, reveal how well a model captures document-level relevance.","pith_inferences":["A direct testable extension is replacing the static transition counts with context-dependent transition probabilities; the paper itself flags its fixed transitions as likely to degrade smoothing, and a dynamic version would show whether sequence structure is the remaining bottleneck.","Ranking metrics may be a more honest evaluation target than token $F_2$ for triage tasks, since two models with nearly identical $F_2$ can rank documents differently; future comparisons would be more informative reporting both.","The ranking interface could be embedded in an active-learning loop: use the ranker to select the least-certain documents for expert review, retrain, and repeat, which would lower the annotation cost of reaching a usable ranking."],"forward_implications":["A new corpus can be triaged without a domain ontology: label a few hundred documents for the construct of interest, train the token tagger, and rank the rest by highlighted segments.","Viterbi smoothing should be treated as a standard post-processing step for ranking tasks, since it improved Spearman $\\rho$ for all three embedding types in the reported experiments while leaving token $F_2$ essentially unchanged.","Contextualized embeddings are not automatically better for this workflow: static in-domain embeddings produced the highest ranking correlation, so embedding choice should be validated on ranking quality rather than assumed.","The same binary pipeline extends to multi-label and multi-class settings by training independent relevance taggers or switching to multi-class cross-entropy, as described in the paper's appendix.","Qualitative tools—threshold curves, score distributions, and per-token lexicalization—allow model comparisons to happen before costly manual annotation review."],"supporting_citations":[{"why":"Supplies the 400 clinical notes with gold token-level mobility annotations that HARE is trained and evaluated on.","marker":"Thieu et al. (2017)"},{"why":"Supplies the FastText static embeddings used as one of the three feature sets in the mobility experiments.","marker":"Bojanowski et al. (2017)"},{"why":"Supplies the ELMo contextualized features compared against static and BERT embeddings.","marker":"Peters et al. (2018)"},{"why":"Supplies the BERT architecture whose final-layer outputs are used as contextualized features.","marker":"Devlin et al. (2019)"},{"why":"Supplies clinicalBERT, the BERT variant used for the clinical notes.","marker":"Alsentzer et al. (2019)"},{"why":"Prior mobility extraction work that frames the task and documents the difficulty HARE takes on.","marker":"Newman-Griffis and Zirikly (2018)"},{"why":"Established annotation tool that HARE contrasts with by adding ranking and exploration.","marker":"Stenetorp et al. (2012)"},{"why":"Vocabulary-driven clinical NLP system whose terminology dependence motivates HARE's label-only approach.","marker":"Savova et al. (2010)"}],"fun_headline_variants":["Simple embeddings rival clinical BERT for note ranking","Viterbi smoothing lifts ranking correlation on clinical notes","HARE: lightweight tagger matches heavy clinical NLP","Ranking, not token F2, exposes model quality in clinical notes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that token relevance can be judged from local embedding features plus a fixed transition table learned from training data, so if real mobility passages have longer-range structure that those averages miss, the highlighted segments and rankings will be systematically off.","fun_headline_variants_meta":{"raw":{"variants":["Simple embeddings rival clinical BERT for note ranking","Viterbi smoothing lifts ranking correlation on clinical notes","HARE: lightweight tagger matches heavy clinical NLP","Ranking, not token F2, exposes model quality in clinical notes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000735,"raw_usage":{"total_tokens":3227,"prompt_tokens":825,"completion_tokens":2402,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":2336}},"tokens_in":441,"tokens_out":2402,"duration_ms":16485,"temperature":1.0,"reasoning_tokens":2336,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:18:24.639107+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a new clinical corpus with gold mobility annotations and compare HARE's document ranking against a sequence tagger that models token transitions contextually; if the context-aware model ranks documents markedly better while token-level $F_2$ is unchanged, HARE's conditional-independence-plus-static-transition assumption is the bottleneck.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 400 clinical notes with gold token-level mobility annotations that HARE is trained and evaluated on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the BERT architecture whose final-layer outputs are used as contextualized features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies clinicalBERT, the BERT variant used for the clinical notes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior mobility extraction work that frames the task and documents the difficulty HARE takes on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Established annotation tool that HARE contrasts with by adding ranking and exploration."}],"review_version":1}