Pith. sign in

REVIEW 4 major objections 6 minor 15 references

Development of the user-friendly decision aid Rule-based Evaluation and Support Tool (REST) for optimizing the resources of an information extraction task

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read REST is a decision aid that lets a non-NLP annotator decide per entity whether rules are feasible before any algorithm is built.

desk verdict A plausible, clearly described decision aid whose central validation is only agreement between two judges using the same checklist—there is no evidence yet that its rule-versus-ML recommendations actually match real extraction performance. read the letter →

arxiv 2506.13177 v1 pith:Q6FTRI5A submitted 2025-06-16 cs.CL

Guillaume Bazin , Xavier Tannier , Fanny Adda , Ariel Cohen , Akram Redjdal , Emmanuelle Kempf This is my paper · ORCID
classification cs.CL
keywords rule-basedinformationextractiondecisionsupporttoolannotationfeasibilityclinicaltextminingexplainableartificialintelligencesustainablelinguistichomogeneityentity-levelNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

REST is a decision aid that lets a person with no NLP background decide, before any algorithm is built, whether each entity in an information extraction task should be handled by hand-written rules or by machine learning. The decision is made entity by entity, based on a checklist of textual features computed from expert highlighting of a small representative corpus subset. The paper's central claim is that this checklist reliably predicts rule-development feasibility, and the evidence offered is a validation on a 12-entity oncology use case in which two field experts agreed on 11 of 12 decisions (91.67%). If the claim holds, IE teams can avoid unnecessary annotation and rule-building effort, using rules as a sustainable default and ML only as a backup for entities where rules are not feasible.

What carries the argument

REST's decision machinery is the per-entity checklist with four numeric gates: text-highlight count share $TH \geq 25\%$, linguistic homogeneity $LH \geq 10\%$, entity recall $ER \geq 75\%$, and entity precision $EP \geq 75\%$. The homogeneity score underneath is $H_e = (T_e - U_e)/T_e$, where $T_e$ is total word occurrences and $U_e$ unique words in that entity's highlights, transformed by a sigmoid $\sigma(x,k)=1/(1+e^{-kx})$ with $k=10$ to widen score gaps. REST generates candidate categories from a modified tf-idf over entity highlights, checks each category's regular expressions against the corpus to count true and false positives, and lets the user ban words or adjust term spacing to improve precision. The work the machinery does is to convert raw expert highlighting into a small, inspectable set of quantitative signals on which the final rules-versus-ML recommendation rests.

What would settle it

Build rule-based extractors for the 12 entities from the categories REST creates, run them on held-out reports, and compare their actual precision and recall against the checklist's 75% recall and 75% precision gates; if entities REST marks as feasible consistently fall below the gates, or infeasible entities exceed them, the thresholds are not doing the predictive work the paper assigns to them.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes a workflow: a field expert highlights, in one session, a representative subset of the corpus for all entities; REST then computes a linguistic homogeneity score, surfaces the most informative n-grams, and lets the user group highlights into a small number of categories that generate regular expressions. From these categories REST estimates entity-level recall and precision. A final checklist—at least 25% of highlights present, at least 10% homogeneity, at least 75% entity recall and 75% entity precision—converts those estimates into a binary recommendation: rules or machine learning. The discovery is that this procedure is reproducible across experts: the junior and senior field experts agreed on 11 of the 12 entities, with the single disagreement traced to a difference in understanding the highlight guide. REST is therefore presented as a validated way to make the rules-versus-ML choice explicit before any algorithm is developed.

Load-bearing premise

The load-bearing premise is that the fixed thresholds (25% highlights, 10% homogeneity, 75% recall, 75% precision) really do predict how easily rules can be written and how well they will perform—an assumption the paper supports only by inter-expert agreement, not by comparing REST's verdicts to measured rule-based extraction or to ML baselines.

Editorial extensions

If this is right

  • An IE team can decide the method for each entity before writing any extractor, avoiding wasted rule development on entities whose surface forms are too heterogeneous.
  • Manual annotation effort is reduced because highlighting is faster than full annotation and ML training data is only needed for the entities where rules fail.
  • Rules become the default method, which lowers the carbon footprint and improves interpretability and transferability of the resulting IE system.
  • The 91.67% inter-expert agreement suggests that, with a clear highlighting guide, the checklist decision is reproducible enough to be delegated to non-NLP annotators.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper leaves open is to build the actual rules for each recommended entity and check whether measured precision and recall meet the 75% gates; agreement between two experts does not by itself show the gates predict performance.
  • The same checklist logic could be transferred to deciding when few-shot or zero-shot LLM extraction will work, since lexical homogeneity and entity specificity are plausibly the same features that make a surface-form prompt reliable.
  • Because the homogeneity score is computed over raw word counts, it can drift with highlight length and corpus size; normalizing per sentence or per report would give a fairer cross-corpus comparison.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces REST, a decision-support tool intended to help non-NLP annotators choose, per entity, between rule-based and machine-learning information extraction. The workflow consists of expert highlighting of entities in a small representative corpus, visual exploration via a homogeneity score and tf-idf n-grams, user-defined highlight categories, category-level precision estimates, and a final checklist with thresholds TH>=25%, LH>=10%, ER>=75%, EP>=75%. The authors report a 12-entity use case on 35 translated CANTEMIST reports, with 91.67% inter-expert agreement on rule feasibility, and frame this as external validation. The central claim is that REST yields 'expected rule development feasibility and IE performance metrics' before any algorithm is built.

Significance. The tool addresses a real problem: making sustainable rule-based IE decisions in resource-constrained settings. The interface design, tutorial video, and checklist are concrete contributions, and the inter-annotator agreement result is honestly reported as a number. However, the paper does not establish that the checklist or its thresholds predict actual rule-based extraction performance; agreement between two judges applying the same instrument is a reproducibility measure, not a validity measure. The ER/EP estimates are in-sample, computed by applying the user-created categories to the same highlight corpus used to create them, so the claimed 'expected performance' lacks an external anchor. As a design study and usability report the paper is useful, but as a validated decision aid the evidence is currently insufficient.

major comments (4)
  1. [4.2 / Table 1] The validation reported in Section 4.2 and Table 1 measures agreement between two experts applying the same checklist (11/12, 91.67%). This establishes reproducibility of the decision procedure, but not that the procedure's feasible/infeasible calls match actual rule-development effort or extraction quality. Both experts could share a systematically miscalibrated threshold set, and agreement would remain high. To support the central claim, the authors should compare REST's recommendation to measured precision/recall of a simple rule-based extractor built from the categories, or to an ML/LLM baseline on held-out text, and show that entities passing the checklist indeed yield acceptable rule performance.
  2. [3.3.2 / 3.4.2, Figures 9 and 12] The ER/EP metrics used in the checklist are in-sample: regular expressions built from the user's categorization are applied to the same highlight corpus that defined the categories, and the highlight-correction step (Section 3.3.3) retroactively turns false positives into true positives. The resulting entity precision and recall therefore measure fit to the development highlights, not performance on unseen text. The thresholds (TH>=25%, LH>=10%, ER>=75%, EP>=75%) are asserted without derivation or sensitivity analysis; since every decision in Table 2 hinges on these thresholds, the paper needs either an external performance benchmark or a sensitivity analysis over threshold values.
  3. [3.2.1, Eq. (1)] The homogeneity score H_e=(T_e-U_e)/T_e and the sigmoid transform with fixed k=10 are presented without justification or robustness checks. There is no evidence that H_e correlates with rule-development complexity or with the number of categories needed to reach acceptable recall as illustrated in Figure 11. At minimum, the authors should report the distribution of H_e and H_e^sigma across the 12 entities, state how k was chosen, and test whether the ordering induced by H_e matches the difficulty observed in the categorization step.
  4. [4.1] The 'external validation' is based on 35 reports machine-translated from Spanish to French, with no reported translation quality check. Given that the tool's output depends on lexical surface forms, translation artifacts could directly influence the feasibility judgments. The authors should either validate on original-language data or justify why translation does not affect the conclusions.
minor comments (6)
  1. [4.1] Typo: 'datataset' should be 'dataset'; also state how many of the 500 reports were translated and how the 35 reports were selected.
  2. [3.1] 'whereas it could be formalized' should be 'whether it could be formalized'; consider defining 'highlightment' at first use.
  3. [3.3.2 / 3.3.3] Figure 9A/B captions mention 'metastatic stage with localization' while Section 3.3.3 says entity precision increased from 0.63 to 0.8; clarify which entity and whether the figure shows before and after the correction step.
  4. [3.4.2 / Table 2] The table would be easier to read if threshold names were spelled out in full and if the 'Ø' symbol in Table 1 were defined.
  5. [Abstract / Conclusion] The phrase 'external validity' is stronger than the evidence presented; suggest 'reproducibility on a use case' to match what was actually measured.
  6. [References] The CANTEMIST dataset is not cited directly; reference [15] is an AMIA abstract and does not provide the corpus. Please add the appropriate data citation.

Circularity Check

2 steps flagged · score 6.0 of 10

REST's 'expected IE performance' is computed in-sample on the same highlights used to build the categories, and its external validation measures agreement on the tool's own checklist rather than agreement with actual rule-based performance; the central claim is therefore only partially anchored.

  1. fitted input called prediction [Section 3.3.2 ('Visualization of the precision of each text highlight category') and Abstract]
    "REST enables the assessment of the precision of each highlight category by the following method. After categorizing the highlights, the same underlying regular expressions are applied on the whole text corpus. If the match corresponds to an existing highlight, it will be considered as TP. ... We developed and validated the feasibility and the performance metrics of the REST decision tool ... REST makes the annotator visualize ... the expected rule development feasibility and IE performance metrics."

    The 'expected rule development feasibility and IE performance metrics' are presented as predictions of how a rule-based extractor would behave before any algorithm is built. But the ER/EP numbers are obtained by applying the user-created regular expressions back to the very same expert highlights from which those categories and expressions were derived. Any match to an existing highlight is counted as TP, so recall and precision are, by construction, training-set fit statistics rather than held-out extraction performance. The categories were created from the highlights (Section 3.3.1), so the metric cannot fail to reflect the input; it is an in-sample summary of the categorizer, not a prediction of rule-based IE performance on unseen text.

  2. self definitional [Section 4.2 ('External evaluations') and Table 2]
    "The assessment of rule feasibility was realized by the same junior field expert (FA) ... using the same checklist of rule feasibility. ... Among the 12 evaluated entities, the junior and senior expert agreed on 11 entities, resulting in an overall agreement of 91.67% (Table 1). ... Entity Criteria ... TH ≥ 25%, LH ≥ 10%, ER ≥ 75%, EP ≥ 75%."

    Rule feasibility is operationalized by the authors' own checklist thresholds in Table 2; the external validation consists of a second expert applying that same checklist. Agreement between two raters using the same decision rule demonstrates reproducibility of the checklist, not that the checklist predicts the effort or performance of building actual rules. Since 'feasibility' is defined as passing the checklist, two experts agreeing on the checklist is the tool judging itself: a systematically miscalibrated threshold set would still yield high agreement. No measured rule-based precision/recall on held-out text, no ML/LLM baseline, and no external ground truth is used to validate the thresholds.

full rationale

The main circularity is in the performance claim. The paper calls ER/EP 'expected rule development feasibility and IE performance metrics', but computes them by applying regexes to the same highlights used to create the regexes, so they are in-sample fit measures. The validation section then treats inter-annotator agreement on the tool's own checklist as 'external validity'; that is a reproducibility check, not an independent confirmation that REST's feasible/infeasible calls correspond to real rule-based IE outcomes. The thresholds and the sigmoid k parameter are fixed without derivation or sensitivity analysis, which is a correctness/validity concern rather than a circularity per se, but it reinforces that the decision procedure has no external anchor. I do not find a load-bearing self-citation chain: reference [14] supports the default-rule premise but the tool's operation does not depend on it. The circularity is therefore partial: the visualization and interaction modules could still be useful, but the central 'prediction' of rule feasibility/performance reduces, by construction, to the tool's own in-sample metrics and to agreement on its own checklist. Score 6.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The decision workflow relies on four hand-set parameters and constants and several unverified assumptions. The free parameters are k=10 in the sigmoid and the four checklist thresholds. The core assumptions are that highlighting is a valid proxy, that the homogeneity ratio predicts rule difficulty, that rules are preferable by default, and that the translated 35-report subset is representative. None of these is validated against measured extraction performance or ML baselines.

free parameters (2)
  • Sigmoid steepness k = 10
    Equation for sigma(x,k) in Section 3.2.1; hand-chosen to increase score separation; no justification for the value.
  • Checklist thresholds = TH >= 25%, LH >= 10%, ER >= 75%, EP >= 75%
    Table 2 and Section 3.4; arbitrary feasibility cutoffs applied to all entities; no external or statistical basis described.
assumptions (4)
  • domain assumption Rules are a better default than ML/LLM for IE when entity terminology is homogeneous.
    Introduction and Section 2; the entire tool assumes this preference order and no comparison is provided.
  • ad hoc to paper Entity highlightment is a quick and reliable proxy for full annotation and for rule feasibility.
    Section 2, step 1; the paper asserts that highlighting is sufficient for rule-feasibility assessment without evidence that highlights capture the same distribution as full annotations.
  • ad hoc to paper The homogeneity score H_e = (T_e - U_e)/T_e predicts rule development complexity.
    Section 3.2.1; this specific ratio is proposed without validation against any actual rule-building outcome.
  • domain assumption The 35-report corpus translated by Google Translate is representative of the full corpus and preserves the linguistic features relevant to rule feasibility.
    Section 4.1; no sampling or translation quality check is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Development of the user-friendly decision aid Rule-based Evaluation and Support Tool (REST) for optimizing the resources of an information extraction task." pith.science (2026). https://pith.science/paper/Q6FTRI5A

@misc{pith2026250613177,
  author       = {Pith},
  title        = {Pith review of: Development of the user-friendly decision aid Rule-based Evaluation and Support Tool (REST) for optimizing the resources of an information extraction task},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q6FTRI5A}},
  note         = {Machine review of arXiv:2506.13177}
}
read the original abstract

Rules could be an information extraction (IE) default option, compared to ML and LLMs in terms of sustainability, transferability, interpretability, and development burden. We suggest a sustainable and combined use of rules and ML as an IE method. Our approach starts with an exhaustive expert manual highlighting in a single working session of a representative subset of the data corpus. We developed and validated the feasibility and the performance metrics of the REST decision tool to help the annotator choose between rules as a by default option and ML for each entity of an IE task. REST makes the annotator visualize the characteristics of each entity formalization in the free texts and the expected rule development feasibility and IE performance metrics. ML is considered as a backup IE option and manual annotation for training is therefore minimized. The external validity of REST on a 12-entity use case showed good reproducibility.

Figures

Figures reproduced from arXiv: 2506.13177 by the authors.

Figure 7
Figure 7. Visualization of the uncategorized text highlights for the entity ‘ [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 12
Figure 12. Exploration, interaction and decision steps as offered by REST, [PITH_FULL_IMAGE:figures/full_fig_p012_12.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages

  1. [1]

    Natural language processing: state of the art, current trends and challenges

    Khurana D, Koli A, Khatter K, Singh S. Natural language processing: state of the art, current trends and challenges. Multimed Tools Appl. 2023;82(3):3713–44

    work page 2023

  2. [2]

    Evaluation of ChatGPT on Biomedical Tasks: A Zero -Shot Comparison with Fine-Tuned Generative Transformers

    Jahan I, Laskar MTR, Peng C, Huang J. Evaluation of ChatGPT on Biomedical Tasks: A Zero -Shot Comparison with Fine-Tuned Generative Transformers. In: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks [Internet]. Toronto, Canada: Association for Computational Linguistics; 2023. p. 326–36. Available from: https://aclantholo...

    work page 2023

  3. [3]

    A Zero-shot and Few-shot Study of Instruction-Finetuned Large Language Models Applied to Clinical and Biomedical Tasks

    Labrak Y, Rouvier M, Dufour R. A Zero-shot and Few-shot Study of Instruction-Finetuned Large Language Models Applied to Clinical and Biomedical Tasks. 2023; Available from: http://arxiv.org/abs/2307.12114

    work page Pith review arXiv 2023

  4. [4]

    Benchmarking large language models for biomedical natural language processing applications and recommendations

    Chen Q, Du J, Hu Y, Keloth VK, Peng X, Raja K, et al. Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations. 2023;(2). Available from: http://arxiv.org/abs/2305.16326

    work page Pith review arXiv 2023

  5. [5]

    Rule-based information extraction is dead! Long live rule-based information extraction systems! EMNLP 2013 - 2013 Conf Empir Methods Nat Lang Process Proc Conf

    Chiticariu L, Li Y, Reiss FR. Rule-based information extraction is dead! Long live rule-based information extraction systems! EMNLP 2013 - 2013 Conf Empir Methods Nat Lang Process Proc Conf. 2013;(October):827–32

    work page 2013

  6. [6]

    Clinical information extraction applications: A literature review

    Wang Y, Wang L, Rastegar-Mojarad M, Moon S, Shen F, Afzal N, et al. Clinical information extraction applications: A literature review. J Biomed Inform. 2018 Jan;77:34 –49

    work page 2018

  7. [7]

    Design of a rule based bio medical entity extractor

    Suganya G, Porkodi R. Design of a rule based bio medical entity extractor. Int J Eng Adv Technol. 2019;8(5 Special Issue 3):245–9

    work page 2019

  8. [8]

    Automatic Annotation Tool to Support Supervised Machine Learning for Scaphoid Fracture Detection

    Foufi V, Lanteri S, Gaudet-Blavignac C, Remy P, Montet X, Lovis C. Automatic Annotation Tool to Support Supervised Machine Learning for Scaphoid Fracture Detection. Stud Health Technol Inform. 2018;255:210 –4

    work page 2018

Show all 15 references
  1. [9]

    Symbolic rule-based classification of lung cancer stages from free-text pathology reports

    Nguyen AN, Lawley MJ, Hansen DP, Bowman R V., Clarke BE, Duhig EE, et al. Symbolic rule-based classification of lung cancer stages from free-text pathology reports. J Am Med Informatics Assoc. 2010 Jul;17(4):440–5

    2010

  2. [10]

    Comparison of Natural Language Processing Techniques i n Analysis of Sparse Clinical Data: Insulin Decline by Patients

    Malmasi S, Ge W, Hosomura N, Turchin A. Comparison of Natural Language Processing Techniques i n Analysis of Sparse Clinical Data: Insulin Decline by Patients. AMIA Jt Summits Transl Sci proceedings AMIA Jt Summits Transl Sci. 2019;2019:610–9

    2019

  3. [11]

    Named Entity Recognition for Elect ronic Health Records: A Comparison of Rule-based and Machine Learning Approaches

    Gorinski PJ, Wu H, Grover C, Tobin R, Talbot C, Whalley H, et al. Named Entity Recognition for Elect ronic Health Records: A Comparison of Rule-based and Machine Learning Approaches. 2019;(2008):12–6. Available from: http://arxiv.org/abs/1903.03985

    2008 arXiv

  4. [12]

    Geevarghese R, Sigel C, Cadley J, Chatterjee S, Jain P, Hollingsworth A, et al. Extraction and classific ation of structured data from unstructured hepatobiliary pathology reports using large language models: A feasibility study compared with rules-based natural language proce...

    2024

  5. [13]

    Context-aware Self-Attention Networks for Natural Language Processing

    Yang B, Wang L, Wong DF, Shi S, Tu Z. Context-aware Self-Attention Networks for Natural Language Processing. Neurocomputing [Internet]. 2021;458:157–69. Available from: https://www.sciencedirect.com/science/article/pii/S0925231221009048

    2021

  6. [14]

    The More, the Better? Modalities of Metastatic Status Extraction on Free Medical Reports Based on Natural Language Processing

    Kempf E, Priou S, Cohen A, Redjdal A, Guével E, Tannier X. The More, the Better? Modalities of Metastatic Status Extraction on Free Medical Reports Based on Natural Language Processing. JCO Clin cancer informatics [Internet]. 2024 Aug [cited 2024 Oct 17];8(8). Available from: ...

    2024

  7. [15]

    How to improve cancer Patients ENrollment within clinical trials from rEal Life databases using the OMOP oncology Extension : the French PENELOPE initiative

    Kempf E, Vaterkowski M, Griffon N, Leprovost D, Breant S. How to improve cancer Patients ENrollment within clinical trials from rEal Life databases using the OMOP oncology Extension : the French PENELOPE initiative. AMIA 2022 Annu Symp. 2022;abstract N

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.