Generating Information Extraction Patterns from Overlapping and Variable Length Annotations using Sequence Alignment

Craig A. Morioka; Danne C. Elbers; Frank Meng

REVIEW 4 major objections 4 minor 10 references

Generating Information Extraction Patterns from Overlapping and Variable Length Annotations using Sequence Alignment

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Modified Smith-Waterman alignment over overlapping, variable-length annotation grids generates information-extraction patterns whose entity-level F1 on CoNLL-2003 is comparable to an established statistical NER system.

desk verdict A genuinely novel alignment-based pattern generator, but the headline NER comparison does not isolate the alignment method from priors and gazetteers — it needs an entity-level pattern-only ablation. read the letter →

arxiv 1908.03594 v2 pith:2YKJUX6J submitted 2019-08-09 cs.CL

Frank Meng , Craig A. Morioka , Danne C. Elbers This is my paper · ORCID

classification cs.CL

keywords sequencealignmentinformationextractionnamedentityrecognitionpatterngenerationSmith-Watermanannotationgridsvariable-lengthannotationscontextwindow

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that information-extraction patterns can be generated automatically by aligning sentences that contain the same type of target, using a sequence-alignment algorithm extended to handle overlapping and variable-length annotations. The alignment does double duty: it produces the patterns themselves and it discovers, for each target, the surrounding context that matters, so no fixed context window has to be chosen in advance. The paper evaluates this on the CoNLL-2003 named-entity task and reports entity-level F1 scores for person, organization, and location names that it judges comparable to an established CRF-based NER system. If this holds, pattern-based extraction, which is transparent and easy to inspect, can be built automatically from the same annotated data used by statistical systems.

What carries the argument

The machinery is a span-aware extension of Smith-Waterman approximate local alignment. Traditional alignment operates on one-dimensional sequences of fixed-length elements; here each matrix cell holds multiple scores, backtracking links, and spans that jump over matrix cells covered by variable-length elements, and elements beginning at the same position are matched pairwise so overlapping annotations are handled. The scoring schedule, with target match $100$, other match $1$, mismatch $-1$, and a high gap penalty, forces every alignment to include the target while preferring gaps to mismatches. Backtracking then yields the common context patterns and target patterns, with co-occurring elements expressed inside individual pattern elements.

What would settle it

Run the same pipeline on a second named-entity corpus from a different genre, using the same annotation types and scoring values. If entity-level F1 drops substantially relative to the reference CRF system while the top patterns remain dominated by genre-specific templates such as scores, parentheses, and country names, that would show the alignments are surfacing incidental structure rather than target-relevant context. A sharper version is already in the paper: a person-name pattern fired on the phrase 'Real Madrid (Spain)' in the test set, so a test corpus with a few such template clashes can directly probe whether the generated patterns are meaning-sensitive.

Watch

Extended reading notes

Core claim

The central claim is that pairwise local alignments of annotation grids can stand in for both feature selection and rule learning in information extraction. Instead of matching raw token strings, the extended Smith-Waterman algorithm aligns two two-dimensional grids in which any position can carry several overlapping annotations of different lengths, such as tokens, parts of speech, gazetteer lookups, and prior extractions. Aligning all sentences containing a target type produces a shared left context, target, and right context, and aligning target spans produces target patterns; the context window is an output of the alignment rather than a preset input. On CoNLL-2003, the paper reports entity-level F1 scores for persons, organizations, and locations that it treats as comparable to the reference CRF system, with prior probabilities adding recall on top of pattern-only extraction.

Load-bearing premise

The load-bearing premise is that aligning whole sentences that contain the same target type, with the paper's chosen match and gap scores, reliably surfaces the words that actually determine the target's meaning; if alignments instead capture incidental shared structure, the generated patterns will not generalize.

Editorial extensions

If this is right

Automatic pattern generation from annotated corpora is sufficient to build a competitive named-entity system, so pattern-based extraction does not require manual rule writing or task-specific feature engineering.
Because the context window is discovered by alignment rather than preset, patterns can adapt to variable syntactic and semantic distances between a target and its evidence.
The resulting patterns are inspectable and filterable: users can retain only high-precision pattern-target pairs and remove redundant longer patterns that are covered by shorter ones.
The same alignment machinery transfers to other annotation types and other extraction tasks, including relations between concepts, by changing the target annotations and the annotation grid.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: the method's reliance on semi-structured surface patterns, visible in the sports-result templates, suggests its advantage will be greatest when the training and test corpora share genre-level textual structure; out-of-genre transfer is a likely failure mode worth testing explicitly.
Inference: because the scoring weights are hand-set, an objective test would be to learn them from held-out data; if learned weights materially change F1, the current results partly measure the chosen scoring schedule rather than the alignment mechanism itself.
Inference: the same span-based alignment could be used to align two different annotation layers or two languages over the same text, which would let patterns be transferred across annotation schemes without retraining.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, the core idea is new and worth engaging with: extending Smith-Waterman to align two-dimensional grids of overlapping, variable-length annotations, and using those alignments to set context windows automatically instead of fixing a window size. That is a real extension beyond the authors' own 2015 lexical-pattern work and beyond ordinary sequence alignment. Second, the evaluation as written does not actually show that the alignment method is what makes the system competitive on CoNLL-2003. The entity-level F1 numbers in Table 2 are for the full system, which includes prior-probability labeling/removal and GATE gazetteer annotations. The only pattern-only result, 'Patterns, Token', is token-level, and token-level precision/recall can diverge badly from entity-level F1. So the abstract's claim that sequence alignments generate the patterns and context windows that drive performance is not directly supported by the reported numbers. The paper does several things well. The motivation for transparent, rule-based IE is clearly stated. The pattern examples in Tables 3–5 are informative, showing how multi-level annotations and co-occurring elements work in practice. The authors are honest about limitations: they note that the patterns are sensitive to semi-structured news contexts, and they explicitly say prior knowledge adds a non-trivial amount of recall. The train/dev/test split follows standard CoNLL-2003 practice, so there is no circularity. The internal logic of the method is coherent, and I don't see signs of sloppy thinking or citation stuffing. Now the soft spots, in proportion. The missing entity-level pattern-only ablation is the biggest one, and it is a missing-ablation problem rather than misconduct—but it matters, because the headline comparison to Stanford CRF may largely reflect the priors and gazetteer rather than the alignment-generated patterns. Additionally, the alignment algorithm is described only informally: no pseudocode, no correctness argument, no complexity analysis. There is no code or data release, which makes reproducibility hard to assess. The evaluation uses a single benchmark and no error bars or significance tests. Several parameters (match scores, gap penalty, precision threshold) are hand-tuned and not analyzed. None of these are deal-breakers for a method paper, but together they shift the paper from 'convincing demonstration' to 'promising work in progress.' Who is this for? People working on interpretive, rule-based IE, especially in domains like biomedicine where transparency matters. It is not a contribution to SOTA NER, and it does not claim to be. I would not cite it in my own work in the next year, but I would bring it to a reading group to discuss what counts as a fair baseline in pattern-based systems. The right path is to send it to peer review with a request for major revision: the reviewers should ask for an entity-level ablation with patterns alone, and the authors should provide at least a precise algorithmic description, ideally with complexity bounds. That would make the paper's central claim testable.

Referee Report

4 major / 4 minor

Summary. The paper proposes a pattern-based information extraction method in which an extended Smith-Waterman local alignment algorithm aligns two-dimensional grids of overlapping, variable-length annotations. From these alignments the system generates extraction patterns that mix raw tokens, part-of-speech tags, gazetteer lookups, and other annotation types, and it uses the alignments to derive a variable-size context window for each target rather than a fixed token window. The method is evaluated on CoNLL-2003 named entity recognition (NER) for person, organization, and location names, and the authors report entity-level F1 scores comparable to those of Stanford's CRF NER system.

Significance. If the method is validated, the contribution is valuable: it offers an interpretable, pattern-based alternative to statistical NER, with the appealing property that context widths are learned from data through sequence alignment rather than fixed in advance. The paper's strengths include a concrete evaluation on a standard benchmark, transparent example patterns in Tables 3-5, and a comparison with an established statistical system. However, the central validation is currently incomplete. The headline entity-level comparison is for the full system, which includes prior-probability labeling and gazetteer annotations, while the only pattern-only result is reported at the token level. As a result, the paper does not yet establish that the alignment-generated patterns alone are responsible for the reported entity-level performance. The algorithm description is also informal, which makes the core technical contribution difficult to reproduce or assess.

major comments (4)

[Section 4, Table 2] The central comparison with Stanford relies on the 'Entity' row, but that row reports the full system, which per Section 3.7 includes prior-probability labeling/removal and GATE gazetteer annotations. The only pattern-only evidence, the 'Patterns, Token' row, is token-level and is not comparable to the entity-level F1 scores used for the headline claim. Because token-level precision/recall can systematically differ from entity-level scores, the paper never reports entity-level F1 for the alignment-generated patterns alone. The claim that sequence alignments generate extraction patterns and context windows that perform comparably to Stanford is therefore not directly supported by the reported evaluation; an entity-level ablation separating patterns-only, patterns+priors, patterns+gazetteer, and the full system is needed.
[Section 3.2] The extended Smith-Waterman algorithm is described only informally. The recurrence for combining multiple scores, backtracking links, and spans at a cell containing overlapping elements is not written down; it is not specified how gap penalties apply when an element spans multiple cells; and the backtracking procedure is described in prose but not defined precisely enough to reimplement. No correctness argument or complexity analysis is provided for the extended algorithm. Since this algorithm is the paper's main technical contribution, the absence of a precise specification is a load-bearing gap that prevents verification and reproducibility.
[Sections 3.3 and 3.6] The method depends on several hand-set choices whose influence on the reported results is not analyzed: the match scores (target match 100, other match 1, mismatch -1, high gap penalty), the precision threshold for pattern retention (typically 0.95 or above), and the annotation-type weights mentioned in Section 5. No sensitivity analysis is reported, and the goodness metric M(p,t) is named but not formally defined, nor is it stated how precision is estimated on training examples. Because the method's central premise is that alignments with these scores identify the words and phrases that 'most directly impact' target meaning, the scoring choices need to be justified empirically or shown not to be decisive for the outcome.
[Section 3.5] The procedure for applying patterns to text is underspecified. The text says that 'both are aligned against the text' but does not state whether the matching is exact element-by-element matching or approximate alignment, what scoring is used during this application step, or how partial matches are handled. The requirement that rc appear after lc in token order is clear, but the semantics of 'valid alignment' is not. Since this step determines whether a candidate target is produced, the matching rule must be defined precisely for the method to be reproducible.

minor comments (4)

[Section 4, Table 2] The structure of Table 2 is difficult to parse: the Stanford row appears to contain only three numbers while the Entity row contains nine, and the column headers do not make clear which cell corresponds to PER, ORG, or LOC. The table needs explicit column labels and, ideally, the Stanford scores for each entity type.
[Section 4] The comparison with Stanford is based on single point estimates on one test set (test B). No error bars, bootstrap confidence intervals, or significance tests are reported, so the 'comparable' claim is not statistically supported even for the full system.
[Section 3.6] The filtering rule is ambiguous: the condition 'for every p\', len(p\') < len(p)' could mean that removal requires all shorter patterns to collectively cover the extracted targets, or that any shorter pattern suffices. The intended logic should be stated formally.
[Throughout] There are numerous typographical and formatting issues, including 'ta rgets', 'surrounding co ntext', 'articial', and inconsistent spacing around references and parentheses. A careful proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

Evaluation on held-out CoNLL-2003 test B; pattern generation is supervised on training data, so no derivation step reduces to its own input.

full rationale

No circular derivation chain is present. The paper's pipeline is a supervised pattern learner: annotations such as tokens, parts of speech, and gazetteer lookups are aligned to produce context and target patterns from training plus development data, and the resulting pattern-target pairs are applied to the held-out test B corpus. The entity-level F1 scores in Table 2 are measured against previously unseen gold labels, so the headline result is not constructed from the evaluation labels. The alignment scoring parameters (target match 100, other match 1, mismatch -1, high gap penalty) are hand-set modeling choices, not values fitted to the test set, and they do not define the reported F1 metric. The only self-citation, "Similar to Meng and Morioka (2015)" in Section 1, references the authors' earlier method for using sequence alignments to identify context; the present paper re-describes the algorithm in Section 3 and evaluates it independently, so the citation is not load-bearing and does not import an unverified uniqueness claim. The conflation of gazetteer annotations and prior probabilities with the alignment-generated patterns in the entity-level comparison is a missing-ablation or attribution concern, not a circularity, especially since the paper reports a pattern-only token-level row and explicitly discusses the prior-knowledge contribution to recall (Section 5). No equation or construction in the paper equates an output quantity with an input quantity by definition.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The method relies on several hand-set parameters, an untested algorithmic correctness assumption, and the premise that sentence-level alignments identify informative context. No new physical entities are introduced.

free parameters (5)

Target and other match scores plus gap penalty = 100, 1, -1, high penalty
Section 3.3 sets these scoring values by hand; they shape every generated pattern, and no sensitivity analysis is reported.
Precision threshold for pattern retention = 0.95 typical
Section 3.6 keeps pattern-target pairs whose precision exceeds a user-defined threshold, typically 0.95; no analysis of threshold choice is provided.
Prior probability cutoffs for label assignment and removal = not specified
Section 3.7 applies or removes labels based on high and low priors, but the exact cutoff values are not reported.
Annotation type weights = expert-assigned, not reported
Section 5 states annotation types are weighted according to expert assessment of importance; the weights are subjective and not listed.
Noun repetition rule = manual rule
Section 2 adds a hand-coded heuristic: a noun is likely a person name if other instances of the same noun in the document are person names.

assumptions (6)

domain assumption The extended Smith-Waterman alignment on 2D annotation grids is correct, including span-based backtracking.
The paper describes the algorithm but provides no proof of optimality, correctness, or complexity.
domain assumption The entire sentence containing a target is a reasonable general context.
Section 3.3 states this is reasonable in practice, with no systematic comparison of context choices.
ad hoc to paper Pairwise alignments of general contexts identify the words and phrases that most directly impact target meaning.
This is the central premise of the pattern-generation method and is not independently validated.
domain assumption Existing annotations from GATE are accurate enough to learn patterns from.
Section 3.1 notes annotations have error rates, but the method treats them as ground truth during pattern generation.
domain assumption Prior probabilities computed from training data transfer to the test distribution.
Standard supervised learning assumption, but the thresholds for applying and removing labels are unspecified.
ad hoc to paper Target patterns generated by aligning target grids are sufficient to validate candidate targets.
Section 3.5 says any successful alignment with a target pattern indicates a valid target, which assumes target patterns are accurate filters.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Generating Information Extraction Patterns from Overlapping and Variable Length Annotations using Sequence Alignment." pith.science (2026). https://pith.science/paper/2YKJUX6J

@misc{pith2026190803594,
  author       = {Pith},
  title        = {Pith review of: Generating Information Extraction Patterns from Overlapping and Variable Length Annotations using Sequence Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2YKJUX6J}},
  note         = {Machine review of arXiv:1908.03594}
}

read the original abstract

Sequence alignments are used to capture patterns composed of elements representing multiple conceptual levels through the alignment of sequences that contain overlapping and variable length annotations. The alignments also determine the proper context window of words and phrases that most directly impact the meaning of a given target within a sentence, eliminating the need to predefine a fixed context window of words surrounding the targets. We evaluated the system using the CoNLL-2003 named entity recognition (NER) task.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 10 canonical work pages

[8]

Ellen Riloff

Extracting Product Features and Opinions from Reviews, Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2005). Ellen Riloff . Automatically Generating Extraction Patterns from Untagged Text. 1996 . In Proceedings of the Thirteenth National Conference on Articial Intelligence, pages 1044–1049, Portland, OR, August. T...

work page 2005
[10]

Disease named entity recognition by combining conditional random fields and bidirectional recurrent neural networks, Database, Volume 2016, 1 January

work page 2016
[2003]

In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - Volume 4 (CONLL '03), Vol

Introduction to the CoNLL -2003 shared task: language - independent named entity recognition. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - Volume 4 (CONLL '03), Vol

work page 2003
[2005]

Proceedings of the 43nd Annual Meeting of the Association for Computational Linguistics (ACL 2005), pp

Incorporating Non -local Information into Information Extraction Systems by Gibbs Sampling. Proceedings of the 43nd Annual Meeting of the Association for Computational Linguistics (ACL 2005), pp. 363-370. Jun Gao, Ninghao Liu, Mark Lawley, and Xia Hu,

work page 2005
[2007]

In EMNLP -CoNLL 2007 - Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning

Effective information extraction with semantic affinity patterns and relevant regions. In EMNLP -CoNLL 2007 - Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. p. 717-727. Ana-Maria Popescu and Oren Etzioni

work page 2007
[2011]

Named Entity Recognition from Biomedical Text Using SVM," 2011 5th International Conference on Bioinformatics and Biomedical Engineering, Wuhan, pp. 1-4. Tian Kang, Shaodian Zhang, Youlan Tang, Gregory W. Hruby, Alexander Rusanov, Noemie Elhadad, Chunhua Weng. EliIE: An open-source information extraction system for clinical trial eligibility criteria. J A...

work page 2011
[2012]

Lexical patterns, features and knowledge resources for coreference resolution in clinical notes. J. of Biomedical Informatics 45, 5 (October 2012), 901-912. Ralph Grishman, Beth Sundheim

work page 2012
[2015]

Zhenfei Ju, Jian Wang and Fei Zhu

ICE: Rapid Information Extraction Customization for NLP Novices, Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, Denver, Colorado, Association for Computational Linguistics, 31—35. Zhenfei Ju, Jian Wang and Fei Zhu

work page 2015

Show all 10 references

[2016]

Jenny Rose Finkel, Trond Gr enager, and Christopher Manning

Overview of Linguistic Resources for the TAC KBP 2016 Evaluations: Methodologies and Results, Proceedings of the Ninth Text Analysis Conference (TAC). Jenny Rose Finkel, Trond Gr enager, and Christopher Manning

2016
[2017]

2017, Article ID 2460174, 12 pages

An Interpretable Classification Framework for Information Extraction from Online Healthcare Forums, Journal of Healthcare Engineering, vol. 2017, Article ID 2460174, 12 pages. Phil Gooch and Abdul Roudsari

2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [8]

Ellen Riloff

Extracting Product Features and Opinions from Reviews, Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2005). Ellen Riloff . Automatically Generating Extraction Patterns from Untagged Text. 1996 . In Proceedings of the Thirteenth National Conference on Articial Intelligence, pages 1044–1049, Portland, OR, August. T...

work page 2005

[2] [10]

Disease named entity recognition by combining conditional random fields and bidirectional recurrent neural networks, Database, Volume 2016, 1 January

work page 2016

[3] [2003]

In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - Volume 4 (CONLL '03), Vol

Introduction to the CoNLL -2003 shared task: language - independent named entity recognition. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - Volume 4 (CONLL '03), Vol

work page 2003

[4] [2005]

Proceedings of the 43nd Annual Meeting of the Association for Computational Linguistics (ACL 2005), pp

Incorporating Non -local Information into Information Extraction Systems by Gibbs Sampling. Proceedings of the 43nd Annual Meeting of the Association for Computational Linguistics (ACL 2005), pp. 363-370. Jun Gao, Ninghao Liu, Mark Lawley, and Xia Hu,

work page 2005

[5] [2007]

In EMNLP -CoNLL 2007 - Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning

Effective information extraction with semantic affinity patterns and relevant regions. In EMNLP -CoNLL 2007 - Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. p. 717-727. Ana-Maria Popescu and Oren Etzioni

work page 2007

[6] [2011]

Named Entity Recognition from Biomedical Text Using SVM," 2011 5th International Conference on Bioinformatics and Biomedical Engineering, Wuhan, pp. 1-4. Tian Kang, Shaodian Zhang, Youlan Tang, Gregory W. Hruby, Alexander Rusanov, Noemie Elhadad, Chunhua Weng. EliIE: An open-source information extraction system for clinical trial eligibility criteria. J A...

work page 2011

[7] [2012]

Lexical patterns, features and knowledge resources for coreference resolution in clinical notes. J. of Biomedical Informatics 45, 5 (October 2012), 901-912. Ralph Grishman, Beth Sundheim

work page 2012

[8] [2015]

Zhenfei Ju, Jian Wang and Fei Zhu

ICE: Rapid Information Extraction Customization for NLP Novices, Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, Denver, Colorado, Association for Computational Linguistics, 31—35. Zhenfei Ju, Jian Wang and Fei Zhu

work page 2015