Pith. sign in

REVIEW 1 cited by

A Stochastic Finite-State Word-Segmentation Algorithm for Chinese

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv cmp-lg/9405008 v2 pith:RXOZIYT3 submitted 1994-05-03 cmp-lg cs.CL

Richard Sproat (AT&T Bell Laboratories) , Chilin Shih (AT&T Bell Laboratories) , William Gale (AT&T Bell Laboratories) , Nancy Chang (Harvard University) This is my paper · ORCID
classification cmp-lgcs.CL
keywords chinesefinite-statemodelstochasticwordsaccountagreealgorithm
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We present a stochastic finite-state model for segmenting Chinese text into dictionary entries and productively derived words, and providing pronunciations for these words; the method incorporates a class-based model in its treatment of personal names. We also evaluate the system's performance, taking into account the fact that people often do not agree on a single segmentation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Entropy-Driven Pre-Tokenization for Byte-Pair Encoding

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Entropy-guided pre-tokenization, especially a PMI plus left/right entropy combination with lambda equal to 4, raises BPE word-boundary F1 from 49.30 to 58.73 on a 2,255-sentence PKU subset.

Pith tools