Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models

Bo Du; Hai Zhao; Lefei Zhang; Liqi He; Zihong Zhang; Zuchao Li

REVIEW 3 major objections 6 minor 50 references

Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs beat prior Chinese word segmentation records with simple prompts.

desk verdict The LLACA distillation pipeline is a solid, useful engineering contribution, but the comprehension and SOTA claims rest on an untested pretraining-leakage assumption. read the letter →

arxiv 2505.19631 v1 pith:KQEYSGJE submitted 2025-05-26 cs.CL cs.AI

Zihong Zhang , Liqi He , Zuchao Li , Lefei Zhang , Hai Zhao , Bo Du This is my paper · ORCID

classification cs.CLcs.AI

keywords wordsegmentationlargelanguagemodelsunsupervisedlearningAho-CorasickautomatonChinesevariablen-grammodelpromptfollowingmultilingualNLP

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that word segmentation should be done by understanding first: feed raw text to an LLM with a simple prompt, let the model split it into words using context, then treat the output as a measure of the model's comprehension. It reports that mainstream chatbots, especially the Chinese model Qwen1.5-7B-Chat, follow this instruction across Chinese, Japanese, Korean, and Thai, and that this model already beats all earlier unsupervised segmenters on the MSR and PKU Chinese benchmarks. To make the approach practical, the paper introduces LLACA, an Aho-Corasick automaton whose word list and probabilities come from LLM segmentations rather than human-annotated dictionaries, paired with a variable n-gram model and Viterbi decoding. If the claim holds, word segmentation becomes both a fine-grained probe of LLM understanding and a fast unsupervised segmenter that surpasses traditional methods.

What carries the argument

LLACA is an Aho-Corasick automaton: a trie-based multi-pattern matcher where every node is a possible word or prefix, each recognized word is a weighted edge, and failure links allow linear-time scanning. The automaton's vocabulary comes from LLM segmentations of the raw corpus, filtered by pointwise mutual information to drop hallucinated strings, and each word's probability is modeled as its count divided by the sum of counts over all words sharing the longest prefix that is itself a final state, which yields a context-sensitive variable n-gram model. Viterbi decoding over the directed acyclic graph of recognized patterns then chooses the highest-probability segmentation for each sentence. This construction lets a fast automaton stand in for the slow LLM while retaining and regularizing the LLM's comprehension.

What would settle it

Use the same prompting and LLACA pipeline on a corpus released after the model's training cutoff, with newly produced human segmentations, and compare F-measures. If the fresh-corpus scores fall back to the level of older unsupervised methods or show no parameter-scaling trend, the claim that the benchmark results demonstrate comprehension rather than memorization is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLMs 'comprehend first, segment later': they can segment raw, boundaryless text into words by understanding the sentence, not by dictionary lookup or local statistics, and their segmentation quality tracks model scale. In the LLM-WS setup, a simple prompt asking the model to insert spaces yields high token F-measures, with Qwen1.5-7B-Chat reaching 84.2 and 86.7 on MSR and PKU respectively, exceeding previous unsupervised state-of-the-art scores in Chinese; larger Qwen1.5 models push further, and the proposed LLACA method raises LLM output to 87.7 and 88.9 on those datasets while cutting inference time from hours to seconds. The mechanism is that the LLM's segmentation carries latent semantic decisions, and the automaton's variable n-gram probability model preserves those decisions while filtering out hallucinations.

Load-bearing premise

The scores are interpreted as comprehension only if the standard benchmark corpora were not memorized during LLM pretraining; if those texts or their gold segmentations appear in training, the reported gains could be recall rather than understanding.

Editorial extensions

If this is right

If LLM segmentation reflects comprehension, word segmentation becomes a fine-grained, prompt-driven test of semantic understanding that works across languages without labeled training data.
LLACA preserves the gains of LLM-based segmentation while making inference orders of magnitude faster, making the approach usable in practical NLP pipelines.
Because the dictionary is built from raw text on demand, LLACA can adapt to new domains and handle out-of-vocabulary words better than static dictionary or supervised segmenters.
The measured consistency ceiling among Chinese segmenters is around 93.8% F-measure; the paper argues this, rather than the lowest consistency rate, is the realistic target for unsupervised methods, and LLACA approaches it.
Larger parameter LLMs produce fewer hallucinations and higher segmentation accuracy across languages, supporting scaling as the driver of comprehension-based segmentation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A decisive open question the paper leaves implicit is data leakage: if the benchmark texts or their gold segmentations appear in pretraining, the reported comprehension advantage could be memorization rather than understanding; a test on a corpus released after a model's knowledge cutoff would separate the two.
The same LLM-to-automaton recipe could be applied to other segmentation-adjacent tasks, such as subword induction for tokenizer design, morpheme discovery, or compound splitting, because it requires no language-specific labels.
The approach suggests a new diagnostic: the specific strings a small model mis-segments, after hallucinations are filtered, may pinpoint exactly which ambiguities require larger-scale comprehension.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, the LLACA pipeline is a real and useful engineering contribution: sample segmentations from an LLM, filter with PMI, build an AC automaton with variable n-gram probabilities, decode with Viterbi. It turns hours of LLM inference into seconds, and it consistently beats the raw LLM outputs and the classical unsupervised Chinese baselines in the paper. Code is out, and the perplexity comparison in Table 10 is a nice sanity check. Second, the interpretive story — that these results show LLMs 'comprehend first, segment later' and that LLACA is a new unsupervised SOTA — is not supported until the authors rule out pretraining leakage. That is the load-bearing issue, and it is not a minor one.

The leakage worry is concrete. MSR and PKU are from the 2005 SIGHAN Bakeoff, widely mirrored on the web. The paper says testing and sampling were done on the test sets (Section 4.1) and gives no membership or overlap analysis. Qwen1.5-7B's poor raw Thai F1 (8.9 on BEST) shows it is not a general segmenter; the high Chinese scores are exactly what you would expect from a model that saw those sentences or their boundaries during pretraining. That does not kill the LLACA engineering result — distilling LLM outputs into a fast segmenter is still useful — but it guts the 'comprehension' interpretation and the 'unsupervised' framing, since a model pretrained on labeled benchmarks is not an unsupervised segmenter in the relevant sense.

Other soft spots, in decreasing order of size. The SOTA claim overreaches: Table 1 has no unsupervised baselines for KWDLC, UD_JA, UD_KO, or UD_TH, and the BEST row shows PYHSMM at 82.1 while the best LLACA gets 68.9, contradicting the conclusion's 'surpassing previous benchmarks'. No error bars or variance over shuffles are reported, and Table 1 is one iteration. The top ratio is fixed a priori, so the method itself is not circular with gold labels — that is to their credit.

The Limitations section is honest about annotation mismatch, which helps. This paper deserves a serious referee, but the authors should be asked to run a contamination check (or at least report n-gram overlap between training data and the test sets) and to scale back the 'comprehend first' and SOTA claims to what the data actually shows. A revised version that does that would be a solid contribution; the current version is an interesting system paper with an unearned interpretation.

Referee Report

3 major / 6 minor

Summary. The paper proposes a framework called LLM-WS in which large language models segment raw text into words via simple prompts, and introduces LLACA, an unsupervised segmenter that distills LLM segmentation outputs into an Aho-Corasick automaton with a variable n-gram probability model. Experiments cover Chinese (AS, CITYU, MSR, PKU), Japanese (KWDLC, UD_JA), Korean (UD_KO), and Thai (BEST, UD_TH). The main empirical claims are that Qwen1.5-7B-Chat exceeds previous unsupervised state of the art on MSR and PKU, that larger models generally segment better, and that LLACA improves on direct LLM outputs while being far faster.

Significance. If the pretraining-leakage concern can be resolved, the Chinese MSR/PKU results (raw LLM F1 of 84.2/86.7 and LLACA F1 of 86.7/87.7) are a notable advance over the roughly 80 F1 of older unsupervised methods, and LLACA is a practical and efficient way to retain much of the LLM's segmentation ability without per-sentence inference cost. The paper's strengths include a public code release, a fixed a priori choice of top_ratio (0.99) rather than fitting to gold labels, and a systematic comparison across several languages and model sizes. The conceptual claim that word segmentation can probe LLM comprehension is interesting, but the evidence is currently too weak to support the strong 'comprehend first, segment later' interpretation without additional controls.

major comments (3)

[§4.1, §A.2, Table 1] The central claim that Qwen1.5-7B-Chat surpasses previous unsupervised state of the art on MSR and PKU depends on the assumption that the test sentences and gold segmentations were not present in the model's pretraining data. Section 4.1 states that sampling and testing were performed on the test sets, and Appendix A.2 states that only the test datasets were used for LLM-WS and testing. No contamination or membership analysis is reported. Because MSR and PKU are widely distributed SIGHAN Bakeoff datasets, they are plausible members of Qwen1.5's training corpus. Without a leakage check (e.g., exact-match and near-duplicate detection against pretraining data, or evaluation on newly collected text in the same domains), the raw LLM F1 scores could reflect memorization of benchmark segmentations rather than comprehension-based segmentation, and the 'unsupervised' framing is undermined.
[Table 1, §5] The conclusion that LLACA 'sets new state-of-the-art unsupervised results' is overbroad. On Thai BEST, LLACA-7B scores 48.6 F1, far below the previously reported PYHSMM score of 82.1 and SGB-A-12 score of 80.1 (even with the dagger note on comparability). On KWDLC, UD_JA, UD_KO, and UD_TH, no previous unsupervised baselines are listed, so the claimed superiority is not established for those datasets. The SOTA claim should be restricted to the languages and datasets where the comparison actually supports it, and the large gap on Thai should be discussed explicitly rather than subsumed in a general conclusion.
[§4.2, Table 2] The paper interprets the poor Thai raw scores of Qwen1.5-7B-Chat (BEST 8.9, UD_TH 21.3) versus GPT-4 (BEST 76.3, UD_TH 75.1) as a consequence of pretraining-language mismatch rather than a lack of comprehension. This interpretation, however, also weakens the 'comprehend first, segment later' generalization: if segmentation success is driven by language-specific pretraining exposure, then the high Chinese scores may be explained by domain-specific familiarity with Chinese benchmark text rather than by a general comprehension-based segmentation ability. A control experiment with a Chinese-pretrained model that is not exposed to WS benchmark test sets, or a comparison on newly collected Chinese text, is needed to distinguish these explanations.

minor comments (6)

[§3.2, Eq. (2)] The variable n-gram probability model is central to LLACA, but the notation in Equation 2 is garbled in the PDF: the denominator is rendered as a sum over 'prev(w) ∈ prefix(wi)' and the relationship between prev(w) and the sub-Trie counts is not formally defined. Please rewrite the equation and define the transition probability used in Algorithm 1.
[Table 1] No standard deviations or repeated-run results are reported. Since LLM sampling is stochastic and the datasets are shuffled into batches, reporting results from a single run makes it difficult to assess whether the observed differences among model sizes are meaningful; at least three seeds or prompt variants should be reported.
[§A.4] The top_ratio hyperparameter is fixed at 0.99, but no sensitivity analysis is provided. Given that the paper emphasizes the method's parameter choices, a small sweep over top_ratio values would strengthen the claim that the results are not sensitive to this choice.
[§4.3, Fig. 6] Figure 6 shows performance over several iterations, but the number of iterations and the update rule for the lexicon between iterations are not stated in the caption or the text; please clarify these details.
[Throughout] There are multiple typos and grammar errors, including 'comprehendion' (§4.4), 'integreted' (§4.3), 'probility' (§3.2), 'Trainig' (Table 7), and 'out proposed approach' (§4.3). A careful proofreading pass is needed.
[§4.4, Table 4] The OOV comparison with SLM relies on re-trained SLM results; please state the hyperparameters used for the SLM re-training and confirm that the Viterbi decoding and unknown-word handling are identical between the two methods, since these choices can affect F1.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity by construction; the leakage/contamination worry is an external validity risk, not a by-construction reduction.

full rationale

The derivation chain is not circular. LLM-WS is a frozen-model prompting protocol: raw test sentences plus a simple prompt are fed to the LLM, and the token F-measure is computed against gold annotations without fitting any parameter of the paper to those annotations. LLACA then builds its lexicon from the LLM's segmented outputs, filters with PMI (top ratio fixed a priori at 0.99 in Appendix A.4), estimates word probabilities with the prefix-conditioned count model in Equation 2, and decodes with Viterbi; this is an explicit distillation and re-ranking of LLM outputs, not a claim that the final segmentation is independent of the LLM input. The same unsegmented test corpus is used for lexicon induction and evaluation (Appendix A.2: 'We only used their test datasets for LLM-WS and testing'), but this is disclosed and is consistent with the re-trained SLM baseline in Section 4.4, and it does not inject gold labels into the pipeline. The consistency ceiling in Appendix C is derived from external supervised models, not from the proposed method, and no load-bearing self-citation or imported uniqueness theorem appears. The real threat to the central 'comprehend first, segment later' interpretation is data contamination: Section 4.1 and Appendix A.2 never test whether Qwen1.5's pretraining included the MSR/PKU test strings, so the raw LLM F1 scores (84.2/86.7) and the LLACA lexicon could in principle be derived from the evaluation distribution. That is a missing-support/validity concern, not a by-construction equivalence, so it does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central method introduces no new theoretical entities but relies on the LLM's pretrained knowledge as an external oracle. The key free parameters are the PMI retention ratio and iteration count, and the axioms are dominated by assumptions about what the LLM actually learned and whether the test data leaked into pretraining.

free parameters (3)

top_ratio = 0.99
Fraction of LLM-extracted words retained after PMI filtering; set conservatively to keep OOV candidates, but chosen by hand.
num_iterations = 4 (for Figure 6 and Table 3)
Number of LLACA refinement iterations used when reporting the final F1; the algorithm description does not fully specify the iterative procedure.
batch_size = sqrt(total sentences)
Prompt batching heuristic; not tuned but a free choice.

assumptions (4)

domain assumption LLM segmentations reflect semantic understanding rather than memorization of training data or surface statistics.
The entire 'comprehend first' interpretation rests on this, but it is not validated against leakage (Sections 1, 3.1).
domain assumption PMI filtering with top_ratio 0.99 rejects hallucinations while retaining valid words.
The filtering step is central to building a clean lexicon, yet no analysis of its precision/recall is provided (Section 3.2).
ad hoc to paper The evaluation datasets and their gold segmentations are absent from the LLMs' pretraining corpora.
This is an unstated and untested assumption; if false, the unsupervised and comprehension claims fail (Sections 3.1, 4.2).
standard math Aho-Corasick automaton and Viterbi decoding are correct and applicable as described.
Standard algorithms with well-known correctness, used as building blocks (Section 3.2).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models." pith.science (2026). https://pith.science/paper/KQEYSGJE

@misc{pith2026250519631,
  author       = {Pith},
  title        = {Pith review of: Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KQEYSGJE}},
  note         = {Machine review of arXiv:2505.19631}
}

abstract

Word segmentation stands as a cornerstone of Natural Language Processing (NLP). Based on the concept of "comprehend first, segment later", we propose a new framework to explore the limit of unsupervised word segmentation with Large Language Models (LLMs) and evaluate the semantic understanding capabilities of LLMs based on word segmentation. We employ current mainstream LLMs to perform word segmentation across multiple languages to assess LLMs' "comprehension". Our findings reveal that LLMs are capable of following simple prompts to segment raw text into words. There is a trend suggesting that models with more parameters tend to perform better on multiple languages. Additionally, we introduce a novel unsupervised method, termed LLACA ($\textbf{L}$arge $\textbf{L}$anguage Model-Inspired $\textbf{A}$ho-$\textbf{C}$orasick $\textbf{A}$utomaton). Leveraging the advanced pattern recognition capabilities of Aho-Corasick automata, LLACA innovatively combines these with the deep insights of well-pretrained LLMs. This approach not only enables the construction of a dynamic $n$-gram model that adjusts based on contextual information but also integrates the nuanced understanding of LLMs, offering significant improvements over traditional methods. Our source code is available at https://github.com/hkr04/LLACA

Figures

Figures reproduced from arXiv: 2505.19631 by the authors.

**Figure 2.** Evaluations on datasets MSR (Simplified Chinese), KWDLC (Japanese) and BEST (Thai). [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** The illustration of an Aho-Corasick automaton [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: The construction of LLACA. ods. ESA is a goodness-based method for unsupervised Chinese word segmentation, employing an iterative process with local maximum strategy (Wang et al., 2011). nVBE relies on Variation of Branching Entropy, enhancing performance through n…

**Figure 5.** Figure 5: A Directed Acyclic Graph (DAG) for sequence segmentation, starting from "BOS" (Beginning of Sentence) [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: The evolution across iterations for four methods based on LLM-WS, starting from a baseline of 86.8 for [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: The process of how "sherd" matches patterns "she", "he", and "her" by transitions on an AC automaton. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 38 canonical work pages

[1]

Alfred V Aho and Margaret J Corasick. 1975. Efficient string matching: an aid to bibliographic search. Communications of the ACM, 18(6):333--340

work page 1975
[2]

Galen Andrew. 2006. https://aclanthology.org/W06-1655 A hybrid M arkov/semi- M arkov conditional random field for sequence segmentation . In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 465--472, Sydney, Australia. Association for Computational Linguistics

work page 2006
[3]

J-I Aoe. 1989. An efficient implementation of static string pattern matching machines. IEEE Transactions on Software Engineering, 15(8):1010--1016

work page 1989
[4]

Miguel Arevalillo - Herr \' a ez, Pablo Arnau - Gonz \' a lez, and Naeem Ramzan. 2022. https://doi.org/10.1109/ACCESS.2022.3213061 On adapting the DIET architecture and the rasa conversational toolkit for the sentiment analysis task . IEEE Access , 10:107477--107487

arXiv 2022
[5]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

arXiv 2023
[6]

Deng Cai and Hai Zhao. 2016. https://doi.org/10.18653/v1/P16-1039 Neural word segmentation learning for C hinese . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 409--420, Berlin, Germany. Association for Computational Linguistics

work page doi:10.18653/v1/p16-1039 2016
[7]

Bob Carpenter. 2006. https://aclanthology.org/W06-0129/ Character language models for chinese word segmentation and named entity recognition . In Proceedings of the Fifth Workshop on Chinese Language Processing, SIGHAN@COLING/ACL 2006, Sydney, Australia, July 22-23, 2006, pages 169--172. Association for Computational Linguistics

work page 2006
[8]

Paulo Roberto Massa Cereda, Newton Kiyotaka Miura, and Jo \ a o Jos \' e Neto. 2018. https://doi.org/10.1016/J.PROCS.2018.04.164 Syntactic analysis of natural language sentences based on rewriting systems and adaptivity . In The 9th International Conference on Ambient Systems, Networks and Technologies (ANT 2018) / The 8th International Conference on Sust...

work page doi:10.1016/j.procs.2018.04.164 2018

Show all 50 references

[9]

Miaohong Chen, Baobao Chang, and Wenzhe Pei. 2014. https://doi.org/10.3115/V1/D14-1092 A joint model for unsupervised chinese word segmentation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Q...

2014 doi
[10]

Xinchi Chen, Xipeng Qiu, Chenxi Zhu, and Xuanjing Huang. 2015. https://doi.org/10.3115/v1/P15-1168 Gated recursive neural network for C hinese word segmentation . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Internation...

2015 doi
[11]

Thomas Emerson. 2005. The second international chinese word segmentation bakeoff. In Proceedings of the fourth SIGHAN workshop on Chinese language Processing

2005
[12]

Sharon Goldwater, Thomas L Griffiths, and Mark Johnson. 2009. A bayesian framework for word segmentation: Exploring the effects of context. Cognition, 112(1):21--54

2009
[13]

Masatsugu Hangyo, Daisuke Kawahara, and Sadao Kurohashi. 2012. https://aclanthology.org/Y12-1058 Building a diverse document leads corpus annotated with semantic relations . In Proceedings of the 26th Pacific Asia Conference on Language, Information, and Computation, pages 535...

2012
[14]

Liqi He, Zuchao Li, Xiantao Cai, and Ping Wang. 2024. Multi-modal latent space learning for chain-of-thought reasoning in language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18180--18187

2024
[15]

Changning Huang and Hai Zhao. 2007. Chinese word segmentation: A decade review. Journal of Chinese Information Processing, 21(3):8--20

2007
[16]

Hongjiang Jing, Zuchao Li, Hai Zhao, and Shu Jiang. 2021. Seeking common but distinguishing difference, a joint aspect-based sentiment analysis model. arXiv preprint arXiv:2111.09634

2021 arXiv
[17]

Dan Jurafsky and James H Martin. 2014. Speech and language processing. vol. 3

2014
[18]

Donald E Knuth, James H Morris, Jr, and Vaughan R Pratt. 1977. Fast pattern matching in strings. SIAM journal on computing, 6(2):323--350

1977
[19]

Krit Kosawat, Monthika Boriboon, Patcharika Chootrakool, Ananlada Chotimongkol, Supon Klaithin, Sarawoot Kongyoung, Kanyanut Kriengket, Sitthaa Phaholphinyo, Sumonmas Purodakananda, Tipraporn Thanakulwarapas, et al. 2009. Best 2009: Thai word segmentation software contest. In ...

2009
[20]

Lafferty, Andrew McCallum, and Fernando C

John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williams...

2001
[21]

Zuchao Li, Shexia He, Jiaxun Cai, Zhuosheng Zhang, Hai Zhao, Gongshen Liu, Linlin Li, and Luo Si. 2018. A unified syntax-aware framework for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2401--2411

2018
[22]

Zuchao Li, Rui Wang, Kehai Chen, Masao Utiyama, Eiichiro Sumita, Zhuosheng Zhang, and Hai Zhao. 2020. Explicit sentence compression for neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8311--8318

2020
[23]

Pierre Magistry and Beno \^ t Sagot. 2012. https://aclanthology.org/P12-2075 Unsupervized word segmentation: the case for M andarin C hinese . In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 383--387, J...

2012
[24]

Udi Manber and Gene Myers. 1993. Suffix arrays: a new method for on-line string searches. siam Journal on Computing, 22(5):935--948

1993
[25]

Andrew McCallum, Dayne Freitag, and Fernando C. N. Pereira. 2000. Maximum entropy markov models for information extraction and segmentation. In Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June...

2000
[26]

Daichi Mochihashi, Takeshi Yamada, and Naonori Ueda. 2009. https://aclanthology.org/P09-1012 B ayesian unsupervised word segmentation with nested P itman- Y or language modeling . In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th Internat...

2009
[27]

Yasmin Moslem. 2024. https://doi.org/10.48550/ARXIV.2401.14559 Language modelling approaches to adaptive machine translation . CoRR, abs/2401.14559

2024 doi arXiv
[28]

o khan T \

Mahdi Namazifar, Alexandros Papangelis, G \" o khan T \" u r, and Dilek Hakkani - T \" u r. 2021. https://doi.org/10.1109/ICASSP39728.2021.9413810 Language model is all you need: Natural language understanding as question answering . In IEEE International Conference on Acousti...

2021
[29]

Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Jan Haji c , Christopher D Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal dependencies v2: An evergrowing multilingual treebank collection. arXiv preprint arXiv:2004.10643

2020 arXiv
[30]

Fuchun Peng, Fangfang Feng, and Andrew McCallum. 2004. https://aclanthology.org/C04-1081 C hinese segmentation and new word detection using conditional random fields . In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics , pages 562--56...

2004
[31]

Ponte and W

Jay M. Ponte and W. Bruce Croft. 2017. https://doi.org/10.1145/3130348.3130368 A language modeling approach to information retrieval . SIGIR Forum , 51(2):202--208

2017
[32]

Xu Sun, Houfeng Wang, and Wenjie Li. 2012. https://aclanthology.org/P12-1027/ Fast online training with frequency-adaptive learning rates for chinese word segmentation and new word detection . In The 50th Annual Meeting of the Association for Computational Linguistics, Proceed...

2012
[33]

Xu Sun, Yao - zhong Zhang, Takuya Matsuzaki, Yoshimasa Tsuruoka, and Jun'ichi Tsujii. 2009. https://aclanthology.org/N09-1007/ A discriminative latent variable chinese segmenter with hybrid word/character information . In Human Language Technologies: Conference of the North Am...

2009
[34]

Zhiqing Sun and Zhi - Hong Deng. 2018. https://aclanthology.org/D18-1531/ Unsupervised neural word segmentation for chinese via segmental language modeling . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 3...

2018
[35]

Jordan, Matthew J

Yee Whye Teh, Michael I. Jordan, Matthew J. Beal, and David M. Blei. 2004. https://proceedings.neurips.cc/paper/2004/hash/fb4ab556bc42d6f0ee0f9e24ec4d1af0-Abstract.html Sharing clusters among related groups: Hierarchical dirichlet processes . In Advances in Neural Information ...

2004
[36]

Kei Uchiumi, Hiroshi Tsukahara, and Daichi Mochihashi. 2015. https://doi.org/10.3115/V1/P15-1171 Inducing word and part-of-speech with pitman-yor hidden semi-markov models . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th ...

2015 doi
[37]

Andrew Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE transactions on Information Theory, 13(2):260--269

1967
[38]

Chong Wang, Yining Wang, Po - Sen Huang, Abdelrahman Mohamed, Dengyong Zhou, and Li Deng. 2017. http://proceedings.mlr.press/v70/wang17j.html Sequence modeling via segmentations . In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, ...

2017
[39]

Chunqi Wang and Bo Xu. 2017. https://aclanthology.org/I17-1017 Convolutional neural network with word embeddings for C hinese word segmentation . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 163--172,...

2017
[40]

Hanshi Wang, Jian Zhu, Shiping Tang, and Xiaozhong Fan. 2011. https://doi.org/10.1162/COLI\_A\_00058 A new unsupervised approach to word segmentation . Comput. Linguistics, 37(3):421--454

2011 doi
[41]

Lihao Wang and Xiaoqing Zheng. 2022. Unsupervised word segmentation with bi-directional neural language model. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(1):1--16

2022
[42]

Xingyu Yan, Xiaofan Xiong, Xiufeng Cheng, Yujing Huang, Haitao Zhu, and Fang Hu. 2021. https://doi.org/10.1016/J.COMPELECENG.2021.107354 Hmm-bimm: Hidden markov model-based word segmentation via improved bi-directional maximal matching algorithm . Comput. Electr. Eng., 94:107354

2021
[43]

Yao Yao, Zuchao Li, and Hai Zhao. 2023. Beyond chain-of-thought, effective graph-of-thought reasoning in language models. arXiv preprint arXiv:2305.16582

2023 arXiv
[44]

Yushi Yao and Zheng Huang. 2016. https://doi.org/10.1007/978-3-319-46681-1\_42 Bi-directional LSTM recurrent neural network for chinese word segmentation . In Neural Information Processing - 23rd International Conference, ICONIP 2016, Kyoto, Japan, October 16-21, 2016, Proceed...

2016 doi
[45]

Zhuosheng Zhang, Hai Zhao, Kangwei Ling, Jiangtong Li, Zuchao Li, Shexia He, and Guohong Fu. 2019. Effective subword segmentation for text comprehension. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(11):1664--1674

2019
[46]

Hai Zhao, Deng Cai, Changning Huang, and Chunyu Kit. 2019. https://arxiv.org/abs/1901.06079 Chinese word segmentation: Another decade review (2007-2017) . CoRR, abs/1901.06079

2019 arXiv
[47]

Hai Zhao, Chang-Ning Huang, and Mu Li. 2006. https://aclanthology.org/W06-0127 An improved C hinese word segmentation system with conditional random field . In Proceedings of the Fifth SIGHAN Workshop on C hinese Language Processing , pages 162--165, Sydney, Australia. Associa...

2006
[48]

Hai Zhao and Chunyu Kit. 2008. An empirical comparison of goodness measures for unsupervised chinese word segmentation with a unified framework. In Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-I

2008
[49]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[50]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Alfred V Aho and Margaret J Corasick. 1975. Efficient string matching: an aid to bibliographic search. Communications of the ACM, 18(6):333--340

work page 1975

[2] [2]

Galen Andrew. 2006. https://aclanthology.org/W06-1655 A hybrid M arkov/semi- M arkov conditional random field for sequence segmentation . In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 465--472, Sydney, Australia. Association for Computational Linguistics

work page 2006

[3] [3]

J-I Aoe. 1989. An efficient implementation of static string pattern matching machines. IEEE Transactions on Software Engineering, 15(8):1010--1016

work page 1989

[4] [4]

Miguel Arevalillo - Herr \' a ez, Pablo Arnau - Gonz \' a lez, and Naeem Ramzan. 2022. https://doi.org/10.1109/ACCESS.2022.3213061 On adapting the DIET architecture and the rasa conversational toolkit for the sentiment analysis task . IEEE Access , 10:107477--107487

arXiv 2022

[5] [5]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

arXiv 2023

[6] [6]

Deng Cai and Hai Zhao. 2016. https://doi.org/10.18653/v1/P16-1039 Neural word segmentation learning for C hinese . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 409--420, Berlin, Germany. Association for Computational Linguistics

work page doi:10.18653/v1/p16-1039 2016

[7] [7]

Bob Carpenter. 2006. https://aclanthology.org/W06-0129/ Character language models for chinese word segmentation and named entity recognition . In Proceedings of the Fifth Workshop on Chinese Language Processing, SIGHAN@COLING/ACL 2006, Sydney, Australia, July 22-23, 2006, pages 169--172. Association for Computational Linguistics

work page 2006

[8] [8]

Paulo Roberto Massa Cereda, Newton Kiyotaka Miura, and Jo \ a o Jos \' e Neto. 2018. https://doi.org/10.1016/J.PROCS.2018.04.164 Syntactic analysis of natural language sentences based on rewriting systems and adaptivity . In The 9th International Conference on Ambient Systems, Networks and Technologies (ANT 2018) / The 8th International Conference on Sust...

work page doi:10.1016/j.procs.2018.04.164 2018