N-th Order Ergodic Multigram HMM for Modeling of Languages without Marked Word Boundaries 2 citations ? 0 self

Abstract:  rgodic llMMs have been successhilly used for modeling sentence production. However for some oriental languages such as Chinese, a word can consist of multiple characters without word boundary markers between adjacent words in a sentence. This makes word- segmentation on the [raining and testing data necessary before ergodie IIMM can be applied as the langnage model. This paper introduces the N-th order 13rgodic Mnltigram IIMM for language modeling of such languages. Each state of the IIMM can generate a variable number of characters corresponding to one word. The model can be trained without word- segmented and tagged corpus, and both segmentation and tagging are trained in one single model. Results on its application on a Chinese corpus are reported. 1 Motivation Statistical language modeling offers advantages including minimal domain specific knowledge and hand-written rules, trainability and sealability given a langnage corpus. Language models, such as N-gram class models (Brown et. al., 1992) and Ergodie Hidden Markov Models (Kuhn el, al., 1994) were proposed and used in applications such as syntactic class (POS) tagging for English (Cutting et al., 1992), clustering and scoring of recog- nizer sentence hypotheses.