Overview of Chinese Word Segmentation Method

2013 ◽  
Vol 427-429 ◽  
pp. 2568-2571
Author(s):  
Shu Xian Liu ◽  
Xiao Hua Li
Keyword(s):  
Information Processing ◽  
Natural Language Processing ◽  
Language Processing ◽  
Word Segmentation ◽  
Basic Knowledge ◽  
Chinese Characters ◽  
Chinese Word ◽  
Segmentation Method ◽  
Chinese Word Segmentation ◽  
Chinese Information Processing

This article provides a brief introduction to Natural Language Processing and basic knowledge of Chinese Word Segmentation at first. Chinese Word Segmentation is a process of turning a series of Chinese characters into a series of Chinese words with some rules. As the fundamental component of Chinese information processing, it is wildly used in correlative areas. Accordingly, research on Chinese Word Segmentation has important theoretic and realistic meaning. In this paper, we mainly introduces the challenge in Chinese Word Segmentation, and presented the categories of Chinese Word Segmentation method.

The Review of Automatic Chinese Word Segmentation

2014 ◽  
Vol 701-702 ◽  
pp. 386-389 ◽  
Author(s):  
Xiao Yan Ren ◽  
Yun Xia Fu
Keyword(s):  
Information Processing ◽  
Formal Model ◽  
Word Segmentation ◽  
Chinese Word ◽  
Chinese Word Segmentation ◽  
Research Status ◽  
Chinese Information Processing ◽  
Great Progress

As the fundamental work of Chinese information processing, Chinese word segmentation has achieved great progress since its birth. This paper reviews the research status of the CWS, discusses the formal model of automatic word segmentation, and analyzes the difficulties of word segmentation.

Splitting-merging model of Chinese word tokenization and segmentation

1998 ◽  
Vol 4 (4) ◽  
pp. 309-324 ◽  
Author(s):  
YUAN YAO ◽  
KIM TEN LUA
Keyword(s):  
Natural Language Processing ◽  
Mutual Information ◽  
Natural Language ◽  
Language Processing ◽  
Word Segmentation ◽  
Chinese Characters ◽  
Chinese Word ◽  
Blank Space ◽  
New Words ◽  
Segmentation Methods

Currently, word tokenization and segmentation are still a hot topic in natural language processing, especially for languages like Chinese in which there is no blank space for word delimitation. Three major problems are faced: (1) tokenizing direction and efficiency; (2) insufficient tokenization dictionary and new words; and (3) ambiguity of tokenization and segmentation. Most existing tokenization and segmentation methods have not dealt with the above problems together. To tackle the three problems in one basket, this paper presents a novel dictionary-based method called the Splitting-Merging Model (SMM) for Chinese word tokenization and segmentation. It uses the mutual information of Chinese characters to find the boundaries and the non-boundaries of Chinese words, and finally leads to a word segmentation by resolving ambiguities and detecting new words.

Identification of Chinese Unknown Word Based on Finite Multi-List Method

2011 ◽  
Vol 474-476 ◽  
pp. 460-465
Author(s):  
Bo Sun ◽  
Sheng Hui Huang ◽  
Xiao Hua Liu
Keyword(s):  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Word Identification ◽  
Difficult Problem ◽  
Word Segmentation ◽  
Chinese Word ◽  
Unknown Word ◽  
List Method ◽  
Low Rate

Unknown word is a kind of word that is not included in the sub_word vocabulary, but must be cut out by the word segmentation program. Peoples’ names, place names and translated names are the major unknown words.Unknown Chinese words is a difficult problem in natural language processing, and also contributed to the low rate of correct segmention. This paper introduces the finite multi-list method that using the word fragments’ capability to composite a word and the location in the word tree to process the unknown Chinese words.The experiment recall is 70.67% ,the correct rate is 43.65% .The result of the experiment shows that unknown Chinese word identification based on the finite multi-list method is feasible.

A cyclic self-learning Chinese word segmentation for the geoscience domain

GEOMATICA ◽  
2020 ◽  
Author(s):  
Qinjun Qiu ◽  
Zhong Xie ◽  
Liang Wu
Keyword(s):  
Language Processing ◽  
Domain Knowledge ◽  
Short Term Memory ◽  
Word Segmentation ◽  
Training Data ◽  
Local Context ◽  
Chinese Word ◽  
Chinese Word Segmentation ◽  
Benchmark Datasets ◽  
Self Learning

Unlike English and other western languages, Chinese does not delimit words using white-spaces. Chinese Word Segmentation (CWS) is the crucial first step towards natural language processing. However, for the geoscience subject domain, the CWS problem remains unresolved with many challenges. Although traditional methods can be used to process geoscience documents, they lack the domain knowledge for massive geoscience documents. Considering the above challenges, this motivated us to build a segmenter specifically for the geoscience domain. Currently, most of the state-of-the-art methods for Chinese word segmentation are based on supervised learning, whose features are mostly extracted from a local context. In this paper, we proposed a framework for sequence learning by incorporating cyclic self-learning corpus training. Following this framework, we build the GeoSegmenter based on the Bi-directional Long Short-Term Memory (Bi-LSTM) network model to perform Chinese word segmentation. It can gain a great advantage through iterations of the training data. Empirical experimental results on geoscience documents and benchmark datasets showed that geological documents can be identified, and it can also recognize the generic documents.

Research on Sustainable Mining Engineering

2013 ◽  
Vol 340 ◽  
pp. 126-130 ◽  
Author(s):  
Xiao Guang Yue ◽  
Guang Zhang ◽  
Qing Guo Ren ◽  
Wen Cheng Liao ◽  
Jing Xi Chen ◽  
...  
Keyword(s):  
Information Processing ◽  
Natural Language Processing ◽  
Natural Language ◽  
Language Processing ◽  
Social Economy ◽  
Scientific Research ◽  
The Other ◽  
Mining Engineering ◽  
Chinese Information Processing ◽  
Development Tendency

The concepts of Chinese information processing and natural language processing (NLP) and their development tendency are summarized. There are different comprehension of Chinese information processing and natural language processing in China and the other countries. But the work appears to emerge in the study of key point of languages processing. Mining engineering is very important for our country. Though the final task of languages processing is difficult, Chinese information processing has contributed substantially to our scientific research and social economy and it will play an important part for mining engineering in our future.

The Critical Technology Development Status of Machine Translation

2013 ◽  
Vol 791-793 ◽  
pp. 1622-1625
Author(s):  
Dan Han ◽  
Zhi Han Yu
Keyword(s):  
Machine Translation ◽  
Technology Development ◽  
Statistical Machine Translation ◽  
Word Segmentation ◽  
Translation System ◽  
Chinese Word ◽  
Segmentation Method ◽  
Chinese Word Segmentation ◽  
Critical Technology ◽  
Translation Machine

In this article, we mainly introduce some basic concepts about machine translation. Machine translation means translating a natural language text to another by software. It can be divided into two categories: rule-based and corpus-based. IBM's statistical machine translation, Microsoft's multi-language machine translation project, AT & T's voice translation system and CMUs PANGLOSS system are three typical machine translation systems. Due to sentences are constructed by words continuously in Chinese. Chinese word segmentation is very essential. Three methods of Chinese word segmentation: segmentation methods based on string matching, segmentation method based on the understanding and segmentation method based on the statistics.

scholarly journals Chinese Word Segmentation and Named Entity Recognition: A Pragmatic Approach

2005 ◽  
Vol 31 (4) ◽  
pp. 531-574 ◽  
Author(s):  
Jianfeng Gao ◽  
Mu Li ◽  
Chang-Ning Huang ◽  
Andi Wu
Keyword(s):  
Language Processing ◽  
Named Entity Recognition ◽  
Word Segmentation ◽  
Computer Applications ◽  
Entity Recognition ◽  
Chinese Word ◽  
Chinese Word Segmentation ◽  
Pragmatic Approach ◽  
Named Entity ◽  
Test Sets

This article presents a pragmatic approach to Chinese word segmentation. It differs from most previous approaches mainly in three respects. First, while theoretical linguists have defined Chinese words using various linguistic criteria, Chinese words in this study are defined pragmatically as segmentation units whose definition depends on how they are used and processed in realistic computer applications. Second, we propose a pragmatic mathematical framework in which segmenting known words and detecting unknown words of different types (i.e., morphologically derived words, factoids, named entities, and other unlisted words) can be performed simultaneously in a unified way. These tasks are usually conducted separately in other systems. Finally, we do not assume the existence of a universal word segmentation standard that is application-independent. Instead, we argue for the necessity of multiple segmentation standards due to the pragmatic fact that different natural language processing applications might require different granularities of Chinese words. These pragmatic approaches have been implemented in an adaptive Chinese word segmenter, called MSRSeg, which will be described in detail. It consists of two components: (1) a generic segmenter that is based on the framework of linear mixture models and provides a unified approach to the five fundamental features of word-level Chinese language processing: lexicon word processing, morphological analysis, factoid detection, named entity recognition, and new word identification; and (2) a set of output adaptors for adapting the output of (1) to different application-specific standards. Evaluation on five test sets with different standards shows that the adaptive system achieves state-of-the-art performance on all the test sets.

Sign in / Sign up

Export Citation Format

Share Document