1、 Reference number ISO 24614-2:2011(E) ISO 2011INTERNATIONAL STANDARD ISO 24614-2 First edition 2011-09-01 Language resource management Word segmentation of written texts Part 2: Word segmentation for Chinese, Japanese and Korean Gestion des ressources langagires Segmentation des mots dans les textes
2、 crits Partie 2: Segmentation des mots pour le chinois, le japonais et le coren ISO 24614-2:2011(E) COPYRIGHT PROTECTED DOCUMENT ISO 2011 All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic or mechanical,
3、including photocopying and microfilm, without permission in writing from either ISO at the address below or ISOs member body in the country of the requester. ISO copyright office Case postale 56 CH-1211 Geneva 20 Tel. + 41 22 749 01 11 Fax + 41 22 749 09 47 E-mail copyrightiso.org Web www.iso.org Pu
4、blished in Switzerland ii ISO 2011 All rights reservedISO 24614-2:2011(E) ISO 2011 All rights reserved iiiContents Page Foreword . v Introduction vi 1 Scope 1 2 Normative references 1 3 Terms and definitions . 2 4 Overview . 4 4.1 Introduction 4 4.2 Markup convention 4 4.3 Review of the concept of w
5、ord segmentation unit 5 4.4 Features common to Chinese, Japanese and Korean . 5 5 General rules for identifying WSUs in Chinese, Japanese and Korean 6 5.1 Words 6 5.2 Derivationally formed words 6 5.3 Word compounds 7 5.4 Phrasal compounds 8 5.5 Idioms . 8 5.6 Fixed expressions . 9 5.7 Abbreviations
6、 . 10 5.8 Transliterated loanwords 10 5.9 Strings of foreign or special characters . 11 5.10 Components of a WSU 11 6 Specific rules for identifying WSUs in Chinese . 12 6.1 Lexical items followed by the suffix 儿(r) 12 6.2 Lexical items 12 6.2.1 Nouns 12 6.2.2 Verbs . 17 6.2.3 Adjectives . 20 6.2.4
7、Pronouns 22 6.2.5 Numerals 23 6.2.6 Measure words 25 6.2.7 Adverbs 25 6.2.8 Prepositions . 26 6.2.9 Conjunctions 26 6.2.10 Auxiliary words 26 6.2.11 Modal words . 27 6.2.12 Exclamations . 27 6.2.13 Imitative words 27 7 Specific rules for identifying WSUs in Japanese text . 27 7.1 Bunsetsus 27 7.2 Le
8、xical items 27 7.2.1 General rule 27 7.2.2 Nouns 28 7.2.3 Verbs . 32 7.2.4 Adjectives . 33 7.2.5 Adnouns . 34 7.2.6 Adverbs 34 7.2.7 Conjunctions 35 7.2.8 Exclamations . 35 ISO 24614-2:2011(E) iv ISO 2011 All rights reserved7.2.9 Particles 35 7.2.10 Auxiliary verbs .35 8 Specific rules for identifyi
9、ng WSUs in Korean text 36 8.1 Eojeols 36 8.2 Lexical items 36 8.2.1 General rule 36 8.2.2 Nouns 37 8.2.3 Pronouns 38 8.2.4 Numerals .39 8.2.5 Verbs .39 8.2.6 Adjectives .39 8.2.7 Adnouns .40 8.2.8 Adverbs .40 8.2.9 Exclamations 40 8.3 Grammatical affixes .40 Annex A (informative) Comparative table o
10、f parts of speech in Chinese, Japanese and Korean 42 Bibliography 43 ISO 24614-2:2011(E) ISO 2011 All rights reserved vForeword ISO (the International Organization for Standardization) is a worldwide federation of national standards bodies (ISO member bodies). The work of preparing International Sta
11、ndards is normally carried out through ISO technical committees. Each member body interested in a subject for which a technical committee has been established has the right to be represented on that committee. International organizations, governmental and non-governmental, in liaison with ISO, also
12、take part in the work. ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of electrotechnical standardization. International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2. The main task of technical committees
13、 is to prepare International Standards. Draft International Standards adopted by the technical committees are circulated to the member bodies for voting. Publication as an International Standard requires approval by at least 75 % of the member bodies casting a vote. Attention is drawn to the possibi
14、lity that some of the elements of this document may be the subject of patent rights. ISO shall not be held responsible for identifying any or all such patent rights. ISO 24614-2 was prepared by Technical Committee ISO/TC 37, Terminology and other language and content resources, Subcommittee SC 4, La
15、nguage resource management. ISO 24614 consists of the following parts, under the general title Language resource management Word segmentation of written texts: Part 1: Basic concepts and general principles Part 2: Word segmentation for Chinese, Japanese and Korean ISO 24614-2:2011(E) vi ISO 2011 All
16、 rights reservedIntroduction This part of ISO 24614 focuses on word segmentation in Chinese, Japanese and Korean written texts. As far as typography is concerned, there is no white space between words in Chinese, Japanese or pre-modern Korean texts. This makes it hard to segment a text into words, u
17、nless there is a consistent way of identifying word segmentation units for those languages. On the other hand, in modern-day Korean text, word forms or verbal stems that are agglutinated with grammatical affixes, called eojeol or malmadi, are separated by white space as in English written texts. Hen
18、ce, it is much easier to identify words or other word segmentation units in a Korean text. Nevertheless, a large number of words in Korean as well as in Japanese are borrowed or derived from Chinese words; their internal structures are also based on the word formation principles of Chinese. As a con
19、sequence, general rules for identifying word segmentation units (WSUs) in Chinese, especially internal WSUs embedded in larger WSUs, are also applicable to some extent to the processing of Japanese and Korean texts. The use of characters does not play a real role in identifying WSUs in a text. Many
20、Korean words can be written either in Chinese or in Korean characters, but the same principles of analysing Chinese-derived words and identifying sub-WSUs of those words apply. A newspaper published in Beijing is written in simplified Chinese characters, while a Hong Kong newspaper may be written in
21、 traditional Chinese characters. Here again, the same principles of identifying WSUs apply to both newspapers. This part of ISO 24614 first sets out the general rules for identifying WSUs in Chinese, Japanese and Korean, then addresses the specific rules for each language. INTERNATIONAL STANDARD ISO
22、 24614-2:2011(E) ISO 2011 All rights reserved 1Language resource management Word segmentation of written texts Part 2: Word segmentation for Chinese, Japanese and Korean 1 Scope The basic concepts and general principles of word segmentation as defined in ISO 24614-1 apply to Chinese, Japanese and Ko
23、rean. Text needs to be segmented into tokens, words, phrases or some other types of smaller textual units in order to perform certain computational applications on language resources, such as natural language processing, information retrieval (IR) and machine translation (MT). This part of ISO 24614
24、 is restricted to the segmentation of a text into words or other word segmentation units (WSUs). This task is distinct from morphological or syntactic analysis per se, although it greatly depends on morphosyntactic analysis. It is also different from the task of laying out a framework for constructi
25、ng a lexicon and identifying its lexical entries, namely lemmas and lexemes. The frameworks for the latter tasks are provided by ISO 24611, ISO 24613 and ISO 24615. The main objective of this part of ISO 24614 is to specify rules for delineating WSUs for Chinese, Japanese and Korean. Some rules are
26、common to all three languages, though each language also has its own distinct rules for identifying WSUs. The common features are discussed in Clause 5, then the distinct rules are laid out in Clause 6 for Chinese, Clause 7 for Japanese and Clause 8 for Korean. 2 Normative references The following r
27、eferenced documents are indispensable for the application of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies. ISO 24611, Language resource management Morpho-syntactic annotat
28、ion framework ISO 24613:2008, Language resource management Lexical markup framework (LMF) ISO 24614-1:2010, Language resource management Word segmentation of written texts Part 1: Basic concepts and general principles ISO 24614-2:2011(E) 2 ISO 2011 All rights reserved3 Terms and definitions For the
29、purposes of this document, the terms and definitions given in ISO 24611, ISO 24613 and ISO 24614-1 and the following apply. 3.1 adnoun ADN non-conjugating word that modifies a noun NOTE Adnouns modify nouns, as adverbs modify verbs. EXAMPLE 1 a. 国 arayuru kuni ADN N every country b. 好 花 suki+na hana
30、 ADNst+SX N favourite flower EXAMPLE 2 a. sae ot ADN noun new clothes b. bbalga+n ot ADJst+GX N red clothes 3.2 bunsetsu phrase (3.8) without internal modifying relations EXAMPLE The sentence 私学校早行(I went to school early) consists of four bunsetsus: 私(watashiwa), 学校 (gakkoue), 早(hayaku) and 行(ikimas
31、hita) in which 私(watashi) is a pronoun, (wa) is a particle, 学校(gakkou) is a noun, (e) is a particle, 早(hayaku) is an adjective in adverbial usage, 行(iki) is a verbal stem followed by (mashi) is an auxiliary verb denoting politeness, and (ta) is an auxiliary verb indicating the past tense. NOTE A bun
32、setsu normally consists of a noun plus its particle(s) or a verb plus its ending(s), auxiliary verb(s) or particle(s) as shown in the example above. ISO 24614-2:2011(E) ISO 2011 All rights reserved 33.3 ending agglutinative affix of a verb or adjective NOTE Verbs and adjectives end with agglutinativ
33、e forms, called “endings”. These endings may be a negative form, an adverbial form, a base form, an adnominal form, an assumption form or an imperative form. 3.4 eojeol malmaldi word or its variant word form agglutinated with grammatical affixes NOTE 1 White space (space between characters) helps to
34、 segment text into eojeols. EXAMPLE nae+ga sagwa+reul meok+eot+da pronoun+GX noun+GX Vst+GX+GX I+SBJ apple+OBJ eat+PST+DCL = I ate (an) apple NOTE 2 This sentence consists of three eojeols: , and , each of which is separated by white space. The acronyms GX, SBJ, OBJ, PST and DCL in the example above
35、 stand for grammatical affix, subject, object, past tense and declarative sentential type, respectively. The pronoun is a variant form of the pronoun referring to the speaker. is an eojeol and at the same time is a word form agglutinated with two grammatical affixes and to a verb stem . 3.5 lexical
36、item entry in a lexicon that is a lexeme or one of its variant forms NOTE Headed by a lemma, each lexical item may be either a free-standing word (or one of its variant word forms) or a bound (non-free-standing) form such as stems and affixes. See ISO 24614-1:2010 for the definitions of lexeme, lemm
37、a and lexicon. 3.6 measure word part of speech defining, along with numbers, the quantity of a given object, or identifying specific objects with demonstrative pronouns such as “this” and “that” NOTE 1 Whereas English speakers say “one person” or “this person”, Chinese speakers say respectively 一个人(
38、yi ge ren; numeral + measure word + noun; one person) or 这个人 (zhe ge ren; demonstrative pronoun + measure word + person; this person), where 个 (ge) is a measure word. NOTE 2 A set of “verbal measure words” is used to count the number of times an action occurs, rather than the number of items. For ex
39、ample, in the sentence 我去过三次北京(wo qu guo san ci Beijing; pronoun + verb + auxiliary word + numeral + measure word + proper noun; I have been to Beijing three times), the 次(ci) functions as a measure word to combine with a numeral 三 to derive the adverb三次 (sanci) that modifies the verb 去(qu). 3.7 par
40、ticle grammatical affix agglutinated mostly to nominal forms but sometimes to other free-standing lexical items (3.5) NOTE The grammatical category particle can be treated as a part of speech. EXAMPLE The noun phrase 学校(gakkoue) is analysed into a noun 学校 (gakkou) and a particle (e). The verb phrase
41、 寒(samuine, It is very cold, isnt it?) is analysed into a verb 寒 (samui) and a particle (ne) which corresponds to the tag isnt it ?. ISO 24614-2:2011(E) 4 ISO 2011 All rights reserved3.8 phrase group of words that perform a grammatical function and that form a conceptual unit within a sentence 4 Ove
42、rview 4.1 Introduction This clause first introduces a markup convention for word segmentation units, then reviews the concept of word segmentation unit (WSU) which was introduced in ISO 24614-1. Some features shared by Chinese, Japanese and Korean are discussed in 4.3. A comparative table of parts o
43、f speech is given in Annex A. 4.2 Markup convention The following clauses contain a very large number of examples of WSUs. A simple way of representing WSUs is introduced here. NOTE This markup convention is introduced here just for the sake of simple illustration in this part of ISO 24614. First, a
44、 stand-off annotation is adopted; this allows primary data to be kept intact from markup notations. (More information on linguisitic annotation can be found in ISO 24612.) Exceptions are made concerning this requirement when primary data in Chinese or some other language are not provided with a roma
45、nized version or when the identification of syllables is not easy. Second, a citation format consisting of four lines is adopted. Line 1 introduces primary text fragment in its original script. Line 2 represents annotated fragment in romanized form. Line 3 assigns morpho-syntactic descriptions. Line
46、 4 provides an English equivalent. The following symbols are used (optionally) for marking up primary data in a romanized form: 1) the dot . for syllable boundaries, if ambiguity arises; 2) the sign + (plus) for boundaries between a word and an affix; 3) the underscore _ for word segmentation units
47、(WSUs); 4) the square brackets for WSUs, if ambiguity arises; 5) the parentheses ( ) for non-WSUs; 6) the symbol := represents combined resultant information. EXAMPLE (heot)+dol_da prefix + verbStem_GX := verb in vain spin := to spin in vain ISO 24614-2:2011(E) ISO 2011 All rights reserved 5The verb
48、 consists of a prefix (heot) and a verb (dol+da). The verb is then analysed into a stem (dol) and a verbal grammatical affix (da). The example as a whole is a WSU, while its subpart is also a WSU embedded in it. In Korean, as will be discussed, the prefix (heot) is not treated as a WSU. For computat
49、ional purposes, namely character encoding schemes, the characters in Chinese, Japanese and Korean are all treated as being syllabic. Hence, each of the characters in primary data can easily be identified with its corresponding romanized string of alphabetic characters. The word given in the above example, for instance, consists of three syllable char
