BS ISO 24614-2-2013 Language resource management Word segmentation of written texts Word segmentation for Chinese Japanese and Korean《语言资源管理 书面文本的单词分割 汉语、日语和韩语的词语切分》.pdf

《BS ISO 24614-2-2013 Language resource management Word segmentation of written texts Word segmentation for Chinese Japanese and Korean《语言资源管理 书面文本的单词分割 汉语、日语和韩语的词语切分》.pdf》由会员分享，可在线阅读，更多相关《BS ISO 24614-2-2013 Language resource management Word segmentation of written texts Word segmentation for Chinese Japanese and Korean《语言资源管理 书面文本的单词分割 汉语、日语和韩语的词语切分》.pdf（54页珍藏版）》请在麦多课文档分享上搜索。

1、raising standards worldwideNO COPYING WITHOUT BSI PERMISSION EXCEPT AS PERMITTED BY COPYRIGHT LAWBSI Standards PublicationBS ISO 24614-2:2011Language resourcemanagement Wordsegmentation of written textsPart 2: Word segmentation for Chinese,Japanese and KoreanBS ISO 24614-2:2011 BRITISH STANDARDNatio

2、nal forewordThis British Standard is the UK implementation of ISO 24614-2:2011.The UK participation in its preparation was entrusted to TechnicalCommittee TS/1, Terminology.A list of organizations represented on this committee can beobtained on request to its secretary.This publication does not purp

3、ort to include all the necessaryprovisions of a contract. Users are responsible for its correctapplication. The British Standards Institution 2013. Published by BSI StandardsLimited 2013ISBN 978 0 580 66328 4ICS 01.140.10Compliance with a British Standard cannot confer immunity fromlegal obligations

4、.This British Standard was published under the authority of theStandards Policy and Strategy Committee on 30 April 2013.Amendments issued since publicationDate Text affectedBS ISO 24614-2:2011Reference numberISO 24614-2:2011(E)ISO 2011INTERNATIONAL STANDARD ISO24614-2First edition2011-09-01Language

5、resource management Word segmentation of written texts Part 2: Word segmentation for Chinese, Japanese and Korean Gestion des ressources langagires Segmentation des mots dans les textes crits Partie 2: Segmentation des mots pour le chinois, le japonais et le coren BS ISO 24614-2:2011ISO 24614-2:2011

6、(E) COPYRIGHT PROTECTED DOCUMENT ISO 2011 All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the addre

7、ss below or ISOs member body in the country of the requester. ISO copyright office Case postale 56 CH-1211 Geneva 20 Tel. + 41 22 749 01 11 Fax + 41 22 749 09 47 E-mail copyrightiso.org Web www.iso.org Published in Switzerland ii ISO 2011 All rights reservedBS ISO 24614-2:2011ISO 24614-2:2011(E) ISO

8、 2011 All rights reserved iiiContents Page Foreword . v Introduction vi 1 Scope 1 2 Normative references 1 3 Terms and definitions . 2 4 Overview . 4 4.1 Introduction 4 4.2 Markup convention 4 4.3 Review of the concept of word segmentation unit 5 4.4 Features common to Chinese, Japanese and Korean .

9、 5 5 General rules for identifying WSUs in Chinese, Japanese and Korean 6 5.1 Words 6 5.2 Derivationally formed words 6 5.3 Word compounds 7 5.4 Phrasal compounds 8 5.5 Idioms . 8 5.6 Fixed expressions . 9 5.7 Abbreviations . 10 5.8 Transliterated loanwords 10 5.9 Strings of foreign or special chara

10、cters . 11 5.10 Components of a WSU 11 6 Specific rules for identifying WSUs in Chinese . 12 6.1 Lexical items followed by the suffix儿(r) 12 6.2 Lexical items 12 6.2.1 Nouns 12 6.2.2 Verbs . 17 6.2.3 Adjectives . 20 6.2.4 Pronouns 22 6.2.5 Numerals 23 6.2.6 Measure words 25 6.2.7 Adverbs 25 6.2.8 Pr

11、epositions . 26 6.2.9 Conjunctions 26 6.2.10 Auxiliary words 26 6.2.11 Modal words . 27 6.2.12 Exclamations . 27 6.2.13 Imitative words 27 7 Specific rules for identifying WSUs in Japanese text . 27 7.1 Bunsetsus 27 7.2 Lexical items 27 7.2.1 General rule 27 7.2.2 Nouns 28 7.2.3 Verbs . 32 7.2.4 Adj

12、ectives . 33 7.2.5 Adnouns . 34 7.2.6 Adverbs 34 7.2.7 Conjunctions 35 7.2.8 Exclamations . 35 BS ISO 24614-2:2011ISO 24614-2:2011(E) iv ISO 2011 All rights reserved7.2.9 Particles 35 7.2.10 Auxiliary verbs .35 8 Specific rules for identifying WSUs in Korean text 36 8.1 Eojeols 36 8.2 Lexical items

13、36 8.2.1 General rule 36 8.2.2 Nouns 37 8.2.3 Pronouns 38 8.2.4 Numerals .39 8.2.5 Verbs .39 8.2.6 Adjectives .39 8.2.7 Adnouns .40 8.2.8 Adverbs .40 8.2.9 Exclamations 40 8.3 Grammatical affixes .40 Annex A (informative) Comparative table of parts of speech in Chinese, Japanese and Korean 42 Biblio

14、graphy 43 BS ISO 24614-2:2011ISO 24614-2:2011(E) ISO 2011 All rights reserved vForeword ISO (the International Organization for Standardization) is a worldwide federation of national standards bodies (ISO member bodies). The work of preparing International Standards is normally carried out through I

15、SO technical committees. Each member body interested in a subject for which a technical committee has been established has the right to be represented on that committee. International organizations, governmental and non-governmental, in liaison with ISO, also take part in the work. ISO collaborates

16、closely with the International Electrotechnical Commission (IEC) on all matters of electrotechnical standardization. International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2. The main task of technical committees is to prepare International Standards.

17、Draft International Standards adopted by the technical committees are circulated to the member bodies for voting. Publication as an International Standard requires approval by at least 75 % of the member bodies casting a vote. Attention is drawn to the possibility that some of the elements of this d

18、ocument may be the subject of patent rights. ISO shall not be held responsible for identifying any or all such patent rights. ISO 24614-2 was prepared by Technical Committee ISO/TC 37, Terminology and other language and content resources, Subcommittee SC 4, Language resource management. ISO 24614 co

19、nsists of the following parts, under the general title Language resource management Word segmentation of written texts: Part 1: Basic concepts and general principles Part 2: Word segmentation for Chinese, Japanese and Korean BS ISO 24614-2:2011ISO 24614-2:2011(E) vi ISO 2011 All rights reservedIntro

20、duction This part of ISO 24614 focuses on word segmentation in Chinese, Japanese and Korean written texts. As far as typography is concerned, there is no white space between words in Chinese, Japanese or pre-modern Korean texts. This makes it hard to segment a text into words, unless there is a cons

21、istent way of identifying word segmentation units for those languages. On the other hand, in modern-day Korean text, word forms or verbal stems that are agglutinated with grammatical affixes, called eojeol or malmadi, are separated by white space as in English written texts. Hence, it is much easier

22、 to identify words or other word segmentation units in a Korean text. Nevertheless, a large number of words in Korean as well as in Japanese are borrowed or derived from Chinese words; their internal structures are also based on the word formation principles of Chinese. As a consequence, general rul

23、es for identifying word segmentation units (WSUs) in Chinese, especially internal WSUs embedded in larger WSUs, are also applicable to some extent to the processing of Japanese and Korean texts. The use of characters does not play a real role in identifying WSUs in a text. Many Korean words can be w

24、ritten either in Chinese or in Korean characters, but the same principles of analysing Chinese-derived words and identifying sub-WSUs of those words apply. A newspaper published in Beijing is written in simplified Chinese characters, while a Hong Kong newspaper may be written in traditional Chinese

25、characters. Here again, the same principles of identifying WSUs apply to both newspapers. This part of ISO 24614 first sets out the general rules for identifying WSUs in Chinese, Japanese and Korean, then addresses the specific rules for each language. BS ISO 24614-2:2011INTERNATIONAL STANDARD ISO 2

26、4614-2:2011(E) ISO 2011 All rights reserved 1Language resource management Word segmentation of written texts Part 2: Word segmentation for Chinese, Japanese and Korean 1 Scope The basic concepts and general principles of word segmentation as defined in ISO 24614-1 apply to Chinese, Japanese and Kore

27、an. Text needs to be segmented into tokens, words, phrases or some other types of smaller textual units in order to perform certain computational applications on language resources, such as natural language processing, information retrieval (IR) and machine translation (MT). This part of ISO 24614 i

28、s restricted to the segmentation of a text into words or other word segmentation units (WSUs). This task is distinct from morphological or syntactic analysis per se, although it greatly depends on morphosyntactic analysis. It is also different from the task of laying out a framework for constructing

29、 a lexicon and identifying its lexical entries, namely lemmas and lexemes. The frameworks for the latter tasks are provided by ISO 24611, ISO 24613 and ISO 24615. The main objective of this part of ISO 24614 is to specify rules for delineating WSUs for Chinese, Japanese and Korean. Some rules are co

30、mmon to all three languages, though each language also has its own distinct rules for identifying WSUs. The common features are discussed in Clause 5, then the distinct rules are laid out in Clause 6 for Chinese, Clause 7 for Japanese and Clause 8 for Korean. 2 Normative references The following ref

31、erenced documents are indispensable for the application of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies. ISO 24611, Language resource management Morpho-syntactic annotatio

32、n framework ISO 24613:2008, Language resource management Lexical markup framework (LMF) ISO 24614-1:2010, Language resource management Word segmentation of written texts Part 1: Basic concepts and general principles BS ISO 24614-2:2011ISO 24614-2:2011(E) 2 ISO 2011 All rights reserved3 Terms and def

33、initions For the purposes of this document, the terms and definitions given in ISO 24611, ISO 24613 and ISO 24614-1 and the following apply. 3.1 adnoun ADN non-conjugating word that modifies a noun NOTE Adnouns modify nouns, as adverbs modify verbs. EXAMPLE 1 a. 国 arayuru kuni ADN N every country b.

34、 好 花 suki+na hana ADNst+SX N favourite flower EXAMPLE 2 a. sae ot ADN noun new clothes b. bbalga+n ot ADJst+GX N red clothes 3.2 bunsetsu phrase (3.8) without internal modifying relations EXAMPLE The sentence 私学校早行(I went to school early) consists of four bunsetsus: 私(watashiwa), 学校 (gakkoue), 早(hay

35、aku) and 行(ikimashita) in which 私(watashi) is a pronoun, (wa) is a particle, 学校(gakkou) is a noun, (e) is a particle, 早(hayaku) is an adjective in adverbial usage, 行(iki) is a verbal stem followed by (mashi) is an auxiliary verb denoting politeness, and (ta) is an auxiliary verb indicating the past

36、tense. NOTE A bunsetsu normally consists of a noun plus its particle(s) or a verb plus its ending(s), auxiliary verb(s) or particle(s) as shown in the example above. BS ISO 24614-2:2011ISO 24614-2:2011(E) ISO 2011 All rights reserved 33.3 ending agglutinative affix of a verb or adjective NOTE Verbs

37、and adjectives end with agglutinative forms, called “endings”. These endings may be a negative form, an adverbial form, a base form, an adnominal form, an assumption form or an imperative form. 3.4 eojeol malmaldi word or its variant word form agglutinated with grammatical affixes NOTE 1 White space

38、 (space between characters) helps to segment text into eojeols. EXAMPLE nae+ga sagwa+reul meok+eot+da pronoun+GX noun+GX Vst+GX+GX I+SBJ apple+OBJ eat+PST+DCL = I ate (an) apple NOTE 2 This sentence consists of three eojeols: , and , each of which is separated by white space. The acronyms GX, SBJ, O

39、BJ, PST and DCL in the example above stand for grammatical affix, subject, object, past tense and declarative sentential type, respectively. The pronoun is a variant form of the pronoun referring to the speaker. is an eojeol and at the same time is a word form agglutinated with two grammatical affix

40、es and to a verb stem . 3.5 lexical item entry in a lexicon that is a lexeme or one of its variant forms NOTE Headed by a lemma, each lexical item may be either a free-standing word (or one of its variant word forms) or a bound (non-free-standing) form such as stems and affixes. See ISO 24614-1:2010

41、 for the definitions of lexeme, lemma and lexicon. 3.6 measure word part of speech defining, along with numbers, the quantity of a given object, or identifying specific objects with demonstrative pronouns such as “this” and “that” NOTE 1 Whereas English speakers say “one person” or “this person”, Ch

42、inese speakers say respectively 一个人 (yi ge ren; numeral + measure word + noun; one person) or 这个人 (zhe ge ren; demonstrative pronoun + measure word + person; this person), where 个 (ge) is a measure word. NOTE 2 A set of “verbal measure words” is used to count the number of times an action occurs, ra

43、ther than the number of items. For example, in the sentence 我去过三次北京 (wo qu guo san ci Beijing; pronoun + verb + auxiliary word + numeral + measure word + proper noun; I have been to Beijing three times), the 次 (ci) functions as a measure word to combine with a numeral 三 to derive the adverb 三次 (sanc

44、i) that modifies the verb 去(qu). 3.7 particle grammatical affix agglutinated mostly to nominal forms but sometimes to other free-standing lexical items (3.5) NOTE The grammatical category particle can be treated as a part of speech. EXAMPLE The noun phrase 学校(gakkoue) is analysed into a noun 学校 (gak

45、kou) and a particle (e). The verb phrase 寒(samuine, It is very cold, isnt it?) is analysed into a verb 寒 (samui) and a particle (ne) which corresponds to the tag isnt it ?. BS ISO 24614-2:2011ISO 24614-2:2011(E) 4 ISO 2011 All rights reserved3.8 phrase group of words that perform a grammatical funct

46、ion and that form a conceptual unit within a sentence 4 Overview 4.1 Introduction This clause first introduces a markup convention for word segmentation units, then reviews the concept of word segmentation unit (WSU) which was introduced in ISO 24614-1. Some features shared by Chinese, Japanese and

47、Korean are discussed in 4.3. A comparative table of parts of speech is given in Annex A. 4.2 Markup convention The following clauses contain a very large number of examples of WSUs. A simple way of representing WSUs is introduced here. NOTE This markup convention is introduced here just for the sake

48、 of simple illustration in this part of ISO 24614. First, a stand-off annotation is adopted; this allows primary data to be kept intact from markup notations. (More information on linguisitic annotation can be found in ISO 24612.) Exceptions are made concerning this requirement when primary data in

49、Chinese or some other language are not provided with a romanized version or when the identification of syllables is not easy. Second, a citation format consisting of four lines is adopted. Line 1 introduces primary text fragment in its original script. Line 2 represents annotated fragment in romanized form. Line 3 assigns morpho-syntactic descriptions. Line 4 provides an Eng