欢迎来到麦多课文档分享! | 帮助中心 海量文档,免费浏览,给你所需,享你所想!
麦多课文档分享
全部分类
  • 标准规范>
  • 教学课件>
  • 考试资料>
  • 办公文档>
  • 学术论文>
  • 行业资料>
  • 易语言源码>
  • ImageVerifierCode 换一换
    首页 麦多课文档分享 > 资源分类 > PDF文档下载
    分享到微信 分享到微博 分享到QQ空间

    ISO 24614-2-2011 Language resource management - Word segmentation of written texts - Part 2 Word segmentation for Chinese Japanese and Korean《语言资源管理 书面文本的单词分割 第.pdf

    • 资源ID:1252776       资源大小:1,022KB        全文页数:50页
    • 资源格式: PDF        下载积分:10000积分
    快捷下载 游客一键下载
    账号登录下载
    微信登录下载
    二维码
    微信扫一扫登录
    下载资源需要10000积分(如需开发票,请勿充值!)
    邮箱/手机:
    温馨提示:
    如需开发票,请勿充值!快捷下载时,用户名和密码都是您填写的邮箱或者手机号,方便查询和重复下载(系统自动生成)。
    如需开发票,请勿充值!如填写123,账号就是123,密码也是123。
    支付方式: 支付宝扫码支付    微信扫码支付   
    验证码:   换一换

    加入VIP,交流精品资源
     
    账号:
    密码:
    验证码:   换一换
      忘记密码?
        
    友情提示
    2、PDF文件下载后,可能会被浏览器默认打开,此种情况可以点击浏览器菜单,保存网页到桌面,就可以正常下载了。
    3、本站不支持迅雷下载,请使用电脑自带的IE浏览器,或者360浏览器、谷歌浏览器下载即可。
    4、本站资源下载后的文档和图纸-无水印,预览文档经过压缩,下载后原文更清晰。
    5、试题试卷类文档,如果标题没有明确说明有答案则都视为没有答案,请知晓。

    ISO 24614-2-2011 Language resource management - Word segmentation of written texts - Part 2 Word segmentation for Chinese Japanese and Korean《语言资源管理 书面文本的单词分割 第.pdf

    1、 Reference number ISO 24614-2:2011(E) ISO 2011INTERNATIONAL STANDARD ISO 24614-2 First edition 2011-09-01 Language resource management Word segmentation of written texts Part 2: Word segmentation for Chinese, Japanese and Korean Gestion des ressources langagires Segmentation des mots dans les textes

    2、 crits Partie 2: Segmentation des mots pour le chinois, le japonais et le coren ISO 24614-2:2011(E) COPYRIGHT PROTECTED DOCUMENT ISO 2011 All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic or mechanical,

    3、including photocopying and microfilm, without permission in writing from either ISO at the address below or ISOs member body in the country of the requester. ISO copyright office Case postale 56 CH-1211 Geneva 20 Tel. + 41 22 749 01 11 Fax + 41 22 749 09 47 E-mail copyrightiso.org Web www.iso.org Pu

    4、blished in Switzerland ii ISO 2011 All rights reservedISO 24614-2:2011(E) ISO 2011 All rights reserved iiiContents Page Foreword . v Introduction vi 1 Scope 1 2 Normative references 1 3 Terms and definitions . 2 4 Overview . 4 4.1 Introduction 4 4.2 Markup convention 4 4.3 Review of the concept of w

    5、ord segmentation unit 5 4.4 Features common to Chinese, Japanese and Korean . 5 5 General rules for identifying WSUs in Chinese, Japanese and Korean 6 5.1 Words 6 5.2 Derivationally formed words 6 5.3 Word compounds 7 5.4 Phrasal compounds 8 5.5 Idioms . 8 5.6 Fixed expressions . 9 5.7 Abbreviations

    6、 . 10 5.8 Transliterated loanwords 10 5.9 Strings of foreign or special characters . 11 5.10 Components of a WSU 11 6 Specific rules for identifying WSUs in Chinese . 12 6.1 Lexical items followed by the suffix 儿(r) 12 6.2 Lexical items 12 6.2.1 Nouns 12 6.2.2 Verbs . 17 6.2.3 Adjectives . 20 6.2.4

    7、Pronouns 22 6.2.5 Numerals 23 6.2.6 Measure words 25 6.2.7 Adverbs 25 6.2.8 Prepositions . 26 6.2.9 Conjunctions 26 6.2.10 Auxiliary words 26 6.2.11 Modal words . 27 6.2.12 Exclamations . 27 6.2.13 Imitative words 27 7 Specific rules for identifying WSUs in Japanese text . 27 7.1 Bunsetsus 27 7.2 Le

    8、xical items 27 7.2.1 General rule 27 7.2.2 Nouns 28 7.2.3 Verbs . 32 7.2.4 Adjectives . 33 7.2.5 Adnouns . 34 7.2.6 Adverbs 34 7.2.7 Conjunctions 35 7.2.8 Exclamations . 35 ISO 24614-2:2011(E) iv ISO 2011 All rights reserved7.2.9 Particles 35 7.2.10 Auxiliary verbs .35 8 Specific rules for identifyi

    9、ng WSUs in Korean text 36 8.1 Eojeols 36 8.2 Lexical items 36 8.2.1 General rule 36 8.2.2 Nouns 37 8.2.3 Pronouns 38 8.2.4 Numerals .39 8.2.5 Verbs .39 8.2.6 Adjectives .39 8.2.7 Adnouns .40 8.2.8 Adverbs .40 8.2.9 Exclamations 40 8.3 Grammatical affixes .40 Annex A (informative) Comparative table o

    10、f parts of speech in Chinese, Japanese and Korean 42 Bibliography 43 ISO 24614-2:2011(E) ISO 2011 All rights reserved vForeword ISO (the International Organization for Standardization) is a worldwide federation of national standards bodies (ISO member bodies). The work of preparing International Sta

    11、ndards is normally carried out through ISO technical committees. Each member body interested in a subject for which a technical committee has been established has the right to be represented on that committee. International organizations, governmental and non-governmental, in liaison with ISO, also

    12、take part in the work. ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of electrotechnical standardization. International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2. The main task of technical committees

    13、 is to prepare International Standards. Draft International Standards adopted by the technical committees are circulated to the member bodies for voting. Publication as an International Standard requires approval by at least 75 % of the member bodies casting a vote. Attention is drawn to the possibi

    14、lity that some of the elements of this document may be the subject of patent rights. ISO shall not be held responsible for identifying any or all such patent rights. ISO 24614-2 was prepared by Technical Committee ISO/TC 37, Terminology and other language and content resources, Subcommittee SC 4, La

    15、nguage resource management. ISO 24614 consists of the following parts, under the general title Language resource management Word segmentation of written texts: Part 1: Basic concepts and general principles Part 2: Word segmentation for Chinese, Japanese and Korean ISO 24614-2:2011(E) vi ISO 2011 All

    16、 rights reservedIntroduction This part of ISO 24614 focuses on word segmentation in Chinese, Japanese and Korean written texts. As far as typography is concerned, there is no white space between words in Chinese, Japanese or pre-modern Korean texts. This makes it hard to segment a text into words, u

    17、nless there is a consistent way of identifying word segmentation units for those languages. On the other hand, in modern-day Korean text, word forms or verbal stems that are agglutinated with grammatical affixes, called eojeol or malmadi, are separated by white space as in English written texts. Hen

    18、ce, it is much easier to identify words or other word segmentation units in a Korean text. Nevertheless, a large number of words in Korean as well as in Japanese are borrowed or derived from Chinese words; their internal structures are also based on the word formation principles of Chinese. As a con

    19、sequence, general rules for identifying word segmentation units (WSUs) in Chinese, especially internal WSUs embedded in larger WSUs, are also applicable to some extent to the processing of Japanese and Korean texts. The use of characters does not play a real role in identifying WSUs in a text. Many

    20、Korean words can be written either in Chinese or in Korean characters, but the same principles of analysing Chinese-derived words and identifying sub-WSUs of those words apply. A newspaper published in Beijing is written in simplified Chinese characters, while a Hong Kong newspaper may be written in

    21、 traditional Chinese characters. Here again, the same principles of identifying WSUs apply to both newspapers. This part of ISO 24614 first sets out the general rules for identifying WSUs in Chinese, Japanese and Korean, then addresses the specific rules for each language. INTERNATIONAL STANDARD ISO

    22、 24614-2:2011(E) ISO 2011 All rights reserved 1Language resource management Word segmentation of written texts Part 2: Word segmentation for Chinese, Japanese and Korean 1 Scope The basic concepts and general principles of word segmentation as defined in ISO 24614-1 apply to Chinese, Japanese and Ko

    23、rean. Text needs to be segmented into tokens, words, phrases or some other types of smaller textual units in order to perform certain computational applications on language resources, such as natural language processing, information retrieval (IR) and machine translation (MT). This part of ISO 24614

    24、 is restricted to the segmentation of a text into words or other word segmentation units (WSUs). This task is distinct from morphological or syntactic analysis per se, although it greatly depends on morphosyntactic analysis. It is also different from the task of laying out a framework for constructi

    25、ng a lexicon and identifying its lexical entries, namely lemmas and lexemes. The frameworks for the latter tasks are provided by ISO 24611, ISO 24613 and ISO 24615. The main objective of this part of ISO 24614 is to specify rules for delineating WSUs for Chinese, Japanese and Korean. Some rules are

    26、common to all three languages, though each language also has its own distinct rules for identifying WSUs. The common features are discussed in Clause 5, then the distinct rules are laid out in Clause 6 for Chinese, Clause 7 for Japanese and Clause 8 for Korean. 2 Normative references The following r

    27、eferenced documents are indispensable for the application of this document. For dated references, only the edition cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies. ISO 24611, Language resource management Morpho-syntactic annotat

    28、ion framework ISO 24613:2008, Language resource management Lexical markup framework (LMF) ISO 24614-1:2010, Language resource management Word segmentation of written texts Part 1: Basic concepts and general principles ISO 24614-2:2011(E) 2 ISO 2011 All rights reserved3 Terms and definitions For the

    29、purposes of this document, the terms and definitions given in ISO 24611, ISO 24613 and ISO 24614-1 and the following apply. 3.1 adnoun ADN non-conjugating word that modifies a noun NOTE Adnouns modify nouns, as adverbs modify verbs. EXAMPLE 1 a. 国 arayuru kuni ADN N every country b. 好 花 suki+na hana

    30、 ADNst+SX N favourite flower EXAMPLE 2 a. sae ot ADN noun new clothes b. bbalga+n ot ADJst+GX N red clothes 3.2 bunsetsu phrase (3.8) without internal modifying relations EXAMPLE The sentence 私学校早行(I went to school early) consists of four bunsetsus: 私(watashiwa), 学校 (gakkoue), 早(hayaku) and 行(ikimas

    31、hita) in which 私(watashi) is a pronoun, (wa) is a particle, 学校(gakkou) is a noun, (e) is a particle, 早(hayaku) is an adjective in adverbial usage, 行(iki) is a verbal stem followed by (mashi) is an auxiliary verb denoting politeness, and (ta) is an auxiliary verb indicating the past tense. NOTE A bun

    32、setsu normally consists of a noun plus its particle(s) or a verb plus its ending(s), auxiliary verb(s) or particle(s) as shown in the example above. ISO 24614-2:2011(E) ISO 2011 All rights reserved 33.3 ending agglutinative affix of a verb or adjective NOTE Verbs and adjectives end with agglutinativ

    33、e forms, called “endings”. These endings may be a negative form, an adverbial form, a base form, an adnominal form, an assumption form or an imperative form. 3.4 eojeol malmaldi word or its variant word form agglutinated with grammatical affixes NOTE 1 White space (space between characters) helps to

    34、 segment text into eojeols. EXAMPLE nae+ga sagwa+reul meok+eot+da pronoun+GX noun+GX Vst+GX+GX I+SBJ apple+OBJ eat+PST+DCL = I ate (an) apple NOTE 2 This sentence consists of three eojeols: , and , each of which is separated by white space. The acronyms GX, SBJ, OBJ, PST and DCL in the example above

    35、 stand for grammatical affix, subject, object, past tense and declarative sentential type, respectively. The pronoun is a variant form of the pronoun referring to the speaker. is an eojeol and at the same time is a word form agglutinated with two grammatical affixes and to a verb stem . 3.5 lexical

    36、item entry in a lexicon that is a lexeme or one of its variant forms NOTE Headed by a lemma, each lexical item may be either a free-standing word (or one of its variant word forms) or a bound (non-free-standing) form such as stems and affixes. See ISO 24614-1:2010 for the definitions of lexeme, lemm

    37、a and lexicon. 3.6 measure word part of speech defining, along with numbers, the quantity of a given object, or identifying specific objects with demonstrative pronouns such as “this” and “that” NOTE 1 Whereas English speakers say “one person” or “this person”, Chinese speakers say respectively 一个人(

    38、yi ge ren; numeral + measure word + noun; one person) or 这个人 (zhe ge ren; demonstrative pronoun + measure word + person; this person), where 个 (ge) is a measure word. NOTE 2 A set of “verbal measure words” is used to count the number of times an action occurs, rather than the number of items. For ex

    39、ample, in the sentence 我去过三次北京(wo qu guo san ci Beijing; pronoun + verb + auxiliary word + numeral + measure word + proper noun; I have been to Beijing three times), the 次(ci) functions as a measure word to combine with a numeral 三 to derive the adverb三次 (sanci) that modifies the verb 去(qu). 3.7 par

    40、ticle grammatical affix agglutinated mostly to nominal forms but sometimes to other free-standing lexical items (3.5) NOTE The grammatical category particle can be treated as a part of speech. EXAMPLE The noun phrase 学校(gakkoue) is analysed into a noun 学校 (gakkou) and a particle (e). The verb phrase

    41、 寒(samuine, It is very cold, isnt it?) is analysed into a verb 寒 (samui) and a particle (ne) which corresponds to the tag isnt it ?. ISO 24614-2:2011(E) 4 ISO 2011 All rights reserved3.8 phrase group of words that perform a grammatical function and that form a conceptual unit within a sentence 4 Ove

    42、rview 4.1 Introduction This clause first introduces a markup convention for word segmentation units, then reviews the concept of word segmentation unit (WSU) which was introduced in ISO 24614-1. Some features shared by Chinese, Japanese and Korean are discussed in 4.3. A comparative table of parts o

    43、f speech is given in Annex A. 4.2 Markup convention The following clauses contain a very large number of examples of WSUs. A simple way of representing WSUs is introduced here. NOTE This markup convention is introduced here just for the sake of simple illustration in this part of ISO 24614. First, a

    44、 stand-off annotation is adopted; this allows primary data to be kept intact from markup notations. (More information on linguisitic annotation can be found in ISO 24612.) Exceptions are made concerning this requirement when primary data in Chinese or some other language are not provided with a roma

    45、nized version or when the identification of syllables is not easy. Second, a citation format consisting of four lines is adopted. Line 1 introduces primary text fragment in its original script. Line 2 represents annotated fragment in romanized form. Line 3 assigns morpho-syntactic descriptions. Line

    46、 4 provides an English equivalent. The following symbols are used (optionally) for marking up primary data in a romanized form: 1) the dot . for syllable boundaries, if ambiguity arises; 2) the sign + (plus) for boundaries between a word and an affix; 3) the underscore _ for word segmentation units

    47、(WSUs); 4) the square brackets for WSUs, if ambiguity arises; 5) the parentheses ( ) for non-WSUs; 6) the symbol := represents combined resultant information. EXAMPLE (heot)+dol_da prefix + verbStem_GX := verb in vain spin := to spin in vain ISO 24614-2:2011(E) ISO 2011 All rights reserved 5The verb

    48、 consists of a prefix (heot) and a verb (dol+da). The verb is then analysed into a stem (dol) and a verbal grammatical affix (da). The example as a whole is a WSU, while its subpart is also a WSU embedded in it. In Korean, as will be discussed, the prefix (heot) is not treated as a WSU. For computat

    49、ional purposes, namely character encoding schemes, the characters in Chinese, Japanese and Korean are all treated as being syllabic. Hence, each of the characters in primary data can easily be identified with its corresponding romanized string of alphabetic characters. The word given in the above example, for instance, consists of three syllable char


    注意事项

    本文(ISO 24614-2-2011 Language resource management - Word segmentation of written texts - Part 2 Word segmentation for Chinese Japanese and Korean《语言资源管理 书面文本的单词分割 第.pdf)为本站会员(eastlab115)主动上传,麦多课文档分享仅提供信息存储空间,仅对用户上传内容的表现方式做保护处理,对上载内容本身不做任何修改或编辑。 若此文所含内容侵犯了您的版权或隐私,请立即通知麦多课文档分享(点击联系客服),我们立即给予删除!




    关于我们 - 网站声明 - 网站地图 - 资源地图 - 友情链接 - 网站客服 - 联系我们

    copyright@ 2008-2019 麦多课文库(www.mydoc123.com)网站版权所有
    备案/许可证编号:苏ICP备17064731号-1 

    收起
    展开