1、EISI 3404583 0069325 550 REPORT ETR 070 June 1993 Source: ETSI TC-HF Reference: DTR/HF-1 O1 OB UDC: 621.39 Key words: Human factors, Multiple Index Approach Human Factors (HF); The Multiple Index Approach (MIA) for the evaluation of pictograms ETSI Europzan Telecommunications Standards Institute ETS
2、I Secretariat Postal address: 06921 Sophia Antipolis Cedex - FRANCE Office address: Route des Lucioles - Sophia Antipolis - Valbonne - FRANCE Tel.: ,+33 92 94 42 O0 - Fax: +33 93 65 47 16 European Telecommunications Standards Institute 1 993. All rights reserved. No part may be reproduced except as
3、authorised by written permission. The copyright and the foregoing restriction on reproduction extend to all media in which the information may be embodied. m 3404583 0069326 497 m Page 2 ETR 070:1993 Whilst every care has been taken in the preparation and publication of this document, errors in cont
4、ent, typographical or otherwise, may occur. If you have comments concerning its accuracy, please write to “ETSI Editing and Standards Approval Dept.“ at the address shown on the title page. 3Y04583 0069327 323 H Contents Page 3 ETR 070: 1993 Foreword . 5 1 Scope and aim of the document 7 2 Introduct
5、ion . 7 2.1 Definition 7 2.2 Pictograms in user controls 7 2.3 Rationale of an empirical testing method of pictograms 7 3 Evaluating pictograms by multiple indices . 8 3.1 The seven MIA indices 8 4 Structure of the MIA questionnaire 9 4.1 The Introduction Section . 9 4.2 The Test of Pictogram Associ
6、ativeness . 10 4.3 The Test of Pictogram Preference . 10 4.4 The Test of Pictogram Set Preference . 10 4.5 Control variables 11 5 Analysis 11 Test of order and learning effects . 11 5.1 5.2 Analysis of the Test of Associativeness data 11 5.2.1 Pictogram selection . 11 5.2.2 Subjective Certainty . 12
7、 5.2.3 Subjective Suitability . 12 5.3 Analysis of the Test of Pictogram Preference and Test of Pictogram Set Preference Data . 12 6 Making a decision based on the results 12 7 The special case of testing one pictogram set only 12 8 Conclusion . 13 Annex A (informative): Examples for the different s
8、ections of a MIA-questionnaire . 14 History 23 . 3404583 00b9128 2bT Page 5 ETR 070: 1993 Foreword This ETSI Technical Report (ETRI has been produced by the Human Factors (HF) Technical Committee of the European Telecommunications Standards Institute (ETSI). ETRs are informative documents resulting
9、from ETSI studies which are not appropriate for European Telecommunication Standard (ETS) or Interim European Telecommunication Standard (LETS) status. An ETR may be used to publish material which is either of an informative nature, relating to the use or application of ETSs or I-ETSs, or which is i
10、mmature and not yet suitable for formal adoption as an ETS or LETS. Previous page is blank H 3404583 0069329 LTb Page 7 ETR 070: 1993 1 Scope and aim of the document This ETR describes a method for the evaluation of pictograms, the Multiple Index Approach (MIA). This method has been developed, teste
11、d and employed in the context of an ETSI (TC Human Factors) study on pictograms for basic videophone functions and the examples given in this ETR are taken from this study. This method has been found to be suitable as a general testing method for pictograms from all areas. As presented in this ETR,
12、the method takes the form of a questionnaire test but it can be administered by other means as well (e.9. on a personal computer). The main purpose of a pictogram evaluation study using the Multiple Index Approach is to collect data with the help of which the best suited pictograms of a number of pi
13、ctogram proposals (.e. alternative pictogram sets) can be selected for use on products, or for standardisation. Depending on the evaluators aims, aesthetic criteria may or may not play a role. The Multiple Index Approach provides seven indices or parameters that support the evaluator in making a sel
14、ection. It does not, however, provide a formula that computes one best solution - the task of weighing the importance of the various indices is left to the evaluator. In most cases, performance data (Hit rate, False alarm rate and Missing values) will be the prime criteria. 2 Introduction 2.1 Defini
15、tion In this ETR, the term “pictogram“ is used for the graphical representation of a function or element of a user interface and includes both “icons“ (concrete representations and “symbols“ (abstract representations). 2.2 Pictograms in user controls Pictograms and icons are used more and more frequ
16、ently in the context of the controls and indications of a large variety of devices. Recently, the trend towards pictograms has experienced an additional boost through the advent of a new generation of graphical interfaces on personal computers and workstations. In other areas, such as in telecommuni
17、cations and transport, pictograms have been used for a long time, sometimes officially standardised, sometimes following quasi-standards. Pictograms and icons have the potential of easing the use of telecommunications devices. Well designed pictograms allow the user to intuitively understand which f
18、unction of a device is supposed to be represented. In many cases, pictograms require less learning time and effort than text based alternatives. They are “international“ in the sense that they are not bound to a particular language and no level of literacy is required. Standardised pictograms for th
19、e functions of widely used devices allow the user to recognise the basic functions of any such device without the need for extra instruction. Unlike in areas with well established design guidelines, e.g. menu structures in computer software, design recommendations for pictograms are somewhat vague l
20、eaving the designer a great deal of artistic freedom. Some of the resulting pictogram designs are highly ambiguous and lessen the usability of the device. This state of affairs makes it imperative that pictograms be empirically tested in order to establish whether the user does indeed associate the
21、function, location, etc. to be represented (from this point known “referent“). If the pictogram is intended to be used on interfaces of devices for the international market, or if it is a candidate for international standardisation, the empirical testing has to be conducted in several languages in o
22、rder to ascertain that the pictogram does not draw on language mediated associations that work in some languages only, thus losing the pictograms potential benefit of being free from language and culture biases. 2.3 Rationale of an empirical testing method of pictograms Before comparing the differen
23、t methodological options available for testing pictograms, the criteria by which the pictograms are to be judged have to be made explicit, or in other words, the question of what establishes a good set of pictograms needs to be addressed (in the following, it is assumed that a set of pictograms is t
24、o be tested which represents a number of functions of a device). Previous page is blank = 3404583 00bL30 L Page 8 ETR 070:1993 A set of pictograms will optimise learning and user performance if: - - - - each of its elements is associated with the corresponding referent; none of its elements is assoc
25、iated with any referent other than the corresponding one; users feel subjectively certain in their choice of pictogram; users like their aesthetic appeal. Therefore, the testing method should focus on both correct associations and on errors and it should take into account the respondents subjective
26、certainty. In addition, it is desirable that the method provides data on the pictograms aesthetic appeal, a criterion which, inter alia, may be important in product design. The most realistic evaluation approach is one that tries to represent an actual usage situation, .e. a recognition situation in
27、 which a user with a certain intention is confronted with the controls of a device and has to make a choice as to which control will bring about the desired effect. There are basically four ways of assessing the associativeness of pictograms. Display One pictogram at a time. 1 Task Name referent. 2
28、Set of pictograms and one referent. Pick pictogram that represents the referent. 3 One pictogram and the list of referents. Pick the referent that is represented by the pictogram. 4 Set of pictograms and list of referents. To map the elements of each list. The four options meet the criteria specifie
29、d above to different degrees. The first test is one of recall rather than recognition processes. The third one, in which all referents and only one pictogram are presented at a time, is equally unsuited for the present aim since in a real-life situation, the user of a videophone will have all pictog
30、rams visually present but he/she will not necessarily have a complete cognitive representation of all functions of the terminal as defined by the referents. The same applies to Option 4 which has the additional disadvantage that it is a one-to-one mapping and that certain errors (like one pictogram
31、being associated with two referents) do not occur. Option 2, .e. the test in which the complete set of pictograms is presented to the subject (as would be in the case of a real videophone call situation in which the pictograms are placed on the terminal) and only one referent is presented at a time,
32、 is the testing method on which the Multiple Index Approach is based. In addition to its greater validity, it has the advantage of allowing for all four kinds of outcomes of a signal detection situation (Hit, Miss, False Alarm and Correct Rejection) to occur thus making possible a detailed analysis
33、of the respondents selections. 3 Evaluating pictograms by multiple indices 3.1 The seven Multiple Index Approach indices The Multiple Index Approach was developed on the basis of the above considerations. It enables the evaluator to collect data on seven indices on which his final selection of picto
34、grams can be based and which are presented as follows. 1 The Hit rate This index is the main parameter of performance and it is equivalent to the score of correct associations between the referent and pictogram. c 2 The False alarm rate The False alarm rate tells the evaluator in how many cases a pi
35、ctogram has been associated with the wrong referent. Depending on the pictogram context, False alarm errors (sometime referred to as Type II or p-errors) can be more hazardous than a Miss (incorrectly rejecting an association, Type I or a-error). W 3404583 OOb9L3L 854 Page 9 ETR 070:1993 3 Missing v
36、alues The percentage of Missing values tells us in how many instances a respondent did not answer a question presumably because he/she did not know the answer. Missing values represent usage situations in which the user does not know which control to use to bring about a certain effect. 4 Subjective
37、 certainty The Subjective certainty index indicates how certain the respondents feel in their association between a pictogram and referent. If the users of a device are extremely uncertain about the effects of the controls of a device, they may decide not to use it at all, which in turn may seriousl
38、y hamper the uptake of the device. 5 Subjective suitability In addition to making the association between pictogram and referent and to indicating how certain they are in this association, the respondents can tell us their subjective impression as to how well a pictogram represents its referent. 6 P
39、ictogram Preference The respondents indicate, which of the candidate pictograms for one referent represents best the referent in question. In this, we do not know which criteria (aesthetic or functional) the respondents apply. 7 Pictogram Set Preference This index is an indicator for which pictogram
40、 set is preferred in toto mainly on aesthetic grounds. The seven indices are collected with a questionnaire that is organised in three tests (it is, of course, possible to implement the test on computers with sufficiently high resolution screens): - test of pictogram associativeness (Hit rate, False
41、 Alarm Rate, Missing values, Subjective certainty, and Subjective suitability). In this part of the questionnaire, one referent (name and description of a function) is presented at a time with all pictograms of one set. The respondents task is to choose the appropriate pictogram for the function in
42、question. In addition, Subjective certainty and suitability ratings are required for each rating; - test of pictogram preference. Here, the respondent is asked to give preference ratings on the level of function, .e. all candidates for one function are shown and the most suitable one is to be indica
43、ted; - test of pictogram set preference. Preference ratings are requested on the level of sets, .e. all pictogram sets are displayed and the preferred one is to be indicated. The results of the test of pictogram associativeness are the main indicator for the usability of the sets to be tested. The t
44、ests of pictogram preference and of pictogram set preference are to be used mainly to verify that a pictogram set fulfils not only the associativeness criterion, but also aesthetic criteria. Furthermore, these indices can be used in cases in which there are competing sets with similar results for as
45、sociativeness. Finally, order and learning effects should be controlled by employing versions of the questionnaire with a different presentation order of the pictograms. 4 Structure of the MIA questionnaire 4.1 The introduction section In the introduction section of the questionnaire, the purpose of
46、 the study should be made explicit and the referents should be described in detail. For the evaluation study to yield meaningful results, it is important that the referents be understood by the respondents. Ideally, the respondents are shown a model or mock-up of the device on which the pictograms w
47、ill be used. Different models, W 3404583 0069132 790 Page 10 ETR 070:1993 each with one of the pictogram sets to be tested can then be used for the MIA tests. However, this procedure will rarely be feasible. An alternative way of familiarising the respondents with the referents is to give them a vid
48、eotape presentation. In many cases, graphical representations can be used to support the description. In addition, it may be useful to present the referents with the help of a usage scenario - this is recommended particularly in the case of relatively novel devices. An example of the introduction se
49、ction for an evaluation questionnaire (in this case of pictograms for basic videophone functions) can be found in Annex A. 4.2 The test of pictogram associativeness This test collects data on the individual pictograms associativeness with regards to its corresponding referent and in the context of the pictogram set it belongs to. This test provides data on the Hit rate, False alarm rate and Missing Values indices. Secondly, Subjective certainty and Subjective suitability ratings are given in this subclause. Each pictogram is tested on a separate page (.e. if there are r
