1、raising standards worldwide NO COPYING WITHOUT BSI PERMISSION EXCEPT AS PERMITTED BY COPYRIGHT LAW BSI Standards Publication BS ISO/IEC 19795-6:2012 Information technology Biometric performance testing and reporting Part 6: Testing methodologies for operational evaluationBS ISO/IEC 19795-6:2012 BRIT
2、ISH STANDARD National foreword This British Standard is the UK implementation of . The UK participation in its preparation was entrusted to Technical Committee IST/44, Biometrics. A list of organizations represented on this committee can be obtained on request to its secretary. This publication does
3、 not purport to include all the necessary provisions of a contract. Users are responsible for its correct application. The British Standards Institution 2012. Published by BSI Standards Limited 2012 ISBN 978 0 580 78356 2 ICS 35.040 Compliance with a British Standard cannot confer immunity from lega
4、l obligations. This British Standard was published under the authority of the Standards Policy and Strategy Committee on 29 February 2012. Amendments issued since publication Date 29 February 2012 Correction to BSI title Text affectedBS ISO/IEC 19795-6:2012Reference number ISO/IEC 19795-6:2012(E) IS
5、O/IEC 2012INTERNATIONAL STANDARD ISO/IEC 19795-6 First edition 2012-02-01 Information technology Biometric performance testing and reporting Part 6: Testing methodologies for operational evaluation Technologies de linformation Essais et rapports de performances biomtriques Partie 6: Mthodologies des
6、sai pour lvaluation oprationnelle BS ISO/IEC 19795-6:2012 ISO/IEC 19795-6:2012(E) COPYRIGHT PROTECTED DOCUMENT ISO/IEC 2012 All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic or mechanical, including phot
7、ocopying and microfilm, without permission in writing from either ISO at the address below or ISOs member body in the country of the requester. ISO copyright office Case postale 56 CH-1211 Geneva 20 Tel. + 41 22 749 01 11 Fax + 41 22 749 09 47 E-mail copyrightiso.org Web www.iso.org Published in Swi
8、tzerland ii ISO/IEC 2012 All rights reservedBS ISO/IEC 19795-6:2012 ISO/IEC 19795-6:2012(E) ISO/IEC 2012 All rights reserved iiiContents Page Foreword . v Introduction vi 1 Scope 1 2 Conformance . 1 3 Normative references 1 4 Terms and definitions . 2 5 Operational evaluation overview . 3 5.1 Operat
9、ional evaluation goals . 3 5.2 Operational performance metrics 4 5.3 Operational evaluation methods 4 5.4 Determining operational performance 4 5.5 Use of technology and scenario evaluation methodologies in evaluating operational systems 5 6 Operational evaluation 5 6.1 Purpose and scope . 5 6.1.1 G
10、eneral . 5 6.1.2 Criteria for system inclusion 5 6.1.3 System specification . 5 6.1.4 Biometric functionality 6 6.1.5 Performance measures . 6 6.2 Application characteristics 6 6.2.1 General . 6 6.2.2 Concept of operations 7 6.2.3 Guidance and instruction . 7 6.2.4 Levels of effort and decision poli
11、cies . 8 6.2.5 Multiple-instance systems 8 6.2.6 Environment . 9 6.2.7 Deployment factors . 9 6.2.8 Acclimatization 10 6.2.9 Habituation . 10 6.3 Test Plan . 10 6.3.1 General . 10 6.3.2 System implementation and configuration. 11 6.3.3 Test population 11 6.3.4 Test transactions . 12 6.4 Performance
12、measurement 14 6.4.1 Throughput 14 6.4.2 Enrolment analysis 15 6.4.3 Recognition analysis . 15 6.5 Reporting 16 6.5.1 Reporting planned test results 16 6.5.2 Reporting additional analyses . 16 6.5.3 Reporting observations 17 6.5.4 Report structure 17 6.6 Record keeping 17 Annex A (informative) Non-m
13、andatory performance metrics and reporting 18 Annex B (informative) Sub-transaction events in operational testing 20 BS ISO/IEC 19795-6:2012 ISO/IEC 19795-6:2012(E) iv ISO/IEC 2012 All rights reservedAnnex C (informative) Sample operational test specification 21 Annex D (informative) Methods to dete
14、rmine test size .23 Annex E (informative) Operational system monitoring .25 Annex F (informative) Operational habituation testing .27 Annex G (informative) Sample operational test report outline .28 Bibliography 30 BS ISO/IEC 19795-6:2012 ISO/IEC 19795-6:2012(E) ISO/IEC 2012 All rights reserved vFor
15、eword ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission) form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC participate in the development of International Standards through technical
16、 committees established by the respective organization to deal with particular fields of technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in
17、 the work. In the field of information technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1. International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2. The main task of the joint technical committee is to prepare Internati
18、onal Standards. Draft International Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as an International Standard requires approval by at least 75 % of the national bodies casting a vote. Attention is drawn to the possibility that some of t
19、he elements of this document may be the subject of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights. ISO/IEC 19795-6 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology, Subcommittee SC 37, Biometrics. ISO/IEC 19795 co
20、nsists of the following parts, under the general title Information technology Biometric performance testing and reporting: Part 1: Principles and framework Part 2: Testing methodologies for technology and scenario evaluation Part 3: Modality-specific testing Technical Report Part 4: Interoperability
21、 performance testing Part 5: Access control scenario and grading scheme Part 6: Testing methodologies for operational evaluation Part 7: Testing of on-card biometric comparison algorithms BS ISO/IEC 19795-6:2012 ISO/IEC 19795-6:2012(E) vi ISO/IEC 2012 All rights reservedIntroduction Operational test
22、s evaluate complete biometric systems in the targeted operational environment with the target population. Tests may encompass performance monitoring of operational systems or assessment of performance in operational trials. Operational performance assessment may be based on: data collected by the op
23、erational system in the course of normal operation; additional data collected during normal system use, but with the system running in an “evaluation mode” allowing extra data to be collected; data collected with a set of test subjects considered separately from the subject base of the operational s
24、ystem. Operational evaluation differs from technology or scenario evaluation in that the subject base, environment, and system design are no longer controlled for the purpose of repeatable testing, but vary in accordance with operational use. Examples of uncontrolled variables include the legitimacy
25、 of the subjects identity claim, environmental effects from weather or lighting, or the variability of system use by different individuals. The overarching goals of operational testing are to measure or monitor operational biometric system performance over a period of time. Subgoals of operational t
26、esting may include: to determine if performance meets the requirements specified for a particular application or the claims asserted by the supplier; to determine the need to adjust or configure the system to improve performance; to predict performance as the numbers of subjects, locations, or devic
27、es increase; to obtain information on the target population and environmental parameters found to affect system performance; to obtain performance data from a pilot implementation; to obtain performance data to benchmark future systems. This part of ISO/IEC 19795 provides the test planning, test con
28、duct, performance measurement, test reporting, and record keeping requirements to be followed during a biometric systems operational evaluation. BS ISO/IEC 19795-6:2012 INTERNATIONAL STANDARD ISO/IEC 19795-6:2012(E) ISO/IEC 2012 All rights reserved 1Information technology Biometric performance testi
29、ng and reporting Part 6: Testing methodologies for operational evaluation 1 Scope This part of ISO/IEC 19795: provides guidance on the operational testing of biometric systems; specifies performance metrics for operational systems; details data that may be retained by operational systems to enable p
30、erformance monitoring; and specifies requirements on test methods, recording of data, and reporting of results of operational evaluations. NOTE Some operational biometric systems perform a single biometric function. For example, in the initial stages of rollout of biometric passports, the operationa
31、l system might be performing biometric enrolment only. Operational evaluation of such systems is within the scope of this part of ISO/IEC 19795. This part of ISO/IEC 19795 does not: cover testing of operational systems in the laboratory or address vulnerability testing. 2 Conformance An operational
32、evaluation is in conformance with this part of ISO/IEC 19795 if it is planned, executed and reported in accordance with the requirements of Clause 6. 3 Normative references The following referenced documents are indispensable for the application of this document. For dated references, only the editi
33、on cited applies. For undated references, the latest edition of the referenced document (including any amendments) applies. ISO/IEC 19795-1, Information technology Biometric performance testing and reporting Part 1: Principles and framework BS ISO/IEC 19795-6:2012 ISO/IEC 19795-6:2012(E) 2 ISO/IEC 2
34、012 All rights reserved4 Terms and definitions For the purposes of this document, the terms and definitions given in ISO/IEC 19795-1 and the following apply. 4.1 acclimatization change, over the course of one or more transactions, of a biometric characteristic that might impact the ability of a syst
35、em to process a sample NOTE Acclimatization is primarily associated with a subjects temporal adjustment to environmental effects, such as skin temperature. 4.2 attendant agent of the biometric system operator who directly interacts with the biometric capture subject 4.3 biometric capture subject ind
36、ividual who is the subject of a biometric capture process 4.4 biometric data subject individual whose individualized biometric data is within the biometric system 4.5 biometric probe biometric data input to an algorithm for comparison to a biometric reference(s) 4.6 biometric operational personnel i
37、ndividuals, other than the biometric capture subjects, who take an active role in the operation of the biometric system NOTE Biometric operational personnel includes biometric system administrators, attendants, and examiners. 4.7 biometric system administrator person who executes policies and proced
38、ures in the administration of a biometric system 4.8 biometric system operator organization responsible for defining policies and procedures in the operation of a biometric system 4.9 biometric reference one or more stored biometric samples, biometric templates or biometric models attributed to a bi
39、ometric data subject and used for comparison 4.10 comparison attempt limit maximum allowed number or duration of attempts in a comparison transaction 4.11 enrolment attempt limit maximum allowed number or duration of attempts in an enrolment transaction 4.12 habituation familiarity a subject has wit
40、h the biometric device, system and application NOTE The level of habituation can affect biometric sample presentation and acquisition device. BS ISO/IEC 19795-6:2012 ISO/IEC 19795-6:2012(E) ISO/IEC 2012 All rights reserved 34.13 subject base set of individuals whose biometric data is intended to be
41、enrolled or compared in operational use of a biometric system 4.14 system acceptance rate proportion of recognition transactions in an operational system in which the subject is recognized NOTE 1 Though the acceptance of an impostor is an incorrect recognition, it can still count as a system accepta
42、nce. NOTE 2 System acceptance rate = 1 system rejection rate. 4.15 system identification rate proportion of identification transactions in an operational system in which one or more subjects are identified 4.16 system rejection rate proportion of recognition transactions in an operational system in
43、which the subject is not recognized NOTE The system rejection rate differs from the false reject rate in that, in addition to false rejections, it also includes any rejected impostor transaction and any improper genuine transaction. 4.17 test crew member selected biometric data subject whose use of
44、the operational system is controlled or monitored as part of the evaluation NOTE In an operational evaluation, test subjects can be subjects of the operational system or they can be members of a test crew using the system specifically for evaluation purposes 5 Operational evaluation overview 5.1 Ope
45、rational evaluation goals The overarching goals of operational testing are to measure or monitor operational biometric system performance. Subgoals of operational testing may include: to determine if performance meets the requirements specified for a particular application or the claims asserted by
46、the supplier; to determine the need to adjust or configure the system to improve performance; to predict performance as the numbers of subjects, locations, or devices increase; to obtain information on the target population and environmental parameters found to affect system performance; to obtain p
47、erformance data from a pilot implementation; to obtain performance data to benchmark future systems. Operational evaluation considers the performance of people as well as the equipment, algorithms, and environment. Consequently, operational testing includes aspects of social science in addition to p
48、hysical science, whereas technology testing does not. In general, operational performance will vary over time due to uncontrolled conditions in people, equipment and environment. For example, if the majority of subjects are enrolled at the start of operations, with few new enrolments, performance of
49、 the system might improve as BS ISO/IEC 19795-6:2012 ISO/IEC 19795-6:2012(E) 4 ISO/IEC 2012 All rights reservedsubjects habituate or degrade as subjects biometric characteristics age over time away from their enrolled references. The performance observed in testing can depend on the operational personnel, such as attendants or biometric examiners, as well as the biometric subjects. Operational personnel based factors should be taken into consideration in a
