ASTM E3000-2018 Standard Guide for Measuring and Tracking Performance of Assessors on a Descriptive Sensory Panel《评定员在描述性感官板上的测量和跟踪性能的标准指南》.pdf

《ASTM E3000-2018 Standard Guide for Measuring and Tracking Performance of Assessors on a Descriptive Sensory Panel《评定员在描述性感官板上的测量和跟踪性能的标准指南》.pdf》由会员分享，可在线阅读，更多相关《ASTM E3000-2018 Standard Guide for Measuring and Tracking Performance of Assessors on a Descriptive Sensory Panel《评定员在描述性感官板上的测量和跟踪性能的标准指南》.pdf（13页珍藏版）》请在麦多课文档分享上搜索。

1、Designation: E3000 18Standard Guide forMeasuring and Tracking Performance of Assessors on aDescriptive Sensory Panel1This standard is issued under the fixed designation E3000; the number immediately following the designation indicates the year oforiginal adoption or, in the case of revision, the yea

2、r of last revision. A number in parentheses indicates the year of last reapproval. Asuperscript epsilon () indicates an editorial change since the last revision or reapproval.1. Scope1.1 This guide provides guidelines for measuring and track-ing the performance of individual assessors on a descripti

3、vesensory panel.1.2 This guide provides guidelines to assist sensory profes-sionals in measuring performance for given assessors. Measur-ing performance will form the basis for (1) determining thereliability of the results, and (2) establishing remedial actionsfor an individual assessor.1.3 This gui

4、de examines various aspects of trained assessorperformance; such as repeatability, discrimination, and agree-ment and demonstrates some ways to measure them. Theprocedures will help the sensory professional determine areasof good performance as well as those that require improve-ment.1.4 Individual

5、assessor performance is tracked using estab-lished statistical procedures. These procedures depend onwhether replicates are collected and if they are collected overmultiple sessions or within a single session.1.5 This guide provides suggested procedures, includingstatistical procedures that can be d

6、one using standard statisticalsoftware, for evaluating performance and is not meant toexclude other methods that may be effectively used for asimilar purpose.1.6 Methods for training and screening assessors are notwithin the scope of this guide. This guide does not address howto communicate performa

7、nce feedback information to indi-vidual assessors. This monitoring of panel reproducibility, ameasure of the panels ability to reproduce the results of otherpanels, is also not within the scope of this guide.1.7 This standard does not purport to address all of thesafety concerns, if any, associated

8、with its use. It is theresponsibility of the user of this standard to establish appro-priate safety, health, and environmental practices and deter-mine the applicability of regulatory limitations prior to use.1.8 This international standard was developed in accor-dance with internationally recognize

9、d principles on standard-ization established in the Decision on Principles for theDevelopment of International Standards, Guides and Recom-mendations issued by the World Trade Organization TechnicalBarriers to Trade (TBT) Committee.2. Referenced Documents2.1 ASTM Standards:2E253 Terminology Relating

10、 to Sensory Evaluation of Mate-rials and ProductsE456 Terminology Relating to Quality and Statistics2.2 Other Documents:2ASTM STP 758 Guidelines for the Selection and Training ofSensory Panel MembersASTM MNL13 Manual on Descriptive Analysis for SensoryEvaluation2.3 ISO Standards:3ISO 11132:2012 Sens

11、ory Analysis MethodologyGuide-lines for Monitoring the Performance of a QuantitativeSensory Panel3. Terminology3.1 Please refer to Terminologies E253 and E456, ASTMSTP 758, ASTM MNL13 and ISO 11132:2012 for any termsrelated to assessor performance that are not listed below.3.2 Definitions:3.2.1 agre

12、ementability of an assessor to give similarscores (rate) or to order the intensity of stimuli similarly to therest of the panel (rank) on a given attribute.3.2.2 performanceability of an assessor to make repeat-able assessments that are in agreement with other assessors on1This guide is under the ju

13、risdiction of ASTM Committee E18 on SensoryEvaluation and is the direct responsibility of Subcommittee E18.03 on SensoryTheory and Statistics.Current edition approved April 1, 2018. Published April 2018. Originallyapproved in 2017. Last previous edition approved in 2017 as E3000 17. DOI:10.1520/E300

14、0-18.2For referenced ASTM standards, visit the ASTM website, www.astm.org, orcontact ASTM Customer Service at serviceastm.org. For Annual Book of ASTMStandards volume information, refer to the standards Document Summary page onthe ASTM website.3Available from International Organization for Standardi

15、zation (ISO), ISOCentral Secretariat, BIBC II, Chemin de Blandonnet 8, CP 401, 1214 Vernier,Geneva, Switzerland, http:/www.iso.org.Copyright ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959. United StatesThis international standard was developed in accordance

16、with internationally recognized principles on standardization established in the Decision on Principles for theDevelopment of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.1the panel and discriminate perceptible

17、 differences betweenattributes when they are present.3.2.3 scale usagethe extent to which the assessor(s) usesthe scale with respect to the intensities of the attributes beingmeasured.4. Summary of Practice4.1 The protocols described in this guide provide a proce-dure for quantitatively establishing

18、 the performance of indi-vidual assessors by discussing the minimum level of goodperformance, determining when a performance problem exists,and detailing specific procedures to address those problems.5. Significance and Use5.1 This guide is meant to be used with and applied toindividual trained desc

19、riptive assessors.5.2 The procedures recommended in this guide can be usedby the panel leader to periodically appraise the performance ofindividual descriptive assessors.5.3 Tracking assessor performance will provide informationas to the quality of the data being generated. Performanceinformation ma

20、y be used to decide whether to use the data tointerpret product profiles.5.4 Monitoring assessor performance will enable the panelleader to identify retraining needs or to identify assessors whoare not performing well enough to continue participating on apanel.6. Performance6.1 Introduction:6.1.1 Th

21、is section provides sensory approaches for theassessment of assessor performance. It is assumed that goodsensory practices are being followed in order to allow for goodassessor performance. Panel members must be motivated tocarry out the job conscientiously, be in good health bothphysically and ment

22、ally, and must be willing and able to followinstructions. Standard procedures to reduce random variabilityand systematic bias, including robust experimental design toreduce order and carry over effects must be followed by thesensory professional.6.1.2 Assessor performance is the measure of the abili

23、ty ofan assessor to make reliable attribute assessments across theproducts being evaluated. It has been recognized as an impor-tant component of descriptive analysis since the method wasfirst developed (1, 2). It can be measured at a given time pointor tracked over time. Performance is compromised i

24、f anassessor cannot repeat their own results (repeatability), dis-criminate among the products (discrimination) and assessstimuli similarly to other assessors on a given attribute (agree-ment). This guide will focus on these three key measures.These measures can allow the sensory professional to dia

25、gnosesources of poor performance such as the inability to use a scaleto correctly indicate intensity (scale usage), and failure to useattributes similarly to other assessors (lexicon usage).6.1.3 It is important to track panel performance as a whole,since panel data are used for decision making; how

26、ever, thisguide describes how to measure and interpret performancecriteria for the individual assessors, since assessor performanceinfluences panel results.An assessor who does not discriminateamong the samples may impact the panel data, causing themean values for a specific attribute to be close to

27、gether andpreventing overall discrimination between samples. In somecases, a poorly performing assessor can cause panel data to beinconsistent and non-repeatable. All of the assessors must beusing the vocabulary in the same way and utilizing the scalesin a consistent manner in order for the panel to

28、 succeed.6.2 Individual Assessor Performance:6.2.1 In the early stages of training, performance evaluationshould be analyzed for each individual assessor prior toparticipation in a panel. During this phase, the panel leadertypically monitors panel agreement on ranking or rating stimulifor intensity

29、and on scale usage. Specific examples for eachattribute need to be introduced, experienced and defined toensure that all assessors understand the sensory qualities andrange of intensity of the attribute. During training, attributedefinitions and references should be reviewed and possiblyrevised, to

30、ensure that attributes are understood and usedconsistently by individual assessors across all samples. Asses-sors should be selected for continued panel participation basedupon performance.6.2.2 Once assessors are trained, they should be monitoredfor the three key measures of performance (repeatabil

31、ity,discrimination, agreement). It is important to evaluate assessorperformance periodically in order to detect any change inindividual performance over time and to identify an assessor orassessors who are not performing well. Assessor performanceon an individual study should be monitored if you are

32、 makinghigh value or high risk business decisions with your panel data.The panel member should be engaged for a sufficient period tohave established a history of performance which has beenmonitored.6.2.3 In cases of poor performance, initially check with theindividual assessor for any reasons they m

33、ay not have beenperforming as usual. This would indicate the need to eliminatetheir responses on relevant data sets.6.2.4 Rule out the possibility that assessor variation may bedue to potential variability within the samples. Some types ofproducts such as meat, seafood, or crop-based products can be

34、quite variable and this variability must also be understoodbefore concluding that there is an issue with assessor or panelperformance.6.2.5 Verification of test procedures, such as correct samplesevaluated, correct instructions given, no data transcriptionerrors, should also be done.6.2.6 Fundamenta

35、l issues such as insufficient training (is-sues with scale or lexicon usage), not understanding theprocedure, boredom, over-use, or being unable to perceivecertain attributes of the stimuli (physiological differences) canalso contribute to poor assessor performance. It is important toidentify early

36、signs of performance inconsistency and correctthe problem before the assessor has an impact on the overallpanels results. Additional training should be given as a part ofpanel maintenance to address these issues. By correcting theproblem of the inconsistent assessor, one can achieve the aimof having

37、 a consistent panel.E3000 1826.3 Key Measures of PerformanceThere are three mainelements of poor performancelack of repeatability, inabilityto discriminate, and lack of agreementthat should be exam-ined regularly.6.3.1 RepeatabilityLack of repeatability occurs when as-sessor(s) cannot replicate thei

38、r ratings from one evaluation toanother of the same sample. It should be noted that assessmentof repeatability is only possible if assessors evaluate the samesample on at least two occasions, either during the samesession or on different sessions.6.3.1.1 Inadequate training, inconsistent scale usage

39、, lexi-con usage, and various psychological and physiological factorscan impair repeatability. Assessor fatigue, improper spacing ofsamples, poor instructions, inconsistent reference samples, andsample variation can also contribute to the problem. Studyredesign or retraining, or both, may be necessa

40、ry to reduce thevariation in repeatability.6.3.2 DiscriminationAn assessors inability to find sig-nificant differences among samples that are found to bedifferent by the panel as a whole may occur for several reasons,such as general or specific ageusia and anosmia or differencesin lexicon usage.6.3.

41、2.1 Using the same ratings across all samples for anattribute may indicate low sensory acuity resulting in theassessors inability to use the scale as they were trained. Poorlydiscriminating assessors may use similar ratings across allsamples in a “safe scale range” to cover their inability todiscern

42、 the attribute.6.3.2.2 The non-discriminating attributes should be identi-fied and training provided to the assessor on those attributes forwhich the samples are expected to differ. It may be necessaryto change the reference standards to better represent theattribute if previously used references ar

43、e not helpful for thepanel.6.3.3 AgreementAgreement is obtained when assessorsrate samples similarly in relation to each other. Similar ratingsindicate that the assessors are scoring the samples consistentlyfor each attribute.6.3.3.1 The data set should be carefully examined to deter-mine which indi

44、vidual assessors contribute to the dissimilarityof the attribute ratings. A lack of agreement may be due to adifference in the assessors discrimination, differences in scaleor lexicon usage, or both.6.3.3.2 Sometimes the main cause of a lack of agreementmay not be due to a poor assessor, but rather,

45、 an assessor whomay be more discriminating or more sensitive to an attribute.The identification of the origin of a disagreement is thereforeessential for identification of the appropriate corrective action.6.3.3.3 Assessors who vary on the perceived intensity inrelation to other assessors, but still

46、 show the same sampleranking pattern as the other assessors (magnitude typeinteractions), usually differ in scale usage. A disagreement inassessor ratings may also indicate that assessors do notassociate the same sensory perception with the attributes orvary on the perceived intensity due to individ

47、ual differences insensory acuity, thus causing cross-over interactions. A cross-over interaction occurs when an assessors mean score for aspecific sample is reversed in response pattern from those ofother panel members. All cross-over interactions should becarefully examined since they reduce chance

48、s of the panelfinding significant sample differences.6.3.3.4 Lexicon usage may also contribute to agreementissues when one assessor understands the attribute to meansomething different from the other panel members.6.4 Performance DiagnosticsScale usage and lexicon us-age are two diagnostics that can

49、 be examined to understandwhat is causing issues with agreement, discrimination, andrepeatability.6.4.1 Scale UsageInconsistent scale usage occurs whendifferent assessors use different ranges of the scale and alsodifferent areas or locations of the scale while rating the samesample (note: this is an assessor effect in ANOVA). Inconsis-tent scale usage for an overall panel can be consideredacceptable to a certain degree as long as assessors are consis-tent with their own behavior (across all samples) and are inagreement with the rest of the panel (for example, ra