1、Interacting with Technology,Lecture 5. Laboratory versus Field: the Evaluation DebateDr. Dana Stanton Fraser,Interacting with Technology 2008,Contents,Methodological quantitative versus qualitativePractical constraintsChanges in society,Interacting with Technology 2008,Many Methods,Experiments Think
2、 aloud studies Questionnaires Focus groups Interviews Observational work coding Ethnographic work ,Interacting with Technology 2008,More typically lab-based (but not always, e.g. spatial cognition studies on mobile devices)QUANTITATIVE: E.G. EXPERIMENTS,Interacting with Technology 2008,Experiments:
3、Aim,To answer a question or test an hypothesis that predicts a relationship between two or more events, known as variables.E.g. Will spatial knowledge be superior following exploration of a VR simulation of a building or from exploring a model of a building?,Interacting with Technology 2008,Variable
4、s,Such hypotheses are tested by manipulating one or more of the variables. The variable that is manipulated is called the independent variable (the conditions to test this variable are setup independently before the experiment starts). In the study mentioned media type (VR vs model) is the independe
5、nt measure The dependent variable would be accuracy of spatial information (ie time to reach point a from point B) because the time to carry out the task depends on the media explored.,Interacting with Technology 2008,Variables and Conditions,In order to test an hypothesis the experimenter sets up t
6、he experimental conditions e.gCondition 1 Pretest Explore model Post test Condition 2 Pretest Explore VR simulation Post testControl condition against which to compare the results?,Interacting with Technology 2008,Participants,Within participants Counterbalancing requiredBetween participants Two dra
7、wbacks no. of participants needed, individual differences Advantage no order effectsMatched pairs on characteristics such as gender and expertise Difficult to match across all variables,Interacting with Technology 2008,Data Collection and Analysis,Data performance measures are taken e.g response tim
8、es, no. of errors Use graphs The data should be averaged across conditions to examine any differences Statistical tests such as t-tests and ANOVAs can reveal if the differences are significant Software packages such as SPSS are often used If there is no significant difference then the hypothesis is
9、refuted.,Interacting with Technology 2008,Summary of Experimental Design,State a causal hypothesis Manipulate independent variable Assign participants randomly to groups Use systematic procedures to test hypothesised causal relationships Use specific controls to ensure validity,Interacting with Tech
10、nology 2008,Typically out of the labQUALITATIVE: E.G. ETHNOGRAPHY,Interacting with Technology 2008,Ethnography by date,Anthropology (c. 1920s) e.g. E. E. Evans Pritchard Workplaces (c. 1950s) Chicago School E. C. Hughes, Becker etc. Computer-Supported Cooperative Work (c. 1990s) J. Hughes, C. Heath,
11、Interacting with Technology 2008,Ethnography by issue,Long history with its roots in anthropologyIs ethnography a method? eclectic group of techniques does not predefine a theoretical or conceptual status has been used in support of many methodological standpoints, e.g. distributed cognition etc,Int
12、eracting with Technology 2008,What does an ethnographer do?,assume human activities are socially organised commit to inquiring into patterns of interaction go into the field learn the ropes (question, listen, watch, talk, etc., with practitioners) take their time write a report,Interacting with Tech
13、nology 2008,Ethnography principles,Ethnography is naturalistic studies should be studies of real people and their activities, operating in their natural environment doesnt deal with artificial worlds & controlled versions of work but what is artificial? Mike Lynch: “Stop talking about science. Go to
14、 a laboratory - any laboratory will do - hang around a while, listen to conversations, watch the technicians at work, ask them to explain what they are doing, read their notes, observe what they say when they examine data, and watch how they move equipment around.“,Interacting with Technology 2008,E
15、thnography principles (2),Ethnography understands the world from the point of view of those who inhabit itit is behavioural, i.e. interested in the detail of the behaviour to a greater or lesser extent it is not behaviourist, i.e. it does not consider the behaviour itself as the appropriate level of
16、 analysis,Interacting with Technology 2008,Ethnographic data (1),Ethnographic data can include: general descriptions of behaviours, descriptions of physical layouts, close descriptions of conversation, thoughts and feelings, work sequences, anecdotes, examples, common occurrences, hypotheses etc.ana
17、lysis is skilful (but not complex) ethnographer usually provides examples,Interacting with Technology 2008,Ethnographic data (2),Technologies mostly used for data collection analysis is skilful! Questions, notebook, pen Tape recording less intrusive than video but data is less detailed awareness of
18、being recorded not a problem useful to record long explanations, especially highly technical or domain-specific kinds,Interacting with Technology 2008,Ethnographic data (3),Video Data can be analysed repeatedly off-site Time-consuming data collection and analysis Difficulty setting up and using equi
19、pment in some domains Missing visual actions that are peripheral to the video field-of-view Programs to support analysis e.g. Ethnograph, NUDIST,Interacting with Technology 2008,Carrying out an ethnographic study (1),individuals and organisations have ideas, expectations and fears about ethnography
20、Gaining access and an ability to be simultaneously likeable and anonymous will help Gaining acceptance may include working shifts, sharing conditions, a non-intrusive demeanour, sharing dress codes, but not sharing opinions unless absolutely necessary,Interacting with Technology 2008,Practical Const
21、raints,An example: The Living Exhibition Imagine you have been asked to study a museum exhibition You are working with both schools and designers who have certain goals for the project Curriculum and Pedagogical aims Technical innovation You need to study the exhibition in such a way as to achieve i
22、nteresting and coherent results taking into account the other stakeholders aims What do you test and how do you test it?,Interacting with Technology 2008,Variables A Control Books? Data collection participants dispersed over space and time simultaneously Unstable software performance and appearance
23、may change at any time,Interacting with Technology 2008,The living exhibition,Interacting with Technology 2008,Clear ability to design experiment in this very situation is impaired by practical constraintsExperiments could not cover the whole picture, only one or more selected constituent featuresOt
24、her methods of data collection and analysis are possible (e.g. ethnography), but do not specifically provide a thesis for understanding the data, unlike an experimentAny results are skilfully produced to inform stakeholders, and arent available according to a given procedure,Interacting with Technol
25、ogy 2008,Societal change (1),Evolving technological standards, e.g. large-scale systems developmentEvolving technological approaches, e.g. ubiquitous computing moving away from the desktop,Interacting with Technology 2008,Societal change (2),Changing face on human-human interaction, e.g. internet, m
26、obile phones (often only able to see one side of conversation)Interdisciplinary research a push towards disciplines collaborating e.g. psychology and computer science necessitating a mix of methods/approaches,Interacting with Technology 2008,Conclusions,Are traditional psychological methods appropri
27、ate?Are traditional psychological methods sufficiently adaptable?Should we be developing new methods in line with technological change?Can exploratory qualitative research lead into hypotheses for more controlled experimentation?,Interacting with Technology 2008,References,Crabtree, A. (2003) Design
28、ing Collaborative Systems: A Practical Guide to Ethnography, London: Springer-Verlag Fraser M, Stanton D, Ng M, Benford S, OMalley C, Bowers J, Taxen G, Ferris K, Hindmarsh J, (2003) Assembling History: Achieving Coherent Experiences with Diverse Technologies Proceedings of ECSCW 2003 Helsinki, Finl
29、and, Kluwer Heath, C.C., Hindmarsh, J. & P. Luff. (1999) Isolation and interaction: the fragmented world of the train driver on London Underground. Sociology 33, 3, pp. 555-575 Heath, C.C. and P. Luff (2000) Technology in Action Cambridge: Cambridge University Press. pp 1-269. Hemmings T, Crabtree A
30、, ( 2002 ) Ethnography for design?. The 1st International Workshop on “Interpretive“ Approaches to Information Systems, pages 122-124. Association of Information Systems Hughes, J. A., King, V., Rodden, T., and Andersen, H. (1994) “Moving out from the control room: Ethnography in system design“. In Proceedings of CSCW 94, Chapel Hill, North Carolina.,
