Hook HCI (2008)

Knowing, Communicating, and Experiencing through Body and Emotion

Kristina Höök (2008)

IEEE Transactions on Learning Technologies

OCTOBER-DECEMBER 2008 (Vol. 1, No. 4) pp. 248-259

1939-1382/08/$26.00 © 2008 IEEE

Published by the IEEE Computer Society

Three trends

·      New wearable technologies

·      Third wave HCI

·      New approach in learning research  -emotion and cognition  are interrelated.

Second wave of HCI

P248 ‘To deal with the complexities of collaboration, sociologists and ethnographers were consulted, providing richer descriptions of what people do when they work ‘together.

Third wave of HCI: ‘a movement that aims to design for experiences involving users emotionally, bodily, and providing for aesthetic experiences.

P248 ‘The goal of this new movement is to try and design for experiential values rather than efficiency, for entertainment and fun rather than work. This has brought a whole new dimension to the field……….. HCI researchers now have to deal with highly elusive, subjective, and holistic qualities of interaction—qualities that are hard to design for, but also hard to validate through traditional measurements. How can you, for example, measure the tenderness of a touch?’

Three examples of third wave HCI that are ‘non reductioist, do not try to measure emotion and respond with a technological intervention

(1) eMoto

Backgrounds for text messages

The user writes the text message and then chooses which expression to have in the background from a big palette of expressions mapped on a circle. The expressions are designed to convey emotional content along two axes: arousal and valence. For example, aggressive expressions have high arousal and negative valence and are portrayed as sharp, edgy shapes, in strong red colors, with quick, sharp animated movements. Calm expressions have low arousal and positive valence which is portrayed as slow, billowing movements of big, connected shapes in calm blue-green colors. To move around in the circle, the user has to perform a set of gestures using the stylus pen (that comes with some mobile phones) which we had extended with sensors that could pick up on pressure and shaking movements.

Studies of eMoto showed that the circle was not used in a simplistic one-emotion-one-expression manner, mapping emotions directly to what you are experiencing at the time of sending an emoto [ ⁵⁰ ]. Instead, the graphical expressions were appropriated and used innovatively to convey mixed emotions, empathy, irony, expectations of future experiences, surrounding environment (expressing the darkness of the night), and, in general, a mixture of their total embodied experiences of life and, in particular, their friendship. The "language" of colors, shapes, and animations juxtapositioned against the text of the message was open-ended enough for our users to understand them and express themselves and their personality with them. There was enough expressivity in the colors, shapes, and animations to convey meaning, but at the same time, their interpretation was open enough to allow our participants to convey a whole range of messages. We look upon the colors, shapes, and animations as an open "surface" that users may ascribe meaning to.

(ii) Affector

Affector is a distorted video window connecting the neighboring offices of two friends (and colleagues). A camera located under the video screen captures video as well as "filter" information  (Senger et al)

‘While the designers originally intended for this to communicate the emotional moods of the two participants to one another, it turned out that what was needed and what they ended up designing throughout the two-year process was to communicate something else. It became a tool for companionable awareness of the other in an aesthetically pleasing and creative way. It was not a simple identification of the partner's emotional mood, but a complex reading of what was going on in the other person's office, highlighting bodily movements, figuring out how this related to what they already knew about each others work life, and interpreting this.’

‘The distortions of the video became the "surface" that was open enough to invite creative use, and allowed the two participants to put meaning to the expressions based perhaps not only on the visual expression, but also on all the other knowledge they had of each other's work life. Pressing deadlines, late night work, getting papers accepted, or knowledge of each other's private life was mixed into their interpretation and meaning-making processes in using

(iii) Affective Diary: A Personal Logging System

‘As a person starts her day, she puts on the body sensor armband. During the day, the system collects time-stamped sensor data picking up movement and arousal. At the same time, the system logs various activities on the mobile phone: text messages sent and received, photographs taken, and the presence of Bluetooth in other devices nearby. Once the person is back at home, she can transfer the logged data into her Affective Diary. The collected sensor data as shown in Fig. 4 is presented as somewhat abstract, ambiguously shaped, and colored characters placed along a timeline’.

e.g. For Ulrica ( one of the participants)  then, her reflections using the diary provided an explanation of why people sometimes misunderstood her and her emotional reactions. Further, it led her to conclude that she should let more of her inner feelings be expressed in the moment. In short, Ulrica used the diary to reflect on her past actions and, as a consequence, to decide to change some of her behaviors; a process of reflection, learning, and change appeared to result from using the diary.

Themes and lessons learned

All three examples make use of sensor technologies as a means to capture something else than what we normally express through written text.

‘None of the systems try to represent these emotion processes inside the system or diagnose users' emotions based on their facial expressions or some other human emotion expression. Instead, they build upon the users own capabilities as meaning making, intelligent, active coconstructors of meaning, emotional processes, and bodily and social practices. In that sense, they are nonreductionist.’

‘An important lesson from these designs is that they have all left space, or "inscribable surfaces," open for users to fill with content [ ²¹ ]. If users recognize themselves or others through the activities they perform at the interface—if they look familiar to the user through the social or bodily practice they convey—they can learn how to appropriate these open surfaces. The activities of others need to be visible and what can be expressed users should be allowed to shape over time.’

Emotion in HCI: Three Design Approaches

(1) Affective Computing

‘The most discussed and widespread approach in the design of affective computing applications is to construct an individual cognitive model of affect from first principles and implement it in a system that attempts to recognize users' emotional states through measuring the signs and signals we emit in face, body, voice, skin, or what we say related to the emotional processes going on in inside. Emotions, or affect, are seen as identifiable states. Based on the recognized emotional state of the user, the aim is to achieve an as life-like or human-like interaction as possible, seamlessly adapting to the user's emotional state and influencing it through the use of various affective expressions. This model has its limitations, both in its requirement for simplification of human emotion in order to model it, and its difficult approach into how to infer the end-users emotional states through interpreting our sign and signals. This said, it still provides for a very interesting way of exploring intelligence, both in machines and in people.’

(ii) Hedonistic Usability

(iii) The Interactional Approach ( the approach adopted for the above three examples

‘An interactional approach to design tries to avoid reducing human experience to a set of measurements or inferences made by the system to interpret users' emotional states. While the interaction of the system should not be awkward, the actual experiences sought might not only be positive ones. eMoto may allow you to express negative feelings about others. Affector may communicate your negative mood. Affective Diary might make negative patterns in your own behavior painfully visible to you. An interactional approach is interested in the full range of human experience possible in the world’

de Laat & Lally (2003)

Maarten de Laat & Vic Lally (2003)

Complexity, theory and praxis: Researching collaborative learning and tutoring processes in a networked learning community

Instructional Science 31: 7-39,

P29 ‘ a kind of exploratory coversation between theory and praxis that may be mediated by methodology’

Theory

P8 Stenhouse 1983 ‘ argued that the development of theoretical understanding of educational action and doing educational research into the practical problems of education are inseparable.’

On complexity the authors argue that ‘no single theoretical framework is yet capable of offering us a sufficiently powerful articulation of description, rhetoric, inference or application’ ( the 4 ways in which theory can contribute). Theories differ in focus and some are, to an extent, complimentary.

Praxis ( teaching and learning - of both the teacher and the learner is also complex)

Methodology

Then there is the question of methodology. Theory influences what we focus on but does not tell us how. P10 ‘ Methodology assists the conversation between theory and praxis by providing rules for their interaction. However, as a community of researchers, we are still confronted with the methodological challenge of agreeing the rules’

Comment on Content analysis

· Can be cumbersome and time consuming

· Choice of coding strategy

· Can be issues of reliability and validity due to subjectivity when interpreting codes

· Multiplicity of meaning in one unit of analysis

· P11 ‘ more fundamentally, the available theoretical framework may not be sufficiently robust to enable valid inferences to be made about any of these processes from the textual traces’

· ‘Furthermore, what does one do about those aspects of learning that are

not expressed in, and therefore not amenable to, content analysis’

get Hara, Bonk and Angeli (2000) Content analysis of online discussion in an applied educational psychology course, Instructional Science 28(2) 115-152

p18 there is an important question when deciding on methodology p18 ‘ it could be argued that a more grounded approach, ( compared to content analysis) using categories that emerge from a reading of the messages, would provide a more ‘authentic’ summary of the intentions of the participants. In outr view this is a valid and important way of approaching the analysis. However we wanted to connect with some of the conceptual and theoretical ideas about learning and tutoring in the literature using schemas that were already in use’ It also means that results can be more easily shared and compared with other researchers.

Theoretical frameworks (of the study)

1. Social constructivist

Therefore search for evidence that learners are linking new knowledge to their prior knowledge to actively construct new internal representations.

2. the situativity of learning and meaningfulness

p13 ‘ learning as a set of processes by which a learner personalizes new ideas by giving meaning to them, based upon earlier experiences. However meaning is also rooted in, and indexed by experience ( Brown, Collins & Duguid, 1989). Therefore, each experience with an idea, and the environment of which that idea is part, becomes part of the meaning of that idea ( Duffy and Jonassen, 1992). Learning is therefore viewed and considered by us as situated by the activity in which it takes place ( Brown et al., 1989; Lave and Wenger, 1991). This view has led us to also seek evidence , in the online texts, for the cognitive, social, and affective processes in which participants engage in trying to make meaning of the ideas presented to them by the tasks they are undertaking’ We have also used Critical Event Recall (CER) to try to access the meaning making, and awareness of context, that participants use to make judgements and engage in activities in their course of study’

3. group dimensions of learning

( Lave, 1996, Levine, Resnick & Higgins, 1996, Wegerith, Mercer & Dawes, 1999) – the role of the group in shaping and driving individual cognition. There are three aspects

· the group situation p13 the situation itself may exert a strong mediating effect on individual cognitive and conceptual processes; the thinking of individuals is influenced by the group in which they are working’

· merger of intellectual and social processes

· ‘tension between the conceptual structure or understanding ( of the problem or ideas under discussion) of the group of individuals within it. These individual understanding may vary from each other as well as from the group. This tension may be the driving force for the collective processing of the group. So, for example when an individual member of the group expresses his or her opinion in relation to the shared public understanding of the group, this may be based on an attempt to synthesise the understanding with the public ( that is group or shared) one. The other members of the group might compare this new synthesis with their own understandings of the group-accepted version and their own disagreements with it. Depending on the outcome of this process there may be further interaction and negotiation until a new meaning or understanding is accepted by the group. ’

p14 ‘ In this way interaction between individuals, as well as their shared and individual cognitions, can be viewed as key aspects of the co-construction of knowledge, meaning and understanding

4. socio-cultural

p14 ‘ whereas the social-constructivist perspectives makes a distinction between the individual cognitive activities and the environment in which the individual is present, the socio-cultural perspective regards the individual as part of that environment’ According to this view , learning cannot be understood as a process that is solely in the mind of the learner (Van der Linden & Kanselaar, 2000). Learning is a process of participating in cultural practices, a process that structures and shapes cognitive activity ( Lave & Wenger, 1991)’ Therefore negotiation is an important aspect p14 ‘ additionally , they can motivate each other and help each other to finish the task’

Research approach

‘attempt to reveal and understand the richness of processes beyond the capability of any one of the methods, when used by itself’

p12 ‘ it is the interaction between teaching and learning that is of central importance in all educational endeavours. Therefore one of our key aims is to enquire systematically into this key educational interaction’ ‘Learning and tutoring, as ongoing sets of processes, happening in time and space, within an individual or a group, do not feature in detail in this general analysis’

‘ a parallel focus on group and individual processes’

Method

Content analysis of individual ( student and tutor) contributions and differences in learning and tutoring processes. (KRO but note that tutor contributions do not include any social or affective)

Integrated with the use of Critical Event Recall (CER) to probe learning and teaching that may not be expressed in the text records used for the data analysis

Participants

Masters in E-Learning based on an action research approach to professional development. A sophisticated group of professionals who communicate outside the formal Web CT environment. Many are mature learners who bring more than one relevant body of expertise to the course’

Focus on one group, 7 students and 1 tutor ( does not say how selected) engaged in a 10 week course.

RQ

What is the relationship between knowledge construction (learning) and tutoring processes as these evolved over time within the workshop?

Content analysis

Used NVivo p16 ‘ to search, by individual participant, for his or her contributions to each category of the learning and tutoring coding schemes’

Approx 1000 messages , sampled 10% from three phases, beginning, middle, and end.

Unit of analysis – p17 ‘ using semantic features such as ideas, argument chains, topics of discussion, or by regulative activities such as making a plan or explaining unclear information. ( Chi, M.T.H. (1997) Quantifying qualitative analyses of verbal data: A practical guide. Journal of the Learning Sciences 6, 271-313.). Thus, the content of the messages had to be read for meaning to determine segment boundaries.’ i.e. used semantic rather than syntactic boundaries.

Learning ( content analysis of student text)

Used code ( 23 items) developed by Veldhuis-Diermanse for a PhD thesis (unpublished)

Cognitive

Debating (5 items)

Using external information and experiences(4 items)

Linking or repeating internal information ( 2 items )

Affective

General: reacting emotionally to notes of peers without recting directly to the content of that note. This reaction can be +,_- or neutral

Asking for (general) feedback, responses or opinions by fellow-students

‘Chatting’ or ‘social talks ‘ contributions that are not relevant to solve the case/task

metacognitive ( some of these include social processes)

planning ( 3 items)

keeping clarity ( 3 items)

monitoring (2 activities)

rest activities ( units that cannot be decoded by using the above categories

Teaching ( 19 indicators) from ( Anderson, T., Rourke, L., Garrison, D.R. & Archer, W. (2000). Assessing Teaching Presence in a Computer Conference Context. http:///www.atl.ualberta.ca/cmc/publications.html)

Instructional design and organization ( 6 indicators)

Facilitating discourse (6 indicators)

Direct instruction (7 indicators)

Critical Event Recall p32 ‘has the potential to access asepects of learning and tutoring that are not directly available in the discussion transcripts’

P18 ‘ the recall enables the articulation of many previously unexpressed aspects of learning’

Trigger for tutor recall was

1. Summary based on the content analysis of the student interactions – ‘ an overview of the patterns of learning’

2. Full summary ( therefore tutor able to select what were the critical events for him)

Results

Content analysis

Learning (students)

Descriptive , presented by items and individuals in tabular form.

	beginning	mid	end
Cognitive	60%	81%	42%
metacognitive	20%	14%	20%
affective	14%	3%	8.9%
miscellaneous		1.9&	28.8%

Tutor/teaching

Instructional design codes at high level initially (51%), then 22%,33%

Facilitation initially (41%) then rises to (75%), 64%.

Some of the students make contributions that are rated as high for tutor processes, including two student who maintains this instructional mode in the mid phase whereas for another this declines during the mid and final stages.

Critical Incident Recall

Tutor agenda

· Concern with the learning process

· Constant awareness of the dynamics of the group

In particular

Containing Charles who is mostly thinking about getting task completed rather than the process of doing the task

Andrea also strong but supportive, also trying to hold Charles, so tutor tried to support her in doing this but staying neutral and keeping a small footprint

‘trying to avoid them making any decisions before the rest of the group appeared’

at one point offered up two ideas as catalysts.

During the middle phase p26 ‘conscious of sitting back’,’but I was tracking it, like a hawk really, but I was in a more relaxed mode’

Then started to ‘focus more on the dynamics within the group’

‘sense of two groups forming’ two, maybe three cracking on whereas ‘ the others were more shadowy figures’

Kate struggled the most and seemed able to find a niche , she articulated her struggle, ‘Andrea responded academically’…..’ whereas Pauline gave a more emotional response’ and ‘seemed to respond to her more as a person’

Articulates some assumptions about members of the group – e.g. Bill new to this form of learning ( note from learning analysis Bill’s participation level increased as the course progressed ? as the purpose of the exercise clarified for him.

Kate – difficult personal circumstances – p28 ‘her model of the experience to come was possibly of a more directive nature, so it was a struggle to accept a new mind set of knowledge being distributed amongst themselves’

Felicity ‘ she felt more comfortable thinking about the task than providing tutoring support for others’ – ‘ she was wuite a strong person but did not dominate’

Conclusions

P29 the tutor was actively engaged in a social process of actively designing his involvement through interaction, with a view to actively enhancing the learning of the group members. He was concerned ……. To maintain balance and integration within the group’

‘the complexity of practice in networked collaborative learning envionments is such that models of social-constructivism, situated learning and socio-cultural theory are not, separately capable of providing an account of the role of learning in meaning making, the function of context or the power of the interaction between tutoring and learning processes’

‘results show …. How professionals collaborate successfully to develop their own practice