Knowing, Communicating, and Experiencing through Body
and Emotion
Kristina Höök
(2008)
IEEE
Transactions on Learning Technologies
OCTOBER-DECEMBER
2008 (Vol. 1, No. 4) pp. 248-259
1939-1382/08/$26.00
© 2008 IEEE
Published by the
IEEE Computer Society
Three trends
·
New wearable technologies
·
Third wave HCI
·
New approach in learning
research -emotion and cognition are interrelated.
Second wave of HCI
P248 ‘To deal with the
complexities of collaboration, sociologists and ethnographers were consulted,
providing richer descriptions of what people do when they work ‘together.
Third wave of HCI: ‘a
movement that aims to design for experiences involving users emotionally,
bodily, and providing for aesthetic experiences.
P248 ‘The goal of this
new movement is to try and design for experiential values rather than
efficiency, for entertainment and fun rather than work. This has brought a
whole new dimension to the field……….. HCI researchers now have to deal with
highly elusive, subjective, and holistic qualities of interaction—qualities
that are hard to design for, but also hard to validate through traditional
measurements. How can you, for example, measure the tenderness of a touch?’
Three examples of third wave HCI that are ‘non
reductioist, do not try to measure emotion and respond with a technological
intervention
(1) eMoto
Backgrounds for text
messages
The user writes the text
message and then chooses which expression to have in the background from a big
palette of expressions mapped on a circle. The expressions are designed to
convey emotional content along two axes: arousal and valence. For example,
aggressive expressions have high arousal and negative valence and are portrayed
as sharp, edgy shapes, in strong red colors, with quick, sharp animated
movements. Calm expressions have low arousal and positive valence which is
portrayed as slow, billowing movements of big, connected shapes in calm
blue-green colors. To move around in the circle, the user has to perform a set
of gestures using the stylus pen (that comes with some mobile phones) which we
had extended with sensors that could pick up on pressure and shaking movements.
Studies of eMoto showed
that the circle was not used in a simplistic one-emotion-one-expression manner,
mapping emotions directly to what you are experiencing at the time of sending
an emoto [ 50 ].
Instead, the graphical expressions were appropriated and used innovatively to
convey mixed emotions, empathy, irony, expectations of future experiences,
surrounding environment (expressing the darkness of the night), and, in
general, a mixture of their total embodied experiences of life and, in
particular, their friendship. The "language" of colors, shapes, and
animations juxtapositioned against the text of the message was open-ended
enough for our users to understand them and express themselves and their personality
with them. There was enough expressivity in the colors, shapes, and animations
to convey meaning, but at the same time, their interpretation was open enough
to allow our participants to convey a whole range of messages. We look upon the
colors, shapes, and animations as an open "surface" that users may
ascribe meaning to.
(ii) Affector
Affector is a distorted
video window connecting the neighboring offices of two friends (and
colleagues). A camera located under the video screen captures video as well as
"filter" information (Senger
et al)
‘While the designers originally intended for
this to communicate the emotional moods of the two participants to one another,
it turned out that what was needed and what they ended up designing throughout
the two-year process was to communicate something else. It became a tool for
companionable awareness of the other in an aesthetically pleasing and creative
way. It was not a simple identification of the partner's emotional mood, but a
complex reading of what was going on in the other person's office, highlighting
bodily movements, figuring out how this related to what they already knew about
each others work life, and interpreting this.’
‘The distortions of the
video became the "surface" that was open enough to invite creative
use, and allowed the two participants to put meaning to the expressions based
perhaps not only on the visual expression, but also on all the other knowledge
they had of each other's work life. Pressing deadlines, late night work, getting
papers accepted, or knowledge of each other's private life was mixed into their
interpretation and meaning-making processes in using
(iii) Affective Diary: A Personal Logging System
‘As a person starts her
day, she puts on the body sensor armband. During the day, the system collects
time-stamped sensor data picking up movement and arousal. At the
same time, the system logs various activities on the mobile phone: text
messages sent and received, photographs taken, and the presence of Bluetooth in
other devices nearby. Once the person is back at home, she can transfer the
logged data into her Affective Diary. The collected sensor data as shown in Fig. 4 is
presented as somewhat abstract, ambiguously shaped, and colored characters
placed along a timeline’.
e.g. For Ulrica ( one of
the participants) then, her reflections
using the diary provided an explanation of why people sometimes misunderstood
her and her emotional reactions. Further, it led her to conclude that she
should let more of her inner feelings be expressed in the moment. In short,
Ulrica used the diary to reflect on her past actions and, as a consequence, to
decide to change some of her behaviors; a process of reflection, learning, and
change appeared to result from using the diary.
Themes and lessons learned
All three examples make
use of sensor technologies as a means to capture something else than what we
normally express through written text.
‘None of the systems try
to represent these emotion processes inside the system or diagnose users'
emotions based on their facial expressions or some other human emotion
expression. Instead, they build upon the users own capabilities as meaning
making, intelligent, active coconstructors of meaning, emotional processes, and
bodily and social practices. In that sense, they are nonreductionist.’
‘An important lesson
from these designs is that they have all left space, or "inscribable
surfaces," open for users to fill with content [ 21 ].
If users recognize themselves or others through the activities they perform at
the interface—if they look familiar to the user through the social or bodily
practice they convey—they can learn how to appropriate these open surfaces. The
activities of others need to be visible and what can be expressed users should
be allowed to shape over time.’
Emotion in HCI: Three Design Approaches
(1) Affective
Computing
‘The most discussed and
widespread approach in the design of affective computing applications is to
construct an individual cognitive model of affect from first principles and
implement it in a system that attempts to recognize users' emotional states
through measuring the signs and signals we emit in face, body, voice, skin, or
what we say related to the emotional processes going on in inside. Emotions, or
affect, are seen as identifiable states. Based on the recognized emotional state
of the user, the aim is to achieve an as life-like or human-like interaction as
possible, seamlessly adapting to the user's emotional state and influencing it
through the use of various affective expressions. This model has its
limitations, both in its requirement for simplification of human emotion in
order to model it, and its difficult approach into how to infer the end-users
emotional states through interpreting our sign and signals. This said, it still
provides for a very interesting way of exploring intelligence, both in machines
and in people.’
(ii) Hedonistic
Usability
(iii)
The Interactional Approach ( the approach adopted for the above three examples
‘An interactional
approach to design tries to avoid reducing human experience to a set of measurements
or inferences made by the system to interpret users' emotional states. While
the interaction of the system should not be awkward, the actual experiences
sought might not only be positive ones. eMoto may allow you to express negative
feelings about others. Affector may communicate your negative mood. Affective
Diary might make negative patterns in your own behavior painfully visible to
you. An interactional approach is interested in the full range of human
experience possible in the world’
