agent design echoes

Echoes - design of the agent

Building autonomous agents for children with autism

Aims for the agent -

p 46 ' to be able to act credibly both as a peer and as a tutor'

 achieve appropriately designed transactional support. 

Autonomy p 47 ' an agent that is able to decide independently how to act best in order to achieve a set if high level goals that have been delegated to it'

Be pro-active - engender motivation & attention

Reactive - adaptive

Have social ability - so that it can maximise the chances of the child experiencing a sense of self-efficacy

An ideal social agent with socio-emotional competence

ECHOES environment

12 learning activities that focus on two sub components that are challenging for children on the autistic spectrum -

(i) joint attention ( the ability to coordinate and share attention and emotions

(ii) symbol use -p48 ' understanding of meaning expressed through conventional gestures and words and ability to use non verbal means to share intentions'

Design for the agent

Derived from OCC and appraisal theory. OCC identifies 22 emotions. Agents provided with an affective system composed of emotional reaction rules, action tendencies, emotional thresholds, emotion decay rates

P 499 ' The agent experiences one or more of the 22 emotions of the OCC model based on its appraisal of the current external events and IRS subjective tendencies to experience certain emotions instead of others. The agents deal with these emotions by applying problem-focused or emotion-focused coping strategies

The agent

Advantages of an agent - tireless, persistent, consistent & positive

Positive, motivating, and supportive. Tends to be happy, does not get frustrated easily.

Actions - verbal (using simple language or key words), non-verbal (eye gaze and gesture)

Facial expression, a range of positive facial expressions ( involving lips, eyes, & eyebrows)

Evaluation

Response to bids for interaction ( post intervention cf Pre intention)

From the agent -  slight increase but not SS

From the human practitioner- did increase and almost reached SS

Initiation of bids

To the agent - numerical increase but not SS

To the human participant - very low and remained low

Sanger et al (2011) notes

Sanger, Lindenberger, Muller (2011)

Interactive brains, social minds

Communicative & Integrative Biology

P 655  difficult ' studying the complexities of social interaction in tightly controlled experimental settings' p 661 'real-life social interactions are spontaneous, reciprocal , and multimodal, and thereby pose great challenges to experimental design and the ability to draw causal inferences'

Definitions

Social cognition ' the mechanism that allows us to understand others '

Mentalizing , theory of mind ' the ability to represent other people's mental states (Frith & Frith (2002) as well as the knowledge needed for interaction and formation of social relationships

Social interaction is more narrowly defined 'turn taking among active, autonomous agents who follow social rules and control their action and reactions according to. Their perceptions'

Joint action ' any form of coordinated action bringing about change'

Coordination ' non accidental correlation between the behaviours of two or more systems that are in sustained coupling, or have been coupled in the past, or have been coupled to another, common system'

Interpersonal action coordination  occurs in ' the context of joint actions and coordination' note synchronisation of speech and movements does not qualify. '

Interpersonal action coordination

Discussed in terms of musicians and dancers but could apply to collaborative learning, especially face-to-face.

'interpersonal action coordination requires the perception,representation and anticipation of one's own and  partner actions'

Joint goal, (task) required. The task determines individual intentions (which may be very different especially in learning contexts)

Investigating the neural basis of social interaction

P 656 'Currently little is known about the brain areas that are involved and the neural mechanisms that implements interpersonally coordinated behaviour'

Designs and methods

Collectively the following  implicate fronto parietal areas

Focus

•    Agency

•   Cooperation &competition

•   Intentional stance

•   Self relevance and interpretation of relational stance

Single subjects intact interacting with

•   Computers

•   Virtual counterparts

•   Real counterparts

Methods and techniques involving  EEG

•   Formation of shared action representations' (40)

•   Movement coordination (41)

•   Different forms of action coordination (50,51)

Conclusions and outlook

P 661

' reconcile the dynamics of e phenomenon with the requirements of experimental control'

' there is a need for studies that assess the target behaviour as well as the behavioural cues exchanged between  interaction partners in real time, and relate these measures to neural synchronisation within and between brains' ....  ' Interbrain synchronisation during interpersonal interaction (KRO how important is this online?) coordination clearly depends on multimodal perceptual cues ( e.g. Gestures, facial expressions, movements), but the relation between these cues and Interbrain synchronisation is rarely assessed or analysed' 

Balaam et al (2011)

Balaam, M., Fitzpatrick, G., Good, J., &Harris, E. (2011)

Enhancing Interactional Synchrony with a ambient display

CHI 2011, May7-12, Vancouver, BC, Canada

Paper brings together 'research from social psychology with advances in HCI to explore a new role for pervasive and ambient technologies'  p 1

Pervasive and ambient technologies could provide  support both for encoding and decoding in authentic environments

Most of the technologies that have been developed to support social interaction (face-to-face) are informational  or support the verbal aspects e.g. RFID badges, iBand, GroupWear with mixed reactions from those involved.

Other approaches use para linguistics information to provide a view of an  interactional context.

Non verbal communication ' Research suggests that it communicates information about emotion, interpersonal relationships, and is an essential aspect of social competence' p 2 (2,7,20,22,37)

Rapport

'Rapport is one element of social interaction that is strongly encoded through nonverbal communication, and additionally theorised as key to the development of interpersonal relationships, and the success of an interaction. Rapport is believed to be crucial to develop trust, to create closeness, and be included in a  group(22) as well as help interactions feel smooth (36) .' P 2

Has three components ( see ref 37 )

1.    Mutual attentiveness

2.    Positivity

3.    Coordination ( sometimes referred to as interactional synchrony) - postural and movement. Much of this work done by Bernieri who foregrounds the importance of interactant proximity. Also that physical aspects (physical artefacts) of the task are important.

Each of which is encoded through non verbal communication with p7 ' with positivity most important at the outset  whilst coordination becomes more important later.

'there can be a disconnect between how rapport is encode and how observers decode rapport from NV signals' p 2. That  it is ' expressivity that drives social judgements' p 3

Ethics of using persuasive ambient technology is discussed on p 8.

the implication he is that more subtle NV communication can be missed  and led to the research aim - to use ambient technology to amplify (make more salient) the the more subtle NV that is sometimes missed by the observer.

'There are very few guidelines available' p 3 technologically speaking that is.

Scenario

Holiday planning in duos . Map display surrounded by representation of water that could be animated according to a wizard of Oz observer who took the role of the ambient technology of the future.

Ripples activated by

•   Simultaneous movement

•   Tempo similarity

•   Coordination

•   Posture matching

•   Back channel responses ( e.g. Head nodes, utterances showing agreement,/listening such as 'uh ha'

•   Eye contact

Explored ' the relationship between self reports of rapport, observer reports of rapport, and the body language of the participants in the holiday planning.' P 3 did the task twice (for personal use, for an older pair), once with the animation and once without. Untrained observers rated a video of the exchange , the video record was subject to a mosaic procedure that removed some of the expressive codes ( the  ones that are attended too at the expense of more subtle NVs)

Rapport scale by Bernieri (

ref 7) based on theory of 37.

'The data presented in the Results section indicates that our amplification of rapport through an ambient display increased the amount of interactional synchrony, and more specifically, the amount of coordination and posture similarity seen within an interaction by untrained observers.  At the same time, interact ants themselves were seemingly unaware of these increased behavioural manifestations of rapport, with no significant differences in their self reports of rapport between conditions'

Barron and mutual attention p 9 ' To aid this mutual attention, groups coordinated themselves and their work through the use of physical artefacts in the environment, conversational devices, and coordinated physical movement'   Overlaps with Salt and Bernieri

Discussion section contains more detailed account of HCI considerations

Refs

23, 34 spontaneous versus impression management

28 empathy

37, 5, 22, 7rapport

12, 18 Pervasive technology to support Group work