multidiscplinary - echoes

Porayanska -Pomsta et al

Key features, benefits challenges of a multi-disciplinary approach

Journal of Personal and ubiquitous Computing

Literature

22. - emotion recognition

25, 40 - understanding the mental states of others

6,18 - LFA engage with robots more than humans (shared attention, turn taking), fail to generalise to a real world context

37, 38,29 - wearables

42 - abilities of autism

46 importance of reciprocity

ECHOES a socio- cognitive intervention

Multidisciplinary - theories, practices, methods, scientific tradition. Establish common ground and draw on strengths p2 Novelty of the approach lies in the way in which different methods and techniques are combined and applied.

Goal- enable social interaction skills.

Aim - develop tools for research in this area

Affective system for an agent - emotion regulation, recognise emotions, categorise emotions, express emotions

Objects in a garden as the focus of joint attention

Monitors - head posture, eye gaze, facial expression, screen touch data

Retain the development of resources within the users community of practice

Pilot - as a small scale intervention. Subjective also contributes to the design of the resource

Theory of mind - impute others mental state

Joint attention - there is a strong visual component , both the object and the other

How are objects in ECHOES linked into a narrative?

Two challenges p122 SCERTS, on which ECHOES as an intervention is based was developed for a human-human intervention context, in which practitioners use their long term experiences. Multiple data sources on which to base the decision. ' Another challenge relates to whether the child perceives the agent as an intentional being or merely an inanimate object' reciprocity is important 

Krach et al (2008)

Krach, S., Hegel, F., Sagerer, G, Binkofski, F., & Kircher, T (2008)

Can machines think? Interaction and Perspective taking with robots investigated via fMRI

PloS ONE 3, 7, e2597

A key question for robotic dialogue design is ' how to communicate the internal system state of a robot in a way that is understandable to the human user'

 Research question

do we attribute human like properties to machines? Even those that look and/or behave like humans?

Specifically

Activity of right TPC and medial Pre frontal cortex is hypothesised to linearly increase with the perceived grade of human likeness of the interactants

Studies investigating TOM with fMRI 

•   usually asked participants to take the perspective of various stimuli types, cartoon characters, persons on a photograph, ie ps asked to explicitly evaluate TOM in a highly controlled context

•   More recently used reciprocal interactive games between human participants  in order to access a more implicit perspective

Design

Highly interactive game scenario

4 opponents, ' all hypothetically differing liberally in the perceived grade of human likeness. 'Human likeness was operationalised by increasing the degree of anthropomorphism and embodiment'. (  Embodiment refers to the need for physicality in attribution processes, anthropomorphism as a way of explaining things in a way that we understand). It might be that a robot gives a greater sense of presence especially if it engages with the shared environment. Robots anthropomorphise more easily when they more like humans they are interacting with

•   A computer CP no human shape, no perceivable button pressing

•   A functionally designed Lego robot FR no human shape, button pressing with artificial hands

•   An anthropomorphic model. Human like shape, button pressing with human like hands

•   Human partner HP human shape, button pressing with human like hands

Video of the image beamed into the scanner. P was in fact always playing with the same confederate but did not know that assumed opponent was as per video image

Findings

' as a prerequisite to derive meaningful interpretations of the behavioural and functional imaging data on-line response behaviour and questionnaires indicated that all 20 participants

•   believed in the setting I.e they believed to really interact with the partners online'

•   ' neither reaction times nor button passing differed significantly between conditions'

•   'Overall, participants played rather competitive with a ratio of around 60/40 (competitive/cooperative) decisions, irrespective of the partner being played'

•   Debriefing questionnaire

•   Fun Intelligence CP

•   Competiveness CP

•   Human likeness and sympathy  rated only for AR, FR

fMRI findings

' participants increasingly engaged cortical regions corresponding to the classical TOM network the more respective games partners exhibited human like features'

TPJ - each comparison

MPFC pro innately dorsal - AP & HP only

Implications and conclusions

' To summarise the present study provides first evidence that the degree of human-likeness of a counterpart modulates its perception, influences the communication and behaviour, biases mental state attribution, and, finally , affects cortical activity during such interactions'

Shen et al (2009) notes

Shen, L., Wang, M., & Shen, R. (2009)

Affective e-Learning using "emotional" data to improve learning in pervasive learning environment

Educational technology & Society, 12, 2, 176-189

p 176 'Pervasive leaning is supported by wireless communication and wearable computing'

p 176, 177 Affective computing ' computer methods that are related to, derived from or deliberately designed to influence emotions. It involves two areas: emotion synthesis used to artificially imitate some of the physical or behavioural explosions associated with affective states , and emotion analysis which is often employed in decision making for interactive systems' e.g. synthesis (robots) analysis (monitoring).

 Theoretical background

' according to Fowler (1997) study, the relationship between learning performance and the arousal is a type of inverted -U curve, and people learn best when their emotions are at a moderate optimal arousal level'

Circumflex model (arousal, valence)

OCC model - cognitive appraisal model for emotions. 22 emotion categories to goals, relevant events, actions of an agent, attitudes of attractive or unattractive objects..........

Kort's Learning spiral model a state of evolution model Kort, Reilly and Picard (2001). p 180 proposed a four quadrant learning spiral model in which emotions change while the learner moves through quadrants and up the spiral. In quadrant 1 the learner  is experiencing positive affect and constructing knowledge. At this point, the learner is working through the material with ease. Once discrepancies arise between the learner's knowledge he moves to Q2 which consists of constructive knowledge and negative effect. Affect as  confusion. If this is not resolved learner may move to Q3 , frustration. If any misconceptions get discarded then moves onto Q4 'unlearning and positive affect' then propelled back to Q1 looking for new learning.

In Affective computing detection rates for non verbal cues is about 80%  therefore

' because physiological signals are more difficult to conceal or manipulate ....... And potentially less intrusive to text and measure, they are a more reliable representation of inner feelings '

Affective e-Learning model

Use two sets of data

1.     Collected Biophysical data HR, Skin conductance, BP, EEG. Incorporates elements from Circumflex - to describe biophysical in terms of  learner's emotions ( interest,  engagement, confusion, frustration, boredom, hopefulness, satisfaction, disappointment)

2.  OCC cognitive appraisal model for emotions.  considers contextual information of the learner and the learning setting. Infers learner emotions from this data

So in this model 1 represents the emotional states and 2 the appraisal mechanism . Assumes appraisal leads to emotions which in turn are reflected biologically. A Baysian method was used to model the relations between the emotional states and their causal variables.

Method

One subject, 20 trials, 40 minute biophysical recordings while student studies for examination in otherwise ,normal context. Self paced learning

EEG 3 electrodes, PCz, A1,A2.

Proposed Outcome of the system:  generates responses to the learner p181 ' based on his/her emotional states, cognitive abilities, and learning goals.'

 This model can also be used to customise the interaction between the learner and learning system, to predict learner responses to system behaviour, and to predict learner's future interaction with the learning system'

Results

Tested user satisfaction by comparing the number of  user adjustments required for the 'emotion aware' system v  a 'non emotion aware system' latter required more users interventions

Adding EEG moved best classification rate ( for emotional involvement) from 68.1 to 86.3%. P 187 ' this suggests the close relationship between brain waves and emotions during learning'