Antonenko & Niederhauser

Antonenko, P.D., and Niederhauser, D.S. (2010)

The influence of cognitive load and learning in a hypertext environment

Computers in Human Behavior

Participants

Researchers have suggested that gender, handedness, and age can differentially affect brain wave activity (Andreassi, 2007; Fisch, 1999). Further,

EEG patterns can be influenced by brain disorders, or medications to treat brain disorder conditions (Andreassi, 2007). Thus, the subject pool was limited to right-handed, 18–23 year-old females with no known brain disorders.

Materials

 descriptions of 4 different learning theories (500 words + or – 5) . 7 hypertext links at each node (90 + or – 5 words). identical grammatical and syntactical structures were used across all texts (e.g., sentence length, number of clauses, stylistic devices). Also checked for readability and matched conceptual difficulty.

The hypertext presentation system was designed using guidelines

suggested by current web usability research (Nielsen, 2006).

Nodes were presented as a single frame using Georgia serif font,

with black lettering on a white background and no tracing images

or watermarks. Font size was set to 100 percent of the default

browser font size, which translated to approximately 16 point font

on the monitor used for the study.

EEG data

Electroencephalogram. EEG data were acquired using a Biopac

MP30 connected to a Macintosh G4 MiniMac computer. Electrode

placement was confined to one set of electrodes over the

pre-frontal cortex (F7) and one-over the parietal lobe (P3) in the

left hemisphere of the right-handed subjects. EEG data were collected

at a sampling rate of 500 Hz as each subject read each of

the four hypertexts. Electrode placement followed the Modified

Combinatorial Nomenclature expanded 10–20 system, as proposed

by the American Clinical Neurophysiology Society (Jasper, 1958).

The EEG software recorded brain wave rhythms as separate channels,

allowing identification of the following wave components: (a)

raw EEG signal, (b) alpha rhythm, (c) beta rhythm, and (d) theta

rhythm.

Event-Related Desynchronization percentage (ERD%) for alpha,

beta, and theta rhythms were used as online measures of brain

activity (Pfurtscheller & Lopes de Silva, 1999). Increased cognitive

load is associated with higher brain wave desynchronization for alpha

and beta rhythms, and higher brain wave synchronization for

the theta rhythm, when subjects move from a relaxed, eyes-closed

state (baseline) to an eyes-open, active-reading state (Basar, 2004;

Klimesch, 2005). Consequently, ERD%, which compares brain wave

power in the test condition with the brain wave power in the baseline

condition, is represented by a positive number for the subjects’

alpha and beta rhythms (reflecting wave desynchronization), and a

negative number for the theta rhythm (reflecting synchronization).

Thus, larger desynchronization percent values for alpha and beta

waves, and larger synchronization percent values for theta waves

indicate increased cognitive load.

The following formula was used to compute ERD% (Pfurtscheller

& Lopes de Silva, 1999):

ERD% =baseline interval band power-test interval band power

baseline interval band power x 100

Band power values of the subject’s alpha, beta, and theta brain

waves were estimated with the Biopac psychophysiometrical system

software. The Area function was used to calculate alpha, beta,

and theta wave power. This function computes the total area of

waveform under a straight line drawn between the endpoints.

Thus, this measure takes into account both wave frequency (Hz)

and amplitude (microV). Area under the curve for alpha, beta, and theta

brain wave rhythms was computed for a 20-s segment of the baseline

condition, which was obtained before a given subject started

reading each of the four texts. This yielded a unique baseline interval

band power of brain activity for the time period immediately

preceding reading each text that was used in the ERD% formula.

Area was then computed for the test interval band power. Using

the marker placed on the EEG recording as a referent, a 20-s test interval was established which included the 10 s spent reading the

primary passage immediately before selection of given link to a subordinate

node, and the 10 s spent reading immediately after clicking

the link to the subordinate node. Test interval band power was calculated

using this 20-s interval because subjects read and processed

lead information before they clicked on the link to open the subordinate

node in the lead-augmented hypertext condition. ERD% values

were computed for alpha, beta, and theta brain wave rhythms

on each of the seven links within each of the four hypertexts.

Average ERD% value was then computed for each of the three

brain wave rhythms under each of the two experimental conditions.

For example, the ERD% values for the 20-s intervals for the

alpha rhythm were averaged for each subject, yielding one alpha

ERD% value for each text. The alpha ERD% values for the two nolead

texts were then averaged, as were the alpha ERD% values for

the two lead-augmented texts. This yielded one grand alpha

ERD% value for the lead-augmented hypertext condition, and a second

alpha ERD% value for the no-lead condition. This procedure

was then used to compute grand averages for lead and no-lead

conditions for beta and theta rhythms. Since cognitive load is reflected

by more negative ERD% for the theta rhythm (Klimesch,

2005), and more positive ERD% for alpha and beta rhythms, we

used absolute values for measures of theta waves to help make

interpretation of results more intuitive. Grand average ERD% values

for alpha, beta, and theta rhythms served as dependent measures.

Procedure

Each sat at a table with an adjustable-height high-back chair with armrests.

Subjects faced a blank white wall, which extended beyond

peripheral vision on both sides. A full-size 110-key keyboard, wireless

mouse, and 17-inch LCD monitor were on the table in front of

the subject. The same computer and monitor were used for all

subjects.

The monitor surface was approximately 50 cm from the subject, and was set to 1280 by 854 pixels (the highest available resolution) and maximum level of brightness. Luminescence was measured at eye-level and at a distance of approximately 50 cm from the computer screen using a Tenma Digital Lux meter. Luminosity of hypertext displayed on the LCD monitor was equal across text conditions.

With the subject comfortably seated at the computer, the researcher

read a scripted verbal overview of treatment procedures to begin the session. He then attached disposable vinyl electrodes (Ag/AgCl) to two recording sites on the subject’s skull: (a) pre-frontal lobe (F7) to collect data on the power of beta and theta waves and (b) parietal lobe (P3) to collect data on the power of alpha waves. Measurements were referenced to the left mastoid with

the earlobe serving as ground. Electrode impedance was below

10 kX. The EEG signal passed through an Infinite Impulse Response

(IIR) bandpass filter to remove unintended artifacts of movement,

allowing us to retain only the frequency components that were

of interest in the present study: theta (4–7 Hz), alpha (8–13 Hz),

and beta (14–30 Hz). Two more sets of electrodes were attached

to collect the subject’s electrooculogram and electromyogram of

the dominant (right) hand, which was used to filter artifacts associated

with eye movement, blinking, hand movement and mouse clicking. Subjects were instructed to minimize unnecessary movement

during the hypertext reading and browsing task.

After all electrodes were placed and the EEG equipment was

activated, the subject sat in a relaxed state with her eyes closed until

an extended alpha pattern was noted. At that point the 20-s baseline brain wave rhythm sample was recorded and the subject was instructed to open her eyes (blocking the alpha rhythm) and read the first experimental hypertext. When the subject had finished reading she said ‘‘done,” completed a self-report of mental effort measure, and closed her eyes and relaxed until her brainwave patterns returned to the baseline condition (extended alpha).

Returning to baseline helped circumvent carryover effects from one treatment to the next.

The researcher brought up the next experimental text while the

subject was returning to baseline. After establishing and recording

the next baseline brain wave sample, the subject was instructed to open her eyes and read the second text, complete the mental effort

scale, and close her eyes until she returned to the baseline condition.

This procedure was repeated for the remaining two texts. The sequence of presenting texts to subjects was counterbalanced. When a subject had finished reading the fourth hypertext, she was instructed to close her eyes to return to an alpha state, which provided an end point for the brain wave

recording.

Analysis

A 2 x 5 repeated measures MANOVA was conducted to determine

the effect of leads on learners’ cognitive load. Presence of

leads (Lead vs. No-lead) served as a within-subject factor, and

the five measures of cognitive load: (a) reading time; (b) self-reported

mental effort; and Event-Related Desynchronization percentages

of (c) alpha; (d) beta, and (e) theta brain wave rhythms were used as dependent measures.

Results

The MANOVA for cognitive load measures was significant

(F(5,12) = 58.94, p < 0.01), prompting further analysis through a series

of univariate repeated measure ANOVAs. For each of the ensuing

ANOVAs, presence of leads served as the independent variable,

with each of the cognitive load measures serving as a dependent

measure. A main effect was found for reading time (F(1,16) = 5.55,

p < 0.05, MSE = 427.63). Subjects spent more time reading in the

lead condition than in the no-lead condition (X = 587.23,

SD = 116.06 and X = 571.83, SD = 126.20 s, respectively). Main effects

were also found for alpha, beta, and theta ERD%

(F(1,16) = 103.47, p < 0.01, MSE = 7.12; F(1,16) = 35.71, p < 0.01,

MSE = 15.34; F(1,16) = 252.56, p < 0.01, MSE = 1.03, respectively). Table

1 shows that mean alpha, beta, and absolute value of theta

ERD% in the no-lead condition was higher than in the lead condition.

These findings reveal lower cognitive load in the Lead condition.

No other results reached significance (p > 0.08).

Discussion

The aim of this study was to determine the influence of leads on

cognitive load and learning in hypertext. The study produced several

important findings. First, EEG-based cognitive load measures

showed that subjects’ brain wave activity was less intense when

they were accessing hypertext nodes via leads. Conversely, the

self-report of mental effort measure did not detect significant differences

in cognitive load between the two experimental conditions,

and subjects tended to spend more time reading lead augmented

hypertext. Second, use of leads appeared to produce a

positive effect on learning outcomes relative to domain and structural

knowledge acquisition.

The discrepancy between the results of EEG-based measures of

cognitive load, self-report of mental effort,  ie EEG refelects what goes on in real time whereas self report is retrospective

Hook HCI (2008)

Knowing, Communicating, and Experiencing through Body and Emotion

Kristina Höök (2008)

IEEE Transactions on Learning Technologies

OCTOBER-DECEMBER 2008 (Vol. 1, No. 4) pp. 248-259

1939-1382/08/$26.00 © 2008 IEEE

Published by the IEEE Computer Society

Three trends

·      New wearable technologies

·      Third wave HCI

·      New approach in learning research  -emotion and cognition  are interrelated.

Second wave of HCI

P248 ‘To deal with the complexities of collaboration, sociologists and ethnographers were consulted, providing richer descriptions of what people do when they work ‘together.

Third wave of HCI: ‘a movement that aims to design for experiences involving users emotionally, bodily, and providing for aesthetic experiences.

P248 ‘The goal of this new movement is to try and design for experiential values rather than efficiency, for entertainment and fun rather than work. This has brought a whole new dimension to the field……….. HCI researchers now have to deal with highly elusive, subjective, and holistic qualities of interaction—qualities that are hard to design for, but also hard to validate through traditional measurements. How can you, for example, measure the tenderness of a touch?’

Three examples of third wave HCI that are ‘non reductioist, do not try to measure emotion and respond with a technological intervention

(1) eMoto

Backgrounds for text messages

The user writes the text message and then chooses which expression to have in the background from a big palette of expressions mapped on a circle. The expressions are designed to convey emotional content along two axes: arousal and valence. For example, aggressive expressions have high arousal and negative valence and are portrayed as sharp, edgy shapes, in strong red colors, with quick, sharp animated movements. Calm expressions have low arousal and positive valence which is portrayed as slow, billowing movements of big, connected shapes in calm blue-green colors. To move around in the circle, the user has to perform a set of gestures using the stylus pen (that comes with some mobile phones) which we had extended with sensors that could pick up on pressure and shaking movements.

Studies of eMoto showed that the circle was not used in a simplistic one-emotion-one-expression manner, mapping emotions directly to what you are experiencing at the time of sending an emoto [ ⁵⁰ ]. Instead, the graphical expressions were appropriated and used innovatively to convey mixed emotions, empathy, irony, expectations of future experiences, surrounding environment (expressing the darkness of the night), and, in general, a mixture of their total embodied experiences of life and, in particular, their friendship. The "language" of colors, shapes, and animations juxtapositioned against the text of the message was open-ended enough for our users to understand them and express themselves and their personality with them. There was enough expressivity in the colors, shapes, and animations to convey meaning, but at the same time, their interpretation was open enough to allow our participants to convey a whole range of messages. We look upon the colors, shapes, and animations as an open "surface" that users may ascribe meaning to.

(ii) Affector

Affector is a distorted video window connecting the neighboring offices of two friends (and colleagues). A camera located under the video screen captures video as well as "filter" information  (Senger et al)

‘While the designers originally intended for this to communicate the emotional moods of the two participants to one another, it turned out that what was needed and what they ended up designing throughout the two-year process was to communicate something else. It became a tool for companionable awareness of the other in an aesthetically pleasing and creative way. It was not a simple identification of the partner's emotional mood, but a complex reading of what was going on in the other person's office, highlighting bodily movements, figuring out how this related to what they already knew about each others work life, and interpreting this.’

‘The distortions of the video became the "surface" that was open enough to invite creative use, and allowed the two participants to put meaning to the expressions based perhaps not only on the visual expression, but also on all the other knowledge they had of each other's work life. Pressing deadlines, late night work, getting papers accepted, or knowledge of each other's private life was mixed into their interpretation and meaning-making processes in using

(iii) Affective Diary: A Personal Logging System

‘As a person starts her day, she puts on the body sensor armband. During the day, the system collects time-stamped sensor data picking up movement and arousal. At the same time, the system logs various activities on the mobile phone: text messages sent and received, photographs taken, and the presence of Bluetooth in other devices nearby. Once the person is back at home, she can transfer the logged data into her Affective Diary. The collected sensor data as shown in Fig. 4 is presented as somewhat abstract, ambiguously shaped, and colored characters placed along a timeline’.

e.g. For Ulrica ( one of the participants)  then, her reflections using the diary provided an explanation of why people sometimes misunderstood her and her emotional reactions. Further, it led her to conclude that she should let more of her inner feelings be expressed in the moment. In short, Ulrica used the diary to reflect on her past actions and, as a consequence, to decide to change some of her behaviors; a process of reflection, learning, and change appeared to result from using the diary.

Themes and lessons learned

All three examples make use of sensor technologies as a means to capture something else than what we normally express through written text.

‘None of the systems try to represent these emotion processes inside the system or diagnose users' emotions based on their facial expressions or some other human emotion expression. Instead, they build upon the users own capabilities as meaning making, intelligent, active coconstructors of meaning, emotional processes, and bodily and social practices. In that sense, they are nonreductionist.’

‘An important lesson from these designs is that they have all left space, or "inscribable surfaces," open for users to fill with content [ ²¹ ]. If users recognize themselves or others through the activities they perform at the interface—if they look familiar to the user through the social or bodily practice they convey—they can learn how to appropriate these open surfaces. The activities of others need to be visible and what can be expressed users should be allowed to shape over time.’

Emotion in HCI: Three Design Approaches

(1) Affective Computing

‘The most discussed and widespread approach in the design of affective computing applications is to construct an individual cognitive model of affect from first principles and implement it in a system that attempts to recognize users' emotional states through measuring the signs and signals we emit in face, body, voice, skin, or what we say related to the emotional processes going on in inside. Emotions, or affect, are seen as identifiable states. Based on the recognized emotional state of the user, the aim is to achieve an as life-like or human-like interaction as possible, seamlessly adapting to the user's emotional state and influencing it through the use of various affective expressions. This model has its limitations, both in its requirement for simplification of human emotion in order to model it, and its difficult approach into how to infer the end-users emotional states through interpreting our sign and signals. This said, it still provides for a very interesting way of exploring intelligence, both in machines and in people.’

(ii) Hedonistic Usability

(iii) The Interactional Approach ( the approach adopted for the above three examples

‘An interactional approach to design tries to avoid reducing human experience to a set of measurements or inferences made by the system to interpret users' emotional states. While the interaction of the system should not be awkward, the actual experiences sought might not only be positive ones. eMoto may allow you to express negative feelings about others. Affector may communicate your negative mood. Affective Diary might make negative patterns in your own behavior painfully visible to you. An interactional approach is interested in the full range of human experience possible in the world’

Meltz paper at Stllar

Meltzoff, A.N., Kuhl, P.K., Movellan, J., and Sejnowski., T.J. (2009)

Foundations for a new science of learning

Science, 325, 284-288

p284 'Human learning and cultural evolution are supported by a paradoxical adaptation. We are born immature. During the first year if life , the brain of an infant is teeming with structural activity' with sensory processes developing before higher activity'

'Three principles are emerging from cross-disciplinary work in psychology, neuroscience, machine learning, and education, contributing to a new science of learning'  and, in particular, are useful for explaining,, language and social understanding.

1.    Learning is computational, implicit

2.    Learning is social, implicit

3.    Learning  is supported by brain circuits linking perception and action

1. Learning is computational

' infants and young children possess powerful computational skills that allow them to automatically infer structural models of their environment from the statistical patterns they experience' eg 'before they are three, children use frequency distributions to learn which phonetic units distinguish words in their native language' p 285 ' Statistical regularities and co variations in the world thus provide a richer source of information than previously thought' and the learning    running around these regularities is implicit. ' Learning from probabilistic input provides an alternative to Skinnerian reinforcement learning and Chomskian nativist accounts' of learning

2. Learning is social

p285 'Children do not compute statistics indiscriminately. Social cues highlight what and when to learn'  young infants 'more readily learn and enact an event when it is produced by a person than be an inanimate device. Machine learning studies show that systematically increasing a robot's social-like behaviours and contingent responsivity elevates young children's willingness to connect with it and learn from it'

3. Learning is supported by brain circuits linking perception and action

' Human social and language learning are supported by neural-cognitive systems that link the actions of self and other.'  The brain areas responsible for initiation of movement and its action overlap. ' Social learning, imitation, and sensorimotor experience may initially generate, as well as modify and refine, shared neural circuitry for perception and action'.  KRO to what extent and what is the nature of 'the close coupling and attunement between self and other, which is the hallmark of seamless social communication and interaction'

Social learning and understanding

Three social skills are foundational

1.    Imitation

2.    Shared attention

3.    Empathy and social emotions

 Imitation

'Learning by observing and imitating experts in the culture is a powerful social learning mechanism' ' Imitation if faster than individual discovery and safer than trial and error learning' ' Children can use third person information ( observation of others) to create first person knowledge. This is an accelerator for learning: Instead of having to work out causal relationships themselves children can learn from watching experts' ' Imitative learning is valuable because the behavioural actions of others "like me" serve as a proxy for one's own' ' Children do not slavishly duplicate what they see but reenact a person's goals and intentions' ie ' they produce the goal that the adult was striving to achieve, not the unsuccessful attempts. Children choose whom, when, and what to imitate and seamlessly mix imitation and self discovery to solve novel problems'  attempts in robotics to emulate infant imitation include direct (input-action) and more recently goal based approaches .

 Shared attention

'Social learning is facilitated when people share attention. Shared attention to the same object or event provides a common ground for communication and teaching. An early component of shared attention is gaze following' experimental evidence to show that ' we project our own experience onto others'. P286  ' The ability to interpret the behaviour the behaviour and experience of others by using oneself as a model is a highly effective learning strategy that may be unique to human........It would be useful if this could be exploited in machine  learning'

Empathy and social emotions

' The capacity to feel and regulate emotions is critical '  ' In humans, many affective       processes are uniquely social'. Children will even help and comfort a social robot that was crying Tanaka,Cicourel,Movellan, 2007) 'Brain imaging studies in adults show an overlap in the neural systems activated when people  receive a painful stimulus themselves or perceive that another person is in pain  Hein & Singer (2008) These neural reactions are modulated by cultural experience, training, and perceived similarity between self and other Hein & Singer (2008)

Language Learning  - as shedding light on the interaction between computational learning, social facilitation of learning, and shared neural circuitry for perception and production.

Evidence to show that developing infants pick up the statistical regularities of a language leading to neural commitment. ' However, experiments also show that the computations involved in language learning are "gated" by social processes (Kuhl, 2007). In foreign language learning experiments, social interaction strongly influenced infants' statistical learning. Infants exposed to a foreign language at 9 months learn rapidly, but only when experiencing the new language during social interchanges with other humans. 'Temporal contingencies may be critical'.

Idea of neural commitment

A similar pattern , ' passerine  birds learn conspecific song by listening to and imitating adult birds' ' In birds, as in humans, a social context enhances vocal learning'.