participant level analysis

Wise, A., Perera, N., Hsiao,Y-T., Speer, J., Marbouti, F. (2011)

Microanalytic case studies of individual participation patterns in an asynchronous online discussion in an undergraduate blended course

Internet and Higher Education

doi:10.1016/j.iheduc.2011.11.007

Looks at temporility and individuality cf aggregated data

p1 ( of online version)

‘Despite the importance placed on asynchronicity only limited research has investigated online discussions from a temporal perspective (Dringus & Ellis, 2010)’

‘Research has not yet probed in detail the individual experience of participation in asynchronous discussions over time’

p2(of online version)

‘the majority of research on how students interact with discussions aggregates data over students and time’ ‘there is no reason to presume that students all interact as the “average” one does.’

‘Case studies are useful in generating understanding of complex phenomena that have not yet been studied extensively (Yin,2003).

The study

Makes use of log file data , unpacks the temporal pattern for each students and links it to the forum content’

‘Whilst the study takes a data driven approach to understand student behaviours , to ground the work conceptually the empirical patterns found are considered in relation to a theoretical taxonomy for discussion participation proposed by Knowlton(2005).’

Taxonomy for participating in online discussions -5 kinds of student participation (Knowlton, 2005)

1. passive
2. developmental – social conversations and locus for community building
3. generative- space to develop one ideas individually and report them to the instructor.
4. Dialogic – forum for interacting with others and their ideas to build and clarify understandings
5. Metacognitive forum for interacting with others and their ideas to build and clarify understandings and an opportunity to reflect on the process of knowledge development. n.b page 4 ‘ the characteristics of metacognitive participation manifest internally; one external indicator might be additional time spent reviewing one’s own and others’ posts.

P3 ( of online version)

RQs

1. What are different patterns of behaviour that individual student exhibit as they participate in an asynchronous online discussion?
2. How do these patterns relate to the theoretical levels of participation in Knowlton’s taxonomy?

Method

Blended, with students required to participate in three asynchronous discussions worth 9% of grade. There was an Introductions week in which students were given a chance to get to know each other and the discussion tool. In each discussion , students were asked to work with their group to collectively solve an organisational behaviour challenge. To support participation students were provided with some seed questions.

Data extraction and processing log of activities – view, post, make post, edit one’s own post with a time date stamp

Based on this data 4 students were selected as representing extremes ( e.g. many frequent visits, few but extended sessions. Etc)

Analysis -Stages of microanalytic case study construction

1. Overview of activity

‘Overall sense of each student’s behaviour by looking at their aggregate behaviour in the Motivation discussion. Pie charts were created for each student (edits, reads, posts, reviews, scans) to examine their relative proportion. In addition, a table was created to outline each student’s activity broken out by session. From this data, an initial portrait was created; this description was treated as a tentative characterisation about the learner’s behaviours, one that would be tested and refined as the data was examined in more depth.’

2. Temporal microanalysis of log-file data

The log file was processed temporally; and thus the combination of this processed log-file data and the dynamic discussion maps supported a meaningful reconstruction of students’ behaviours in the forum. In addition, the contents of the students’ posts were used to help contextualize their actions in the discussion. Their resulting narrative deppened the characterization of each students’ partiaiption

3. Comparison with participation taxonomy

Results & Discussion

P7 ( of online version)

• ‘ The three students ….. exhibit distinct, complex, and seemingly purposeful patterns of behaviour. In particular, the temporal microanalysis provided insight into individual students’ experiences in the discussions, deepening and often dramatically altering the interpretation suggested by the aggregate data’

• The way students interact with the discussion is ‘more varied and nuanced than previously known’ eg selection strategy of which posts to read.

• ‘The students also differed in how they expressed their voice in the discussions. Eg used language that referred to thecontributions of others cf simply acknowledging’.

• ‘The participation patterns ……. Align well with three of the levels of Knowlton’s (2005) taxonomy.

Recommendations

P8 (of the online version)

• ‘instructors should help students to understand the purpose of online discussion’ …
• students do better when given clear guidelines ( Hew, Cheung & Ng, 2008). Reflective journal on whether/how/what they have learned from peers. Dashboard to indicate students own performance against others, class averages ( KRO need to be real others)

social presence rourke 1999

Liam Rourke, Terry Anderson, D. Randy Garrison, Walter Archer (1999)

Assessing Social Presence In Asynchronous Text-based Computer Conferencing

The Journal of Distance Education / Revue de l'Éducation à Distance, Vol 14, No 2 (1999)

Community of inquiry Model -

was specifically designed to guide the use of computer conferencing to support critical thinking in higher education.

Models learning occurring in the space where cognitive, social and teacher influences overlap

Cognitive presence - the extent to which the participants in any particular configuration of a community of inquiry are able to construct meaning through sustained communication.

social presence, defined as the ability of learners to project themselves socially and emotionally in a community of inquiry. – therefore it supports the cognitive and affective elements of learning. IN CMC, unlike FTF instructional contexts, It is a function of both learners and teachers

“Teacher immediacy is conceptualized as those nonverbal behaviors that reduce physical and/or psychological distance between teachers and students” (p. 544). She found that engaging in eye contact with students, adopting a relaxed body posture, using gestures, and smiling improved students affect toward the practices promoted in the course, the subject matter of the course, and the course instructor.

Gorham (1988) expanded the definition of teacher immediacy behaviors to include oral behaviors such as talking about experiences that have occurred outside class, using humor, addressing students by name, and praising students’ work or comments. Her results suggest that these types of behavior also contributed significantly to students’ affective learning

immediacy

traced back to Mehrabian (1969, p203 ) ‘those communication behaviors that enhance closeness to and nonverbal interaction with another” - not implies that immediacy is given by the non verbal

Sproull and Keisler (1986) regarded the implications of Mehrabian’s (1969) work from a different perspective. They argued that the critical difference between face-to-face communication and mediated communication was the absence of social context cues. Their research indicated that the lack of cues to define the nature of a social situation led to uninhibited communication such as hostile and intense language (i.e., flaming), greater self-absorption, and a resistance to defer speaking turns to higher-status participants.

CMC & immediacy

The capacity of CMC to support highly affective interpersonal interactions is supported by studies that focus on its use in educational settings. Angeli, Bonk, and Hara (1998) conducted a content analysis of a course conducted entirely through CMC. They found that 27% of the total message content consisted of expressions of feeling, self-introductions, jokes, compliments, greetings, and closures. McDonald (1998) studied the development of group dynamics in educational computer conference settings and found that expressions of openness and solidarity were significant elements, rising from 18% and 40% of the total respectively when the conference began to 36% and 54% at its conclusion.

Analysis Framework for social presence

An important step in content analysis is the development of categories and indicators that researchers can then use to analyze the transcripts. We arrived at the categories for social presence that we used through an iterative process. First, behavioral indices were derived from the three categories of social presence articulated by Garrison et al. (2000, i.e., emotional expression, open communication, and group cohesion). Second, indicators of social interaction that had been derived from the media capacity, teacher presence, and group interaction literature were applied deductively to the analysis. Third, additional indices were deduced from careful readings of the transcripts and then added to the coding scheme. This process culminated in the formation of three broad categories of communicative responses that contribute to social presence. These categories correspond directly to Garrison et al.’s original categories. However, they were relabeled to reflect better the nature of the emergent indicators that define them in this study. Open communication is now referred to as interactive responses, referring to indices of threaded interchanges combined with messages of a socially appreciative nature. Emotional presence has been renamed as affective responses and group cohesion as cohesive responses.

Affective

Kuehn (1993) noted that text-based, asynchronous interlocutors employ unconventional symbolic representations such as emoticons to facilitate expressiveness in the medium. Gunawardena and Zittle (1997) found that conference participants “enhanced their socioemotional experience by using emoticons to express missing nonverbal cues in written form” (p. 8).

Teacher immediacy literature has identified the use of humor as a contributory factor to immediacy and subsequently to learning (Christenson & Menzel, 1998;

Cutler (1995) explains that “the more one discloses personal information, the more others will reciprocate, and the more individuals know about each other the more likely they are to establish trust, seek support, and thus find satisfaction” (p. 17)

Interaction

Eggins and Slade (1997) add that responses and rejoinders serve several beneficial purposes in conversation. They build and sustain relationships, express a willingness to maintain and prolong contact, and tacitly indicate interpersonal support, encouragement, and acceptance of the initiator. Using the “reply” feature to post messages, quoting directly from the conference transcript, and referring explicitly to the content of others’ messages are all types of interactive response in CMC.

Walberg (1984) conducted a meta-analysis of 3,000 studies that examined the effects of educational interventions designed to improve academic achievement. Reinforcement was at the top of his list and had an effect size of 1.17.

Complimenting and acknowledging, and expressing appreciation are ways of communicating reinforcement in a text-based medium.

Cohesive

This category is exemplified by activities that build and sustain a sense of group commitment.

Phatics are defined as communication “used to share feelings or to establish a mood of sociability rather than to communicate information or ideas” (GuruNet, 1999)

Vocatives, that is, addressing participants by name, are also an important expression of cohesion.

A variation of the vocative effect occurs at the group level when participants refer to the group with inclusive pronouns such as we, our, us, or group.

Eggins and Slade (1997) note, disagreement and critical evaluation are more characteristic of those who share strong bonds, rather than of new or transient acquaintances.

Method

Corpus extracted from week 5 of two( 14 members in 1 and 17 in the other) text based 13 week modules. New topic each week,

Discussion

the presence of replies and quoted messages may be a superficial artifact of conferencing communication rather than a defining indicator of social presence. The remaining indicators represent a more labor-intensive, and thus a more conscious and willful, effort on the part of the student to interact with others. According to this hypothesis, referring to other students by name and referring explicitly to the contents of another’s message are better indicators of interaction than having another student’s name or message appear automatically due to software features.

overview of articles reviewing content analysis

Som Naidu & Sanna Jarvela (2006)

Analyzing CMC content for what?

Computers and Education, 46,1, 96-103

Review of the articles in the special edition

What needs to be understood?

Leads to the question of what constitutes collaborative learning and the processes that determine its success

What is it? P99 ‘Collaboration involves sharing information, understanding, experience and expertise. There can be many types of collaborative activity which may manifest in several ways while serving many different functions. For example collaborative could serve as a ‘ culturative process which helps participants to become members of a knowledge community that are different from knowledge communities they already belong to’

KRO Why is collaborative learning desirable ? –CF – an approach to learning based on doing something and sharing the doing with peers

P97 ‘ provides a natural setting for self explanation and explaining to others’

P97 ‘Can interaction be more rich in the service of learning when it is not spatial ( in terms of teacher-student interaction or discourse) but virtual, asynchronous, multimediated and not linear?’

‘What are the interactions for supporting productive joint engagement and shared understanding?’

Collaborative traits . Authors claim that these are learned behaviours p100 ‘ characteristics of this trait include a willingness to share, demonstration of respect towards alternative views and dispositions, and the ability to listen carefully and attentively (Koschmann, 1996).

Methodological considerations

P98 ‘ Strategies for the analysis of a particular type of CMC application will have to be carefully selected or developed in alignment with the purposes that the conversation is designed to engender. Furthermore, care needs to be exercised against over analyzing CMC content, or attempting to apply overly reductionist strategies to the study os a rather complex communication channel ( Hemelo-Silver, 2003; Schrire, 2005)

Articles in the special edition identified the following as important steps

P98

· ‘Determination of a unit of analysis

· Development of a segmentation procedure

· Determination of the reliability of the segmentation procedure

· Development of coding categories and rules and

· Determination of the reliability of the coding strategies’

What kinds of RQs

What are successful and critical attributes of CSCL environments

What are the successful and critical attributes of collaborative learners?

How can one generate and engender these ‘attributes and behaviours’ in CSCL environments

Need to consider whether the process of collaboration is facilitated or directed

Theoretical frameworks

P99 ‘strategies should have a strong theoretical base’ ‘ It is arguable that a search for a theoretical base for research into computer mediated communications is rather far fetched and unrealistic. After all CMC is not much more than writing short letters via networked computers. Yet, it is different ……. Because it has a unique set of forms, norms and conventions’

P100 ‘During the recent years there has been a lively discussion on whether we should follow an individual or social perspectives on activity and should we conceptualize the current understanding on situative or cognitive approaches ( e.g. Anderson, Greeno, Reder and Simon, 2000). Through the influence of sociocultural and situated cognition theories, it has been recognized that individual learners are also influenced by social values and context in which the learning takes place. Cognition and social interaction is no more a separate variable or a distinct factor, which can be applied in explanation of an individual readiness to act or learn – but reflective of the social and cultural environment’

Conclusion

Complex interaction of cognitive, social, emotional, motivation

Need p101 ‘data from multiple sources’ e.g. Schrire(2005) –in the special issue.

de Laat & Lally (2003)

Maarten de Laat & Vic Lally (2003)

Complexity, theory and praxis: Researching collaborative learning and tutoring processes in a networked learning community

Instructional Science 31: 7-39,

P29 ‘ a kind of exploratory coversation between theory and praxis that may be mediated by methodology’

Theory

P8 Stenhouse 1983 ‘ argued that the development of theoretical understanding of educational action and doing educational research into the practical problems of education are inseparable.’

On complexity the authors argue that ‘no single theoretical framework is yet capable of offering us a sufficiently powerful articulation of description, rhetoric, inference or application’ ( the 4 ways in which theory can contribute). Theories differ in focus and some are, to an extent, complimentary.

Praxis ( teaching and learning - of both the teacher and the learner is also complex)

Methodology

Then there is the question of methodology. Theory influences what we focus on but does not tell us how. P10 ‘ Methodology assists the conversation between theory and praxis by providing rules for their interaction. However, as a community of researchers, we are still confronted with the methodological challenge of agreeing the rules’

Comment on Content analysis

· Can be cumbersome and time consuming

· Choice of coding strategy

· Can be issues of reliability and validity due to subjectivity when interpreting codes

· Multiplicity of meaning in one unit of analysis

· P11 ‘ more fundamentally, the available theoretical framework may not be sufficiently robust to enable valid inferences to be made about any of these processes from the textual traces’

· ‘Furthermore, what does one do about those aspects of learning that are

not expressed in, and therefore not amenable to, content analysis’

get Hara, Bonk and Angeli (2000) Content analysis of online discussion in an applied educational psychology course, Instructional Science 28(2) 115-152

p18 there is an important question when deciding on methodology p18 ‘ it could be argued that a more grounded approach, ( compared to content analysis) using categories that emerge from a reading of the messages, would provide a more ‘authentic’ summary of the intentions of the participants. In outr view this is a valid and important way of approaching the analysis. However we wanted to connect with some of the conceptual and theoretical ideas about learning and tutoring in the literature using schemas that were already in use’ It also means that results can be more easily shared and compared with other researchers.

Theoretical frameworks (of the study)

1. Social constructivist

Therefore search for evidence that learners are linking new knowledge to their prior knowledge to actively construct new internal representations.

2. the situativity of learning and meaningfulness

p13 ‘ learning as a set of processes by which a learner personalizes new ideas by giving meaning to them, based upon earlier experiences. However meaning is also rooted in, and indexed by experience ( Brown, Collins & Duguid, 1989). Therefore, each experience with an idea, and the environment of which that idea is part, becomes part of the meaning of that idea ( Duffy and Jonassen, 1992). Learning is therefore viewed and considered by us as situated by the activity in which it takes place ( Brown et al., 1989; Lave and Wenger, 1991). This view has led us to also seek evidence , in the online texts, for the cognitive, social, and affective processes in which participants engage in trying to make meaning of the ideas presented to them by the tasks they are undertaking’ We have also used Critical Event Recall (CER) to try to access the meaning making, and awareness of context, that participants use to make judgements and engage in activities in their course of study’

3. group dimensions of learning

( Lave, 1996, Levine, Resnick & Higgins, 1996, Wegerith, Mercer & Dawes, 1999) – the role of the group in shaping and driving individual cognition. There are three aspects

· the group situation p13 the situation itself may exert a strong mediating effect on individual cognitive and conceptual processes; the thinking of individuals is influenced by the group in which they are working’

· merger of intellectual and social processes

· ‘tension between the conceptual structure or understanding ( of the problem or ideas under discussion) of the group of individuals within it. These individual understanding may vary from each other as well as from the group. This tension may be the driving force for the collective processing of the group. So, for example when an individual member of the group expresses his or her opinion in relation to the shared public understanding of the group, this may be based on an attempt to synthesise the understanding with the public ( that is group or shared) one. The other members of the group might compare this new synthesis with their own understandings of the group-accepted version and their own disagreements with it. Depending on the outcome of this process there may be further interaction and negotiation until a new meaning or understanding is accepted by the group. ’

p14 ‘ In this way interaction between individuals, as well as their shared and individual cognitions, can be viewed as key aspects of the co-construction of knowledge, meaning and understanding

4. socio-cultural

p14 ‘ whereas the social-constructivist perspectives makes a distinction between the individual cognitive activities and the environment in which the individual is present, the socio-cultural perspective regards the individual as part of that environment’ According to this view , learning cannot be understood as a process that is solely in the mind of the learner (Van der Linden & Kanselaar, 2000). Learning is a process of participating in cultural practices, a process that structures and shapes cognitive activity ( Lave & Wenger, 1991)’ Therefore negotiation is an important aspect p14 ‘ additionally , they can motivate each other and help each other to finish the task’

Research approach

‘attempt to reveal and understand the richness of processes beyond the capability of any one of the methods, when used by itself’

p12 ‘ it is the interaction between teaching and learning that is of central importance in all educational endeavours. Therefore one of our key aims is to enquire systematically into this key educational interaction’ ‘Learning and tutoring, as ongoing sets of processes, happening in time and space, within an individual or a group, do not feature in detail in this general analysis’

‘ a parallel focus on group and individual processes’

Method

Content analysis of individual ( student and tutor) contributions and differences in learning and tutoring processes. (KRO but note that tutor contributions do not include any social or affective)

Integrated with the use of Critical Event Recall (CER) to probe learning and teaching that may not be expressed in the text records used for the data analysis

Participants

Masters in E-Learning based on an action research approach to professional development. A sophisticated group of professionals who communicate outside the formal Web CT environment. Many are mature learners who bring more than one relevant body of expertise to the course’

Focus on one group, 7 students and 1 tutor ( does not say how selected) engaged in a 10 week course.

RQ

What is the relationship between knowledge construction (learning) and tutoring processes as these evolved over time within the workshop?

Content analysis

Used NVivo p16 ‘ to search, by individual participant, for his or her contributions to each category of the learning and tutoring coding schemes’

Approx 1000 messages , sampled 10% from three phases, beginning, middle, and end.

Unit of analysis – p17 ‘ using semantic features such as ideas, argument chains, topics of discussion, or by regulative activities such as making a plan or explaining unclear information. ( Chi, M.T.H. (1997) Quantifying qualitative analyses of verbal data: A practical guide. Journal of the Learning Sciences 6, 271-313.). Thus, the content of the messages had to be read for meaning to determine segment boundaries.’ i.e. used semantic rather than syntactic boundaries.

Learning ( content analysis of student text)

Used code ( 23 items) developed by Veldhuis-Diermanse for a PhD thesis (unpublished)

Cognitive

Debating (5 items)

Using external information and experiences(4 items)

Linking or repeating internal information ( 2 items )

Affective

General: reacting emotionally to notes of peers without recting directly to the content of that note. This reaction can be +,_- or neutral

Asking for (general) feedback, responses or opinions by fellow-students

‘Chatting’ or ‘social talks ‘ contributions that are not relevant to solve the case/task

metacognitive ( some of these include social processes)

planning ( 3 items)

keeping clarity ( 3 items)

monitoring (2 activities)

rest activities ( units that cannot be decoded by using the above categories

Teaching ( 19 indicators) from ( Anderson, T., Rourke, L., Garrison, D.R. & Archer, W. (2000). Assessing Teaching Presence in a Computer Conference Context. http:///www.atl.ualberta.ca/cmc/publications.html)

Instructional design and organization ( 6 indicators)

Facilitating discourse (6 indicators)

Direct instruction (7 indicators)

Critical Event Recall p32 ‘has the potential to access asepects of learning and tutoring that are not directly available in the discussion transcripts’

P18 ‘ the recall enables the articulation of many previously unexpressed aspects of learning’

Trigger for tutor recall was

1. Summary based on the content analysis of the student interactions – ‘ an overview of the patterns of learning’

2. Full summary ( therefore tutor able to select what were the critical events for him)

Results

Content analysis

Learning (students)

Descriptive , presented by items and individuals in tabular form.

	beginning	mid	end
Cognitive	60%	81%	42%
metacognitive	20%	14%	20%
affective	14%	3%	8.9%
miscellaneous		1.9&	28.8%

Tutor/teaching

Instructional design codes at high level initially (51%), then 22%,33%

Facilitation initially (41%) then rises to (75%), 64%.

Some of the students make contributions that are rated as high for tutor processes, including two student who maintains this instructional mode in the mid phase whereas for another this declines during the mid and final stages.

Critical Incident Recall

Tutor agenda

· Concern with the learning process

· Constant awareness of the dynamics of the group

In particular

Containing Charles who is mostly thinking about getting task completed rather than the process of doing the task

Andrea also strong but supportive, also trying to hold Charles, so tutor tried to support her in doing this but staying neutral and keeping a small footprint

‘trying to avoid them making any decisions before the rest of the group appeared’

at one point offered up two ideas as catalysts.

During the middle phase p26 ‘conscious of sitting back’,’but I was tracking it, like a hawk really, but I was in a more relaxed mode’

Then started to ‘focus more on the dynamics within the group’

‘sense of two groups forming’ two, maybe three cracking on whereas ‘ the others were more shadowy figures’

Kate struggled the most and seemed able to find a niche , she articulated her struggle, ‘Andrea responded academically’…..’ whereas Pauline gave a more emotional response’ and ‘seemed to respond to her more as a person’

Articulates some assumptions about members of the group – e.g. Bill new to this form of learning ( note from learning analysis Bill’s participation level increased as the course progressed ? as the purpose of the exercise clarified for him.

Kate – difficult personal circumstances – p28 ‘her model of the experience to come was possibly of a more directive nature, so it was a struggle to accept a new mind set of knowledge being distributed amongst themselves’

Felicity ‘ she felt more comfortable thinking about the task than providing tutoring support for others’ – ‘ she was wuite a strong person but did not dominate’

Conclusions

P29 the tutor was actively engaged in a social process of actively designing his involvement through interaction, with a view to actively enhancing the learning of the group members. He was concerned ……. To maintain balance and integration within the group’

‘the complexity of practice in networked collaborative learning envionments is such that models of social-constructivism, situated learning and socio-cultural theory are not, separately capable of providing an account of the role of learning in meaning making, the function of context or the power of the interaction between tutoring and learning processes’

‘results show …. How professionals collaborate successfully to develop their own practice