Argumeinkntative Reasoning Assessments l

Toward assessment of knowledge-building practices in technology-mediated work group interactions
Derry, Gance, Gance, & Schlager (2000)

Case Studies link
Collaborative Practices Evaluation link
Mental Model Analysis link
 


what was studied

In this study, the authors were interested in developing a comprehensive framework for studying social knowledge construction in online communities. Specifically, they were interested in the affordances and constraints of online community environments in fostering high levels of discussion and interaction within professional collaborative teams. Toward this end, the researchers conducted a reflexive case study, examining their own online bi-weekly collaborative sessions (12 members total) held in a virtual online professional development institutute for teachers (the Teacher Professional Development Institute, TAPPED IN) over a one-year period. This case study served as a pilot test of their theoretical framework and generated ideas for how one might go about automating the evaluation of online collaborative work.

how effectiveness was measured
The extent to which the online collaborative work served as a mechanism for social knowledge construction was assessed within an over-arching framework that united four major theoretical perspectives (below). In this case study, the researchers examined the transcripts generated by the online evironment for evidence of social knowledge construction within the group.

The four theoretical perspectives included the following:

  1. Situative-Sociocultural Theory, which conceptualizes community activities as a complex social system rather than as isolated events within individual minds. From this perspective, groups should (a) exhibit a reasonable level of apprenticing episodes, events in which more experienced members of the group spend time mentoring new members. They should also (b) develop boundary constructs –conceptual points of intersection between different group members' thinking that (c) are negotiated and refined over time. In addition, groups should (d) develop normative practices and procedures for using tools and other resources so that a shared set of community practices can evolve over time.

  2. Sociocognitive Theory, which focuses attention on the role of conceptual conflict and its resolution in the development of shared understanding. From this perspective, learning is the result of incorporating new knowledge into one's current understanding (when the information in consistent with what one previously thought) or reconceptualizing one's previous understanding (when the new information is in conflict). Therefore, the resolution of mismatches among group members' understanding is another important characteristic of knowledge constructing social entities; members' task-related mental models should become more and more aligned over the course of their collaboration.

  3. Reasoned Argumentation as a Normative Standard for Interaction, which emphasizes that the deliberative process of work groups should be well-reasoned and based on evidence. Groups' decision-making should be based on a careful weighing of the arguments on all sides, making dialogic argumentation a key feature of group process.

  4. Group Information Processing Theory, which highlights the information-sharing practices that best serve collaborative work. From this perspective, broad sharing of knowledge across social, cultural, and professional boundaries is one key indicator of the extent to which group members are engaging with one another in productive ways. Productive attentional processes (e.g., avoiding distractions and information overload) are another key indicator; groups that exhibit more attention-supporting practices, avoid attention-blocking ones, and/or develop practices to compensate for the sometimes-entangled nature of multi-thread talk help insure that the group as a unit functions in productive ways. In addition, the trajectory of information transformation – from private, unshared knowledge to shared ephemeral ideas to tangible group product – sheds light on how the group builds and maintains shared understanding and artifacts over time.

 
 
  Analytic Framework: Group as Apprenticeship Systems link  
  Analytic Framework: Monitoring Boundary Constructs link  
  Analytic Framework: Discourse as Negotiation link  
  Analytic Framework: The Development of Normative Social Practices link  
  Analytic Framework: Mental Model Alignment link  
  Analytic Framework: Reasoned Argumentation as a Normative Standard for Interaction link  
  Analytic Framework: Group as Information Processor link  
 


what the findings were
The authors' framework for assessing the productivity of online collaborative groups as social knowledge construction entities is indeed feasible for the analysis of interaction protocols, providing a richer picture and more subtle assessment of online group work. However, understanding the alignment of members' task-related mental models (an important indicator according to sociocognitive theory) requires access to individual thinking as well; retrospective online interviews are suggested to supplement the analysis of online transcripts. In addition, teasing out the argumentative structure of group interaction is difficult and may be better supported by the development of specific online tools or commands to structure and support such interaction. Together, the four theoretical perspectives highlight the various features and functions of productive online collaborative work, providing a more comprehensive research and assessment framework that no single theory, in isolation, can provide.

 

Derry, S. J., Gance, S., Gance, L. L., & Schlager, M. (2000). Toward assessment of knowledge-building practices in technology-mediated work group interactions. In S. P. Lajoie, (Ed). Computers as cognitive tools: No more walls, Vol. II. (pp. 29-68). Mahwah, NJ: Erlbaum.