Measuring the Importance of Collaborative Learning for the Effectiveness of ALN: A Multi-Measure, Multi-Method Approach

Starr Roxanne Hiltz1, Nancy Coppola, Naomi Rotter, Murray Turoff
New Jersey Institute of Technology
Raquel Benbunan-Fich
Seton Hall University

1 The first author is responsible for the content of this paper and for any errors it contains. The collaborators listed have contributed to the three NJIT evaluation studies described. More complete descriptions of the individual studies and results can be found in the following sources: Hiltz and Benbunan-Fich, 1999; Benbunan-Fich and Hiltz, 1999; Benbunan-Fich, 1997; Coppola, Hiltz & Rotter, 1998; Hiltz & Wellman, 1997.

Abstract
Are there any differences in outcomes between traditional classroom-based university courses and courses delivered via ALN, which feature extensive on-line interaction among students?  Under what conditions are ALN courses most effective?  What can be done to improve the publishability of ALN evaluations, and counter the attacks of critics?

After providing background on the New Jersey Institute of Technology (NJIT) Virtual ClassroomÒ (VC) projects, this paper describes three studies that address the issue of the importance of collaborative learning strategies to the success of ALN for students. A three-year longitudinal field study of 26 courses that are part of an undergraduate degree in Information Systems compared the process and outcomes of learning using an on-line anytime/anywhere environment to those for comparison sections taught in the traditional classroom. An embedded field experiment looked at the separate and joint effects of working on-line versus in the classroom and of working individually versus in groups. Semi-structured interviews with experienced ALN faculty probed their pedagogy and their perceptions of whether or not students learned, on the average, more, less, or about the same as in their traditional sections. The results support the premise  that when students are actively involved in collaborative (group) learning on-line, the outcomes can be as good as or better than those for traditional classes, but when individuals are simply receiving posted material and sending back individual work, the results are poorer than in traditional classrooms.


I. INTRODUCTION

On-line courses and distance learning in general have come under attack in the press recently. These attacks are often based on misconceptions that they necessarily include little instructor-student and student-student interaction. For instance, in an article titled "Wiring the Ivory Tower: But Will On-line Courses Lower Standards?" (Business Week, August 9, 1999), virtual universities are described as having no dorms, no sports fields, and NO COSTLY PROFESSORS (emphasis added.)  The article goes on to describe Unext as the probable model of the future. This for-profit virtual university plans to spend $1 million per course for video-streamed lectures by "stars," then use (low-paid) part-time instructors to answer E-mail and grade assignments. This is not what those involved in on-line learning networks over the past decade have had in mind. Unless we do better at proving that there are vast differences in the quality of on-line courses related not to how much was paid for a star in a video but rather to the skill and effort of the professor, it could very well become a dominant model. It is certainly attractive to the venture capitalists who see higher education as one of the last great untapped markets.

Feeling threatened by the deconstruction of the university as we know it, the American Federation of Teachers and the National Education Association commissioned a recent well-publicized report called "What's the Difference?" on research regarding the effectiveness of distance learning in higher education as it compares to traditional classroom learning [31]. Of course, they asked the wrong question, or at best, only part of the right question. The important questions include

  • How do the outcomes for ALN courses compare to those for other distance-learning modes and for traditional classroom-based courses?  Overall, on the average and for what kinds of courses and students and why?
  • How can ALN and other technologies best be used to improve the effectiveness of educational delivery?  This includes access, efficiency (cost in time and money), as well as learning outcomes.

The Phipps and Merisotis report [31] actually has little to do with ALN or with research on the effectiveness of ALN, except as it forms public discourse. ALNs have been defined as "groups of people who use computer-mediated communication networks to learn together, at the times, places, and pace that best suit them…" [16, p. 4]. A more current operational definition might be, courses that use the World Wide Web as a means of accessing learning resources and which use Computer-Mediated Communication (CMC) to support teacher-student and student-student communication. There are two types of CMC related to two different kinds of pedagogy. E-mail can be used for one-to-one communication between teacher and student, for handing in assignments, and for asking individual questions. Computer conferencing (also known as discussion forums, bulletin boards, and several other names) creates an ordered, stored transcript of group discussion, and is best suited to collaborative learning pedagogical models. This paper will focus on the premise that a very important determinant of desirable outcomes for ALN is the use of collaborative learning strategies.

Though the Phipps and Merisotis [31] report is critical of the quality of research that has been performed for distance learning, it commits many methodological errors and itself requires a very critical, if not skeptical, reading [6]. Among the most fundamental of its inadequacies is a failure to actually list the 40 studies, out of the thousands of studies that exist, that were the basis of the review, or to describe and defend the criteria for choosing these 40, practically none of which appeared in refereed journals.

Assuming that most of the studies used are listed in the "references" in the back, however, one soon realizes that few of them are about ALNs as they are commonly defined. Most of the studies listed are from the late 1980s or early 1990s and, judging from the titles, are of older, non-network-based technologies, such as video. According to the graph of technologies, 26% were "computer-mediated learning;" one cannot tell from this how many used communication systems rather than CAI tutorials or simulations. Nevertheless, the press reported this report as an attack on on-line courses and virtual universities. Obviously, we still do have to prove conclusively that there is no significant difference between on-line classes and on-campus classrooms in terms of outcomes. More importantly, we need to further develop theories and methods to be used in ALN research that will stand up to critical examination.

After describing the NJIT projects that form the context for the studies presented, this paper briefly reviews the importance of theory in framing research that can inform the practice of teaching on-line, and the strengths and weaknesses of the major research methods that can be used. Then it describes the theoretical framework used and a multi-method approach to answering a key question about the effectiveness of on-line courses: How important is the group discussion and collaboration component to successful outcomes?


II. BACKGROUND: THE VIRTUAL CLASSROOM® AND VIRTUAL UNIVERSITY PROJECTS AT NJIT

The idea of creating software structures to support teaching and learning on-line within an asynchronous computer-mediated communication system first occurred to the senior author in 1977. While portions of courses and not-for-credit courses were delivered on-line during the early 1980s, the Hiltz-Turoff research group's first funded research project on the VC began in 1986 and involved the design and implementation of the initial version of the software, course design, and evaluation design based on comparison of a large number of courses delivered in various modes over a period of two years in many different disciplines. For some courses, there were matched sections offered by the same instructor in a traditional classroom and using the VC (as the sole means of delivery, or in combination with a reduced number of face-to-face meetings). For other courses, there was no match, and the comparison was subjectively made by the students and instructors to previous, traditional courses. The purpose of this first project was to establish the feasibility of this approach both technically and in terms of course outcomes. It is most fully chronicled in the book, The Virtual Classroom [20].

We have continued to develop and use our own (text-based) software in subsequent projects, and are currently using the third, Web-based version of VC facilities on Electronic Information Exchange System (EIES), a computer-conferencing system. A second project, from 1994-96, was designed to develop, offer, and assess the effectiveness of entire undergraduate degree programs in Information Systems and Computer Science delivered via VC plus videotapes of lectures.

The third project, From Virtual Classroom to Virtual University, (1997-1999) has the same objectives as above, but also the objective of spreading the innovation begun in the Computer Science department to disciplines throughout NJIT and to graduate and certificate programs as well as undergraduate programs. Another objective of the third project is to encourage faculty to replace videotaped lectures with other media such as CD-ROM or active Web pages.

Course development was always by an individual faculty member with some assistance available from the project director, student laboratory assistants, and the instructional media department. They have been relatively low-budget courses. Faculty developers were given the equivalent of a month's summer pay (second project) or only the equivalent of teaching a summer course ($2500 stipend, current project), and the budget for videotaping a live class in the "candid classroom" was about $7,500 a course. Projects two and three were partially supported by the Sloan Foundation; it is the completed, second project that is the basis for most of the data that will be reported here.

Over the course of the five and one-half years of these projects, enrollments in ALN courses at NJIT have grown from 50 per semester in two courses, to approximately 500 per semester in 25 or more courses. Though the courses in these projects are available through the Internet to students anywhere, the majority who enrolled have been close-to-campus-New Jersey students who mix on-campus and on-line courses in completing their degrees.

For those within driving distance, ALN students are encouraged to attend an orientation session at the beginning of the semester and are required to take any midterms or finals in a proctored, on-campus setting.


III. THEORETICAL FRAMEWORKS

Building and testing theory should be the purpose of any empirical study. Measurement in the absence of theory is generally worthless. Theory consists of a set of concepts, the relationships among them, and most importantly, the "why" that explains those relationships. A good theory leads to a study that asks new questions, or old questions in a new way. It provides the framework and the story line that holds together the entire study, from design of measures and data collection methods to the presentation of results. Usually, a causal model that shows the predicted relationships among concepts can summarize the theory.

There are three major sources of theory for ALN-- pedagogical theories from educational research, media effect theories from communications research, and group interaction/social influence theories from social psychology and sociology. Each of these can be adapted, applied, and integrated to help to explain what happens and why in on-line classes.

From pedagogical theory, one of the major themes is the difference between objectivist approaches and constructivist approaches [14, 24]. The former holds that there is a body of objective knowledge that can be delivered to students through presentation and explanation (lectures, CAI, etc.). The purpose of teaching is to transfer knowledge from archival sources and the brain of the teacher to the brain of the learner. The constructivist theory holds that knowledge has to be discovered, constructed, practiced, and validated by each learner; learning involves "active struggling by the learner" [13, p. 174]. Pedagogical methods using this approach, including collaborative learning, create learning situations that enable learners to engage in active exploration and/or social collaboration, such as laboratories, field studies, simulations, and case studies with group discussion. This distinction will be further elaborated upon below since it is central to the theme of this paper.

One of the best known of the media characteristics theories is media richness, conceived and popularized by Daft and Colleagues [9]. This holds that characteristics of media vary in terms of their ability to support task uncertainty and equivocality; face-to-face is the richest medium and others fall along a continuum. Furthermore, task performance will be improved when task needs are matched to a medium's ability to convey information. A related set of concepts is social presence theory [33, 34], the ability of a medium to give the impression of the presence of others. Recent scholarship has critiqued this concept stating that all media have an inherent degree of richness; for instance, Dennis and Valacich [10] suggest media synchronicity theory as a more comprehensive replacement. According to media synchronicity theory, there are five important media characteristics (feedback, symbol variety, parallelism, rehearsability, and reprocessability). No medium is richest on all media characteristics, and the relationships between communication processes and media capabilities will vary between established and newly formed groups, and will change over time.

Among the group interaction theories that can be applied to on-line classes is the process gains and process losses approach to analyzing group meetings [35, 28]. According to time, interaction, and performance (TIP) theory [26], groups are a complex, intact social system that engage in multiple, interdependent functions on multiple, concurrent projects while nested within and loosely coupled to surrounding systems. You cannot apply something like an ALN technology to groups and expect them all to react the same way. This is a similar concept to adaptive structuration theory [32], which states that a group may choose to faithfully or unfaithfully appropriate the structures and tools provided by the technology, heuristic, environment, etc.

A. Collaborative Learning Theory
Passive (objectivist) approaches to learning assume that students learn by receiving and assimilating knowledge individually, independent from others [5]. In contrast, active (constructivist) approaches present learning as a social process that takes place through communication with others [27]. The learner actively constructs knowledge by formulating ideas into words, and these ideas are built upon through reactions and responses of others [5, 1]. In other words, learning is not only active but also interactive.

In particular, collaborative or group learning refers to instructional methods that encourage students to work together on academic tasks. Collaborative learning is fundamentally different from the traditional direct-transfer or one-way knowledge transmission model in which the instructor is the only source of knowledge or skills [15]. Some examples of collaborative learning activities are seminar-style presentations and discussions (in which students are the teachers), debates, group projects, simulation and role-playing exercises, and collaborative composition of essays, exam questions, web pages, stories, research plans, or other artifacts that demonstrate the knowledge and skills that are the subject of the course [22]. Collaborative learning pedagogy shifts the focus from the teacher-student interaction to the role of peer relationships in educational success (Johnson, 1981).

There are two major explanations for how participating in a group endeavor helps members learn [37]. Group members learn by virtue of mediating socio-emotional variables (such as motivation, reduced anxiety, or satisfaction) that create an emotional or intellectual climate favorable to learning. When working with peers instead of alone (or with the instructor), anxiety and uncertainty are reduced as learners find their ways through complex or new tasks [15]. These effects tend to increase motivation and satisfaction with the learning process in general.

As reviewed by Dillenbourg and Schneider [12], several collaborative learning mechanisms directly affect cognitive processes, including

  • Conflict or Disagreement - When disagreement occurs between peers, social factors prevent learners from ignoring conflict and force them to seek additional information and find a solution.
  • Internalization - The concepts conveyed by the interactions with more knowledgeable peers are progressively integrated into the learner's knowledge structures. When integrated, they can be used in his or her own reasoning.
  • Self-Explanation - Less knowledgeable members learn from the explanations of more advanced peers. But, surprisingly, the more able peer also benefits because providing an explanation improves the knowledge of the explainer (self-explanation effect). Explaining to others may be more beneficial to the explainer when the material is complex than when the material is simple [37]. In collaborative learning, explanation occurs naturally or spontaneously.


B. Theoretical Model
The theoretical framework adopted is based on Hiltz's [19] systems contingency model. In this model, characteristics of the system, the individual, the group (course or class), and the organizational setting (college or university, and department) are expected to influence the amount and style of use of the system, which in turn will determine outcomes. These variables interact to form a complex system of determinants. Favorable outcomes are contingent upon adequate levels of technological infrastructure, organizational support, student ability, and motivation [25], and upon the pedagogical approach, skill, and level of effort of the teacher [20]. The theoretical model is presented in Figure 1.

 
Figure 1: A Causal Model for the Virtual Classroom® Study

The analyses included in this paper will focus on the middle and bottom portions of Figure 1. The intervening variables include the amount and type of use of the system. For example, students may procrastinate and only sign on just before an assignment is due or exam dates rather than participating regularly. Or they may sign on and passively browse rather than contributing to on-line discussions. They may or may not engage in collaborative assignments with other students, depending upon the way the course is structured by the instructor and their own regularity in interacting with their peers. They may or may not perceive the class interactions as a rich medium of participation that includes social-emotional interaction as well as task-oriented interaction, and conveys a sense of social presence of others. These intervening variables, in turn, are conceptualized as leading to the presence or absence of the various (desired) outcomes, such as better access to the instructor, ability to complete more courses during a calendar year, and subjectively and objectively measured quality of learning.

C. Propositions and Hypotheses
Based on the theoretical model in Figure 1, below are some of the major propositions and hypotheses that were derived. For propositions, the assertions could be tested with questionnaire data only, using self-reports by those who used the VC conferencing system, which was utilized. For hypotheses, data contrasting students in different modes were available for statistical tests or qualitative summaries.

  • P1: ALNs can improve ACCESS to education, as compared to traditional face-to-face classrooms.
  • P2: ALNs can improve the rate of progress towards the degree.
  • P3: ALNs can improve the quality of learning as self-reported by students.
  • H1: ALNs can improve quality of learning as measured by grades or similar assessments of quality of student mastery of course material.

Such improvements will be contingent upon a favorable set of circumstances characterizing the use of the ALN; in particular, they will be more likely if

  • H2: The student actively participates in on-line learning.
  • H3: The instructor utilizes collaborative pedagogical strategies.
  • H4: Participating in a collaborative (group vs. individual) assignment will increase an on-line student's motivation, and thus both the amount of active participation and the quality of learning.


    IV. RESEARCH METHODS AND FINDINGS

    Different research methods have different strengths and weaknesses. Quantitative methods measure variables in a standard manner such as a questionnaire using structured scales or detailed counts of behavior episodes. Qualitative methods probe more deeply into the processes and outcomes in a situation by collecting more naturalistic data, but cannot easily be turned into statistics to measure statistical significance of apparent relationships. The three most commonly used quantitative methods are the controlled experiment, the field study relying upon surveys of participants, and the field or quasi-experiment in which participants to some extent self-select into different conditions, e.g., decide to sign up for a traditional classroom or an ALN section of a course. The controlled experiment, in which all subjects are randomly assigned to conditions and one or more independent variables are deliberately manipulated to produce these conditions while everything else is held constant, has the obvious advantage in terms of clear control over the independent variable(s) such as mode of delivery of a course and of being able to statistically isolate and measure cause-effect relationships. However, it is low in realism since in order to be fully controlled, the experiment must take place in a laboratory where all conditions are under the control of the experimenter and are the same for all subjects (except for the deliberately created differences in treatments). It also suffers from poor generalizability since only certain kinds of subjects will be willing to volunteer to come to a laboratory for a study and the tasks assigned must be relatively simple and short term, since they have to be able to be completed in the laboratory session. The field study employing large samples of subjects in a survey permits large numbers of subjects, which may be the basis for generalizability if the sample is representative of a larger population. The field experiment in which subjects to some extent self-select into conditions and the task is a natural part of the group's activities, has the potential for the greatest realism [11].

    Any one method can be attacked for being weak on control, generalizability, or realism. Thus, studies ideally use "triangulation," combining two or more research designs with different strengths and weaknesses, in order to test key hypotheses. If one obtains similar results from different methods, then there is greater confidence in the conclusions. In the sections that follow, brief descriptions will be given of a set of studies using different methods to test our key hypotheses.

    A. A Field Study of ALN
    From 1993-1997, we undertook the design, delivery and evaluation of  the effectiveness of an undergraduate major in Information Systems delivered in a distance ALN mode via a combination of videotaped lectures plus VC (NJIT's computer conferencing system with special features to support asynchronous learning). Designed to serve both students who normally take their classes on campus and distance students, objectives included

    • Faster progress towards the undergraduate degree by facilitating self-paced learning and solving major educational logistics problems.
    • Improved quality of learning through the increased collaborative learning and faculty-student interaction facilitated by computer conferencing.
    • Increased access to educational opportunities for working adults or those trying to re-enter the work force, particularly women.

    A multi-method approach to evaluation of outcomes for the 26 courses in the project included pre (N= 1048) and post?course questionnaires (N= 855) completed by students, direct observation of on-line activities, automatic counts of amount of on-line activity, comparison of test or course grades or other objective measures of performance, an ongoing computer conference for faculty discussion of problems and solutions, and course reports by faculty using a standard format.

    The summary of results presented here is based primarily on the completed questionnaire and grade distribution data. The questionnaires were generally obtained by mail, though in some sections they were distributed in class or at the final exam. Note that both questionnaire and grade data were collected for sections of courses taught by the same instructor(s), for comparison purposes, in three modes of delivery¾traditional face-to-face, video plus VC, and mixed (face-to-face plus VC).

    1. Improving Access
    One of the primary hypothetical benefits of ALNs is to allow anytime/anyplace access to courses. This should improve the ability of students with work and family responsibilities, in particular, to be able to make progress toward the degree. Besides increasing the convenience of scheduling and thus of access, ALNs should improve access to a student's professors or tutors by making them available every day, rather than just during limited on-campus office hours.

    Post-course questionnaire items relating to improved access are shown in Table 1. These questions asked students to compare their experiences in the VC course they had just taken to experiences with traditional face-to-face courses, using Likert-type scales that ranged from Strongly Agree to Strongly Disagree. Seventy-three percent agree that on-line courses are more convenient than traditional courses; 71% say that they provide better access to the instructors.


    Table 1: Results for Post Course Ratings Related to Access

    2. Facilitating Faster Progress Towards the Degree
    One objective measure related to time to degree is the relative proportions of students who withdraw from or fail a course; this represents a waste of time and money. Withdrawals seem to be higher for the on-line sections than for traditional face-to-face sections, but lower than for video-only distance sections. The difference in the withdrawal rate is significant but not alarming   (24% in VC courses vs. 17% in traditional courses; those who drop out are most likely to name inadequate time due to factors such as work and family responsibilities as the reason). In terms of failure rates, there is no difference (VC courses are actually a percentage point lower overall). It is interesting that the lowest failure rates, overall, are for mixed media courses using VC in combination with face-to-face meetings.

    When students were asked if the availability of the ALN courses sped up their progress towards a degree, 63% said it did:

    To what extent has the availability of this telecourse enabled you to complete more  credits this semester than would have been possible otherwise?

    3. Improving the Quality of Education
    The key subjective student rating relating to this desired outcome is shown below. Fifty-eight percent agree, whereas only 19% disagree, that use of the VC improved the overall quality of their educational experience, as compared to traditional courses:

    Did use of the system increase the quality of your education?


    4. Quality of Learning: Grade Distributions
    An analysis of variance (ANOVA) for differences among the three modes shows that overall, grade-point average accounts for most of the variance in course grades among students, and there are no significant differences among modes of delivery. There are so many differences in grade distributions among courses and instructors, however, that such overall comparisons are not meaningful. Comparisons of grade distributions for the 11 courses for which there were sufficient data to compare modes of delivery resulted in one course for which there was significantly poorer student performance in the distance sections, and one for which there was significantly better grade distributions-in other words, no difference overall. However, one possible explanation for this finding is that instructors may curve grades within sections rather than using the exact same standards across different semesters and media.

    5. Active Participation and Collaborative Learning as Intervening Processes
    All 25 faculty members who taught ALN sections during the field study were urged to be on-line at least once a day and to use collaborative learning strategies. However, the model of on-line professorial behavior advocated in their training sessions was followed more closely by some than others. Using ANOVA tests not reported in detail here, we found that the results for different ALN instructors varied significantly on almost all variables measured. These include measures of overall student satisfaction with the VC, student perceptions of the extent to which the course used collaborative approaches, and perceived course outcomes. In other words, differences in pedagogy are much stronger than the differences among media.

    The theoretical model posits causal relationships between the intervening variables (perceived media richness or social presence, active participation, and collaborative learning) and outcomes such as perceived better learning. In looking for such relationships, our first step is to reduce the number of variables by constructing and testing the internal validity of indexes (several related items added together). The two major indexes measuring quality of outcomes are the course overall and the VC overall scales. (The former set of questions on courses was asked of all students in all modes and thus can be used to compare modes of delivery; the latter was composed of items that pertain only to those with a VC course.)  Both indexes, when refined to drop potential items that did not have a high inter-correlation with other items, reached good levels of internal validity as measured by Chronbach's Alpha. The composition of the indexes is shown in Tables 2 and 3.


    Table 2: The VC Overall Index


    Table 3: The Course Overall Index

    Composite measures were also created for the variables of perceived social presence, active involvement, and collaboration. The Chronbach's Alpha for these indexes was unacceptably low, indicating that better, more internally consistent sets of measures are needed in the future (Table 4). In presenting results, individual items will be displayed, and the names of indexes whose reliability is questionable will be shown in brackets.


    Table 4: Social Presence, Active Involvement and Collaboration Indexes

    Table 5 shows significant bivariate Pearsons' correlations between the intervening and outcome variables. Those who experienced the VC as more convenient than the traditional classroom were most likely to give it high ratings for effectiveness. Those who were more involved, found the comments of others useful (engaged in collaborative learning), communicated more, and developed new friendships (a measure of perceived social presence) were also much more likely to experience positive course outcomes and positive evaluations of the VC experience. There were significant positive relationships between the perceived degree of collaborative learning and both the course outcomes and VC overall indexes. Thus, these correlations support the theoretical model that underlay the project.

    (Pearsons R and N; significant at p= <.01)


    Table 5: Correlations with Outcomes

    Though the correlation between the degrees of perceived collaborative learning in the course correlates significantly with perceived outcomes, correlation is not causation. Being on-line is confounded with collaborative learning; few of the traditional sections used group assignments. All of the on-line courses supposedly used collaborative-learning approaches (though this was implemented better and more consistently in some courses than in others). In addition, course grades and even final exam grades, can be challenged in terms of their validity for measuring the quality of a student's work. A more experimental approach and more valid and specific performance measures than overall course grade, are needed to confirm the finding that collaborative learning is a key mechanism in making ALNs effective.

    B. A Field Experiment on Collaborative Learning
    A field experiment within the Computers and Society course  [2] that was part of the larger field study, compared groups and individuals solving ethical case scenarios with and without computer-mediated communication support. A 2x2 factorial design crossed two modes of communication (manual off-line vs. asynchronous computer conference) and two types of teamwork (individuals working alone vs. individuals collaborating in groups). This design was chosen to assess the separate and joint effects of medium of communication and collaborative vs. individual learning strategies on learning, task performance, and motivation. The task was a case analysis and written report on an ethical scenario.

    1. Hypotheses
    Groups are better at making decisions [17] and more creative at generating options and probing their advantages and disadvantages than are single individuals [36]. In particular, previous research found ethical discussions among group members to be superior to an individual's consideration of a dilemma [30]. Consequently, it was hypothesized that groups would produce higher quality solutions to ethical dilemmas than individuals.

    The use of the ALN was also expected to enhance task performance due to the nature of the asynchronous environment in which participants can reflect in more depth about their contributions and work at whatever time they find most convenient [18, 20]. Some empirical studies, e.g. Ocker et al. [29], have found that computer-supported conditions will tend to produce higher quality solutions than their manual counterparts. Therefore, it was hypothesized that participants working through an ALN would produce longer reports and higher quality solutions to the ethical scenarios than their manual counterparts. Length of reports can be considered to combine aspects of motivation, active participation, and quality of solution (since longer reports are more likely to be thorough). For our hypotheses, we will use it primarily as a measure of the amount of active participation.

    Even when working alone, students who are working in the same room and at the same time as other students are aware of the social presence of others. The use of collaborative group projects can help to overcome the leaner medium of CMC and capitalize on its anytime/anywhere ability to support complex group work. On the other hand, being on-line while working alone on a project can be boring. Therefore, we generally expected interaction effects whereby groups on-line produce disproportionately good results, and/or individuals on-line are disproportionately worse than other conditions. Likewise, it was expected that students working in a group on-line would be more motivated than those working alone.

    2. Procedures
    In all conditions, students received the ethical case scenario comprising the task one week ahead of time, and were permitted to use whatever written or other materials they wished while discussing or working on the case. In the individual off-line condition, students solved the case individually, in an in-class exercise like an open-book quiz, and received individual grades based on their own performance. In the individual on-line condition, students submitted their individual responses by using the question-response activity software on VC; they were neither required nor encouraged to subsequently look at other responses. In the group off-line condition, team members discussed and solved the case by interacting face-to-face and prepared their group report manually. In the group on-line condition, team members interacted asynchronously using the computer conference as the only means of communication and submitted a group report by posting it in the group conference.

    3. Subjects
    The subjects were NJIT undergraduate students in the core course Computers and Society, and the ethics scenario that was the experimental task was one of the assignments in the course. (See Benbunan-Fich, [3] for a more detailed description of the task.) Assignment to experimental conditions was done as close to randomly as possible. Most of the students were in a combination face-to-face plus VC course, but some were in the VC +video condition and could not be assigned to come to campus. Students assigned to a group condition were then randomly assigned to a specific group.

    The sample was composed of 140 students distributed across conditions as follows: 42 in individual/manual, 42 in individual/on-line, 28 in groups/manual and 28 in groups/on-line. Due to scheduling constraints and the loss of groups in both conditions because of no-shows, fewer participants completed the experiment in-group conditions than in individual conditions. Five teams completed the experiment in groups/manual condition and seven teams completed the experiment in groups/on-line. Group size ranged from three to six members. It is worthy of note that the a-priori size of the groups was five to six members, but due to no-shows, two groups ended up with only three participating members. It would have been desirable to have more subjects and more groups to increase statistical power, but this was the total number of students available to participate in the five sections of the course conducted by the experimenters during the three semesters of the study.

    4. Measures of Variables
    Perceived learning was measured immediately after the experiment in a post-test questionnaire, using an eight-item scale adapted from Hiltz [20]; Chronbach's Alpha = .92). All reports were transferred or transcribed into Word files; length of the reports was measured by the number of words in each report as computed using the Word Count function of Microsoft Word for WindowsÒ (V. 6.0). This word count was used to compare the length of the solutions submitted by groups and individual participants. The quality of the analysis produced was rated by three expert judges (blind to condition) on a number of dimensions including the extent to which the correct legal and ethical principles were identified and applied to the scenario. Judges' scores were analyzed to assess the level of agreement (inter-rater reliability = .85) and then the scores were averaged to produce a measure of quality.

    Because this was a field experiment with a limited number of possible subjects, we chose the .10 level of significance as the minimum for assessing results as worthy of note. A minimum of .05 is required to refer to the results as statistically significant.

    5. Results
    Working in groups and through an ALN system significantly increases learning perception, length of reports, and solution quality. In terms of self-reported learning (Table 6), there is, as hypothesized, an interaction between medium of communication and group vs. individual learning. According to the results, conditions with (or without) both factors, i.e. individuals/manual and groups/on-line, perceived higher learning than conditions in which only one of the factors was present.


    Table 6: Self-Reported Learning Results


    For length of report (the group product and the artifact which measures learning of the material), group reports were significantly longer than individual reports (p < .001). At the same time, on-line conditions submitted significantly longer reports than their manual counterparts (p < .001). The average length of reports produced by computer-supported groups was 756.02 words, almost twice the length of individual manual reports whose average number of words was about 381 words. There is also a significant (p <.01) interaction effect between teamwork and technology, as predicted by our hypotheses (Table 7).


    Table 7: Length of Report Results

    Regarding solution quality (Table 8), the scores submitted by the judges show that participants working through the system (individually or in groups) submitted better reports than their manual counterparts.


    Table 8: Solution Quality Results

    The final results of the experiment that will be included here relate to levels of motivation (Table 9). Though only marginally significant (at .08), it is worthy of note that those in the individual on-line condition reported lower levels of motivation than either students working together in a classroom or working in groups on-line.


    Table 9: Motivation Results

    In sum, one of the implications of this experiment for ALN is that putting individuals on-line to interact with course materials is not as effective as the traditional classroom, but that using collaborative learning approaches can make on-line learning at least as effective as the traditional classroom.

    C. Study 3: Semi-Structured Interviews with Faculty
    As part of the 1997-99 project called "From Virtual Classroom to Virtual University," Coppola, Hiltz and Rotter [7, 8] designed, conducted, transcribed, and coded 20 semi-structured interviews with faculty who have prepared and delivered at least one on-line course. They cover aspects of the amount of work involved in preparing an ALN course, pedagogy, faculty attitudes toward policy issues, and perceived outcomes for both students and faculty. Figure 2 shows some questions from the interview guide, which probe aspects of how on-line group activities were or were not used, and perceived learning outcomes for students. What we notice in reading through the transcripts is that most faculty who successfully used the group discussion and collaborative work aspects of ALN feel that students learned as much or more as in traditional classrooms. By contrast, if faculty members failed to structure activities, incentives, and encouragement so as to elicit on-line group discussion and work, they tend to feel that the experience was not as good, for either students or faculty, as in a traditional classroom.

    Some Questions from the Semi-Structured Faculty Interview

    Start-Up Logistics

    Now let us move on to the first time you actually tried to DELIVER your distance course [repeat course name] using ALN. The first semester you taught this course on-line what kinds of logistical problems, if any, did you encounter? For instance¾getting students enrolled, on-line, having them obtain their books and/or tapes, getting exams proctored, etc.?

    Pedagogy Innovation

    Many faculty have found that to be most effective instructing on-line, they need to devise new kinds of assignments or activities. Are there any kinds of innovative assignments or class activities that you have devised that worked particularly well? Probe responses.

    Pedagogy Individual or Group?

    Consider the various assignments or weekly activities that you include in your course. Do these activities result primarily in individual student work, small group work, or work in which the whole class was involved?

    Pedagogy On-line Discussions

    What techniques have you used for encouraging discussion?  (Probe¾How well did these techniques work?)

    Outcomes

    Do you think that students in your distance ALN sections learn about the same as those in traditional sections, more, or less?  (Probe--Why?)

    Figure 2

    Figure 3 shows some excerpts from two faculty members that illustrate this strong relationship between faculty reports of the extent of collaborative pedagogy, and their perceptions of relative outcomes.

    Instructor A -- “They did not get quite as much out of the class…”                     

    Pedagogy/Logistics

    For this class, what I hoped to do was, using the VC, it was stated as a requirement for the class that students would post their questions, which I would respond to and which other people in the class would respond to. No matter what I did, I could not get the students to use the VC. I sent them reminders that it was required. They would say, “I don’t know how.”  I would send them a message on how to do it, and they would say, “I don’t have an account…”. As a matter of fact, I think I had only two students who posted. That was the only way in which that class did not work. They sent their bi-weekly assignments, they did good projects, but the discussion— [nothing]… It was 4-6 weeks into the class when I realized, “this is not working.”  Then I did not know what to do. I could fail everyone because nobody . . . basically . . . was doing it. I could tell as a teacher that they were doing the reading and were learning because of their 500-word responses that were due. So on one level I didn’t feel negligent because they were learning. That sort of vibrant student-teacher communication that I expected clearly did not happen.

     

    Pedagogy/Individual or Group

    Individual, all assignments. Students in the real classroom worked together.

    Outcome

    My sense of it honestly, is that they did not get quite as much out of the class. How would I prove that?  I don’t know. I had some sense that the students in the classroom were changed, that they had new ideas. I know that the students on-line read and learned a lot of stuff, but I didn’t really think that they got as much. All those dialogues that transpired in the classroom were missing. For me, so much of the knowledge building happens in that live interface. There might be some way that you can translate that on-line.

    Instructor B -- “They learned a lot more…”

    Pedagogy/Logistics

    On every screen in the CD ROM there’s a link to EIES. So at any point a student can write in or query back and forth. A lot of people are on-line quite often and a lot of the time…

    Pedagogy/Individual or Group

     

    One of the great things about using a system like EIES is that it puts enormous peer pressure on these students. There’s no place to hide. To be in the classroom you have to write. Since this is a writing program, it’s terrific, to have everybody writing everything out all the time, in a sense publishing for peer review. I think that works a lot better than face-to-face because they are working on their writing every time they are in the classroom and everybody knows what everybody else is doing.

    Outcome

    I think they learned a lot more than the previous face-to-face course . . .

    Figure 3: Contrasting Reports by Faculty


    V. SUMMARY AND CONCLUSIONS

    Summary of Results
    In this section, we will review the propositions and hypotheses tested, and source and nature of relevant evidence that has been presented.

    • P1: ALNs can improve ACCESS to education, as compared to traditional face-to-face classrooms. This was supported by student self-reports in the field study of 26 courses. Students reported that ALN was more convenient than traditional courses and gave them better access to their professors.

    • P2: ALNs can improve the rate of progress towards the degree. Supported by student self-reports in the field study.

    • P3: ALNs can improve the quality of learning as self-reported by students. Supported by questionnaire results from the field study of 26 courses and from the quasi-experimental study of one course.

    • H1: ALNs can improve quality of learning as measured by grades or similar assessments of quality of student mastery of course material. In the field study, there were no significant differences between modes of delivery for overall course grades, once student grade point average was used as a co-variate. In the quasi-experimental study, on-line students produced significantly better reports (the measure of learning used) than students working in the traditional classroom.

    It was hypothesized that improvements in the quality of learning will be contingent upon a favorable set of circumstances characterizing the use of the ALN; in particular, they will be more likely if

    • H2: The student actively participates in on-line learning. Supported by correlation in the field study.

    • H3: The instructor utilizes collaborative pedagogical strategies. Supported by correlation between perceived extent of collaborative learning and course outcomes in the field study and by a significant relationship between group work on-line and the quality of the report, in the quasi-experimental study of the computers and society course.

    • H4: Participating in a collaborative (group vs. individual) assignment will increase an on-line student's motivation, and thus both the amount of active participation and the quality of learning.


    The longitudinal field study does not allow us to conclude whether better educational outcomes in ALN-supported courses are the result of collaborative learning techniques, ALN use, or both. Additional insight was sought through the 2x2 field experiment, designed to separate the effects of working in a collaborative environment from the effects of using an ALN. Findings of this study indicate that the combination of teamwork and ALN use enhance student perceptions of learning, whereas students working alone and on-line tended to be less motivated and wrote shorter reports than those working in groups. They reported the lowest perception of learning.

    In addition, semi-structured interviews with experienced ALN faculty indicate a strong association between extensive uses of on-line class discussion and reported learning outcomes for students as good or better than those for the traditional classroom.

    Though any one measure or method might be legitimately questioned in terms of its validity, reliability, or generalizability, the weight of several different kinds of studies over a period of five years, is convincing. In summation, the empirical evidence presented in this paper suggests that when students are actively involved in collaborative (group) learning on-line, the outcomes can be as good as or better than those for traditional classes. When individuals are simply receiving posted material and sending back individual work, the results are poorer than in traditional classrooms.


    VI. CONCLUSION

    The presidential election of 1992, when the incumbent President George Bush was defeated, was summarized with an explanation of why he lost, "It's the economy, stupid!"  The shortest summary of our findings about what makes for quality on-line courses is "It's the pedagogy, stupid!"  Far from there being "no more costly professors," on-line courses represent an arena of struggle between those who see them as a way of maximizing profit versus those who see them as a way of improving quality as well as access to education. They also represent a new and generally satisfying challenge to faculty members, to change their pedagogy to best take advantage of the fast-changing technology of the Internet, the World Wide Web, and their successors.


    ACKNOWLEDGMENTS

    The initial development of the Virtual Classroom* was supported by the Annenberg/CPB project of the Corporation for Public Broadcasting. Development and research on the B.A.I.S. degree via a combination of video and VC was supported by the Alfred P. Sloan Foundation, as is the current project, "From Virtual Classroom to Virtual University."  The field experiment reported here, as well as continuing research on appropriate software structures for collaborative work via asynchronous computer-mediated communication, was partially supported by a grant from the National Science Foundation (NSF-IRI-9015236). Support for these efforts has also been provided by the Center for Multi-Media Research at NJIT, the state of New Jersey, and by industrial partners including IBM and Apple Computer. We are also grateful to the many colleagues and student research assistants who made this research possible.


    REFERENCES

    1. Alavi, M. Computer-mediated collaborative learning: An empirical evaluation. MIS Quarterly, Vol. 18, No. 2, pp. 150-174, June, 1994.

    2. Benbunan-Fich, R. Effects of computer-mediated communication systems on learning. Performance and Satisfaction: A Comparison of Groups and Individuals Solving Ethical Case Scenarios. Ph.D. diss., Rutgers University/NJIT Joint Program in the Management of Computer Systems, Newark NJ, 1997.

    3. Benbunan-Fich, R. Guidelines for using case scenarios to teach computer ethics. ACM SIGCAS Bulletin, pp. 20-24, September, 1998.

    4. Benbunan-Fich, R., and Hiltz, S. R. Impacts of asynchronous learning networks on individual and group problem solving: A field experiment. J. Group Decision and Negotiations, 1999.

    5. Bouton, C., and Garth, R. Y. Learning in Groups, San Francisco: Jossey-Bass, Inc., 1983.

    6. Brown, G., and Wack, M. The difference frenzy and matching buckshot with buckshot. Technology Source, Horizon, 1999. http://horizon.unc.edu/ts/reading/1999-05.asp

    7. Coppola, N., Hiltz, S. R., and Rotter, N. Becoming a virtual professor: Preliminary results of semi-structured interviews. Presentation at the Fourth International Conference on Asynchronous Learning Networks, New York, November, 1998.

    8. Coppola, N., Hiltz, S. R., and Rotter, N. Becoming a virtual professor: Pedagogical changes and ALN. Presentation scheduled for the Fifth International Conference on Asynchronous Learning Networks, University of Maryland, October, 1999.

    9. Daft, R. L., and Lengel, R. H. Organizational information requirements, media richness, and structural design. Management Science, Vol. 32, No. 5, pp. 554-571, 1986.

    10. Dennis, A. R., and Valacich, J. S. Rethinking media richness: Towards a theory of media synchronicity. Proceedings, 32nd Hawaii International Conference on System Sciences, 1999.

    11. Dennis, A., and Valacich, J. Doing experimental research on collaboration technology. Tutorial presented at the Hawaii International Conference on System Sciences, Maui, HI, January, 1999.

    12. Dillenbourg, P., and Schneider, D. Collaborative learning in the Internet. Proceedings, Fourth Int. Conference on Computer Assisted Instruction, Taiwan, S10-6 to S10-13, 1994.

    13. Duffy, T. M., and Cunningham, D. J. 1996. Constructivism: Implications for the design and delivery of instruction. Handbook of Research for Educational Communications and Technology, New York: Macmillan.

    14. Glasser, R., and Bassok, M. Learning theory and the study of instruction. Annual Review of Psychology, No. 40, pp. 631-666, 1989.

    15. Harasim, L. Ed. On-Line Education: Perspectives on a New Medium, New York: Praeger/Greenwood, 1990.

    16. Harasim, L., Hiltz, S. R., Teles, L., and Turoff, M. Learning Networks: A Field Guide to Teaching and Learning On-line, Cambridge MA: MIT Press, 1995.

    17. Hill, G. W. Group versus individual performance: Are N+1 heads better than one? Psychological Bulletin, Vol. 91, No. 3, pp. 517-539, 1982.

    18. Hiltz, S. R. The virtual classroom: Using computer?mediated communication for university teaching. Journal of Communication, Vol. 36, No. 2, pp. 95-104, 1986.

    19. Hiltz, S.R. Productivity enhancement from computer-mediated communication: A systems contingency approach. Communications of the ACM, Vol. 31, No. 12, pp. 1438-1454, December, 1988.

    20. Hiltz, S. R. The Virtual Classroom: Learning Without Limits via Computer Networks, Norwood NJ: Ablex Publishing Corp., Human?Computer Interaction Series, 1994.

    21. Hiltz, S. R., and Benbunan-Fich, R. The Importance of Collaborative Learning in Asynchronous Learning Networks. Manuscript submitted for publication, 1999.

    22. Hiltz, S. R., and Turoff, M. The Network Nation: Human Communication via Computer, Revised edition, Cambridge MA: MIT Press, 1978/1993.

    23. Johnson, D. W. Student-student interaction: The neglected variable in education. Educational Research, Vol. 10, No. 1, pp. 5-10, 1981.

    24. Leidner, D., and Jarvenpaa, S. The use of information technology to enhance management school education: A theoretical view. MIS Quarterly, pp. 265-291, September, 1995.

    25. Leuthold, J. Is computer-based learning right for everyone? IEEE Computer Society Press, Proc. 32nd Hawaii Int. Conf. On System Sciences, 1999, CD ROM.

    26. McGrath, J. E., and Hollingshead, A. B. Groups Interacting with Technology, Thousand Oaks, CA: Sage Publications, 1994.

    27. Mead, G. H. Mind, Self and Society, Chicago, University of Chicago Press, 1934.

    28. Nunamaker, J., Dennis, A., Valacich, J., Vogel, D., and George, J. Electronic meeting systems to support group work. Communications of the ACM, Vol. 34, No. 7, pp. 41-61, 1991.

    29. Ocker, R., Hiltz, S. R., Turoff, M., and Fjermestad, J. The effects of distributed group support and process structuring on software requirements development teams: results on creativity and quality. Journal of Management Information Systems, Vol. 12, No. 3, pp. 127-153, Winter, 1995.

    30. Peek, L. E., Peek, G. S., and Horras, M. Enhancing Arthur Andersen business ethics vignettes: Group discussions using cooperative/collaborative learning techniques. Journal of Business Ethics, Vol. 13, pp. 189-196, 1994.

    31. Phipps, R., and Merisotis, J. 1999. What's the difference?  A review of contemporary research on the effectiveness of distance learning in higher education. A report from the Institute for Higher Education Policy. http://www.ihep.com/PUB.htm 

    32. Poole, M. S., and DeSanctis, G. Understanding the use of group decision support systems: The theory of adaptive structuration, in Fulk, J. and Steinfield, C. (Eds.), Organizations and Communication Technology, Sage, Newbury Park, CA, 1990.

    33. Rice, R. E. The New Media, Beverly Hills, Sage, 1984.

    34. Rice, R. E. Task analyzability, use of new media, and effectiveness; A multi-site exploration of media richness. Organization Science, Vol. 3, No. 4, pp. 475-500, November, 1992.

    35. Steiner, I. Group Process and Productivity, New York, Academic Press, 1972.

    36. Turoff, M., and Hiltz, S. R. Computer support for group versus individual decisions. IEEE Transactions on Communications, Vol. 30, No. 1, pp. 82-91, January, 1982.

    37. Webb, N. M. Student interaction and learning in small groups. Review of Educational Research, Vol. 52, No. 3, pp. 421-445, 1982.

    38. Webster, J., and Hackley, P. Teaching effectiveness in technology-mediated distance learning. Academy of Management Journal, Vol. 40, No. 6, pp. 1282-1309, 1997.


    ABOUT THE AUTHORS

    Starr Roxanne Hiltz is Distinguished Professor of Computer and Information Science, New Jersey Institute of Technology, where she also co-directs the Collaborative Hypermedia Systems Laboratory. She received her A.B. from Vassar and her M.A. and Ph.D. from Columbia. She has spent most of the last twenty years engaged in research on applications and social impacts of computer technology. Her research interests include educational applications of computer-mediated communications, human-computer interaction, and computer support for group decision-making. In particular, with major funding from the Corporation for Public Broadcasting and the Alfred P. Sloan Foundation, she has created and experimented with a Virtual Classroom [TM] for delivery of courses. This is a teaching and learning environment that is constructed, not of bricks and boards, but of software structures within a computer-mediated communication system. A prolific writer, her publications include six books, including The Virtual Classroom: Leaning Without Limits Via Computer Networks (Ablex, Human-Computer Interaction Series, 1994; now available through Intellect-net.com); Learning Networks: A Field Guide to Teaching and Learning Online (with Linda Harasim, Lucio Teles and Murray Turoff, MIT Press, 1995); The Network Nation (with Murray Turoff, 1978/1994, MIT Press); and over 200 articles and professional papers.

    Contact: Computer and Information Science, New Jersey Institute of Technology, 19 Meadowbrook, Randolph, New Jersey 07869, Telephone: 973-361-6680, E-mail: roxanne@vc.njit.edu.

    Nancy Coppola is Associate Professor in the Department of Humanities and Social Sciences at NJIT. 

    Contact: Humanities and Social Sciences, New Jersey Institute of Technology, University Heights, Newark, New Jersey 07102; Telephone: 973-334-0075; E-mail: coppola@adm.njit.edu.

    Naomi Rotter is Professor of Management, NJIT. 

    Contact: School of Management, New Jersey Institute of Technology, University Heights, Newark, New Jersey 07102; Telephone: 973-299-6277; E-mail: rotter@adm.njit.edu.

    Murray Turoff is Distinguished Professor of Computer and Information Science at NJIT. 

    Contact: Computer and Information Science, New Jersey Institute of Technology, University Heights, Newark, New Jersey 07102; Telephone: 973-361-6680; E-mail: turoff@vc.njit.edu.

    Raquel Benbunan-Fich, who was Dr. Hiltz's Ph.D. advisee, is Assistant Professor in the Computing and Decision Sciences department at Seton Hall University. 

    Contact: Computing and Decision Sciences Department, Seton Hall University, South Orange, New Jersey 07079; Telephone: 973-275-2958; E-mail: benbunra@shu.edu.