Studies of Effectiveness of Learning Networks
Copyright, 2001
Starr
Roxanne Hiltz, (Hiltz@adm.njit.edu).
Yi
Zhang (yxz1847@njit.edu), and
Murray
Turoff (Turoff@njit.edu)
Department
of Information Systems,
College
of Computing Sciences,
New
Jersey Institute of Technology,
Newark
NJ 07102
Abstract
The WebCenter for
Learning Networks Effectiveness Research includes entries for empirical studies
of Learning Networks published in refereed journals and conference proceedings.
Nineteen of these studies were identified which both measure learning
effectiveness for students, and compare ALN to traditional face-to-face courses
on the same campus. These studies
employ objective measures of student learning (e.g., grades) about as
frequently as subjective measures (survey responses by students). The evidence
is overwhelming that ALN tends to be as effective or more effective than
traditional modes of course delivery, at the university level.
Key Words: ALN, Learning
Effectiveness Measures, E-Learning, CSCL
Learning networks are defined as groups of people who use computer networks (the Internet and World Wide Web) to communicate and collaborate in order to build and share knowledge. The emphasis for studies in the database of empirical research is on asynchronous (anytime, anyplace) use of networks, but the project includes studies of courses that emphasize use of synchronous (same time) technology or which compare face-to-face, synchronous and asynchronous learning processes. Secondly, the emphasis will be on post-secondary, for-credit courses, but information will also be collected about studies of the use of ALN in pre-college courses and in continuing professional education (not for academic credit) courses or learning communities. Effectiveness is defined in this project to focus on both learning outcomes for students, and positive or negative impacts on faculty. To the extent that other measures of effectiveness are reported in empirical studies (e.g., fiscal impacts on educational institutions, cost-benefit analysis, or societal level impacts in terms of educational access and equity), they will be included in a separate database of "other papers" to be created in the future.
1. Papers included in the ALNResearch database
must be empirical studies of the effectiveness of learning networks and have
been published in a refereed journal or conference proceedings, in the English
language. They must be full papers, not
just extended abstracts. This is operationally defined as at least five pages
in length.
2. "Learning
networks" technology and pedagogy refers to the use of computer-mediated
communication among students as well as between instructor and students, for a
substantial part of the course work.
They may be used asynchronously and/or synchronously, though we are
mainly concerned with courses that include substantial use of asynchronous
(anytime) media. They may be used alone
or in combination with other media, such as face-to-face lectures, videotapes,
Web postings of lecture or reading or tutorial material, etc. Not all web based courses use learning
networks; some just post lecture type materials or exams for downloading, do not
involve extensive interaction among students in a class, and therefore do not
qualify as "learning networks" courses. One synonym for learning
networks is "computer-supported cooperative learning" (CSCL).
3. "Effectiveness"
is defined as primarily concerned with learning outcomes for students, but also
includes effectiveness from the instructor's point of view. It thus includes studies that look at
student perceptions, student performance, or faculty perceptions, satisfaction,
or performance in this mode of course delivery.
4. To
be considered as an empirical research study, the paper must include research
questions or hypotheses (at least implicitly), describe some data collection
methods, and report some empirical results.
In order to be considered an adequate empirical study, it must have a reasonable
number of subjects on which conclusions are based. We have operationally
defined this for the time being as a minimum of at least 20 subjects included
in the study.
Most of the coding of
studies is done by Ph.D. students working under the direction of the project
director, who checks them over and ascertains that the study
"qualifies" according to our criteria. One objective is to make this
WebCenter the "first stop" for the literature review of ALN researchers
who are planning a new study or article.
This should save researchers time and assure them a more complete
overview of related prior research than they are liable to obtain on their own.
For this analysis, we
decided to focus on analysis of a key subset of the papers: those that compare
the effectiveness of ALN courses in terms of student outcomes, to that of
traditional "face to face" courses (Appendix 3). We identified 19 of the studies that clearly
meet this criterion. We have included
only the most important study characteristics in the charts and in the analysis
presented here: the research methods, the way effectiveness was measured, and
the results. As for the remaining studies in the database, they tend to be case
studies of ALN rather than comparative studies measuring comparative
effectiveness, or to be concerned primarily with variables that are correlated
with good outcomes in ALN, or to be focussed on different outcomes, such as
faculty satisfaction. We will probably
be adding some more studies to the 19 included here, in the future.
Two of the authors categorized the types of individual measures used, and whether each individual finding reported showed ALN to be better than traditional courses, no different, or worse. The second step was that all of the results for each study were categorized in terms of whether they, in total, showed ALN to better, worse, or no different on the whole.
2. Research
Models, Methods and Measures for Assessing the Effectiveness of ALNs
Asynchronous Learning Networks may be considered to be one type of information system: a computer-based system designed to support the work of teachers and learners. There are two dominant research models in Information Systems and in other fields using social science methodologies to study human subjects: the "positivists" and the "interpretivists." The positivists strive to follow the model of scientific inquiry developed in the natural sciences, with the objective being to specify quantitative measures of all variables, state hypotheses, collect data using random sampling and other procedures that will enable the testing of hypotheses using inferential as well as descriptive statistics, and then analyze the data, report the results and their limitations. The "interpretivists" strive for an "in depth" understanding of the processes that are occurring in human social systems; they generally start with research questions, use qualitative methods such as participant observation, unstructured or semi-structured interviews, and content analysis to obtain a rich description of the phenomenon and arrive at an interpretation of why and how things work-- or do not work (Ngwenyama and Lee, 1997). The most sophisticated research projects combine both quantitative measures (describing quantitatively "what" is happening") and qualitative measures (describing "why" the results are occurring, in terms of the details of behavior and interaction that transfer an "input" of course materials provided in various modes to an "output" of what the student does or does not learn.)
To qualify as an empirical study using generally accepted methods, a research project should start with specific hypotheses and/or research questions, which guide the selection of methods and measures. Unfortunately, the majority of publications related to ALN's do not have any explicit research questions, let alone specific hypotheses. They tend to be accounts by instructors of courses they designed and of their experiences and impressions, whose value in building a scientific body of knowledge is questionable.
The paradigm of positivist research has enshrined experimental design as the most valid method for determining "cause and effect," specifically, the "pre-test, post-test, control group design" using random assignment of individual subjects to conditions. It is basically impossible to randomly assign students to take a traditional section or an ALN section of a course; they may be unable to travel to campus if they live 2000 miles away, or unable to take an ALN section if they have no PC and Internet provider. One is therefore left with "quasi-experimental" designs at best, in most cases, in which student’s self-select mode of course delivery, but the study designer and instructors try to "hold constant" everything else, such as the syllabus, assignments and exams. The "pre-test, post-test" design means that ideally one measures the dependent variable ( such as knowledge about accounting or data bases or English Literature) before the course, then measures again after the course, to determine "amount learned" as the difference between scores. Most ALN studies do not do this either, since it is not usual to give students the equivalent of a final exam on the first day of the course, and in many project-based courses, it is not appropriate, since there will be no "final exam" either. (One exception is the Worrell et. al study of a graduate accounting course [#], for which standardized professional exams are readily available to test "knowledge" of accounting). Thus, we often do not know whether differences in grades at the end of the course are caused by differences in mode of course delivery, or differences among the students who self-selected the different modes. Positivists would thus tend to say that most ALN research to date is not very rigorous.
There are several
different types of measures of effectiveness of ALN's for students that are
commonly used. Objective measures of
performance and subjective assessments by students have been used about equally
(See Table 1). Objective measures
include the following; the number of studies shown in Appendix 3 using each of
these measures is noted in parentheses:
Grades, for specific projects or exams or for the entire course, compared to sections or students using other delivery modes (16).
Measures of the
quality of work (e.g., group projects may be judged on creativity,
completeness, length, etc.) (9). Such
judgements may be biased if they are made by the instructor who designed the
online course, and who knows who did a particular piece of work and the mode
that student was in. Thus, procedures
need to be designed to make the quality of work measure as reliable as possible
by using multiple judges, who are "blind" to the identity and course
delivery condition of the student (e.g., see Fall, 2000).
· Course completion rates (3 studies)
· Counts or measures of activity levels or patterns (5)
"Subjective" measures are frequently used in the current body of ALN research on effectiveness, though they are not usually considered as valid as objective measures. These include student self-assessments (through questionnaires or interviews) of course learning outcomes (absolute or compared to traditional courses; 8 of the studies included in this paper use such a measure); the effectiveness of the mode or system used for delivery (including convenience, motivation, usability, time required, access to professor; 18 studies); or of the quality of the instruction or materials (3 studies). Note that the total number of studies using each type of measure adds up to more than 100% of the studies, since many used more than one measure of effectiveness. Generally, multiple measures enhance the reliability of the conclusions about effectiveness.
Table 1: Summary of Measures and Results for All Nineteen
Studies
|
Measures |
Positive for ALN |
No Difference |
Negative for ALN |
|
Objective Measures Course grade |
2 |
6 |
|
|
Final exam grade |
2 |
1 |
|
|
Midterm/quiz grades |
2 |
3 |
|
|
Quality of work rated by instructor |
3 |
2 |
|
|
Assignment Measures Length of Report |
1 |
|
|
|
Rated by Judges Forgetting |
|
1 |
|
|
Content Quality |
1 |
|
|
|
Completeness |
1 |
|
|
|
Amount of Collaboration |
|
1 |
1 |
|
Amount of Activity / Participation |
2 |
|
|
|
Female participation |
1 |
|
|
|
Course completion |
1 |
|
2 |
|
Use of instructor who does not prepare materials |
1 |
|
1 |
|
Subjective measures via students Learning more |
|
5 |
|
|
Skill development |
|
|
|
|
Quality of work |
|
2 |
1 |
|
Quality of course materials |
|
1 |
|
|
Quality of discussion |
|
|
2 |
|
Motivation/Interest |
3 |
1 |
1 |
|
Progress to degree |
2 |
|
|
|
Access to degree |
2 |
|
|
|
Access to instructor |
1 |
|
|
|
Access to educational resources |
1 |
|
|
|
Usability of technology |
2 |
1 |
2 |
|
Participation |
|
|
1 |
|
Social Presence |
|
|
1 |
|
Totals |
28 |
24 |
12 |
3. Results: ALN vs. Traditional Face-to-Face Course
Delivery
A. Summary
Tables of Results
Most of the studies either measure effectiveness in more than one way (e.g., grade distributions plus subjective student assessments) and/or study different courses, resulting in many "mixed results." Looking at the results, we have classified them as falling into one of two categories, those generally show ALN to have better outcomes than traditional courses (Table 2), and those that tend to show no difference, overall (Table 3).
:
Table 2: Summary for
eight studies with largely positive ALN findings
|
Measures |
Positive for ALN |
No Difference |
Negative for ALN |
|
Objective Measures Course grade |
2 |
2 |
|
|
Final exam grade |
2 |
|
|
|
Midterm/quiz grades |
2 |
|
|
|
Quality of work rated by instructor |
3 |
|
|
|
Assignment Measures Length of Report |
1 |
|
|
|
Rated by Judges Forgetting |
|
1 |