Fostering students’ statistical and scientific thinking: Lessons learned from an innovative college course
Derry, Levin, Osana, Jones, & Peterson (2000)

what was studied
This study investigated the impact of an innovative statistics course on college students' scientific and statistical reasoning skills. The course, designed for undergraduate education majors, focused on (a) students' understanding of key statistical principles and concepts, (b) authentic problem solving, and (c) evidential argumentation. Students participated in collaborative, complex problem-solving anchored in real-world situations and received ongoing assistance and mentorship from faculty members and graduate assistants.

how effectiveness was measured
Evidence of student growth was obtained from pre- and post-course interviews designed to assess students' ability to reason with statistical evidence from everyday sources. Students completed reasoning tasks during the interviews that required the analysis and judgement of arguments adapted from actual news and online media sources.

Two isomorphic versions of the interview protocol were developed and administered in a counterbalanced design both before and after the semester-long course. Interview tasks were designed to assess students' knowledge and performance along the following lines:

1. Recognizing of the limitations of correlational evidence

2. Demonstrating understanding of the components of scientific experimentation

3. Demonstrating understanding of random sampling (and related topics of generalizability and estimation)

4. Demonstrating understanding of estimation based on random sampling from a known population

5. Critical evaluation of advertisements

Quantitative Analysis: An a priori coding scheme, based on a set of norms derived from the cognitive literature on statistical reasoning, was used to evaluate student responses. Interviews were transcribed and coded to identify the positive and negative features of students' explanations in terms of the 5 criteria listed above. Two coders, "blind" to both student identity and pretest/posttest status, coded all the transcripts; discrepancies between coders' judgements, when they arose, were resolved through discussion. Scores were determined such that student explanations exhibiting the greatest number of positive features and the least number of negative features received the highest scores. Students pretest scores were also used as a measure of pre-course course knowledge so that the interaction between initial knowledge and amount of improvement could be assessed.

Qualitative Analysis: A second, qualitative analysis was also conducted in order to capture a richer description of students' reasoning on two selected reasoning tasks: (a) the tasks designed to elicit evidence of students' recognition of the limitations of correlational evidence, and (b) the tasks designed to elicit evidence of students' understanding of the components of scientific experimentation. A coding scheme was developed iteratively from the bottom up (i.e., the taxonomies of codes were developed from multiple passes through the data). One researcher (again, "blind" to both student identity and pretest/posttest status) coded all the transcripts.

what the findings were
Both analyses revealed that students' ability to reason statistically improved. Though conducted without a control group for comparison and students were not randomly assigned to groups, the study did reveal evidence of pre- to post-course improvement of students' knowledge and performance in at least 3 of the course topics: (1) recognizing of the limitations of correlational evidence, (2) demonstrating understanding of the components of scientific experimentation, and (3) demonstrating understanding of random sampling. Students with relatively less pre-course knowledge showed greater gains than among students with relatively more. However, a pervasive confusion between random sampling and random assignment was also revealed.

Qualitative analysis provided a rich picture of the specific ways in which students' reasoning developed. Students showed an increased concern for the need for a comparison in experimental design, a reduced reliance on non-statistical argumentation (for the task designed to demonstrate understanding of the components of scientific experimentation), and an increase cognizance of general issues with sampling. Although students' understanding of some statistical concepts remained vague and incomplete, most of the concepts they leveraged in their explanations were in some sense justifiable and statistical notions did tend to replace inappropriate non-statistical reasoning.