What and how preservice teachers learn: Designing a course that fosters development of useful theoretical knowledge and the assessment methods to capture it
Siegel, Derry, Steinkuehler, Kim, & Seymour (2001)

what was studied
In this study, the authors were interested in assessing the effectiveness of an innovative undergraduate course in teacher education. The novelty of the course stemmed from the combination of instructional approaches used: case-based instruction (the use of videocases as a context for discussion of the course content), problem-based learning (students collaboratively work on problems that prompt research into the domain literature), and hypertext instruction based on Cognitive Flexibility Theory⁴⁸ (the use of a densely interlinked hypertextbook). Throughout the semester, students were given a series of problems to solve. For each problem, students (1) studied a videocase of actual instruction and a related problem to be solved; (2) formulated hypotheses about the cases that prompted investigation into the content of the domain; (3) used the hypertextbook to guide their investigations into educational theory and research; and (4) developed instructional solutions to the problem based on their research. These activities were designed to help students develop useful knowledge about cognitive and educational research to guide their subsequent instructional decisions and design. By the end of the semester, students were expected to have a more elaborated and more research-based mental model of the kind of classroom situations they might face someday in their own teaching practice.

how effectiveness was measured
The growth and development of students' mental models over the course of the semester was assessed by comparing students' use of cognitive principles to reflect on instructional practices before instruction (pretest essay) versus after (posttest essay). Both pre- and posttests were designed to assess the transfer of course concepts and conceptual models to new teaching situations depicted in videocases they had never seen. Two isomorphic tests, each consisting of 2 video segments and 2 related essay reflection questions, were administered in a counterbalanced design.

Three levels of analysis, based on Hierarchical Schema Theory¹⁴, were conducted. According to this theory, student conceptual development from a course moves through three phases or levels of increasing sophistication (Figure 1). Students exhibit different levels of conceptual sophistication and skill when transferring course concepts to the real world situation to which they apply. At the first, most basic level, students spontaneously activate the related course concepts as interpretive frameworks for the real-world situation they face. At the second level, students not only activate such course concepts, but they activate the most relevant ones given the particulars of the situation confronted. At the third, highest level, students not only activate the most appropriate concepts but they are also able to integrate those concepts into a coherent theoretical interpretation of the situation. Under this model, the educational effectiveness of a course can be measured in terms of the level of conceptual sophistication it enables students to attain.

what the findings were
The authors expectations about what students might gain through such activities appear well founded: Following instruction, students spontaneously activated a greater number of course concepts and used a significantly greater number of "appropriate" and "key" course concepts to explain the situations depicted in the video segments. However, there was no evidence suggesting that students were able to develop a more sophisticated mental model of the situations depicted in the video following instruction; on average, students began with the most simplistic mental model of instruction (level one) and ended with a slightly more sophisticated model (a version of level two), but no students exhibited the highest level of sophistication possible (level 3). In addition, there were significant differences between the two tests used: Students had more difficulty with one of the two isomorphic tests that were administered, possibly mitigating the effects of any changes that may have occurred. The authors note that these results are not surprising given the difficult and time-consuming nature of such conceptual change and advise future research using this particular framework for assessing students' conceptual development to consider administering such assessments multiple times throughout the semester so that a more detailed understanding of students' conceptual progression can be ascertained.