Addressing reliability problems in the portfolio assessment of college writing
Nystrand, Cohen, & Dowling (1993)

what was studied
In a series of two studies, the authors assessed the expository writing skills of third-year undergraduates using writing portfolios. Specifically, they were interested in the reliability of portfolio assessments as an tool for measuring verbal skill. In Study 1, writing portfolios from 329 students from four courses (in the College of Agriculture and Life Sciences, the College of Education, the School of Family Resources and Consumer Resources, and the School of Pharmacy) were collected; in Study 2, writing portfolios from 313 students from four courses (in the School of Nursing, the College of Letters and Science, the Medical School, and the College of Engineering) were collected. In both studies, the portfolios included every piece of writing (e.g., papers, exams, lab reports, topic reviews, etc.) that each student wrote as a part of their regular course requirements. All included writings were "expository" in nature, defined as "sustained reflection in which the author focuses and processes information to varying degrees" (p. 53).¹

how effectiveness was measured
In both studies, texts were assessed in terms of (a) degree of reflection and (b) extent of text elaboration. In Study 1, writing was scored on a portfolio-by-portfolio basis: raters scored each text within a given portfolio before moving on to the next portfolio. In Study 2, writing was scored on an assignment-by-assignment basis: raters scored each text that was written in response to a given prompt or assignment within a class before moving to the next writing assignment. Texts from both studies were also classified by genre so that the researchers could examine how different kinds of assignments prompted different kinds of reflection and/or elaboration. (This also insured that they could examine students' writing ability, not just what kind of writing students were assigned). A subsample of students' writings was reassessed in both studies in order to examine rating reliability.

what the findings were
In both studies, there were a large number of assignments could be considered unchallenging (e.g., short-answer exercises) in that they did not require extended or reflective writing, yet this varied considerably by course. The portfolios that were collected were not homogeneous, indicating that students' writing ability varied considerably across the different genres. Reliability estimates were low to moderate in Study 1, indicating substantial disagreement between raters. Given the constantly shifting topics and genres not only across classes but also within each portfolio, raters had difficulty maintaining consistency over time. Reliability estimates were clearly higher in Study 2, indicating that writing portfolios are better assessed by scores for each piece of text than by a single global writing-skill score.