«ASSESSING THE VALUE-ADDED EFFECTS OF LITERACY COLLABORATIVE PROFESSIONAL DEVELOPMENT ON STUDENT LEARNING Gina Biancarosa This ...»
Results demonstrate signiﬁcant gains in student literacy learning beginning in the ﬁrst year of implementation and that the effect’s magnitude grew larger during each subsequent year of implementation. On average, children in participating schools in the ﬁrst year of implementation made 16% larger learning gains than observed during the baseline no-treatment period. In the second year, children learned 28% more compared to the baseline data, and by the third year they had learned 32% more. Our analyses also indicate that these results persisted across summer periods as veriﬁed through the follow-up of students in the fall of the subsequent academic year.
These results contrast with ﬁndings from two other recent studies, which reported that literacy coaching had little (Marsh et al., 2008) or no (Garet et al.,
2008) impact on student learning. One reason for the novel ﬁndings may be that the coaching strategies evaluated in the two previous studies differed signiﬁcantly from the LC model investigated in this article. As noted in the background to the current study, the LC model involves a full year of PD for coaches before they begin working with teachers, whereas coaches in the aforementioned studies received only a few days of training prior to engaging in their new roles. In addition, coaching in the LC model is organized around a detailed and well-speciﬁed literacy instructional system that includes a repertoire of instructional practices. In contrast, the coaching models in the previous studies were designed more as supplements to extant and more varied curricula. In the Garet et al. (2008) study, coaching was literally an add-on to another PD curriculum, and in the Marsh et al.
(2008) study, the coaching framework was set forth at a state level with an expectation of variation in content and implementation across local school districts.
Either of these differences, as well as other possible differences among the coaching programs studied, could account for the signiﬁcant differences in estimated effects found here and in previous research. Continued research on multiple models of coaching across multiple contexts is needed to resolve whether, and under what conditions, coaching can stimulate improvement in student learning.
At a minimum, the current study does suggest that well-speciﬁed and wellsupported coaching initiatives can effect positive changes in student learning.
The overall pattern of effects—increasing over time in both size and variability within and between schools—also merits comment. In each study school, coaching was a new professional undertaking for the individuals who took on this role.
Both LC program documents and other more general clinical accounts of instructional coaching detail a complex professional role that may take several years to learn well (Gibson, 2005, 2006). Just as novice teachers improve during the ﬁrst few years of learning to teach (Borko & Putnam, 1996; Darling-Hammond & Bransford, 2005; Kane, 1993; Snow, Grifﬁn, & Burns, 2005), it is reasonable to hypothesize that coaches do as well. The increasing size of the estimated program effects from the ﬁrst year to the third year is certainly consistent with the aforementioned hypothesis.
Thus, increasing coaching expertise over time is one plausible explanation for the temporal patterns observed in the current study. Unfortunately, reliable and valid measures of coaching expertise do not yet exist and therefore could not be employed in the current study. The current ﬁndings cohere with previous calls for research to develop coaching expertise measures and explore the potential role of a coach’s developing expertise as an explanatory factor for differences in effects on both teachers and students (Neufeld & Roper, 2003).
The observed increasing effects over time may also have resulted from changes in the informal professional networks around literacy instruction within participating schools. If the professional context of a school changes as a result of coaching, teachers, especially new teachers, may beneﬁt not only from the mentoring of a more experienced coach, but also through the social learning that can occur as they interact with increasingly more expert colleagues. This represents another plausible hypothesis for future research to investigate. Some preliminary descriptive evidence drawn from the current study that supports this latter account has been reported elsewhere (Atteberry & Bryk, 2009).
In terms of the increasing variability in effects over time, both within and between schools, this too seems quite plausible given the nature of a coach’s work.
In principle, the effects of coaching accrue over time through the one-on-one interactions that occur between a coach and an individual classroom teacher.
Stated somewhat differently, coaching is a relational practice whose efﬁcacy presumably depends on the quality of the relationship that a coach is able to establish with each individual teacher (Bean & Carroll, 2006). Variations in the quality of these connections, as well as the duration of these interactions, might well account for the increasing variability in teacher value-added effects on student learning reported in this study. This pattern is consistent with the ﬁndings of Marsh et al.
(2008), who reported that larger effects on student literacy learning were associated with coaches who had been coaching for a longer period of time. Individual variability in teacher effects may well be a structural characteristic of instructional coaching efforts; if so, this variability merits further consideration by researchers and coach trainers so that future coaching initiatives effect greater consistency in teachers’ instructional improvement over time.
The overall pattern of increasing variability in teacher effects found in this study is consistent with a work organization where literacy coaches operate with considerable discretion in engaging individual teachers in extended instructional coaching. By design, coaching is an intervention from which we might reasonably expect variable effects to accrue depending on the quality of the coach, the school context in which the coach works, and the varying amounts of coaching that each individual teacher receives. Future studies should include larger samples of schools and coaches in order to examine more thoroughly these possible sources of variability in effects.
This study was undertaken to assess the effects of the LC program on student literacy learning and found statistically and substantively signiﬁcant impacts.
Even though the study involved a rigorous quasi-experimental design, we need to consider the plausibility of the major competing hypotheses for the observed results reported here.
Foremost, we must consider whether the observed improvements in learning gains over time are in fact school improvement effects or might alternatively be attributable to student selectivity effects. Student selectivity is generally less problematic in an accelerated multicohort longitudinal design because value-added estimates are based on changes in student learning as compared to baseline results from the same students, classrooms, and schools. In other words, each student, classroom, and school serves as its own individual control. Only if the student composition in these schools were changing over time coterminous with the study would a plausible alternative explanation exist. However, we found no evidence of this. We collected basic student demographic data each year, and no substantial changes in student characteristics were observed over time. More signiﬁcantly, the data presented in Figure 2 document almost identical mean achievement scores for students at entry into kindergarten over all 4 years of the study. In addition, our model included cohort effects, which captured differences in baseline abilities between each cohort (see Table 6). Therefore, the value-added effects are observed even after controlling for historical differences in cohorts. For all these reasons, taken together, the current empirical results make a student selectivity hypothesis implausible.
Therefore, we conclude that the current results provide sound evidence that sample schools actually did improve performance during the course of the study.
However, we still have to consider plausible competing hypothetical causes, other than the LC program, that might also account for these effects. That is, might there have been a concomitant alternative treatment effect, occurring during the same time period in these same schools, that could explain the observed results? On balance, school districts are complex organizations with multiple sources of programs and funding. In fact, the study was carried out during the period when NCLB and Reading First initiatives were being introduced. These federal initiatives focused districts on improving reading skills through a variety of mechanisms, including structured literacy block times, a focus on skill work in the early grades, and structured accountability procedures. In any given school, initiatives of this sort could in principle account for some or all of the effects observed in this study.
This alternative explanation for our ﬁndings strikes us as unlikely for several reasons. First, the LC program required a substantial ﬁscal commitment on the part of participating schools. While the study did offer districts a partial reduction in costs for the LC training of school coaches, districts still incurred substantial out-of-pocket expenses for residual tuition costs, travel costs to send novice coordinators for training to Boston, Massachusetts, or Columbus, Ohio, and the salary costs required to free a teacher half time to prepare for and assume the literacy coach role. In short, districts were asked to make a major commitment to LC as their literacy reform strategy and to use their available discretionary resources to fund it. That a second equally signiﬁcant reform initiative might have coexisted in these same schools at the same point in time seems dubious.
Second, although base individual teacher effects are included as controls in the value-added model, changes in these effects over time could be inﬂuenced by other exogenous professional improvement opportunities simultaneously being afforded teachers. This theory seems unlikely given the scope of program effects documented above. Speciﬁcally, at the school building level, LC participation makes substantial time demands on teachers’ time. As a result, the likelihood of teachers being engaged simultaneously in some other literacy PD seems improbable. Any exogenous teacher-improvement efforts would have to have been both deep and widespread to possibly account for the observed data.
Third, we conducted annual interviews with each literacy coach about the progress of the initiative in their respective buildings. We had no reports of concurrent competing initiatives that might account for the broad base of results observed.
Fourth, the fact that the effects were broadly based, accruing by the third year in almost all of the 17 schools, adds further doubt to the alternative treatmenteffect hypothesis. Study schools were located in eight different states and nine different districts. For an alternative concurrent treatment to account for the observed results, most of these districts would have had to develop another competing and equally effective literacy-improvement program in the same schools at the same time this initiative took place. While it is possible that other effective local initiatives may have been introduced in one or more schools during this period, achieving such coincidental effective change consistently across the diverse sites engaged in this study seems unlikely.
Finally, the overall magnitude of the effects documented in the current article militates against a plausible alternative treatment effect as a full explanation for the observed outcome. Even if we were to assume that some signiﬁcant alternative program effects existed in some schools, and even if they accounted for half of the observed effects (all of which seems doubtful given the evidence and arguments already outlined), the residual effect sizes would still represent a meaningful contribution to improving student learning.
For all of the aforementioned reasons, we conclude that the study schools’ participation in the LC program led to positive program effects on children’s literacy.
Given the prominent role of coaching in this program as a lever for enacting change in teachers’ practice and consequently in students’ learning, this study contributes important new evidence of the potential for literacy coaching to yield improvements in student literacy outcomes. Nonetheless, the evidence supporting coaching’s effectiveness is still slim, and it is unclear whether the effects found here are speciﬁc to LC or can be observed in other coaching models. Studies of the variations within and between coaching models, particularly with a focus on coach preparation and developing expertise, would help to clarify the mechanisms by which coaching can be an effective lever for change in student achievement.
The work described in the current article was supported by a Teacher Quality Grant from the Institute for Educational Sciences (IES), R305M040086. We are appreciative of the support provided by IES. All errors of fact, omission, and/or interpretation are solely the authors’ responsibility. The research team involved in this study included afﬁliates of the coaching program being investigated. This collaboration informed the study design and the development of tools to measure teacher practice. The analytical team worked independently from those afﬁliated with the program to ensure objectivity. Correspondence regarding this article should be addressed to Gina Biancarosa, College of Education, 5261, University of Oregon, 1655 Alder Street, Eugene, OR 97403. E-mail: email@example.com.
1. LC coaches may remain active teachers either by having a partner teacher who typically covers science and math instruction while coaching occurs, or by co-teaching in others’ classrooms for extended periods of time, ranging anywhere from one month to an entire year.
2. For further details about the LC program, see http://www.literacycollaborative.org/.
3. The initial design involved 18 schools, but one school was subsequently lost because a coach for the school was never certiﬁed by LC due to the candidate coach’s failure to complete the Literacy Coordinator training. Thus, the school never implemented coaching.
4. Technical information that could not be incorporated into the current article due to length constraints is available at The Carnegie Foundation for the Advancement of Teaching’s web site (http://www.carnegiefoundation.org/elibrary) as well as directly from the ﬁrst author.
5. Further details on the functioning of this set of indicators can be found on the Carnegie Foundation web site (http://www.carnegiefoundation.org/elibrary) as well as directly from the ﬁrst author.
6. Further details on the functioning of this set of indicators can be found on the Carnegie Foundation web site (http://www.carnegiefoundation.org/elibrary) as well as directly from the ﬁrst author.
7. Cohort 3, which entered kindergarten during the study’s baseline year, served as a reference category.
8. Further details on the functioning of this set of indicators can be found on the Carnegie Foundation web site (http://www.carnegiefoundation.org/elibrary) as well as directly from the ﬁrst author.
9. An illustration of the simultaneous operation of these ﬁve indicators can be found in the technical documentation on The Carnegie Foundation for the Advancement of Teaching’s web site (http://www.carnegiefoundation.org/elibrary) as well as directly from the ﬁrst author.