«ASSESSING THE VALUE-ADDED EFFECTS OF LITERACY COLLABORATIVE PROFESSIONAL DEVELOPMENT ON STUDENT LEARNING Gina Biancarosa This ...»
Adjusting for possible time of entry effects. As noted earlier, a key feature of an accelerated multiple cohort design is that students entered the data set at different points in time by virtue of their cohort (i.e., Cohort 1 entered in second grade, Cohort 2 began in ﬁrst grade, and Cohorts 3– 6 entered in kindergarten). In addition, some students transferred into a school after the start of the study and were absorbed into their respective cohorts.
To account for the fact that many children did not enter the dataset in the fall of kindergarten, we added another set of dichotomous indicator variables (termed “Entry”) as ﬁxed effects in the child-level model. Each indicator variable corresponds to a speciﬁc grade and time of year and represents the ﬁrst occasion at which a student appears in the dataset.
For example, if a student joined the study in the fall of ﬁrst grade, that student’s Entry would be a three, and the Entry_3 indicator variable would be scored as one and all other Entry variables would be zero. The reference category for this set of indicators is the fall of kindergarten, or Entry_1. As a result, the intercept in the model represents the predicted initial literacy status in the fall of kindergarten and the Entry variables for Entry locations two through six represent the mean differences in literacy status among students, depending on when they ﬁrst entered the dataset.6 Adjusting for cohort differences. Because the earlier descriptive analysis indicated small average differences in initial literacy status among some baseline cohorts, we also included a set of ﬁxed effects for each cohort to absorb these and any other potential residual differences between cohorts.7 These dichotomous indicator variables were also included as ﬁxed effects at the child level. We note that cohort and entry effects are not redundant because students could enter a cohort after data collection was initiated for their speciﬁc cohort.8 The value-added model. The value-added model estimates both the average value added by LC in each year of implementation and random value-added effects associated with each teacher and with each school year of implementation.
We describe these effects below.
Average LC value-added effects. Separate ﬁxed effects were estimated to assess the average value added of LC during each of the 3 years of program implementation. These estimates represent the increments to learning in comparison to the growth trends observed in the baseline period. These are captured through three dichotomous indicator variables (LC_Year_1, LC_Year_2, and LC_Year_3). The coefﬁcients associated with each of these variables estimate average LC effects after the ﬁrst, second, and third years of implementation, respectively. The LC_ Year_1 indicator was set to one for observations collected in the spring of the second year of the study, LC_Year_2 was equal to one in the spring of the third year of the study, and LC_Year_3 was equal to one in the spring of the fourth year of the study; otherwise, all three were set to zero.
We also estimated the average effects after the summers of the ﬁrst and second years of implementation, which are termed LC_Summer_1 and LC_Summer_2, respectively. LC_Summer_1 was equal to one at the fall of the third year of the study and LC_Summer_2 was equal to one at the fall of the fourth year of the study; otherwise, both were set to zero. Note that these LC summer effects allow us to estimate the extent to which the value added observed at the end of the previous year (i.e., the spring testing) was maintained through the summer (i.e., the subsequent fall testing). If, in fact, the LC academic effects are sustained, then the estimate for the summer LC effects should be similar in magnitude to the corresponding academic year LC effects.
All ﬁve of the LC implementation indicators are set at zero at the ﬁrst occasion when the student appears in the data set. In this way, we preserve the meaning of the intercept.9 Random value-added effects. To estimate value-added effects for schools and teachers, students were linked to their school’s identiﬁer at all time points and to their teacher’s identiﬁer each spring. Each fall, students were linked to a schooland grade-speciﬁc identiﬁer. This linking of students with their teachers in the spring allows us to estimate the value-added effect of each teacher on students’ learning from fall to spring each year.
Based on these links, we estimated several random value-added effects for schools and teachers that captured their contributions to student learning during the baseline year and each of the 3 years of the LC program. We review ﬁrst the random effects in baseline growth, followed by the 3 years of implementation.
We incorporated ﬁve random effects to capture possible teacher and school value-added effects to students’ learning during the baseline period. First, to capture differences among schools in the student achievement levels at entry, the intercept was allowed to vary randomly among schools. This speciﬁcation controls for possible selection effects associated with prior achievement. Second, to capture variation among schools in academic year learning rates during the baseline period, the base academic year growth indicator was allowed to vary randomly at the school level. Third, we added a random effect, “Base_Tch_VA,” to represent the variation among teachers within schools in their value added to student learning in the spring of the baseline year. The dichotomous indicator variable, Base_Tch_VA, takes on a value of one only at this time point and only for those children who were also present in the data set in the fall of the baseline year.
The ﬁxed effect for Base_Tch_VA was set to zero so as not to be redundant with the ﬁxed effect for the base academic year growth indicators described earlier.
Finally, we allowed the two summer indicators to vary randomly at the school level. These effects captured the extent to which schools varied in learning (or loss) during the summers. Since there is no assigned teacher during the summer periods, a random teacher effect for these occasions was not deemed sensible.
Finally, we also allowed LC value-added effects in all 3 years of implementation to vary randomly at both the school and teacher levels. This speciﬁcation allowed us to estimate the amount of variation among schools and among teachers within schools in LC effects and generated empirical Bayes estimates of individual teacher and school effects.
Final model. Assembling all of these components together produced the following ﬁnal model. The outcome, Yijk, was deﬁned as literacy status in logits at time i for student j in teacher k’s class in school. Due to space limitations, we
report here only the ﬁnal mixed model, which was as follows:
Table 5 summarizes the interpretation for the parameters in this model that are of primary interest to the research questions.
Results Estimates for the ﬁnal model were derived using the HCM3 subprogram with HLM (version 7.0). HCM3 is a ﬂexible program that allows for estimating a variety of four-level data models. In our application, the model consisted of repeated measures on students crossing teachers nested within schools. Formally, we represented this as repeated measures: (students teachers): schools. All random effects at the teacher and school levels were treated as cumulative within HCM3.
The full ﬁnal ﬁtted model is reported in Tables 6 and 7; however, only the most relevant results are discussed below.
As a check on model ﬁt, we undertook a number of posterior predictive validity tests in which we used the results from the ﬁtted model to predict the overall outcomes and school by school. Residuals between observed data and modelbased outcomes were then examined. We found no evidence of any systematic variation in the posterior predictions as compared to the observed data.
Student-Level Variance in Growth Results indicate that average entry literacy learning status ( 0000) in kindergarten was.87 logits and the average literacy learning rate ( 1000) was 1.02 during the academic year. Controlling for teacher and school effects, there was signiﬁcant variance between children at entry into the dataset and in their growth rates (see Table 7). The student-level standard deviation at entry was 1.17, indicating a wide range of variability in students’ incoming literacy learning. The student-level standard deviation for the academic year literacy learning rate was.25, indicating considerable variability in learning among children even after controlling for their speciﬁc classrooms and schools. Finally, these random effects are moderately and negatively correlated, meaning that children who entered with lower literacy tended to learn at a faster rate than those who enter with higher literacy.
Average LC Value-Added Effects In terms of program effects, the average value added during the ﬁrst year of implementation ( 7000) was.16 logits. This represents a 16% increase in learning as compared to the average baseline growth rate of 1.02. During the second year of implementation, the estimated value added ( 8000) was.28, which represents a 28% increase in productivity over baseline growth. The third year yielded an estimated value added ( 9000) of.33 logits, which represents a 32% increase in productivity over baseline. These value-added effects convert into standard effect sizes of.22,.37, and.43, respectively, based on the residual level 1 variance (eijk ) estimated under the HCM3 model.
Table 5. Interpretation of Coefﬁcients in the Final Model of Primary Interest to Answering the Research Questions
Variation in LC Value-Added Effects The school and teacher random effects in our model enabled the estimation of variation among schools and teachers in their value added (or extra contribution) to student learning each year (see Table 7). All of the variance components were statistically signiﬁcant.
Variance in school-level effects. Figure 3 illustrates the distribution of valueadded effects among schools. We display empirical Bayes estimates for the LC value-added effect in each of the 17 schools for each year of implementation. As reference points, the average ﬁrst-, second-, and third-year effects are also displayed in Figure 3 as three dashed lines. Recall that a value-added effect of.0 signiﬁes no improvement in academic year learning as compared to the baseline rate in that school. In almost every case, the estimated individual school effects are positive. The only exception was school 12 during the ﬁrst year of LC implementation. Notice also that school value-added effects increased over time in most cases, with the most notable exception being school 16. Increasing variation over time in the magnitude of the school effects is also manifest in this display. For example, the largest individual school effect in the ﬁrst year was about.30. In contrast, by the third year, several schools had effects of.35 or higher. This is equivalent to a 35% improvement during the ﬁnal year of LC implementation in these schools as compared to the overall baseline academic learning rate of 1.02.
The variation among schools in academic growth rates during the baseline period was.082. Variance in LC effects across the 3 years of implementation increases from.013 to.036. There is a weak, positive relationship between the baseline academic year growth rate in a school and its ﬁrst-year LC value-added Table 7. Final Hierarchical Crossed-Level Value-Added Model Variance Components and Correlations among Them
Note.—Diagonal values are variance components and off-diagonal values are correlations among random effects.
effect, meaning that schools with higher growth at baseline accrued slightly larger value-added effects that year. In the second year, there is no relationship between the baseline academic year growth rate in a school and the LC value-added effects.
By the third year, the correlation is strong and negative, indicating that larger LC value-added effects accrued in schools that had lower baseline academic year growth rates. Correlations between LC school value-added effects are moderately strong between the ﬁrst and second years of implementation and the second and third years of implementation, but are weak between the ﬁrst and third years of implementation.
Variance in teacher-level effects. Similarly, we found increasing variance between teachers within schools over time from.056 in the baseline year to.217 in the ﬁnal year of implementation. The correlations among the teacher effects over time are moderate to strong, ranging from.38 to.71. These capture the consistency in the teacher effects from one year to the next after controlling for differences in latent growth trajectories of the children educated in each classroom.
Figure 4 illustrates the variability in teacher value-added effects over time. Each box plot represents the empirical Bayes estimates for the teacher effects during each year of LC implementation in each school. The increasing variability over time among teachers within schools is clearly visible in the increasing range of the box plots here. The vast majority of teachers in most of the participating schools Figure 3. Variation in school value added over 3 years of LC implementation compared to the average value added across schools.
showed substantial value-added effects by the end of the study. Moreover, a fair number of teachers show value-added effects of 1.0 or more by the third year.
Persistence of Effects over the Summer The model also allowed examination of whether the value-added effects estimated at the end of the ﬁrst and second years of implementation were maintained over the subsequent summers. The results indicate that program effects did persist through the subsequent fall testing. The average value-added effect after the ﬁrst summer of LC implementation ( 10000) was.21 logits, which is somewhat higher than the ﬁrst-year implementation effect of.16 logits. The average value-added effect after the second summer of LC implementation ( 11000) was.15 logits, which is about half the magnitude of the second-year implementation effect of.28 logits.
An omnibus test comparing LC effects at the end of the summer to those estimated at the end of the preceding academic year yielded statistically indistinguishable results ( 2 1.43, df 1, p.2303). Improvements during the academic year of LC implementation appear to persist through the following fall assessments.
As a check on model ﬁt, we conducted a number of tests of the posterior predictive validity of our model during which we used the results from the ﬁtted model to predict outcomes school by school, as well as across schools. Speciﬁcally, observed data were compared to model-based residuals and we found no evidence of any systematic variation in the predictions as compared to the observed data.
This article presents the results from a 4-year longitudinal study of the effects of the LC program on student learning in 17 schools. A rigorous quasi-experiment was designed to estimate separate program effects by year, school, and teacher.
Inferences about these program effects are based on a value-added analysis that compares student learning gains during LC implementation to those achieved in the same classrooms and schools prior to the program intervention.